4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	a979b5d11c	Rebuild: the coordinator spools replies without inflating them ... On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator used 1.3-1.7 cores, mostly GC of the text it inflated only to know a compressed reply was complete, plus a copy of every reply. A raw reply is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at the same rate. ".\r\n" can also end a network read inside the compressed data (~1 in 16M reads, a few dozen times over the whole target). Such a chunk doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt), the spooled chunk is removed and the range goes back to the queue; the prefetcher fetches it again. The rest of that reply breaks the connection's next command, so the connection is replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 07:40:06 +02:00
Daan MeijerandClaude Opus 5.5	9dc0a27bb4	Rebuild: a lease never waits for prefetching; one-shot hashes ... A lease was only granted for a range with spooled chunks, and the prefetcher kept ranges ahead per active lease: once the spool ran dry a worker got none and slept 30 s, fewer ranges were leased, fewer prefetched. Three workers held one lease between them (~50k headers/s). A lease now takes a spooled range first, else the next planned one (leased ranges are fetched first; the stream waits). crypto.hash instead of createHash().update().digest() for file and release keys: same digests, ~2.5x faster (~4.6% of a worker's time). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 03:08:19 +02:00
Daan MeijerandClaude Opus 5.5	bd1c846f27	Rebuild: prefetching no longer waits for merging, nor stalls on finished ranges ... The coordinator ran the prefetch tick in the same loop as the merger (up to 30 s a minute): its connections went idle once their targets were fetched, and the first leases after a start found nothing spooled. The prefetcher now ticks in a loop of its own. The prefetcher took the first planned ranges with nothing spooled, even ones with nothing left to fetch (next_id past the end): two such ranges (edge:bench's, set back to planned) kept it from fetching anything. It and the lease query now skip those, and edge:bench marks its range done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 03:04:00 +02:00
Daan MeijerandClaude Opus 5.5	d5f168d8d2	Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker ... - groups:add / ranges:plan / nodes:create / coordinator:run commands - the node API: lease, the range's chunks as one framed stream from the spool, batches, heartbeat, finish, release (node guard, token per node) - edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for the rest), posts a batch every 10 chunks, gives its range back on SIGTERM - releases are gathered per batch: 1.04 chunk rows per release at 10 chunks per batch (1.35 when built per XOVER chunk) - partitions are checked per batch instead of remembered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:31:06 +02:00