Commit Graph
2 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 a979b5d11c Rebuild: the coordinator spools replies without inflating them
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.

".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 07:40:06 +02:00
Daan MeijerandClaude Opus 5.5 d5f168d8d2 Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
  spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
  the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
  chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 00:31:06 +02:00