5 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	821de668d0	Rebuild: the merger stops when a pass over the file keys decides nothing ... Waiting files counted as work, so the merge loop re-read them for its whole 30 s budget every minute: Postgres used ~a core even without workers. Now ~4% at rest, and storing a boneless range costs Postgres ~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads are stored EXTERNAL (zstd already; no pglz attempt). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:49:45 +02:00
Daan MeijerandClaude Opus 5.5	80c920f678	Rebuild: settling quiet releases ... Hourly in coordinator:run: a release unchanged for 6 hours is settled, complete when every file its [i/N] counter announced was found, unknown without a counter (phase 2 reads par2 sets for those). Partial index on unsettled releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:33:47 +02:00
Daan MeijerandClaude Opus 5.5	31493a98be	Rebuild: merger rounds with one neighbour query, a cursor and a time budget; planning skips failing groups ... The neighbour check was three queries per file (~2,000 files/minute); now one per round of 20k files, with a cursor so waiting files don't take every round's places, and rounds until nothing is left or 30 s. On the dev backlog (old-style fragments): ~18k files/minute in the 30 s budget; at full throughput the new fragment rule leaves ~1,200 files/minute. A group that fails GROUP (gone from the server) no longer stops planning the others; edge:bench's groups are inactive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:25:45 +02:00
Daan MeijerandClaude Opus 5.5	b862201991	Rebuild: the fragment merger ... Files cut by a range's edge are put together once the ranges around them are done (coordinator:run, every minute): a complete file is judged by the rules and written as a release chunk with probe candidates; one still missing parts while both neighbouring ranges are done is dropped and counted (incomplete/edge). A neighbour not planned yet counts as not done, except before the group's oldest article. BatchStore.writeReleases is the shared write path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:19:53 +02:00
Daan MeijerandClaude Opus 5.5	d5f168d8d2	Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker ... - groups:add / ranges:plan / nodes:create / coordinator:run commands - the node API: lease, the range's chunks as one framed stream from the spool, batches, heartbeat, finish, release (node guard, token per node) - edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for the rest), posts a batch every 10 chunks, gives its range back on SIGTERM - releases are gathered per batch: 1.04 chunk rows per release at 10 chunks per batch (1.35 when built per XOVER chunk) - partitions are checked per batch instead of remembered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:31:06 +02:00