4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	d0f3497ec9	Rebuild: completeness as indexed, file totals from any counter, pausable tails, progress by articles ... - releases.complete is computed with every batch (files >= the counter's total) instead of only when a release settles after 6 quiet hours: the 'complete' filter matched nothing in a fresh catalog. - A release's file total comes from any counter form (counterTotal: '(01/10)', 'File 27 of 132', …), not only [i/M]; the post rule's key keeps the [i/M] total, so keys are unchanged. Both implementations. - groups.tail: a group's live tail can be paused (its history and owner priority stay), e.g. to let other groups' history go first. - Stats: history by articles, and the current pass (history ranges before those deferred 2000 priorities behind) as its own figure; the header showed 0.1% of ranges counting 145k deferred ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 13:06:03 +02:00
Daan MeijerandClaude Opus 5.5	fd0f2bde75	Rebuild: the live tail ... coordinator:run plans the articles after each group's last range every TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every group's newest article, and new articles become ranges at priority -1, leased and prefetched before any history. A group gets a tail range once it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every round. History (planAll, hourly) is now only for groups not planned yet. ranges:plan --tail plans one round. Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and listActive waited for a plain end that never came (groups:add --top too). It reads the compressed reply now; the fake server in the test does both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 10:41:16 +02:00
Daan MeijerandClaude Opus 5.5	31493a98be	Rebuild: merger rounds with one neighbour query, a cursor and a time budget; planning skips failing groups ... The neighbour check was three queries per file (~2,000 files/minute); now one per round of 20k files, with a cursor so waiting files don't take every round's places, and rounds until nothing is left or 30 s. On the dev backlog (old-style fragments): ~18k files/minute in the 30 s budget; at full throughput the new fragment rule leaves ~1,200 files/minute. A group that fails GROUP (gone from the server) no longer stops planning the others; edge:bench's groups are inactive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:25:45 +02:00
Daan MeijerandClaude Opus 5.5	d5f168d8d2	Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker ... - groups:add / ranges:plan / nodes:create / coordinator:run commands - the node API: lease, the range's chunks as one framed stream from the spool, batches, heartbeat, finish, release (node guard, token per node) - edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for the rest), posts a batch every 10 chunks, gives its range back on SIGTERM - releases are gathered per batch: 1.04 chunk rows per release at 10 chunks per batch (1.35 when built per XOVER chunk) - partitions are checked per batch instead of remembered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:31:06 +02:00