4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	cd06f96f85	Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes ... End to end on the workstation (coordinator:run, the API, edge:node, real NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges. - Prefetcher: every connection takes the next missing chunk (leased ranges first, in order); one connection per range (~26k lines/s) kept the worker waiting - executable rule: releases of bare executables (malware floods named after films and software) are dropped; they were 92% of the catalog rows of the test ranges, 21% of the segments - fragments per file (details once) and gzipped batch bodies; the JSON limit (1 MB by default) raised to 256 MB, empty strings kept (a rule count's key is '') - release_chunks.batch_from is a bigint (boneless article numbers) - tests get their own database; edge:bench --store times BatchStore: 1.3 s for a boneless range (1.24 ms per release chunk) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 01:07:11 +02:00
Daan MeijerandClaude Opus 5.5	d5f168d8d2	Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker ... - groups:add / ranges:plan / nodes:create / coordinator:run commands - the node API: lease, the range's chunks as one framed stream from the spool, batches, heartbeat, finish, release (node guard, token per node) - edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for the rest), posts a batch every 10 chunks, gives its range back on SIGTERM - releases are gathered per batch: 1.04 chunk rows per release at 10 chunks per batch (1.35 when built per XOVER chunk) - partitions are checked per batch instead of remembered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:31:06 +02:00
Daan MeijerandClaude Opus 5.5	20910647fa	Rebuild: the batch store (catalog upserts, write-once chunks, candidates, rule counts, fragments, resume point) ... A batch is stored in one transaction and only at its range's resume point, so a retried batch can't count twice. Garbage is judged before per-article: a dropped tool isn't worth reading bodies for either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:21:17 +02:00
Daan MeijerandClaude Opus 5.5	7dc316c987	Rebuild: the edge worker's range indexer ... Overview chunks in, per batch out: releases as encoded chunks, probe candidates, counts per rule (crossposts per owner group, obfuscated, garbage, per-article, incomplete), a sample of what was dropped, and fragments near the range's edges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:10:16 +02:00