4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	fa7d4f3394	Rebuild: end-to-end fixes — spool accounting, keep-alive frames, resume point, overlapped worker ... End to end (workstation, recent a.b.boneless ranges from the spool): a 2M range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on one core: 215k). - Prefetcher: spooled_at was set against a total counted from the range's resume point, which shrinks as the worker advances: the last chunks were never fetched and the worker waited. Now counted against the whole range, only new chunks counted, and leased ranges are always checked against the files. Planned ranges ahead are bounded (it had spooled ever more); the spool size is measured every 30 s (a stat per file). - The stream sends an empty frame every 20 s while waiting for a chunk (Node's fetch gave up on a quiet body after 5 minutes). - A range taken over midway: the first batch starts at the resume point (it started at the range's start and was refused). - edge:node inflates the next chunk on the thread pool while indexing the current one, and doesn't wait for a post before indexing on (posts stay in order; a failed post stops the range). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 01:31:19 +02:00
Daan MeijerandClaude Opus 5.5	638cfa1dcd	Rebuild: a lean overview scan for the edge worker ... The worker reads the XOVER reply's text directly: fields by position, the owner from the Xref without arrays (crossposts leave before any subject parsing), a subject parsed once per file (cached without its segment counter), a file's details as one shared object, dates as numbers. Garbage collection had been ~22% of the time. Same results as parseOverviews + segmentFromHeader on 800k real lines. edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s), ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 01:13:41 +02:00
Daan MeijerandClaude Opus 5.5	cd06f96f85	Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes ... End to end on the workstation (coordinator:run, the API, edge:node, real NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges. - Prefetcher: every connection takes the next missing chunk (leased ranges first, in order); one connection per range (~26k lines/s) kept the worker waiting - executable rule: releases of bare executables (malware floods named after films and software) are dropped; they were 92% of the catalog rows of the test ranges, 21% of the segments - fragments per file (details once) and gzipped batch bodies; the JSON limit (1 MB by default) raised to 256 MB, empty strings kept (a rule count's key is '') - release_chunks.batch_from is a bigint (boneless article numbers) - tests get their own database; edge:bench --store times BatchStore: 1.3 s for a boneless range (1.24 ms per release chunk) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 01:07:11 +02:00
Daan MeijerandClaude Opus 5.5	d5f168d8d2	Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker ... - groups:add / ranges:plan / nodes:create / coordinator:run commands - the node API: lease, the range's chunks as one framed stream from the spool, batches, heartbeat, finish, release (node guard, token per node) - edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for the rest), posts a batch every 10 chunks, gives its range back on SIGTERM - releases are gathered per batch: 1.04 chunk rows per release at 10 chunks per batch (1.35 when built per XOVER chunk) - partitions are checked per batch instead of remembered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:31:06 +02:00