Commit Graph
3 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 babd3d781d Rebuild: test the corrupt-chunk report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 07:48:07 +02:00
Daan MeijerandClaude Opus 5.5 cd06f96f85 Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.

- Prefetcher: every connection takes the next missing chunk (leased
  ranges first, in order); one connection per range (~26k lines/s) kept
  the worker waiting
- executable rule: releases of bare executables (malware floods named
  after films and software) are dropped; they were 92% of the catalog rows
  of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
  limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
  count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
  1.3 s for a boneless range (1.24 ms per release chunk)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:07:11 +02:00
Daan MeijerandClaude Opus 5.5 d5f168d8d2 Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
  spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
  the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
  chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 00:31:06 +02:00