Rebuild: coordinator:run --shard=i/n splits prefetching over processes

The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed
XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so
one process tops out around 60 MB/s: less than 50 connections bring
(~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with
id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges
and settles. Two shards of 5 connections fetched what one of 10 did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 03:36:24 +02:00
1 parent 462da5c225
commit 69f1cc63b2
3 files changed
+30 -11

No files matched your search

+9 -4
View File
@@ -46,14 +46,14 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
account; the gzip stream is passed on as is).
**Prefetch**: the coordinator fetches the next ranges ahead of the
workers, in chunks of 50 k lines and in order, into a spool on disk
workers, in chunks of 100 k lines and in order, into a spool on disk
(~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
sequentially). It keeps about two ranges ahead per active lease, capped
by size (default 10 GB), and fetches with all its header connections all
the time, independent of how fast workers are. A lease goes to a range
that is already (partly) spooled; a worker reads its chunks from the
spool, waiting only for chunks still being fetched, and falls back to a
direct relay fetch for a range that isn't spooled. A worker gets its
that is already (partly) spooled, else to the next planned one (leased
ranges are fetched first); a worker reads its chunks from the spool,
waiting only for chunks still being fetched. A worker gets its
whole range in one streaming request (`GET /ranges/:id/stream`): the
spooled chunks in order, waiting for those still being fetched. It still
parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
@@ -64,6 +64,11 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
range is leased again later.
Order: oldest era first (2008–2018 holds 63% of the readable files on
17% of the lines), then forward.
One coordinator process handles ~60 MB/s of compressed XOVER (its event
loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over
processes with `coordinator:run --shard=i/n` (ranges with id % n = i − 1,
each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and
settles).
2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013,