Rebuild: coordinator:run --shard=i/n splits prefetching over processes
The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so one process tops out around 60 MB/s: less than 50 connections bring (~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges and settles. Two shards of 5 connections fetched what one of 10 did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
462da5c225
commit
69f1cc63b2
3 files changed
+30
-11
No files matched your search
+9
-4
@@ -46,14 +46,14 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||||
account; the gzip stream is passed on as is).
|
||||
|
||||
**Prefetch**: the coordinator fetches the next ranges ahead of the
|
||||
workers, in chunks of 50 k lines and in order, into a spool on disk
|
||||
workers, in chunks of 100 k lines and in order, into a spool on disk
|
||||
(~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
|
||||
sequentially). It keeps about two ranges ahead per active lease, capped
|
||||
by size (default 10 GB), and fetches with all its header connections all
|
||||
the time, independent of how fast workers are. A lease goes to a range
|
||||
that is already (partly) spooled; a worker reads its chunks from the
|
||||
spool, waiting only for chunks still being fetched, and falls back to a
|
||||
direct relay fetch for a range that isn't spooled. A worker gets its
|
||||
that is already (partly) spooled, else to the next planned one (leased
|
||||
ranges are fetched first); a worker reads its chunks from the spool,
|
||||
waiting only for chunks still being fetched. A worker gets its
|
||||
whole range in one streaming request (`GET /ranges/:id/stream`): the
|
||||
spooled chunks in order, waiting for those still being fetched. It still
|
||||
parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
|
||||
@@ -64,6 +64,11 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||||
range is leased again later.
|
||||
Order: oldest era first (2008–2018 holds 63% of the readable files on
|
||||
17% of the lines), then forward.
|
||||
One coordinator process handles ~60 MB/s of compressed XOVER (its event
|
||||
loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over
|
||||
processes with `coordinator:run --shard=i/n` (ranges with id % n = i − 1,
|
||||
each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and
|
||||
settles).
|
||||
|
||||
2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
|
||||
2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013,
|
||||
|
||||
Reference in new issue
Block a user