A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka stopped answering on all 40 of the coordinator's long-lived
connections (quiet sockets; a fresh connection fetched the same chunk in
20 s): with a 10-minute timeout the prefetcher fetched one chunk a
minute. The timeout (silence, not duration) is 2 minutes now: 365-411k
articles/s after the restart. tools/probe_chunk.ts times a chunk
compressed and plain.
README and REBUILD.md: edge-node, the Rust worker, is the one running
(8x less CPU per header, 20x less memory than edge:node in production).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
