A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.
`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lease bounds alone don't select a window: the history planner's oldest
buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026
history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the
2016 corpus window sat at voyager 2's leftover priorities (done when the window
was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next
and the prefetcher's run-ahead take capture ranges only; the run-ahead now also
keeps to the lease bounds. Test: a group with history at 0 only isn't leased,
the history range after the window isn't handed out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 180 s, 60 chunks were given up in 15 minutes: each held a connection
3 minutes, ~30% of the 40 connections' time, while finished chunks
took 24 s on average (Eweka slow this afternoon).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.
The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.
Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.
Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka stopped answering on all 40 of the coordinator's long-lived
connections (quiet sockets; a fresh connection fetched the same chunk in
20 s): with a 10-minute timeout the prefetcher fetched one chunk a
minute. The timeout (silence, not duration) is 2 minutes now: 365-411k
articles/s after the restart. tools/probe_chunk.ts times a chunk
compressed and plain.
README and REBUILD.md: edge-node, the Rust worker, is the one running
(8x less CPU per header, 20x less memory than edge:node in production).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
