One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.
looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.
The prefetcher's corpus fill skips groups that aren't active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.
`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lease bounds alone don't select a window: the history planner's oldest
buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026
history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the
2016 corpus window sat at voyager 2's leftover priorities (done when the window
was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next
and the prefetcher's run-ahead take capture ranges only; the run-ahead now also
keeps to the lease bounds. Test: a group with history at 0 only isn't leased,
the history range after the window isn't handed out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 512 candidates were the lowest ids at the lowest priority; a group's
ranges have consecutive ids, so once a window's small groups were stored
they were all one group's, two targets on two connections (10 ranges an
hour). Ordered by the per-group rank now: the window spans 85 groups.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prefetcher fetched capture ranges into the corpus 16 at a time in id
order, and ids run group by group: the 16 were one or two groups', the
per-group connection limit (4) let 4-8 loops work and left the rest
idle, ~30k articles/s for the first batch of 19,955 capture ranges. Now
at most perGroup / perRange (2) of a group are taken, so the 16 come
from 8+ groups and every loop gets work (and the pool asks for more).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges with a chunk or two left take fewer than perRange connections:
with loops/perRange + 2 ranges, 3.2 of 40 loops were idle on average
over 15 minutes (up to 9.5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 90 s a chunk that always takes longer would be given up forever
(range 155437's worker gave up waiting on its stream). Each deadline a
chunk hits doubles its next one (90, 180, 360 ... up to 16x), reset
once it comes in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 2 connections per range the prefetcher needs 20+ ranges with
chunks left; fully spooled ranges counted as ready (SPOOL_AHEAD_PER_LEASE)
but take no connection, so on small ranges 30 of 40 connections were
busy (13-15 ranges). Fresh ranges now also fill up to loops/perRange + 2
ranges being fetched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.
Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.
Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a second: pool connections with a command in flight, loops
fetching/writing/idle; per request GROUP time, XOVER time to the status
line and the transfer, transfers with a silence over 5 s. The old
"connections busy" (fetch time of the chunks finished in the minute)
said 59-75% while all 40 connections were in flight all the time:
dropped.
Homelab: 39-40 of 40 in flight, no idle loops; the status line comes at
once, the whole 8-35 s is the transfer, with data flowing (few gaps
over 5 s): ~3 MB per chunk at 0.1-0.4 MB/s per connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)
A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed
XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so
one process tops out around 60 MB/s: less than 50 connections bring
(~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with
id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges
and settles. Two shards of 5 connections fetched what one of 10 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator logs a line a minute: chunks fetched, articles/s,
seconds a chunk, how busy its connections were. It showed the
workstation's runs are bound by its 100 Mbit line now (10 connections
99% busy, ~89 Mbit/s: compressed XOVER is ~69 B an article on recent
boneless), not by the code.
The check that a compressed reply is complete (inflating it) ran on the
coordinator's event loop, ~60 ms a chunk; now on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator ran the prefetch tick in the same loop as the merger
(up to 30 s a minute): its connections went idle once their targets
were fetched, and the first leases after a start found nothing spooled.
The prefetcher now ticks in a loop of its own.
The prefetcher took the first planned ranges with nothing spooled, even
ones with nothing left to fetch (next_id past the end): two such ranges
(edge:bench's, set back to planned) kept it from fetching anything. It
and the lease query now skip those, and edge:bench marks its range done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
