probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Range 161820 failed with "doesn't inflate, also fetched plain" though
the chunk was likely the compressed copy again: in flight, or spooled
before the prefetcher's targets were refreshed, when the plain flag was
set. The range now fails only on the third report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges with a chunk or two left take fewer than perRange connections:
with loops/perRange + 2 ranges, 3.2 of 40 loops were idle on average
over 15 minutes (up to 9.5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Range 154218 (a.b.hdtv.tv-episodes) looped 13 times on HTTP 413: a
500k-article batch over the server's 256 MB body limit, though under
100k files. On a 413 the worker gives the range back and posts that
range's batches at half the articles and files from then on (again on
another 413, down to 1/64); stored batches stay. The message says how
large the batch was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some groups' chunks take 20+ minutes while still coming in (range
154153: 17 MB in 22 minutes); the stream's 15-minute wait failed their
workers (two in 15 minutes). Stalled requests are given up separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 90 s a chunk that always takes longer would be given up forever
(range 155437's worker gave up waiting on its stream). Each deadline a
chunk hits doubles its next one (90, 180, 360 ... up to 16x), reset
once it comes in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 180 s, 60 chunks were given up in 15 minutes: each held a connection
3 minutes, ~30% of the 40 connections' time, while finished chunks
took 24 s on average (Eweka slow this afternoon).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.
The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 2 connections per range the prefetcher needs 20+ ranges with
chunks left; fully spooled ranges counted as ready (SPOOL_AHEAD_PER_LEASE)
but take no connection, so on small ranges 30 of 40 connections were
busy (13-15 ranges). Fresh ranges now also fill up to loops/perRange + 2
ranges being fetched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.
Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dockerfile with two targets: app (compiled AdonisJS, production
dependencies, Node 24.16 as in CI) and edge (the Rust worker, cargo
caches as build mounts). deploy/compose.yaml, project usenet-rebuild:
postgres (data on the HDD, catalog tablespace on the NVMe), migrate,
web, coordinator, edge; edge-ts and ace behind profiles.
deploy/deploy.sh (~/bin/deploy-usenet-rebuild on homelab): fetches the
commit (must be on rebuild), builds, migrates, recreates web,
coordinator and edge (the worker stopped first), never postgres.
CI deploys a push to rebuild that passes: the deploy key's forced
command passes "deploy-rebuild <sha>" on. A push to master no longer
deploys from this branch's workflow. deploy/README.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.
Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a second: pool connections with a command in flight, loops
fetching/writing/idle; per request GROUP time, XOVER time to the status
line and the transfer, transfers with a silence over 5 s. The old
"connections busy" (fetch time of the chunks finished in the minute)
said 59-75% while all 40 connections were in flight all the time:
dropped.
Homelab: 39-40 of 40 in flight, no idle loops; the status line comes at
once, the whole 8-35 s is the transfer, with data flowing (few gaps
over 5 s): ~3 MB per chunk at 0.1-0.4 MB/s per connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each chunk fetched with and without XFEATURE COMPRESS GZIP on two
connections at the same moment. 30 chunks on homelab (24 bottom-500,
6 deferred): medians 2.9 s compressed, 3.2 s plain; 59 vs 350 bytes
per article. Slow fetches (27-52 s) hit either mode, rarely both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka stopped answering on all 40 of the coordinator's long-lived
connections (quiet sockets; a fresh connection fetched the same chunk in
20 s): with a 10-minute timeout the prefetcher fetched one chunk a
minute. The timeout (silence, not duration) is 2 minutes now: 365-411k
articles/s after the restart. tools/probe_chunk.ts times a chunk
compressed and plain.
README and REBUILD.md: edge-node, the Rust worker, is the one running
(8x less CPU per header, 20x less memory than edge:node in production).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/edge-bench-rs/src/bin/edge-node.rs: leases ranges, reads the
coordinator's framed stream (one thread reads and inflates a frame
ahead, libdeflate), indexes with the shared RangeIndexer and posts
batches as gzipped JSON in order on a thread of its own (one batch may
wait). As commands/edge_node.ts: retries for network trouble and 5xx,
409 = lease lost, a batch once 500k articles or 100k files wait, a
chunk that doesn't inflate reported (fetched plain next time), the
range given back on SIGTERM/SIGINT, --max-mbit shared by the threads;
plus a heartbeat when no batch went out for a minute. One range per
thread (--threads, default one per core).
End to end against the web server (a.b.models 2017215-4017214 from
the capture, spooled): the server stored the same from both workers
(38,228 releases with every column, chunks, 481 candidates, rule
counts, 57 fragments with their JSON). Rust 9 s / 5.1 CPU-s, TypeScript
17 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/lib.rs: the indexing modules as a library, for edge-bench-rs and
the coming edge-node worker.
- Rules as a lease brings them (priorities, dropped signatures and
tools, per-article parts, sample share, executables) instead of
DEFAULT_RULES hard-coded; RangeIndexer takes them and a resume point.
- Fragments carry everything the TypeScript Fragment has (poster,
subject, groups, release): the server stores them for the merger.
- wire.rs: a batch as toWire makes it (camelCase JSON, payloads in
base64). edge-bench-rs --wire writes it; check.ts --wire compares it
with toWire of the TypeScript batch: identical on models-a and
ath-2013.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BatchStore.store held the range's row (SELECT … FOR UPDATE) for the
whole batch transaction (15-20 s on the HDD), so the prefetcher's
spooled_chunks updates and the heartbeats of that range waited on it
(0.5 s on average): the prefetcher's NNTP connections sat idle. Now an
advisory lock per range serializes its batches, and the resume point is
checked again in the final, conditional update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TABLESAMPLE SYSTEM (0.5) read 30 s of random pages on the HDD at 7M
releases, growing with the catalog; a refresh that never finished kept
the header's totals 3.5 hours old (it blocked every later refresh).
Now TABLESAMPLE SYSTEM_ROWS (5000) (tsm_system_rows), every 5 minutes,
and a refresh is given up after 2 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A picture group's 500k articles made batches over the server's 256 MB
body limit: HTTP 413, the range given back and leased again in a loop
(148597: 39 times in 30 minutes). The worker now also posts once
100,000 completed or edge-cut files wait (RangeIndexer.pending).
A stats refresh that failed kept the previous totals silently; it
logs the error now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A picture group's batch has 30-55k release keys; an advisory lock each
ran Postgres out of shared memory (max_locks_per_transaction), and those
batches failed with HTTP 500 (the ranges went back, nothing stored).
A chunk only matches releases of its range's group, so one lock per
group serializes exactly the writers that could both start an upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)
A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- releases.complete is computed with every batch (files >= the
counter's total) instead of only when a release settles after 6 quiet
hours: the 'complete' filter matched nothing in a fresh catalog.
- A release's file total comes from any counter form (counterTotal:
'(01/10)', 'File 27 of 132', …), not only [i/M]; the post rule's key
keeps the [i/M] total, so keys are unchanged. Both implementations.
- groups.tail: a group's live tail can be paused (its history and owner
priority stay), e.g. to let other groups' history go first.
- Stats: history by articles, and the current pass (history ranges
before those deferred 2000 priorities behind) as its own figure; the
header showed 0.1% of ranges counting 145k deferred ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
So a cover 'x.jpg' joins 'x.part01.rar' in a stem-named post; an image
set like 'IMG_008' keys as the stem 'IMG'. Both implementations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picture groups made a release per image: 501k single-file releases in
the first 24 ranges of the bottom 500 groups (a.b.models 385k), ~120M
for all of them. Counters written out ('File 027 of 132', '[12of42]',
'104 of 472', '[2239-6389]', '02/17', '0097-1574'; only when the first
number is at most the second, so '[1948-12]' stays) and per-file sizes
('76.88 MB', '335555 Bytes') kept every subject unique.
- subjectRemainder removes those counters and sizes, and empty brackets.
- A subject-named release is bounded by its counter's total
(counterTotal: the first form found): uploads a poster names alike
stay apart, an upload's files (par2, nfo) together.
- Images and audio tracks don't take the poster-tag rule: the subject
names the set or album. An image set without a counter: the image's
name up to its last number, without numbers.
- Documents and comics (pdf, epub, mobi, azw3, djvu, cbr, cbz, cb7, chm,
lit, doc, docx) are a release each: collections post hundreds under
one subject ('cw-horror&mystery').
releases:regroup --group re-keys a group's indexed releases together:
cheerleaders 55,741 -> 3,165 releases, models 385,298 -> 40,313, German
audiobooks 4,106 -> 1,724, playstation 3,309 -> 2,345; comics and
e-books a release per item; boneless unchanged. Ported to edge-bench-rs;
check.ts: identical on all captures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release 145637 held four albums: the poster puts '[KiLL.dA.FuCkIn.
EaStErRaBbIt]----' before every upload, and the text outside the quotes
(when it names a release) was the key with the poster. The same made
one release of a poster's albums under '(PowBeertje80 Posted)' (a post
id, with the same [i/M]), of masterposts' films, and of one volume of
every set in a DVD9 post.
Now, when the file's stem describes something itself (12+ letters and
digits, two words of 3+ letters) and the text outside the quotes shares
no word with it (4+ letters as part of the stem, 3-letter words and
numbers whole; boilerplate like www/net/posted not counted; no 5+ digit
per-post id), that text is taken for the poster's tag and the stem
says which release the file is. Albums of track files ('FTA … Ketchup'),
'DIE_ZEIT_35' with '35_China_dl', 'FTD769997' posts, scene names with
abbreviated file names ('grp-sm') and generic sidecars ('file_id.diz')
keep their grouping.
releases:regroup re-keys a sample of the catalog with the current rule:
on homelab's test data 145637 → 4 albums, 111566 → 17 albums of 27
files, the rest of the subject releases that change mostly lose an ad
image ('Flyer-download-2-Day.gif'); stem releases unchanged. Ported to
edge-bench-rs: check.ts still finds all captures identical (ath 2013:
16 → 76 releases on both sides).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's page (search, sort, filters, poster filter, detail drawer,
NZB download; no framework, Usenet text only as textContent) on the new
catalog. Gone for now, as voyager 2 doesn't know them yet: passwords,
encrypted containers, spots, par2 lists, archive contents, the extension
filter (phase 2). New: a category filter, 'Collecting' for releases not
settled yet, history and live-tail progress in the header, a release's
files from its chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api (Newznab: caps, search, tvsearch, movie, music, book, get) for
Prowlarr, Sonarr and Radarr: voyager 1's controller and XML on the new
catalog; readable releases not known to miss files; NEWZNAB_API_KEY.
- /api/v1/releases (q, cat, readable, complete, source, poster, size,
sort, paging), /api/v1/releases/:id with its files (decoded from the
chunks; NzbService.files, shared with the NZB), /api/v1/stats
(estimates, plus history and tail progress), /health. Open, as voyager
1's catalog; the web interface uses it.
- release_category(name): voyager 1's categorizer without the archive
contents, a generated column with a (category, posted_at) index.
- @adonisjs/static for the web interface (next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rules:signatures publishes a rule set version whose dropped signatures
come from a file of tool<TAB>group lines (--add to add to the current
ones). database/data/voyager1_signatures.tsv: voyager 1's dropped ones
(>=100 probes, <=0.5% named), with their probe counts; the two builds
name the posting tools the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Connections fetch chunks round-robin from several starting points of one
group at once, so each era sees the same server load; per era: seconds
per command (mean, median), articles/s per connection, bytes per article.
On a.b.boneless (2 connections, 10 min): 2008 24.9k, 2011 28.0k, 2019
22.6k, 2026 17.9k articles/s per connection; medians 2.3-3.0 s, means set
by a tail of slow commands (up to 41 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libdeflate inflates a chunk whole, in half the time of zlib-rs (1.09 ->
0.51 s per boneless range); memchr finds line ends and tabs with SIMD.
The per-article timers are gone. Boneless 2026 ~700k headers/s, ath 2013
~920k; all three captures still identical to the TypeScript.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each with its regex/chrono version as the reference in tests; check.ts
still finds all three captures identical. Boneless 2026 ~500k headers/s,
ath 2013 ~650k.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suffix and remainder regexes were most of a new file's description
(333k on boneless 2026): now plain byte matching, with the regex versions
kept as the reference in a test of tricky names and subjects. mimalloc
for the many small strings. Boneless 2026: 360k -> 495k headers/s;
check.ts still finds the outputs identical on all three captures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The RangeIndexer path of edge:bench (inflate, scan, owner rule, segments,
file aggregation, cut-off, rules, release identity, chunk encoding v2,
probe candidates) as a Rust binary over a captured range, with JS
semantics where they matter (\s, trim, \b, parseInt, lastIndexOf, UTF-16
lengths). check.ts runs the TypeScript RangeIndexer on the same capture
and compares every output: identical on all three captures (releases and
their decoded payloads, samples, candidates, rule counts, fragments),
after mapping Date headers ending in UTC to GMT (chrono follows RFC
5322; Date.parse takes UTC, a quarter of a.b.boneless). Compressed
payloads differ by <0.1% in size (zstd framing), not in content.
First run on boneless 2026: 360k headers/s and 228 MB peak (TypeScript:
~217k/s, ~0.7-0.8 GB).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
