At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.
Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.
Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api (Newznab: caps, search, tvsearch, movie, music, book, get) for
Prowlarr, Sonarr and Radarr: voyager 1's controller and XML on the new
catalog; readable releases not known to miss files; NEWZNAB_API_KEY.
- /api/v1/releases (q, cat, readable, complete, source, poster, size,
sort, paging), /api/v1/releases/:id with its files (decoded from the
chunks; NzbService.files, shared with the NZB), /api/v1/stats
(estimates, plus history and tail progress), /health. Open, as voyager
1's catalog; the web interface uses it.
- release_category(name): voyager 1's categorizer without the archive
contents, a generated column with a (category, posted_at) index.
- @adonisjs/static for the web interface (next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.
".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nzb (buildNzb, parseNzb) from the current app; NzbService builds a
release's NZB from its chunks; GET /api/v1/releases?q= (every word of the
name) and GET /api/v1/releases/:id/nzb behind the api guard. A test stores
a batch, finds the release and reads its NZB back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:
- before storing (seeding, ingest): random-named uploads of a dropped
signature (posting tool from the Message-ID, first group) or of a
release dropped after its probe aren't stored; 2% of a dropped
signature's releases are still kept and probed. Stored files of dropped
signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
password, names from files or spot, with all its probes done, and
encrypted, a container or missing its first article, is deleted and
tombstoned in dropped_releases.
Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health checks of web, seed-api and seed-relay called /api/v1/stats
every 30 s, whose release statistics query kept about a core busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SeedService looked up every file of a batch across all files partitions
after inserting it (15% of Postgres' active time); the merge job read
every release on each GC sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 naming needs the fingerprint inspection stores on a binary, and the
GC deleted seeded binaries (6 h unfingerprinted, 48 h stale) long before
the inspection queue reached them. binaries.awaiting_inspection protects
them for up to 7 days; the GC deletes use partial indexes so they don't
read past waiting rows. Seeding no longer creates binaries for files that
aren't inspected (~85% of a.b.ath), which only churned the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For 2026 a.b.ath articles Eweka's gzip replies have a long tail (p90 12.4 s
per 20k vs 3.9 s plain; 6.1k vs 10.5k headers/s). SEED_NNTP_COMPRESS turns
compression back on; plain replies are deflated for nodes asynchronously.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
matchPar2Releases checked every same-size release_files row's set and
date (18 s per batch of body-derived binaries); now an index on
release_files (release_id, size) drops the repeated sizes first: 0.16 s.
File inserts updated their release in four row triggers; one statement
trigger now updates each release once per INSERT. Completeness is noted
in release_dirty and recomputed by gc:scheduler every 30 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection lists only the entries whose headers are in the first segment
of a first volume. A walk hops from header to header, fetching one segment
per entry and volume, in its own queue (one job per collection worker) on
NNTP connections inspection isn't waiting for. content:walk-archives queues
releases inspected earlier (CONCEPT D40).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A range reported to its end but not finished (its node went away at
the very end) was leased again with start past end; nodes asked for
"from 49208000000 to 49207999999", got 400, and the range failed after
five leases. The orchestrator now marks such ranges done before
leasing, and runRange reports them done without fetching.
- Some old ranges come so slowly that one XOVER of 20k headers takes over
two minutes: five ranges failed on "NNTP reply timed out". The relay
pool's timeout is SEED_NNTP_TIMEOUT_MS (default 10 minutes, the node's
HTTP timeout).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:node logs in to NNTP before taking a range and keeps the connection
across ranges (replaced after 2 minutes idle or a failed range). A node
that can't reach its news server takes no range: two ranges were set
aside as failed while an office node's tunnel was being set up
(ECONNREFUSED 127.0.0.1:1563, five leases each). After an IP-address
limit refusal it waits 5 minutes.
- NNTP_TLS_SERVERNAME: the hostname the TLS certificate is checked
against when NNTP_HOST is a tunnel ("Hostname/IP does not match
certificate's altnames: Host: localhost").
- seed-node/README: the IP-address limit and tunnelling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
lease without a batch for 15 min passes to the next node and resumes at
next_id), batch (stores files and fragments, moves next_id), finish,
release, plan (from below live indexing / the backfill / earlier plans
down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
status) behind a node-token middleware; JSON bodies up to 64 MB after
gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
connections, gzip batches, retries for network errors and 5xx, drops a
range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
and seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
