Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview chunks in, per batch out: releases as encoded chunks, probe
candidates, counts per rule (crossposts per owner group, obfuscated,
garbage, per-article, incomplete), a sample of what was dropped, and
fragments near the range's edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nntp (XOVER client, Xref, GROUP), lib/headers (subjects, file keys,
segments from overview lines; the yEnc path stays behind for phase 2),
lib/release (identity, random names, now in lib), lib/files (aggregator,
volume names), lib/rules (posting tools, owner by priority, release
verdicts), lib/chunks (release chunk codec). Their unit tests came along;
new ones for codec, rules and volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:
- before storing (seeding, ingest): random-named uploads of a dropped
signature (posting tool from the Message-ID, first group) or of a
release dropped after its probe aren't stored; 2% of a dropped
signature's releases are still kept and probed. Stored files of dropped
signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
password, names from files or spot, with all its probes done, and
encrypted, a container or missing its first article, is deleted and
tombstoned in dropped_releases.
Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep checked 430k noted releases in 51 minutes, nearly all without a
par2 listing or file total, where completeness can't change. Listed names
are now matched against the files themselves rather than through
binaries, which seeding mostly no longer creates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health checks of web, seed-api and seed-relay called /api/v1/stats
every 30 s, whose release statistics query kept about a core busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating a partition deadlocked with file inserts 11 times in an hour, and
each failure ended the GC sweep before its deletes. The parent is now
locked first (lock_timeout 10 s), four partitions are kept ahead, and an
error is logged while the sweep goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka answers 'no such article' after ~450 ms; a quarter of inspection
requests were for missing articles and each failed job was retried up
to three times. Files get content_type 'missing' instead; homelab also
records it from failure reports of nodes not rebuilt yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 sets, NZBs and SFVs were sniffed from their first segment and then
read whole, downloading that segment a second time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 naming needs the fingerprint inspection stores on a binary, and the
GC deleted seeded binaries (6 h unfingerprinted, 48 h stale) long before
the inspection queue reached them. binaries.awaiting_inspection protects
them for up to 7 days; the GC deletes use partial indexes so they don't
read past waiting rows. Seeding no longer creates binaries for files that
aren't inspected (~85% of a.b.ath), which only churned the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Half the live leases go to higher-priority ranges, the rest to the others,
so a slow recent gap runs beside older history. With no article
connections the article relay answers 503 instead of waiting forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured with 20 relay connections, plain and gzip both gave ~1.5 chunks/s
for 2026 a.b.ath articles; plain used many times the CPU and traffic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For 2026 a.b.ath articles Eweka's gzip replies have a long tail (p90 12.4 s
per 20k vs 3.9 s plain; 6.1k vs 10.5k headers/s). SEED_NNTP_COMPRESS turns
compression back on; plain replies are deflated for nodes asynchronously.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:plan plans every part of a group's history no range covers (it
started below the lowest known range and left a.b.ath's 13 most recent
months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
matchPar2Releases checked every same-size release_files row's set and
date (18 s per batch of body-derived binaries); now an index on
release_files (release_id, size) drops the repeated sizes first: 0.16 s.
File inserts updated their release in four row triggers; one statement
trigger now updates each release once per INSERT. Completeness is noted
in release_dirty and recomputed by gc:scheduler every 30 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_complete() matched each par2-listed file through a correlated
subquery that scanned all binaries per listed file, run twice per
inserted release_files row: one insert of a few thousand rows ran for
20+ minutes. It now reads the release's files once by release id, and
release_files refreshes once per INSERT statement.
seed:node's SEED_CONNECTIONS/--connections is now SEED_RANGES/--ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One poster adds the same 22 GB hidden '.1917-….mkv' decoy to every 7z
upload; the largest-entry rule named 840 releases after it. Entries with
a hidden path part are skipped; release:rename --hidden renames the
existing ones from their stored listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
85% of the 7z archives inspected have encrypted headers (-mhe). Their
encoded header lists the 7zAES coder, so the packed header stream is no
longer downloaded and bsdtar no longer runs for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch of a dense range (over ~5,400 files) exceeded Postgres' 65,535
bind parameters per statement, so its post failed with a 500 until the
range was given up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection lists only the entries whose headers are in the first segment
of a first volume. A walk hops from header to header, fetching one segment
per entry and volume, in its own queue (one job per collection worker) on
NNTP connections inspection isn't waiting for. content:walk-archives queues
releases inspected earlier (CONCEPT D40).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The four ranges kept failing on fresh connections too: in gzip mode Eweka
stops some chunks partway (a.b.ath 49701720000: 19,672 of 20,000
overviews, the last line cut, then silence), while the same chunk without
compression comes in 0.3 s. A failed relay chunk is now retried once
without compression on a connection of its own (dead pooled connections are
covered by the same retry).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five ranges "never answered" in 10 minutes, yet their chunk came back in
0.3 s on a fresh connection: the news server drops idle connections
without telling, and the pool's next request on one waited for the
10-minute inactivity timeout. Now: SEED_NNTP_TIMEOUT_MS 90 s (inactivity;
slow ranges keep streaming), one retry on a fresh connection, and pooled
connections idle over 60 s are replaced before use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
About 5% of seeded headers are fragments. They went through staging,
where they waited 30 minutes for parts that were never coming (the
neighbouring range is worked on by another node, often hours later) and
expired: the staging table grew to 22 GB and the NVMe part of the database
reached its 60 GB limit, which stopped leasing and the backfill. Files cut
by a range edge can still be completed by the neighbouring range, so
fragments within 50k articles of an edge go straight into binaries and
parts; the rest, files with articles missing for good, are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A range reported to its end but not finished (its node went away at
the very end) was leased again with start past end; nodes asked for
"from 49208000000 to 49207999999", got 400, and the range failed after
five leases. The orchestrator now marks such ranges done before
leasing, and runRange reports them done without fetching.
- Some old ranges come so slowly that one XOVER of 20k headers takes over
two minutes: five ranges failed on "NNTP reply timed out". The relay
pool's timeout is SEED_NNTP_TIMEOUT_MS (default 10 minutes, the node's
HTTP timeout).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New rows went into free space scattered through the 9.6 GB heap on the HDD
(left behind by inspection's updates): one random HDD read per insert, and
seed batches took a median 13 s to store. The existing table becomes the
first partition as it is (a CHECK scan, no rewrite; indexes, its foreign
key and the triggers are taken over by the partitioned parent); new
partitions of 4M ids go in the same tablespace and stay in memory while
they fill.
- Every partition has a unique file_key index; FileStore.insert looks keys
up in all partitions first (a key can't be unique across partitions).
- FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead.
- Stats count files from the partitions; seed:node waits ~20 minutes for
the API before giving a range up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed NNTP login gives the range back uncounted (no attempt used), and a
node with nothing to lease holds no connection. Replaces log-in-before-lease,
which kept idle connections and logged in again every minute while idle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seeding node that waited over two minutes for the seed API (restarting)
had its idle NNTP connection closed by its own timeout, and gave its range
back. Now the timeout runs only while a reply is awaited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No new range while the database's NVMe part or the hdd tablespace is over
BACKFILL_MAX_NVME_GB / BACKFILL_MAX_HDD_GB; nodes ask again a minute later.
seed:status says when leasing is paused. The NVMe part was at 47 GB of 50
with a plan for all of a.b.ath queued.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
lease without a batch for 15 min passes to the next node and resumes at
next_id), batch (stores files and fragments, moves next_id), finish,
release, plan (from below live indexing / the backfill / earlier plans
down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
status) behind a node-token middleware; JSON bodies up to 64 MB after
gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
connections, gzip batches, retries for network errors and 5xx, drops a
range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
and seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lib/nntp/xover_client: a minimal one-connection client for seed:range
with XFEATURE COMPRESS GZIP (a zlib stream after "224 … [COMPRESS=GZIP]",
then a "." line; 420 for an empty range). About 6x less header traffic.
Overviews come out with the field names nntp-js uses.
- SfvService.claimAll: one sfv_entries lookup per stored batch.
- Fragments go through the regular ingest path beside fetching (at most
two batches in flight); a failure fails the run at the end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seeding node fetches a range over one NNTP connection, parses headers
with the live rules, groups them into files in memory (lib/aggregate) and
stores per complete file its release, the file and a completed binary
(SeedService), instead of one parts row per article. What doesn't complete
within the window goes through the regular ingest path as fragments.
`--dry-run` only fetches and aggregates, to measure a node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version matched extensions in search_text, which was wrong and
slow: file names enter search_text as stems without extension (so plainly
posted .mkv files were missed), and matching the TOASTed text took 3-4 s
for common types.
- Migration …130: releases.extensions text[] (two to four characters with
a letter; .7z.001 and .r00 count as 7z and rar), kept current by triggers
on files (name, release) and on releases (archive listing), filled with
one sequential pass over files, with a GIN index.
- Search: ext matches with `extensions && ?`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- API: min_size and max_size in MB, ext as a comma-separated list. An
extension matches as a whole word in search_text (file and entry names,
dots as spaces), so "iso" finds .iso files but not "ISOLATION", which a
plain search word matched as a substring. The trigram index serves it.
- Web UI: a size range and an extension field in the filter row, kept in
the URL like the other filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Of 5,000 waiting inspection jobs, 3,212 were queued only because their
stem looked random, and 2,044 of those were later RAR volumes
(xyz.part47.rar, .r00) or par2 recovery volumes (.vol03+04.par2). Their
first segment adds nothing: the set is listed from its first volume and
named from its index .par2, as for readable names already. Each one cost a
~700 KB download; inspection pulled ~85 MB/s and fell behind the backfill.
- isCandidate() rejects later volumes (isLaterVolume) before the random-
name rule.
- worker:file only queues files that are candidates, so the ~30% that
inspection skips anyway no longer take queue slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Crossposts, reposts and backfill ranges keep bringing articles of files
that worker:file already created. Ingest inserted their header parts again,
and GC's copied-parts pass (which searched all binaries for parts on every
batch) removed only 150k-450k per sweep: parts grew to 68M rows, 56M of
them belonging to finished files, and the 23 GB table paused the backfill.
- Ingest skips header parts (no per-part CRC) for binaries with a file_id.
- GC drops copied parts with a cursor over binaries, 2000 per statement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
