A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search matches a word in the name or the inner name (trigram index on
inner_name, partial, built concurrently); "Readable names" includes
releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
Contents (per probed article its yEnc name, kind, type, encryption and
the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
with an inner name is included, titled by it (without its extension).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)
A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- releases.complete is computed with every batch (files >= the
counter's total) instead of only when a release settles after 6 quiet
hours: the 'complete' filter matched nothing in a fresh catalog.
- A release's file total comes from any counter form (counterTotal:
'(01/10)', 'File 27 of 132', …), not only [i/M]; the post rule's key
keeps the [i/M] total, so keys are unchanged. Both implementations.
- groups.tail: a group's live tail can be paused (its history and owner
priority stay), e.g. to let other groups' history go first.
- Stats: history by articles, and the current pass (history ranges
before those deferred 2000 priorities behind) as its own figure; the
header showed 0.1% of ranges counting 145k deferred ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api (Newznab: caps, search, tvsearch, movie, music, book, get) for
Prowlarr, Sonarr and Radarr: voyager 1's controller and XML on the new
catalog; readable releases not known to miss files; NEWZNAB_API_KEY.
- /api/v1/releases (q, cat, readable, complete, source, poster, size,
sort, paging), /api/v1/releases/:id with its files (decoded from the
chunks; NzbService.files, shared with the NZB), /api/v1/stats
(estimates, plus history and tail progress), /health. Open, as voyager
1's catalog; the web interface uses it.
- release_category(name): voyager 1's categorizer without the archive
contents, a generated column with a (category, posted_at) index.
- @adonisjs/static for the web interface (next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hourly in coordinator:run: a release unchanged for 6 hours is settled,
complete when every file its [i/N] counter announced was found, unknown
without a counter (phase 2 reads par2 sets for those). Partial index on
unsettled releases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nzb (buildNzb, parseNzb) from the current app; NzbService builds a
release's NZB from its chunks; GET /api/v1/releases?q= (every word of the
name) and GET /api/v1/releases/:id/nzb behind the api guard. A test stores
a batch, finds the release and reads its NZB back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:
- before storing (seeding, ingest): random-named uploads of a dropped
signature (posting tool from the Message-ID, first group) or of a
release dropped after its probe aren't stored; 2% of a dropped
signature's releases are still kept and probed. Stored files of dropped
signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
password, names from files or spot, with all its probes done, and
encrypted, a container or missing its first article, is deleted and
tombstoned in dropped_releases.
Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep checked 430k noted releases in 51 minutes, nearly all without a
par2 listing or file total, where completeness can't change. Listed names
are now matched against the files themselves rather than through
binaries, which seeding mostly no longer creates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health checks of web, seed-api and seed-relay called /api/v1/stats
every 30 s, whose release statistics query kept about a core busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating a partition deadlocked with file inserts 11 times in an hour, and
each failure ended the GC sweep before its deletes. The parent is now
locked first (lock_timeout 10 s), four partitions are kept ahead, and an
error is logged while the sweep goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka answers 'no such article' after ~450 ms; a quarter of inspection
requests were for missing articles and each failed job was retried up
to three times. Files get content_type 'missing' instead; homelab also
records it from failure reports of nodes not rebuilt yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 naming needs the fingerprint inspection stores on a binary, and the
GC deleted seeded binaries (6 h unfingerprinted, 48 h stale) long before
the inspection queue reached them. binaries.awaiting_inspection protects
them for up to 7 days; the GC deletes use partial indexes so they don't
read past waiting rows. Seeding no longer creates binaries for files that
aren't inspected (~85% of a.b.ath), which only churned the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Half the live leases go to higher-priority ranges, the rest to the others,
so a slow recent gap runs beside older history. With no article
connections the article relay answers 503 instead of waiting forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:plan plans every part of a group's history no range covers (it
started below the lowest known range and left a.b.ath's 13 most recent
months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
matchPar2Releases checked every same-size release_files row's set and
date (18 s per batch of body-derived binaries); now an index on
release_files (release_id, size) drops the repeated sizes first: 0.16 s.
File inserts updated their release in four row triggers; one statement
trigger now updates each release once per INSERT. Completeness is noted
in release_dirty and recomputed by gc:scheduler every 30 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_complete() matched each par2-listed file through a correlated
subquery that scanned all binaries per listed file, run twice per
inserted release_files row: one insert of a few thousand rows ran for
20+ minutes. It now reads the release's files once by release id, and
release_files refreshes once per INSERT statement.
seed:node's SEED_CONNECTIONS/--connections is now SEED_RANGES/--ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch of a dense range (over ~5,400 files) exceeded Postgres' 65,535
bind parameters per statement, so its post failed with a 500 until the
range was given up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
About 5% of seeded headers are fragments. They went through staging,
where they waited 30 minutes for parts that were never coming (the
neighbouring range is worked on by another node, often hours later) and
expired: the staging table grew to 22 GB and the NVMe part of the database
reached its 60 GB limit, which stopped leasing and the backfill. Files cut
by a range edge can still be completed by the neighbouring range, so
fragments within 50k articles of an edge go straight into binaries and
parts; the rest, files with articles missing for good, are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A range reported to its end but not finished (its node went away at
the very end) was leased again with start past end; nodes asked for
"from 49208000000 to 49207999999", got 400, and the range failed after
five leases. The orchestrator now marks such ranges done before
leasing, and runRange reports them done without fetching.
- Some old ranges come so slowly that one XOVER of 20k headers takes over
two minutes: five ranges failed on "NNTP reply timed out". The relay
pool's timeout is SEED_NNTP_TIMEOUT_MS (default 10 minutes, the node's
HTTP timeout).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New rows went into free space scattered through the 9.6 GB heap on the HDD
(left behind by inspection's updates): one random HDD read per insert, and
seed batches took a median 13 s to store. The existing table becomes the
first partition as it is (a CHECK scan, no rewrite; indexes, its foreign
key and the triggers are taken over by the partitioned parent); new
partitions of 4M ids go in the same tablespace and stay in memory while
they fill.
- Every partition has a unique file_key index; FileStore.insert looks keys
up in all partitions first (a key can't be unique across partitions).
- FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead.
- Stats count files from the partitions; seed:node waits ~20 minutes for
the API before giving a range up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed NNTP login gives the range back uncounted (no attempt used), and a
node with nothing to lease holds no connection. Replaces log-in-before-lease,
which kept idle connections and logged in again every minute while idle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No new range while the database's NVMe part or the hdd tablespace is over
BACKFILL_MAX_NVME_GB / BACKFILL_MAX_HDD_GB; nodes ask again a minute later.
seed:status says when leasing is paused. The NVMe part was at 47 GB of 50
with a plan for all of a.b.ath queued.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
lease without a batch for 15 min passes to the next node and resumes at
next_id), batch (stores files and fragments, moves next_id), finish,
release, plan (from below live indexing / the backfill / earlier plans
down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
status) behind a node-token middleware; JSON bodies up to 64 MB after
gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
connections, gzip batches, retries for network errors and 5xx, drops a
range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
and seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seeding node fetches a range over one NNTP connection, parses headers
with the live rules, groups them into files in memory (lib/aggregate) and
stores per complete file its release, the file and a completed binary
(SeedService), instead of one parts row per article. What doesn't complete
within the window goes through the regular ingest path as fragments.
`--dry-run` only fetches and aggregates, to measure a node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version matched extensions in search_text, which was wrong and
slow: file names enter search_text as stems without extension (so plainly
posted .mkv files were missed), and matching the TOASTed text took 3-4 s
for common types.
- Migration …130: releases.extensions text[] (two to four characters with
a letter; .7z.001 and .r00 count as 7z and rar), kept current by triggers
on files (name, release) and on releases (archive listing), filled with
one sequential pass over files, with a GIN index.
- Search: ext matches with `extensions && ?`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- API: min_size and max_size in MB, ext as a comma-separated list. An
extension matches as a whole word in search_text (file and entry names,
dots as spaces), so "iso" finds .iso files but not "ISOLATION", which a
plain search word matched as a substring. The trigram index serves it.
- Web UI: a size range and an extension field in the filter row, kept in
the URL like the other filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Crossposts, reposts and backfill ranges keep bringing articles of files
that worker:file already created. Ingest inserted their header parts again,
and GC's copied-parts pass (which searched all binaries for parts on every
batch) removed only 150k-450k per sweep: parts grew to 68M rows, 56M of
them belonging to finished files, and the 23 GB table paused the backfill.
- Ingest skips header parts (no per-part CRC) for binaries with a file_id.
- GC drops copied parts with a cursor over binaries, 2000 per statement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
