59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merges delete releases, so the count of a key's releases fell behind its
numbers (83 releases up to key:129): the 20 numbers tried were all taken
and the range's batch failed for good. Spot releases (spot:key) are not
numbered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The incremental judge counted each changed release's files twice, each
index entry a heap fetch: 24-32 s a chunk. file_hashes by (hash16k,
size, release_id) replaces (hash16k); the other releases of a file are
judged only when the chunk brings it to 20.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The judge's fake rule reads all of it every round: 17-18 min on the HDD,
~3 on the NVMe. Migration 4600 is a no-op now; 4800 moves it back where
4600 ran. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round took 22-29 min an hour: every match went through the insert's
conflict check (60-150 s a rule), scraped and locked each scanned all of
releases. The spot inspector's releases now add their poster to posters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober's output, appended and read a release at a time: 40% of the
NVMe's growth while probing. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jsonb can't hold \u0000, and ON CONFLICT can't take a row twice in one
statement: either failed the whole flush, its outcomes lost and its
candidates taken until a restart. NULs are dropped from strings; the
probe_cache insert takes each message-id once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each claim looks its candidates up in probe_cache by message_id; with the
index on the HDD that took 1.27 s a claim, 48 s of each minute. Moved by
hand on homelab; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
~20 GB of the NVMe, read by key only; the catalog was at its cap and history
paused. Moved on homelab by hand (3 min, writers stopped; the cap raised to
100); the migration does the same on a new deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The October partition's three indexes took 2.9 GB of the NVMe, its pkey
never scanned; chunks are read from the HDD anyway. Moved on homelab by hand
(12 s); the migration does the same on a new deploy, parent indexes too so new
partitions follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
obfuscators give each volume its own extension (or none); for a random name
(no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
dot counts with a digit before it too (5,419 single files of YME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
"….vol4+4.par2": the subject and the total were the key, so every upload of
a total was one release (2,338 files for 20; also partner-of-ssl-news,
brothers-of-usenet). A random stem (no dots or spaces) on a volume file
joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
name up to its first number, when another follows, joins the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
