One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.
A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The name pass put every holder of a set into its merge: a par2 posted
again years later ("Auto Bild Klassik Februar 02-2014", held by a 2014
and a 2026 release) would have joined the repost to the old upload. Now
the newest holder anchors it and only holders in its group within 12
hours take part. Every round, three sets failed on the probe candidates'
unique key: a member held a candidate of an article the target had.
merge() now drops such a candidate (the target's, or a lower member's,
stays), as it already does for par2 sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.
category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merges delete releases, so the count of a key's releases fell behind its
numbers (83 releases up to key:129): the 20 numbers tried were all taken
and the range's batch failed for good. Spot releases (spot:key) are not
numbered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The incremental judge counted each changed release's files twice, each
index entry a heap fetch: 24-32 s a chunk. file_hashes by (hash16k,
size, release_id) replaces (hash16k); the other releases of a file are
judged only when the chunk brings it to 20.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round took 22-29 min an hour: every match went through the insert's
conflict check (60-150 s a rule), scraped and locked each scanned all of
releases. The spot inspector's releases now add their poster to posters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jsonb can't hold \u0000, and ON CONFLICT can't take a row twice in one
statement: either failed the whole flush, its outcomes lost and its
candidates taken until a restart. NULs are dropped from strings; the
probe_cache insert takes each message-id once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
obfuscators give each volume its own extension (or none); for a random name
(no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
dot counts with a digit before it too (5,419 single files of YME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
"….vol4+4.par2": the subject and the total were the key, so every upload of
a total was one release (2,338 files for 20; also partner-of-ssl-news,
brothers-of-usenet). A random stem (no dots or spaces) on a volume file
joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
name up to its first number, when another follows, joins the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spot older than what the catalog has indexed (the spotter's first backfill
reaches back ~3 months) matched nothing and would have become a release of
its own, a duplicate once the history pass builds the upload. Unmatched
spots are matched again in turns (rematched_at) for 30 days; the 24 h before
a standalone release count from the inspection; a later match deletes the
spot's own release and links the header pass's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
