One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A 24 h posting-time window per block instead of write-early-merge-later,
albums for counterless series, poster/group-set ids and binary keys in
releases, cold tables on the HDD, the corpus test before the reset.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.
category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.
Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From "draft for approval" to the concept of voyager 2 (voyager 1's
CONCEPT.md and D1-D42 are superseded). Phase 2: what's built (the
prober), and voyager 1's features not yet here: spots, NZB import,
passwords, SFV, RAR walks, 7z listings, body naming of obfuscated
subjects, after-probe garbage drops; where its exported data is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka stopped answering on all 40 of the coordinator's long-lived
connections (quiet sockets; a fresh connection fetched the same chunk in
20 s): with a 10-minute timeout the prefetcher fetched one chunk a
minute. The timeout (silence, not duration) is 2 minutes now: 365-411k
articles/s after the restart. tools/probe_chunk.ts times a chunk
compressed and plain.
README and REBUILD.md: edge-node, the Rust worker, is the one running
(8x less CPU per header, 20x less memory than edge:node in production).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed
XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so
one process tops out around 60 MB/s: less than 50 connections bring
(~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with
id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges
and settles. Two shards of 5 connections fetched what one of 10 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
