Commit Graph
134 Commits
Author SHA1 Message Date
Daan MeijerandClaude Fable 5.1 aa4022295a Tiny-archive rule; the ladder's discard by index; the namer in chunks
Test and deploy / test (push) Successful in 3m1s
Test and deploy / deploy (push) Successful in 32m21s
The tiny-archive rule (dropTinyArchives, both indexers): a settled single
archive of one article and 2-100 kB under a release-like name, spaces
allowed, outside the exempt groups and without an e-book, subtitle,
crack, cover or sidecar word: the 2015-17 exe-stub flood in miniature,
1.66M catalog rows (LOG.md (131), (133)).

PendingLadder.discard scanned every waiting row against the dropped
signatures and classes, 9+ minutes a sweep inside the claim, stalling
every lane: now driven from the dropped classes and signatures into the
indexes. The namer feeds the indexer the span's overview text chunk by
chunk (one string of it topped V8's limit) and gives up spans over 3M
articles. Migration 7000: a partial index on the taken probe candidates
for the prober's start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 20:23:16 +01:00
Daan MeijerandClaude Fable 5.1 3612bf3e43 Readable names: CamelCase product names and words with numbers stay readable
Test and deploy / test (push) Successful in 2m49s
Test and deploy / deploy (push) Successful in 3m24s
The production dry run of releases:readable flagged 5.37M of 19.8M rows,
and its first sample held real names: "CrossStitcher - Issue 440",
"SchiffsModell", "FastFilms 2.04", "BestCrypt NP v4.0", "SmithsWork",
"Laura15Juli2026". Two CamelCase words with few vowels have four runs and
no lone letter: a scramble now needs five runs, a lone letter or an inner
capital run besides its few vowels; the consonant-run clause is gone
(it counted y); a token whose letter runs are all words or acronyms with
numbers between them is neutral. On a 49k-row production sample the
flagged share went from 27.4% to 26.4% with the letters-only flags all
scrambles; the evaluation sets are unchanged (LOG.md (129)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:22:54 +01:00
Daan MeijerandClaude Fable 5.1 44c0c73b05 Pending ladder by class, twins by content, dropped classes at the edge
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Successful in 40m8s
The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).

- lib/release/classes.ts, names::pending_class, SQL pending_class():
  a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
  target's article; candidates carry bytes now), pending_classes with
  each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
  the unexplored, at most a quarter of a claim from one), the cheapest
  target first within one, then the oldest of the rest; records classes
  like signatures and drops them at the same thresholds; discards their
  waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
  dropped class out of the pending store (counted as garbage), the
  explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
  catalog release's names a pending release the probe didn't, and sets
  the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
  the finished ones; rules:classes publishes the dropped classes;
  releases:readable also marks the rows' candidates unreadable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 12:22:32 +01:00
Daan MeijerandClaude Fable 5.1 81a5dce45a Readable names: a token-by-token classifier for the readable flag
Test and deploy / test (push) Successful in 3m36s
Test and deploy / deploy (push) Skipped
Searches with the readable filter on returned scrambles: with a random
4-letter query nearly every hit was one ("Zx5AjNaK-mh7CFg6Hf8uu",
"Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ", "bd5b...-addon"), and
15% of a 6,000-row sample of the catalog's readable names were. looksRandom
knows single-token shapes only; separators, spaces, a random "extension"
of any length and a packaging word hid the rest.

isUnreadableName (lib/release/readable.ts) and names::is_unreadable_name
read a name token by token: words (one case or Capitalized), scrambles
(letter/digit and case switches, lone letters between capitals, inner
capital runs, few vowels, hex) and neutral tokens (digits, codes like
S02E10 and 1080p, CamelCase, packaging words); a name is unreadable when
its scrambles outweigh its words. Scene tags' "i" for "I" and the
NZBSPLIT hash are read as such. On the evaluation sets: 554 of 559
random-search names caught, 0 of 630 real names flagged (LOG.md (129)).

The edge's readable flag and the prober's inner names use it; the drop
rules keep looksRandom, which is narrower. An unnamed release goes to the
pending store (decision f), so more of the obfuscated groups' releases
wait for a probe now. releases:readable clears the flag on the existing
catalog's rows by name (a dry run by default).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 10:01:31 +01:00
Daan MeijerandClaude Fable 5.1 2643266559 Stub-archive rule: 100 MB cap on every leg, ROT13 volume names are volumes
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Successful in 4m1s
The first five hours in production (LOG.md (128)) dropped 522 passworded
videos of one a.b.erotica poster: ROT13-named volumes (".cnegN.rar" is
".partN.rar") of 1-270 MB in 258 kB articles to two groups, which the
article-size leg took for stubs. The cap that X and W already had now
applies to A as well (the catalog sample has 6 A-leg hits over 100 MB,
4 of them exe-only), and a ROT13 volume name is no single archive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 05:49:37 +01:00
Daan MeijerandClaude Fable 5.1 e470e231d0 Single-archive rules: stub archives and nonsense pieces dropped at the edge
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m21s
The judge gets each file's articles seen and bytes, and whether the window
closed on the release. A settled release of one unsplit archive posted to 2+
groups is dropped as a stub archive when its articles are <= 256 KiB with the
release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two
under 100 MB); releases touching music, e-book, comics, tabs or fonts groups
are exempt. A one-article archive of 100 kB-1 MB under a release-like,
unscrambled name without spaces or an e-book format is dropped as a nonsense
piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces),
mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 11:35:46 +01:00
Daan MeijerandClaude Opus 5.5 cfce8884d4 Lone articles under scrambled names are per-article; inactive groups get no corpus fill
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.

looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.

The prefetcher's corpus fill skips groups that aren't active.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:19:11 +01:00
Daan MeijerandClaude Opus 5.5 3f0b5b8be1 Subject names lose the brackets the counters left unmatched
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 2m13s
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:04:06 +01:00
Daan MeijerandClaude Opus 5.5 8cf965af80 The namer re-reads partial releases as the Rust window writes them
Test and deploy / test (push) Successful in 3m10s
Test and deploy / deploy (push) Successful in 5m38s
The pending namer's TypeScript re-index gave a file short of parts on the
server to the range path's fragment rule, so a partial release the edge
wrote (Bob's hex-named uploads in chello, ratcave, font, storage) never came
out again: 10 of 148k named releases moved. RangeIndexer's partialFiles
keeps such a file in its release, marked partial; tools/namer_check.ts
replays sample rows from the corpus (10 of 10 found with it, 1 without).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 10:41:50 +01:00
Daan MeijerandClaude Fable 5.1 680751e26a Lease lanes for the tail, a mixed lease order, coordinator:status, the edge end to end
Test and deploy / test (push) Successful in 3m14s
Test and deploy / deploy (push) Successful in 2m30s
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.

`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 13:47:31 +01:00
Daan MeijerandClaude Fable 5.1 1f6e3316ec The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m16s
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 18:22:18 +01:00
Daan MeijerandClaude Fable 5.1 e135076fcd The pending ladder in the prober: unnamed releases probed cheapest first, signatures learned, the named moved into the catalog
Test and deploy / test (push) Failing after 48s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Skipped
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 13:41:30 +01:00
Daan MeijerandClaude Fable 5.1 9cf85e09d8 Nothing enters the catalog without a usable name: the pending store (decision f amended)
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Skipped
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 13:18:14 +01:00
Daan MeijerandClaude Fable 5.1 4d0610ca06 Albums in the Rust edge, and the batch-counter rule (design 3)
Test and deploy / test (push) Successful in 1m4s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Skipped
A release the window closed on with one file named by its stem or post, the
name with a number and no archive, volume, sidecar or random name, waits in
the album of its poster and template (the name's digit runs as #, the
extension apart) posted within the window of the album's first file; the
album closes a window after that span and is written as one release
(source album, no file total) named by the members' shared numbers, a lone
member as the release it was. The batch store writes an album's member
names as release_texts kind 6 (member). A counter total of 100 or more under
which one poster started 20 or more releases is a batch counter: those
releases get no file total.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 12:22:30 +01:00
Daan MeijerandClaude Fable 5.1 f8cb17903b voyager 3: the window in the Rust edge
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:28:35 +01:00
Daan MeijerandClaude Fable 5.1 2580f20a43 voyager 3: the batch store writes releases once, by id references and bytea keys
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Skipped
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.

Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:04:03 +01:00
Daan MeijerandClaude Fable 5.1 bc4e17e86f voyager 3: release_texts, the searchable strings of a release (migration 6200)
Test and deploy / test (push) Failing after 52s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Skipped
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 20:33:28 +01:00
Daan MeijerandClaude Fable 5.1 346493b6c3 Voyager 3 schema: the slim catalog (migration 6000, the reset) and blocks (6100)
Test and deploy / test (push) Failing after 50s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Skipped
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 13:44:54 +01:00
Daan MeijerandClaude Fable 5.1 3cc7d9738e TruncatedReply: a compressed reply the server cut at a ".\r\n" fails in 2 s; ranges:dates samples plain
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m4s
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:49:53 +01:00
Daan MeijerandClaude Fable 5.1 1e01702dff nntp:probe: shows the bytes a stalled reply left
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 1m14s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:40:02 +01:00
Daan MeijerandClaude Opus 5.5 03ca96b255 Fix findCounter infinite loop; corpus filter skips chunks past 256 MB
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 2m5s
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.

A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:08:22 +02:00
Daan MeijerandClaude Opus 5.5 a66b0980f4 Corpus: store chunks without the lines the worker only counts
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 16:34:09 +02:00
Daan MeijerandClaude Opus 5.5 e00e4f3988 Join an upload's releases by its par2 set's file names
Test and deploy / test (push) Successful in 1m5s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 1m19s
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.

Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:02:51 +02:00
Daan MeijerandClaude Opus 5.5 61bdd43b34 Categories from the inner name, spots and groups; fix the name rules
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.

category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:07:12 +02:00
Daan MeijerandClaude Opus 5.5 d446bb57df Probe 7z listings and ZIP starts
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.

An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:55:07 +02:00
Daan MeijerandClaude Opus 5.5 ce43099188 Final completeness by time window: ranges keep their posting span, release_final() over blocks of done ranges
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 8m5s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:19:03 +02:00
Daan MeijerandClaude Opus 5.5 3d24db5565 Rules drop posters at the header pass: the CelebMovie family (rule set version 3)
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m47s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:36:46 +02:00
Daan MeijerandClaude Opus 5.5 7d9a81784e A folder's images posted with their sizes are one release: no image set under the subject
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m28s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:13:20 +02:00
Daan MeijerandClaude Opus 5.5 99b72ddd8f Sniff Windows executables (MZ with its PE header)
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:42:29 +02:00
Daan MeijerandClaude Opus 5.5 eddbe2863f Take the padded volume back out: it merged a dump's versions
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 2m34s
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:11:03 +02:00
Daan MeijerandClaude Opus 5.5 208ca1d535 A random name's volumes with extensions of their own; a padded volume after a date
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m31s
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
  obfuscators give each volume its own extension (or none); for a random name
  (no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
  dot counts with a digit before it too (5,419 single files of YME).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:10:16 +02:00
Daan MeijerandClaude Opus 5.5 c624a624df Take the image set head back out of the subject key
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m13s
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:14:17 +02:00
Daan MeijerandClaude Opus 5.5 f1c5af5f36 Under a constant subject: an upload's random volume stem, and an image's set head
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m27s
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
  "….vol4+4.par2": the subject and the total were the key, so every upload of
  a total was one release (2,338 files for 20; also partner-of-ssl-news,
  brothers-of-usenet). A random stem (no dots or spaces) on a volume file
  joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
  name up to its first number, when another follows, joins the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 09:12:17 +02:00
Daan MeijerandClaude Opus 5.5 9f255e6ac0 Images and tracks named by a prefix alone are bounded by the counter's total
Test and deploy / test (push) Successful in 1m3s
Test and deploy / edge (push) Successful in 2m10s
Test and deploy / deploy (push) Successful in 3m17s
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:54:02 +02:00
Daan MeijerandClaude Opus 5.5 96553a5504 Spaces at a dash are no part of a stem key
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m38s
"www.realmom.info - XVID- Farbiges Krabbeltier.r00" beside "… XVID-Farbiges
Krabbeltier.vol03+04.par2": 52 JBinUp uploads were two releases each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:51:47 +02:00
Daan MeijerandClaude Opus 5.5 7df5a638ca Spots: the spotter and the spot inspector, passwords kept apart
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m28s
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
  seen, each spot's full header verified (lib/spotnet from voyager 1, as
  Spotweb), its description, website and NZB articles; the password its
  title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
  its files listed (spot_files), each matched to the header pass's release
  by the same release identity in one of its groups within 48 h
  (release_spots); unmatched spots matched again hourly, after 24 h a
  release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
  spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:10:10 +02:00
Daan MeijerandClaude Opus 5.5 3fd08b598a An image's post token needs a word in it, not only a size
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m23s
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:33:52 +02:00
Daan MeijerandClaude Opus 5.5 229189f35c A date alone isn't an unquoted name; an image under a post token keys by the post
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m12s
Test and deploy / deploy (push) Successful in 3m28s
- "Ballers.2014.S01E01.HDTV.x264-KILLERS[ettv] 15_07_2015.rar", "Computer Bild
  - Ausgabe 152015 - 15_07_2015.rar": the cut after "] " or " - " left
  "15_07_2015.rar" for every item of a Razor1911 dump (1,335 files for 12);
  a date alone after the cut means the whole text is the name.
- "MDG005b - (01/90) "IMG_2307.JPG"": a counter in parentheses makes no post
  key, so every "IMG" set of the poster was one (609 for 90); an image under
  a post token is keyed by it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 09:03:23 +02:00
Daan MeijerandClaude Opus 5.5 590436d02a SCENE_RELEASE only for a track post's lower-case name
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 3m11s
7ad4b70 also keyed rar uploads by artist and total: "VA - Best Of Shamania
Pro 2K16-(SHB 006)-WEB-2016-ZzZz" became 140 files for 20, Ace_Of_Base 72
for 12. Those group by their full name; only a name without capitals or
spaces (Meow's "blaise_mc-in_hellstill_3-ep-web-2022-known.par2") joins
the tracks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 08:05:02 +02:00
Daan MeijerandClaude Opus 5.5 7ad4b70775 A scene music post's par2 under the release name; .par/.part before .par2
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m31s
"[07/14] - "blaise_mc-in_hellstill_3-ep-web-2022-known.par2"": the tracks and the
00- files are keyed by artist and total, the par2 by its stem, so ~1,900 Meow
and Cat posts were two releases. Under a counter a stem that reads as a scene
music release (artist-…-web|cd|ep|single|vinyl|…-…YYYY(-group)) takes the
artist key. And "goh1Phoz5ahXu.par.vol006+08.par2", "X.part.par2" beside
the volumes: .par/.part before a par2 suffix goes too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 06:56:56 +02:00
Daan MeijerandClaude Opus 5.5 6f7a1692d6 runOn: a space after the extension, not a bare dash
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 4m21s
0c5ede9 cut "Angry.Birds.Rio.v1.2.2.cracked.READ.NFO-THETA.rar" at .NFO: the
release lost its group, and two groups' releases of one game would merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 05:06:02 +02:00
Daan MeijerandClaude Opus 5.5 0c5ede99dd A quoted name that runs on into a title; .xNNN/.zNNN volumes
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Successful in 1m32s
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 03:59:24 +02:00
Daan MeijerandClaude Opus 5.5 1c8da1dd12 A long number is a per-post id only under a counter
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m32s
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:41:40 +02:00
Daan MeijerandClaude Opus 5.5 8d5385fdec Split zip volumes (.zNN) and .xNN as volumes, but not .x32/.x64/.x86
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m23s
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 01:47:21 +02:00
Daan MeijerandClaude Opus 5.5 87e15d810d Old rar volumes: .s–.v only, not .x64/.x86
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m36s
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 00:53:38 +02:00
Daan MeijerandClaude Opus 5.5 d4e53ca0d8 More per-file suffixes: old rar volumes, .rev, a part's par2, .bc; a size after an unquoted name
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 1m22s
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
  own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:59:09 +02:00
Daan MeijerandClaude Opus 5.5 7b99cfdb0e A scene music post's 00- files go with its tracks
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m22s
"[01/13] - "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with
"[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the tracks are keyed by
artist and total (SCENE_TRACK), but the cover, nfo, sfv and m3u were releases
of their own (2,630 single files of Meow in a.b.ftn). Under a bare counter a
00- stem takes the same key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:01:12 +02:00
Daan MeijerandClaude Opus 5.5 0fafe71b21 The poster-tag tolerance only without a counter
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m43s
Test and deploy / deploy (push) Successful in 1m24s
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:52:26 +02:00
Daan MeijerandClaude Opus 5.5 62250081ef A poster tag of 4+ words may share one with the stem
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m39s
Test and deploy / deploy (push) Successful in 1m10s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:24:42 +02:00
Daan MeijerandClaude Opus 5.5 cfd29c8de3 A long hex stem without a counter is the upload
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m22s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:53:53 +02:00