22 Commits
Author SHA1 Message Date
Daan MeijerandClaude Fable 5.1 66980829dc Twins by content: at most 500 twins read per pair
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m16s
With the hash typed as the column (f537add) the planner walks every twin of a
pair by index; 55 pairs have over 500 twins, up to 131k, and a flush with such
a pair ran the query for 37 s inside its transaction, the other flushes and
node-api's class counters waiting on it. A LATERAL per pair now reads its
lowest 500 release ids and takes the first named one: 49 ms for 305 pairs with
the five hottest among them. LOG.md (138).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 09:49:40 +01:00
Daan MeijerandClaude Fable 5.1 f537add573 Twins by content: type the hash as the column
Test and deploy / test (push) Successful in 3m49s
Test and deploy / deploy (push) Successful in 1m5s
jsonb_to_recordset declared hash16k as text; against the char(32) column the join
cast the column, so Postgres scanned all of file_hashes for every call (4 s, 20
calls a minute: 44% of the database's time). Declared char(32), the join walks
file_hashes_hash_size_release_idx: 24 ms for 300 pairs. LOG.md (138).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 09:31:06 +01:00
Daan MeijerandClaude Fable 5.1 d7171a5ca4 Pending ladder: the sweep and the record lock their rows in key order
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m4s
discardKeys updated with `release_key = ANY(subquery)`, which locks rows in
the plan's scan order, and deadlocked against a batch upsert that takes its
rows by key (LOG.md (137)). It now updates from a subquery that selects the
rows ORDER BY release_key FOR UPDATE, so the locks are taken sorted (the
plan: LockRows above Sort). record() runs the same SELECT ... FOR UPDATE over
its taken rows before its UPDATE ... FROM jsonb_to_recordset, whose join
order the planner chooses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 00:32:09 +01:00
Daan MeijerandClaude Fable 5.1 eadae82b0c The prober's tables on the NVMe; the prober runs through the night
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m30s
Migration 7400 moves pending_releases (with its two HDD indexes),
release_inspections and probe_cache to the default tablespace (nvme on
homelab): the edge's batch upserts into pending_releases took minutes each
on the HDD and the prober's flushes waited on them (LOG.md (137)). Homelab
moved them by hand with the services stopped: 66 GB in 10.5 minutes.

With everything the prober touches on the NVMe, the nightly pause leaves the
prober running (usenet-rebuild-pause.sh), and a deploy while paused
recreates it with node-api and web (deploy.sh).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 23:57:59 +01:00
Daan MeijerandClaude Fable 5.1 e02a7c89b4 REBUILD.md: probe_candidates back on the NVMe (migration 7300)
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:52 +01:00
Daan MeijerandClaude Fable 5.1 6ef7ddca92 Batch store: one lock order for the pending upsert; probe_candidates on the NVMe
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m33s
writePending upserted pending_releases in chunks of 1,000 rows, each chunk
followed by its class upsert, so one transaction locked releases, classes,
releases, classes, and a later chunk's keys and classes could sort before an
earlier chunk's: the opposite of the ladder's record(), which holds all its
rows and then takes all its classes. Four deadlocks between the two class
upserts in six hours (LOG.md (136)). Now all rows are sorted once and the
classes upserted once, after the last chunk.

Migration 7300 moves probe_candidates and its indexes to the default
tablespace (nvme on homelab): the catalog claim's UPDATE from random points
took 2-3 minutes on the HDD beside the sweep and the autovacuum, with all
lanes waiting. Homelab moved it by hand (83 s, ~8 GB, services stopped); the
migration is a no-op there, and without a default tablespace.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:09 +01:00
Daan MeijerandClaude Fable 5.1 a71de56a7c Judge: the tiny rule, the catalog's side of the edge's tiny-archive rule
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 4m23s
One file of one article, 2–100 kB, under a readable release-like name outside the music, e-book and comic
groups, without an e-book, subtitle, crack or sidecar word: the 2015–17 flood's 1.5M rows that the edge
indexed before its rule. The same tests as isTinyArchive in Postgres's regex dialect; no join on the
candidates (98% of a sample have an archive candidate, the rest lone par2 files and scrambled names).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 13:15:03 +01:00
Daan MeijerandClaude Fable 5.1 411c36eec0 Prober: the signature index by the poster's md5
Test and deploy / test (push) Successful in 3m5s
Test and deploy / deploy (push) Successful in 1h2m47s
A poster of 2.7 kB broke the index build (a btree key is at most a third of a page) and left an invalid index
that IF NOT EXISTS would have taken for the real one: the migration drops it first and indexes md5(poster), and
the sweep asks by it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 12:08:53 +01:00
Daan MeijerandClaude Fable 5.1 8e008da364 Prober: the sweep by a signature index, pending rows locked in one order, deadlocked flushes kept
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Failing after 21m32s
The discard sweep of a dropped signature read the whole pending_releases table per signature (a parallel seq
scan over the poster test, four minutes on the HDD that starved the lanes' claims): a partial index on
(group_id, tool, poster) WHERE status = 0 makes it a range, and the sweep queries a poster's rows by it and a
tool's and group's by the ladder index.

The batch store's pending upsert, the record, and the sweep deadlocked on pending_releases and pending_classes
rows locked in different orders, and a deadlocked flush dropped its outcomes: lockOrder() sorts every
multi-row statement's set the same way, and a flush that deadlocks anyway keeps its outcomes for the next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 10:36:19 +01:00
Daan MeijerandClaude Fable 5.1 5d3b578322 The discard sweep beside the lanes, each dropped signature swept once
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 38m14s
After aa40222 the prober still idled: a dropped signature's range of the
ladder index is minutes of HDD (309 dropped, 242 with a poster tested on
the rows) and the sweep read the same ranges again every five minutes
inside the claim. Now each dropped class and signature is swept once
(swept_at, migration 7100; null again when the record drops it anew),
20 of each per sweep in rounds of 10k rows, in a loop of its own in the
prober beside the lanes and the namer (LOG.md (133)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 21:12:51 +01:00
Daan MeijerandClaude Fable 5.1 aa4022295a Tiny-archive rule; the ladder's discard by index; the namer in chunks
Test and deploy / test (push) Successful in 3m1s
Test and deploy / deploy (push) Successful in 32m21s
The tiny-archive rule (dropTinyArchives, both indexers): a settled single
archive of one article and 2-100 kB under a release-like name, spaces
allowed, outside the exempt groups and without an e-book, subtitle,
crack, cover or sidecar word: the 2015-17 exe-stub flood in miniature,
1.66M catalog rows (LOG.md (131), (133)).

PendingLadder.discard scanned every waiting row against the dropped
signatures and classes, 9+ minutes a sweep inside the claim, stalling
every lane: now driven from the dropped classes and signatures into the
indexes. The namer feeds the indexer the span's overview text chunk by
chunk (one string of it topped V8's limit) and gives up spans over 3M
articles. Migration 7000: a partial index on the taken probe candidates
for the prober's start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 20:23:16 +01:00
Daan MeijerandClaude Fable 5.1 4d46db291e pending:classes without table scans; an index for the prober's startup reset
Test and deploy / test (push) Successful in 2m55s
Test and deploy / deploy (push) Successful in 25m6s
The first run scanned the 58M-row pending table on the HDD once per
6-hour window and starved the prober and the batch store. Now one scan
classes the finished rows and seeds the record, and the waiting rows go
by windows of created_at through their index. The prober's start puts a
previous run's claims back with a status scan of the same table, 20+
minutes after every deploy: a partial index on the taken rows (migration
6900) makes it instant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:33:48 +01:00
Daan MeijerandClaude Fable 5.1 3612bf3e43 Readable names: CamelCase product names and words with numbers stay readable
Test and deploy / test (push) Successful in 2m49s
Test and deploy / deploy (push) Successful in 3m24s
The production dry run of releases:readable flagged 5.37M of 19.8M rows,
and its first sample held real names: "CrossStitcher - Issue 440",
"SchiffsModell", "FastFilms 2.04", "BestCrypt NP v4.0", "SmithsWork",
"Laura15Juli2026". Two CamelCase words with few vowels have four runs and
no lone letter: a scramble now needs five runs, a lone letter or an inner
capital run besides its few vowels; the consonant-run clause is gone
(it counted y); a token whose letter runs are all words or acronyms with
numbers between them is neutral. On a 49k-row production sample the
flagged share went from 27.4% to 26.4% with the letters-only flags all
scrambles; the evaluation sets are unchanged (LOG.md (129)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:22:54 +01:00
Daan MeijerandClaude Fable 5.1 44c0c73b05 Pending ladder by class, twins by content, dropped classes at the edge
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Successful in 40m8s
The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).

- lib/release/classes.ts, names::pending_class, SQL pending_class():
  a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
  target's article; candidates carry bytes now), pending_classes with
  each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
  the unexplored, at most a quarter of a claim from one), the cheapest
  target first within one, then the oldest of the rest; records classes
  like signatures and drops them at the same thresholds; discards their
  waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
  dropped class out of the pending store (counted as garbage), the
  explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
  catalog release's names a pending release the probe didn't, and sets
  the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
  the finished ones; rules:classes publishes the dropped classes;
  releases:readable also marks the rows' candidates unreadable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 12:22:32 +01:00
Daan MeijerandClaude Fable 5.1 81a5dce45a Readable names: a token-by-token classifier for the readable flag
Test and deploy / test (push) Successful in 3m36s
Test and deploy / deploy (push) Skipped
Searches with the readable filter on returned scrambles: with a random
4-letter query nearly every hit was one ("Zx5AjNaK-mh7CFg6Hf8uu",
"Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ", "bd5b...-addon"), and
15% of a 6,000-row sample of the catalog's readable names were. looksRandom
knows single-token shapes only; separators, spaces, a random "extension"
of any length and a packaging word hid the rest.

isUnreadableName (lib/release/readable.ts) and names::is_unreadable_name
read a name token by token: words (one case or Capitalized), scrambles
(letter/digit and case switches, lone letters between capitals, inner
capital runs, few vowels, hex) and neutral tokens (digits, codes like
S02E10 and 1080p, CamelCase, packaging words); a name is unreadable when
its scrambles outweigh its words. Scene tags' "i" for "I" and the
NZBSPLIT hash are read as such. On the evaluation sets: 554 of 559
random-search names caught, 0 of 630 real names flagged (LOG.md (129)).

The edge's readable flag and the prober's inner names use it; the drop
rules keep looksRandom, which is narrower. An unnamed release goes to the
pending store (decision f), so more of the obfuscated groups' releases
wait for a probe now. releases:readable clears the flag on the existing
catalog's rows by name (a dry run by default).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 10:01:31 +01:00
Daan MeijerandClaude Fable 5.1 2643266559 Stub-archive rule: 100 MB cap on every leg, ROT13 volume names are volumes
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Successful in 4m1s
The first five hours in production (LOG.md (128)) dropped 522 passworded
videos of one a.b.erotica poster: ROT13-named volumes (".cnegN.rar" is
".partN.rar") of 1-270 MB in 258 kB articles to two groups, which the
article-size leg took for stubs. The cap that X and W already had now
applies to A as well (the catalog sample has 6 A-leg hits over 100 MB,
4 of them exe-only), and a ROT13 volume name is no single archive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 05:49:37 +01:00
Daan MeijerandClaude Fable 5.1 e470e231d0 Single-archive rules: stub archives and nonsense pieces dropped at the edge
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m21s
The judge gets each file's articles seen and bytes, and whether the window
closed on the release. A settled release of one unsplit archive posted to 2+
groups is dropped as a stub archive when its articles are <= 256 KiB with the
release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two
under 100 MB); releases touching music, e-book, comics, tabs or fonts groups
are exempt. A one-article archive of 100 kB-1 MB under a release-like,
unscrambled name without spaces or an e-book format is dropped as a nonsense
piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces),
mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 11:35:46 +01:00
Daan MeijerandClaude Opus 5.5 2f1bbf2538 REBUILD.md: lone articles under scrambled names
Test and deploy / test (push) Successful in 3m8s
Test and deploy / deploy (push) Successful in 2m10s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:20:04 +01:00
Daan MeijerandClaude Opus 5.5 cfce8884d4 Lone articles under scrambled names are per-article; inactive groups get no corpus fill
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.

looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.

The prefetcher's corpus fill skips groups that aren't active.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:19:11 +01:00
Daan MeijerandClaude Opus 5.5 5132ab9083 Search: a word with a leading ^ matches at a word's start only
Test and deploy / test (push) Successful in 3m5s
Test and deploy / deploy (push) Successful in 2m7s
"^zed" finds "Zed Dead", "The_Zed_Files", not "Facialized": a regex on
release_texts, (^|[^[:alnum:]]) before the word, which the trigram index
serves (homelab: 551 texts for ^zed against 6,606 for zed, 35 ms warm).
Words without ^ match anywhere as before; the search box says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 20:19:52 +01:00
Daan MeijerandClaude Opus 5.5 3f0b5b8be1 Subject names lose the brackets the counters left unmatched
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 2m13s
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:04:06 +01:00
Daan MeijerandClaude Opus 5.5 c6f9409e7f REBUILD.md: the namer's failures were the partial releases, 52k of 210k, not nearly all
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:07:24 +01:00
44 changed files with 2737 additions and 190 deletions

No files matched your search

+97 -16
View File
@@ -151,7 +151,9 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
indexed groups in the article's Xref, the one with the lowest priority
number; see the group list) → group articles into files → files
into releases (`lib/release` rules) → drop garbage (random name +
dropped signature or tool) → set aside per-article random subjects and
dropped signature or tool), bare executables, dropped posters, and the
single-archive fakes (stub archives and nonsense pieces, below) → set
aside per-article random subjects and
obfuscated subjects (counted only; phase 2). Returns per batch: release
chunks, edge fragments, probe candidates, counters. Holds no state beyond
its lease.
@@ -339,8 +341,13 @@ or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
*locked* (passworded, no password from a spot, its NZB or the title),
*unusable* (random names, no inner name, no par2 or inner-name merge, no
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
*tiny* (one file of one article, 2–100 kB, under a readable release-like
name outside the music, e-book and comic groups, without an e-book,
subtitle, crack or sidecar word: the catalog's side of the edge's
tiny-archive rule, 1.5M rows of the 2015–17 flood that the edge indexed
before the rule; LOG.md (131), (135)), *unusable* (random names, no inner
name, no par2 or inner-name merge, no spot, a day after settling).
Verdicts (`release_verdicts`, with the rule
set's version) hide releases from search (`judged` shows them); a verdict
whose rule no longer matches is taken back. Nothing is dropped yet.
@@ -710,10 +717,13 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
`settled_at IS NOT NULL`), the settler, the fragment merger,
`releases_category_newest_idx`.
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
rule read all of `release_inspections` every round, 17 min on the HDD;
the judge must read it by release or keep what it needs on the NVMe.
- `release_inspections` and `probe_candidates` were on `CHUNK_TABLESPACE`
(the HDD) with their indexes; both are back on the NVMe (migrations 4800
and 7300). Note LOG.md 2026-10-19 (81): the judge's fake rule read all of
`release_inspections` every round, 17 min on the HDD; and 2026-11-03
(136): the prober's catalog claim updates `probe_candidates` from random
points every minute, 2–3 min on the HDD beside the pending ladder's sweep.
What a worker reads or updates by random key every minute stays on the NVMe.
### Batch store (built 2026-10-25)
@@ -774,7 +784,49 @@ is before frontier − `WINDOW_MS` are judged and written `finished`, as an
it, else `finished: false, edge: true`. A file short of parts stays in
its release and sets `partial` (the store: `complete = false`). A lone
article of a multi-part random file is still counted `per-article`, not
written. Gone: fragments and the cut-off, `uploads_of` (online now),
written; before the block's end a scrambled name counts as random too
(`looks_scrambled`: a random subject and poster per article, 160k of
alt.binaries.storage's releases; LOG.md (122)).
**Single-archive rules (built 2026-10-31; LOG.md (123)-(127)).** The judge
(`names::judge`, `RuleSet.judge`) gets each file's articles seen and their
bytes, and whether the release is *settled* (the window closed on it: a
par2 can still join an open one; the start edge doesn't count, since a par2
follows its rar and 7% of the catalog's releases are start edges). A settled
release of one unsplit zip, rar or 7z and nothing else is dropped as a
**stub archive** (`stub-archive`, key: the legs tripped) when posted to
2+ groups, under 100 MB (the fakes stop at ~90 MB; hash-named uploads,
real albums and passworded videos above it don't), and (A) its articles
are ≤ 256 KiB with the release ≥ 1 MB (Power-Post's 250 kB default; a
one-article release always has small articles, which exempts e-books and
sidecars), or (X) posted to 4+ groups, or (W) by a webmail poster. A
ROT13 volume name (".cnegN.rar" is ".partN.rar": a.b.erotica's passworded
videos, LOG.md (128)) is a volume, not a single archive. A release
touching a music, e-book, comics, tabs or fonts group is exempt: those
communities post real single small archives the same way (karaoke zips to
a.b.sounds.karaoke + a.b.test with Power-Post's articles; 16 of 32,308
known fakes lost, LOG.md (127)). These are the
exe-stub and recycled-installer floods under film and software names:
55% of the catalog on 2026-10-30, 99.3% of the known fakes, 0 known
content in the 217k sample, verified on 1,500 decoded articles. A settled
single archive of one article and 100 kB–1 MB under a release-like,
unscrambled name without spaces and without an e-book format in it is
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
with spaces are books and albums. Below that, a settled single archive of
one article and 2–100 kB under a release-like name (spaces allowed), outside
the exempt groups and without an e-book, subtitle, crack/keygen/trainer,
cover or sidecar word, is dropped as a **tiny archive** (`tiny-archive`,
key: the upload tool; LOG.md (131), (133)): the 2015–17 flood's RARs
holding "<title>.exe" at 2–60 kB, 1.66M catalog rows from a dozen posters
(12 of 12 fetched fake); the word exemptions keep the real tiny archives
(subtitles, cracks, trainers) at the price of ~50k fakes named "Incl
Keygen" that the prober judges later. The rules are switched by the
rule set (`dropStubArchives`, `dropNonsensePieces`, `dropTinyArchives`),
sampled like every drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
judges as each batch completes files, so it passes settled without the
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
`check.ts` and the fixture comparison (the parity with the TypeScript
indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
(incomplete releases are kept). The wire carries `finished`, `edge` and
@@ -843,7 +895,11 @@ e. Headers through the coordinator's relay, with prefetch (part 1).
f. **Nothing enters the catalog without a usable name** (amended
2026-10-26, the user's direction; before: random-named releases keep
no chunk but their rows). A release the window closed on whose files
all have random names is no release yet: it goes to `pending_releases`
all have random names (`isUnreadableName`, `names::is_unreadable_name`:
a name read token by token, so separators, spaces and random
"extensions" don't hide a scramble; since 2026-11-02, LOG.md (129),
when 15% of the catalog's "readable" names were "Zx5AjNaK-mh7CFg6Hf8uu"
and "Description - Lt5nHjlo3RiuBb3C") is no release yet: it goes to `pending_releases`
(migration 6300, the HDD) with its keys, counts, article span, upload
tool and probe targets, no row in `releases`, `release_texts` or
`posters`, no chunk. The pending ladder works them **from least to most
@@ -857,7 +913,20 @@ f. **Nothing enters the catalog without a usable name** (amended
yields none counts against its signature, and a signature with
≥100 probes and ≤0.5% named (voyager 2's garbage learner, LOG.md
2026-10-08 (5)) is **discarded** from then on, a sample of it still
probed to notice a change. The reason, in the 2016 replay: one poster's
probed to notice a change. **By class** (2026-11-02, LOG.md (130)): a
pending release's class is its tool, group, size bucket, name shape and
partial flag (`lib/release/classes.ts`, `names::pending_class`, the SQL
`pending_class()`; `pending_releases.klass`), and `pending_classes` keeps
each class's climbs and names; the claim takes the classes with the best
record first (`classScore`, a prior for the unexplored, at most a quarter
of a claim from one class), the cheapest target article first within one
(`target_bytes`), and the oldest of the rest; a class with the learner's
thresholds is dropped like a signature, and the rule set's
`droppedClasses` keep its releases out of the pending store at the edge,
the explored sample excepted. A probed file whose first 16 KiB and size
match a named catalog release's (`file_hashes`) takes that name without
another fetch (`app/services/hash_twins.ts`), on the ladder and for
catalog releases without one. The reason, in the 2016 replay: one poster's
hash-named pieces ("Bob", 11 aliases, 3.2 M releases in five small
groups) were 76% of the rows and three quarters of the NVMe bytes for
11% of the segments and nothing anyone can find; the full history has
@@ -874,16 +943,28 @@ f. **Nothing enters the catalog without a usable name** (amended
target or status 3 (exhausted); each finished climb counts on
`pending_signatures` for (tool, group, poster) and (tool, group), and a
signature with PENDING_MIN_PROBES (100) climbs and at most
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
waiting releases of dropped signatures are discarded (status 2), the
rules' `samplePercent` of them left to climb. The namer
PENDING_MAX_NAMED (0.5%) named is dropped: a sweep beside the lanes
(every five minutes, in its own loop) discards the waiting releases
of each dropped signature and class once (`swept_at`; the edge drops
their new releases itself), the rules' `samplePercent` of them left to
climb, each by an index (`pending_releases_signature_idx` for a
poster's rows: tested on the rows, a signature was a scan of the whole
table; LOG.md (133), (134)). The batch store, the record, the claim
and the sweep lock pending rows in one order, by key and the counters
after (`lockOrder` for the multi-row statements; the record and the
sweep lock their rows with `SELECT … ORDER BY release_key FOR UPDATE`
first, since an UPDATE's join or `= ANY(subquery)` locks in plan
order), and a flush that deadlocks anyway keeps its outcomes for the
next (LOG.md (134), (137)). The namer
(`app/services/pending_namer.ts`, in the prober's process): a named
release's article span read again from the corpus (the stored range's
chunks) or by XOVER on a connection of its own, through the TypeScript
chunks) or by XOVER on a connection of its own, a chunk of articles
at a time (one string of a wide span topped V8's limit; spans over 3M
articles are given up), through the TypeScript
indexer with `partialFiles` (a file given up short of parts stays a
partial file of its release, as the Rust window writes it; without it
the range path's fragment rule left nothing to find, and 10 of 148k named
releases were moved by 2026-10-29), written as a release under the pending key with `source`
the range path's fragment rule left nothing to find: by 2026-10-29
about 52k of 210k named releases failed, all partial), written as a release under the pending key with `source`
weak-stem, the name as `inner_name`, its chunk and its candidates (the
prober answers those from `probe_cache`: inspections, par2 sets, hashes,
the password flag). Not built: the spot/NZB match on pending keys (step
+46 -12
View File
@@ -9,6 +9,7 @@ import { SIDECAR } from '#services/sidecar_folder'
import { identityKey, releaseKey } from '#lib/release/keys'
import { sourceCode } from '#lib/release/source'
import { uploadTool } from '#lib/rules/index'
import { byEffort, lockOrder } from '#services/pending_ladder'
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
/**
@@ -179,16 +180,21 @@ class BatchStore {
candidates: ProbeCandidate[]
): Promise<number> {
if (releases.length === 0) return 0
const targets = new Map<string, { kind: string; messageId: string; file: string }[]>()
const targets = new Map<
string,
{ kind: string; messageId: string; file: string; bytes?: number }[]
>()
for (const c of candidates) {
const key = slot(c.releaseKey, c.upload ?? 0)
const list = targets.get(key) ?? []
list.push({ kind: c.kind, messageId: c.messageId, file: c.file })
list.push({ kind: c.kind, messageId: c.messageId, file: c.file, bytes: c.bytes })
targets.set(key, list)
}
let written = 0
for (let i = 0; i < releases.length; i += 1000) {
const rows = releases.slice(i, i + 1000).map((r) => {
// In the record's lock order (LOG.md (136)): all rows sorted, then all classes sorted, in one transaction. The
// upsert locks a finished row too, before its WHERE passes it by; a chunk's classes before the next chunk's rows
// crossed the ladder's record(), which holds its rows and then takes its classes.
const rows = lockOrder(
releases.map((r) => {
const probes = targets.get(slot(r.key, r.upload)) ?? []
return {
release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
@@ -207,25 +213,53 @@ class BatchStore {
file_total: r.fileTotal,
tool: r.tool ?? (probes[0] ? uploadTool(probes[0].messageId) : 'unknown'),
candidates: probes,
// The first target's article (by effort): the cheapest probe first within a class (LOG.md (130)).
target_bytes: byEffort(probes)[0]?.bytes ?? null,
}
})
}),
(r) => r.release_key
)
let written = 0
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
for (let i = 0; i < rows.length; i += 1000) {
const { rows: done } = await trx.rawQuery(
`INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
files, segments, bytes, partial, finished, file_total, tool, candidates)
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes)
SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.name, v.poster, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0),
v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates
v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates,
pending_class(v.name, v.bytes, v.partial), v.target_bytes
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, identity_key text, name text, poster text, first bigint, last bigint, first_article bigint, last_article bigint,
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb)
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb, target_bytes bigint)
ON CONFLICT (release_key) DO UPDATE SET
files = pending_releases.files + excluded.files, segments = pending_releases.segments + excluded.segments, bytes = pending_releases.bytes + excluded.bytes,
last_posted_at = greatest(pending_releases.last_posted_at, excluded.last_posted_at), last_article = greatest(pending_releases.last_article, excluded.last_article),
partial = excluded.partial, finished = excluded.finished, file_total = coalesce(excluded.file_total, pending_releases.file_total),
candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END
candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END,
klass = pending_class(pending_releases.name, pending_releases.bytes + excluded.bytes, excluded.partial),
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes)
WHERE NOT pending_releases.finished
RETURNING release_key`,
[groupId, JSON.stringify(rows)]
RETURNING release_key, tool, klass, (xmax = 0) AS inserted`,
[groupId, JSON.stringify(rows.slice(i, i + 1000))]
)
written += done.length
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
for (const d of done) {
const key = `${d.tool}\t${d.klass}`
const entry = classes.get(key) ?? { tool: d.tool, klass: Number(d.klass), waiting: 0 }
if (d.inserted) entry.waiting++
classes.set(key, entry)
}
}
if (classes.size > 0) {
await trx.rawQuery(
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
[
groupId,
JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${groupId}\t${c.klass}`)),
]
)
}
return written
}
+36
View File
@@ -0,0 +1,36 @@
import db from '@adonisjs/lucid/services/db'
/**
* Twins by content (LOG.md (130)): a probed file's first 16 KiB and size (file_hashes, what the par2 merge and the
* repeated verdict match) in a catalog release that has a name, readable or found inside. The same film posted
* under scrambles by other tools gets that name without another fetch. Releases with a junk verdict don't count.
*/
/** Twins read per pair: the hottest pairs have 131k. */
const TWIN_SCAN = 500
export async function twinNames(
pairs: { hash16k: string; size: number }[],
trx: { rawQuery: typeof db.rawQuery } = db
): Promise<Map<string, string>> {
const out = new Map<string, string>()
if (pairs.length === 0) return out
// Per pair the first named twin among its lowest TWIN_SCAN release ids. hash16k is typed as the column
// (char(32)): as text the join cast the column and scanned all of file_hashes, 4 s a call; and with the index
// the hottest pairs (55 with over 500 twins, up to 131k, all readable) made a call walk every twin, 37 s inside
// the flush's transaction (LOG.md (138)). One name is all a pair needs.
const { rows } = await trx.rawQuery(
`SELECT v.hash16k, v.size, t.name
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k char(32), size bigint)
CROSS JOIN LATERAL (
SELECT coalesce(r.inner_name, r.name) AS name
FROM (SELECT release_id FROM file_hashes h WHERE h.hash16k = v.hash16k AND h.size = v.size ORDER BY release_id LIMIT ?) h
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
ORDER BY r.readable DESC, r.id LIMIT 1) t`,
[JSON.stringify(pairs), TWIN_SCAN]
)
for (const r of rows) out.set(`${r.hash16k}\t${r.size}`, r.name)
return out
}
export const twinKey = (hash16k: string, size: number) => `${hash16k}\t${size}`
+329 -45
View File
@@ -4,6 +4,7 @@ import coordinatorConfig from '#config/coordinator'
import { explored } from '#lib/rules/index'
import { usableName, type Inspection } from '#lib/probe/index'
import RuleSets from '#services/rule_sets'
import { twinKey, twinNames } from '#services/hash_twins'
/** pending_releases.status (migration 6400). */
export const PENDING = {
@@ -22,6 +23,8 @@ export interface PendingTarget {
kind: string
messageId: string
file: string
/** The target article's bytes (candidates from 2026-11-02 on). */
bytes?: number
}
/** One probe the ladder hands the prober: the next target of a pending release. */
@@ -33,6 +36,8 @@ export interface PendingProbe {
file: string
tool: string
groupId: number
/** Its class (lib/release/classes.ts; 0 for a row from before the classes). */
klass: number
poster: string
/** Targets probed before this one. */
tries: number
@@ -53,12 +58,82 @@ export function byEffort(targets: PendingTarget[]): PendingTarget[] {
return [...targets].sort((a, b) => rank(a.kind) - rank(b.kind))
}
/** A class's record (pending_classes), for the claim's order. */
interface ClassRow {
tool: string
groupId: number
klass: number
probed: number
named: number
waiting: number
}
/** The record is read again this often. */
const CLASSES_MS = 60_000
/** A class found short of a claim's share is left alone this long. */
const EMPTY_MS = 5 * 60_000
/**
* Rows in one order for a statement that locks many of them (pending_releases by key, pending_classes and
* pending_signatures by theirs): the batch store, the record, the claim and the sweep all lock the same way, so two
* of them can't deadlock on each other (LOG.md (134)).
*/
export function lockOrder<T>(rows: Iterable<T>, key: (row: T) => string): T[] {
return [...rows].sort((a, b) => {
const ka = key(a)
const kb = key(b)
return ka < kb ? -1 : ka > kb ? 1 : 0
})
}
/** The prior rate of an unexplored class and its weight in probes: the ladder's overall rate on 2026-11-02 (LOG.md (130)). */
const PRIOR_RATE = 0.15
const PRIOR_WEIGHT = 10
/** One class takes at most this share of a claim: the rest explores the classes after it. */
const CLASS_SHARE = 0.25
/** The expected naming rate of a class: its record shrunk towards the prior. */
export function classScore(c: { probed: number; named: number }): number {
return (c.named + PRIOR_RATE * PRIOR_WEIGHT) / (c.probed + PRIOR_WEIGHT)
}
/** An UPDATE returns its rows in no order: the claim's (the cheapest target, then the oldest) restored. */
function claimed(rows: any[]): any[] {
const bytes = (r: any) =>
r.target_bytes === null ? Number.POSITIVE_INFINITY : Number(r.target_bytes)
return [...rows].sort(
(a, b) =>
bytes(a) - bytes(b) ||
new Date(a.created_at).getTime() - new Date(b.created_at).getTime() ||
Number(a.first_article) - Number(b.first_article)
)
}
function toProbe(r: any): PendingProbe {
const targets = byEffort(r.candidates as PendingTarget[])
const tries = Number(r.tries)
const target = targets[Math.min(tries, targets.length - 1)]
return {
releaseKey: r.release_key,
kind: target.kind,
messageId: target.messageId,
file: target.file,
tool: r.tool,
groupId: Number(r.group_id),
klass: Number(r.klass ?? 0),
poster: r.poster,
tries,
targets: targets.length,
}
}
/**
* The pending ladder (REBUILD.md decision f): pending_releases probed from least to most effort, a usable name
* handing the release to the namer (app/services/pending_namer.ts), no name after the last target exhausting it;
* every finished climb counted on its signature (tool, group, poster; and the tool and group together), and a
* signature with enough probes and next to no names dropped: its waiting releases are discarded without a probe,
* a sample of them (the rules' samplePercent) still climbing so the rate keeps being measured.
* every finished climb counted on its signature (tool, group, poster; and the tool and group together) and on its
* class (tool, group, size bucket × name shape × partial; lib/release/classes.ts, LOG.md (130)), and a signature
* or class with enough probes and next to no names dropped: its waiting releases are discarded without a probe,
* a sample of them (the rules' samplePercent) still climbing so the rate keeps being measured. The claim takes the
* classes with the best record first (classScore), the cheapest target first within one.
*/
class PendingLadder {
/** A previous prober's claims back in the queue. */
@@ -70,34 +145,84 @@ class PendingLadder {
return rowCount ?? 0
}
/** The next target of up to `n` waiting releases with a target left, oldest first, claimed. */
/** The ladder's record per class, cached CLASSES_MS between claims, best first; classes found empty, until when. */
private classes: ClassRow[] = []
private classesAt = 0
private empty = new Map<string, number>()
/** The record read again on the next claim (the tests). */
forget() {
this.classesAt = 0
this.empty.clear()
}
/** pending_classes with waiting releases and not dropped, by their expected rate (a prior for the unexplored). */
private async classRows(): Promise<ClassRow[]> {
if (Date.now() - this.classesAt < CLASSES_MS) return this.classes
const { rows } = await db.rawQuery(
`SELECT tool, group_id, klass, probed, named, waiting FROM pending_classes WHERE waiting > 0 AND NOT dropped`
)
this.classes = rows
.map((r: any) => ({
tool: r.tool as string,
groupId: Number(r.group_id),
klass: Number(r.klass),
probed: Number(r.probed),
named: Number(r.named),
waiting: Number(r.waiting),
}))
.sort((a: ClassRow, b: ClassRow) => classScore(b) - classScore(a))
this.classesAt = Date.now()
return this.classes
}
/**
* The next target of up to `n` waiting releases with a target left, claimed: by class, the best record first,
* at most CLASS_SHARE of the claim from one class (the rest explores the classes after it), the cheapest target
* and then the oldest within one; what the classes don't fill, the oldest of the rest (rows from before the
* classes, status 0 of a class without a row).
*/
async claim(n: number): Promise<PendingProbe[]> {
if (n <= 0) return []
const { rows } = await db.rawQuery(
`UPDATE pending_releases p SET status = ? FROM (
SELECT release_key FROM pending_releases
WHERE status = ? AND finished AND tries < jsonb_array_length(candidates)
ORDER BY created_at LIMIT ? FOR UPDATE SKIP LOCKED) c
WHERE p.release_key = c.release_key
RETURNING encode(p.release_key, 'hex') AS release_key, p.candidates, p.tries, p.tool, p.group_id, coalesce(p.poster, '') AS poster`,
[PENDING.taken, PENDING.waiting, n]
)
return rows.map((r: any) => {
const targets = byEffort(r.candidates as PendingTarget[])
const tries = Number(r.tries)
const target = targets[Math.min(tries, targets.length - 1)]
return {
releaseKey: r.release_key,
kind: target.kind,
messageId: target.messageId,
file: target.file,
tool: r.tool,
groupId: Number(r.group_id),
poster: r.poster,
tries,
targets: targets.length,
const out: PendingProbe[] = []
const perClass = Math.max(1, Math.floor(n * CLASS_SHARE))
const now = Date.now()
for (const c of await this.classRows()) {
if (out.length >= n) break
const key = `${c.tool}\t${c.groupId}\t${c.klass}`
if ((this.empty.get(key) ?? 0) > now) continue
const want = Math.min(perClass, n - out.length)
const { rows } = await db.rawQuery(
`UPDATE pending_releases p SET status = ? FROM (
SELECT release_key FROM pending_releases
WHERE status = ? AND tool = ? AND group_id = ? AND klass = ? AND finished AND tries < jsonb_array_length(candidates)
ORDER BY target_bytes NULLS LAST, created_at, first_article LIMIT ? FOR UPDATE SKIP LOCKED) c
WHERE p.release_key = c.release_key
RETURNING encode(p.release_key, 'hex') AS release_key, p.candidates, p.tries, p.tool, p.group_id, p.klass, coalesce(p.poster, '') AS poster, p.target_bytes, p.created_at, p.first_article`,
[PENDING.taken, PENDING.waiting, c.tool, c.groupId, c.klass, want]
)
if (rows.length < want) this.empty.set(key, now + EMPTY_MS)
if (rows.length > 0) {
await db.rawQuery(
`UPDATE pending_classes SET waiting = greatest(0, waiting - ?) WHERE tool = ? AND group_id = ? AND klass = ?`,
[rows.length, c.tool, c.groupId, c.klass]
)
}
})
out.push(...claimed(rows).map(toProbe))
}
if (out.length < n) {
const { rows } = await db.rawQuery(
`UPDATE pending_releases p SET status = ? FROM (
SELECT release_key FROM pending_releases
WHERE status = ? AND finished AND tries < jsonb_array_length(candidates)
ORDER BY created_at, first_article LIMIT ? FOR UPDATE SKIP LOCKED) c
WHERE p.release_key = c.release_key
RETURNING encode(p.release_key, 'hex') AS release_key, p.candidates, p.tries, p.tool, p.group_id, p.klass, coalesce(p.poster, '') AS poster, p.target_bytes, p.created_at, p.first_article`,
[PENDING.taken, PENDING.waiting, n - out.length]
)
out.push(...claimed(rows).map(toProbe))
}
return out
}
/**
@@ -106,6 +231,13 @@ class PendingLadder {
*/
async record(outcomes: PendingOutcome[], trx: TransactionClientContract | typeof db = db) {
if (outcomes.length === 0) return
// A twin by content names what the probe itself didn't (hash_twins.ts).
const twins = await twinNames(
outcomes
.filter((o) => o.status === 'done' && o.inspection?.hash16k && o.inspection.size)
.map((o) => ({ hash16k: o.inspection!.hash16k!, size: o.inspection!.size })),
trx
)
const rows = outcomes.map((o) => {
const p = o.probe
if (o.status === 'waiting') {
@@ -118,7 +250,12 @@ class PendingLadder {
final: null,
}
}
const name = o.status === 'done' && o.inspection ? usableName(o.inspection, p.kind) : null
const probed = o.status === 'done' && o.inspection ? usableName(o.inspection, p.kind) : null
const twin =
o.status === 'done' && o.inspection?.hash16k
? twins.get(twinKey(o.inspection.hash16k, o.inspection.size))
: undefined
const name = probed ?? twin ?? null
const tries = p.tries + 1
const inspection = {
kind: p.kind,
@@ -152,13 +289,26 @@ class PendingLadder {
final: null,
}
})
// The rows locked in key order first: the UPDATE's join below takes them in whatever order the plan does.
await trx.rawQuery(
`SELECT 1 FROM pending_releases WHERE release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
ORDER BY release_key FOR UPDATE`,
[rows.map((r) => r.release_key).sort()]
)
await trx.rawQuery(
`UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
WHERE p.release_key = decode(v.release_key, 'hex')`,
[JSON.stringify(rows.map(({ final: _final, ...row }) => row))]
[
JSON.stringify(
lockOrder(
rows.map(({ final: _final, ...row }) => row),
(r) => r.release_key
)
),
]
)
// The signatures: the poster's own and the tool's and group's together.
const counts = new Map<
@@ -188,42 +338,176 @@ class PendingLadder {
`INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
[JSON.stringify([...counts.values()])]
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`))]
)
await trx.rawQuery(
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float)
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float),
swept_at = CASE WHEN s.dropped THEN s.swept_at END
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
[
pendingMinProbes,
pendingMaxNamed,
JSON.stringify(
[...counts.values()].map(({ tool, group_id, poster }) => ({ tool, group_id, poster }))
lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`).map(
({ tool, group_id, poster }) => ({ tool, group_id, poster })
)
),
]
)
// The classes: the same record per (tool, group, class), and a probe given back is waiting again.
const classes = new Map<
string,
{
tool: string
group_id: number
klass: number
probed: number
named: number
waiting: number
}
>()
for (const [i, o] of outcomes.entries()) {
const key = `${o.probe.tool}\t${o.probe.groupId}\t${o.probe.klass}`
const entry = classes.get(key) ?? {
tool: o.probe.tool,
group_id: o.probe.groupId,
klass: o.probe.klass,
probed: 0,
named: 0,
waiting: 0,
}
const final = rows[i].final
if (final) entry.probed++
if (final === 'named') entry.named++
if (o.status === 'waiting') entry.waiting++
classes.set(key, entry)
}
await trx.rawQuery(
`INSERT INTO pending_classes (tool, group_id, klass, probed, named, waiting)
SELECT v.tool, v.group_id, v.klass, v.probed, v.named, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, probed int, named int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named,
waiting = pending_classes.waiting + excluded.waiting, updated_at = now()`,
[JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
)
await trx.rawQuery(
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float),
swept_at = CASE WHEN c.dropped THEN c.swept_at END
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int)
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
[
pendingMinProbes,
pendingMaxNamed,
JSON.stringify(
lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`).map(
({ tool, group_id, klass }) => ({ tool, group_id, klass })
)
),
]
)
}
/** Waiting releases of a dropped signature discarded without a probe, the explored sample excepted; how many. */
async discard(limit = 10_000): Promise<number> {
/**
* Waiting releases of a dropped class or signature discarded without a probe, the explored sample excepted; how
* many. Each dropped class or signature is swept once (swept_at; null again when the record drops it anew),
* `perSweep` of them at most, each by its index (the claim's for a class, the signature index for a poster's
* rows, the ladder's for a tool's and group's) in rounds of `limit`. The prober runs this beside the lanes, not
* in the claim (LOG.md (133)); the edge drops new releases of dropped signatures and classes itself, so one
* sweep is all a drop needs.
*/
async discard(limit = 10_000, perSweep = 20): Promise<number> {
const { samplePercent } = await RuleSets.current()
const { rows } = await db.rawQuery(
`SELECT encode(p.release_key, 'hex') AS release_key FROM pending_releases p
WHERE p.status = ? AND EXISTS (
SELECT 1 FROM pending_signatures s
WHERE s.dropped AND s.tool = p.tool AND s.group_id = p.group_id AND (s.poster = '' OR s.poster = coalesce(p.poster, '')))
LIMIT ?`,
[PENDING.waiting, limit]
let discarded = 0
const { rows: classes } = await db.rawQuery(
`SELECT tool, group_id, klass FROM pending_classes WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, klass LIMIT ?`,
[perSweep]
)
for (const c of classes) {
let rows: any[]
do {
;({ rows } = await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND tool = ? AND group_id = ? AND klass = ? LIMIT ?`,
[PENDING.waiting, c.tool, c.group_id, c.klass, limit]
))
discarded += await this.discardKeys(rows, samplePercent)
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
await db.rawQuery(
`UPDATE pending_classes SET swept_at = now() WHERE tool = ? AND group_id = ? AND klass = ?`,
[c.tool, c.group_id, c.klass]
)
}
const { rows: signatures } = await db.rawQuery(
`SELECT tool, group_id, poster FROM pending_signatures WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, poster LIMIT ?`,
[perSweep]
)
for (const s of signatures) {
let rows: any[]
do {
// The poster's rows by the signature index (on the poster's md5: a poster can be longer than a btree
// key), a tool's and group's by the ladder's: never a test on the rows, which was a scan of the whole
// table per signature on the HDD (LOG.md (134)).
;({ rows } =
s.poster === ''
? await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND group_id = ? AND tool = ? LIMIT ?`,
[PENDING.waiting, s.group_id, s.tool, limit]
)
: await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND group_id = ? AND tool = ? AND md5(poster) = md5(?::text) AND poster = ? LIMIT ?`,
[PENDING.waiting, s.group_id, s.tool, s.poster, s.poster, limit]
))
discarded += await this.discardKeys(rows, samplePercent)
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
await db.rawQuery(
`UPDATE pending_signatures SET swept_at = now() WHERE tool = ? AND group_id = ? AND poster = ?`,
[s.tool, s.group_id, s.poster]
)
}
return discarded
}
/** The rows discarded (status 2), the explored sample excepted, their classes' waiting counts lowered; how many. */
private async discardKeys(rows: any[], samplePercent: number): Promise<number> {
const keys = rows
.map((r: any) => r.release_key as string)
.filter((key: string) => !explored(key, samplePercent))
.sort()
if (keys.length === 0) return 0
const { rowCount } = await db.rawQuery(
`UPDATE pending_releases SET status = ? WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)`,
// Locked in key order (the record's lock order, LOG.md (137)): a plain `key = ANY(subquery)` locks in scan
// order and deadlocked against a batch upsert; ORDER BY sorts before FOR UPDATE locks.
const { rows: gone } = await db.rawQuery(
`UPDATE pending_releases p SET status = ?
FROM (SELECT release_key FROM pending_releases
WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
ORDER BY release_key FOR UPDATE) k
WHERE p.release_key = k.release_key
RETURNING p.tool, p.group_id, p.klass`,
[PENDING.discarded, PENDING.waiting, keys]
)
return rowCount ?? 0
const counts = new Map<string, { tool: string; group_id: number; klass: number; n: number }>()
for (const g of gone) {
const key = `${g.tool}\t${g.group_id}\t${g.klass}`
const entry = counts.get(key) ?? {
tool: g.tool,
group_id: Number(g.group_id),
klass: Number(g.klass),
n: 0,
}
entry.n++
counts.set(key, entry)
}
if (counts.size > 0) {
await db.rawQuery(
`UPDATE pending_classes c SET waiting = greatest(0, c.waiting - v.n)
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, n int)
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
)
}
return gone.length
}
}
+74 -50
View File
@@ -30,6 +30,13 @@ interface NamedRow {
/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
const MAX_MOVES = 3
/**
* The widest article span the namer reads again (a busy group posts 50M articles a day, a large upload among
* them spans a million or two): one wider is a name the edge saw again much later ("v-lgpc.3ds"), and its span
* of segments would not fit the indexer's memory (LOG.md (133)).
*/
const MAX_SPAN = 3_000_000
/**
* The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
* read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
@@ -94,6 +101,10 @@ export class PendingNamer {
/** One named release into the catalog. */
async move(row: NamedRow): Promise<void> {
if (row.last_article - row.first_article > MAX_SPAN)
throw new Error(
`its articles ${row.first_article}–${row.last_article} span more than ${MAX_SPAN}`
)
const group = await db.from('groups').where('id', row.group_id).firstOrFail()
const range = await db
.from('ranges')
@@ -103,19 +114,20 @@ export class PendingNamer {
.orderBy('id')
.first()
if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
const text = await this.overviewText(group.name, row)
const rules = await RuleSets.current()
const ix = new RangeIndexer({
group: group.name,
start: row.first_article,
end: row.last_article,
rules: { ...rules, samplePercent: 0 },
idleChunks: 0,
// No file given up before the end: the span is read chunk by chunk (one text of it topped V8's string).
idleChunks: Number.MAX_SAFE_INTEGER,
edgeArticles: 0,
// The edge wrote it partial when its file is short of parts on the server: so must the re-read.
partialFiles: true,
})
ix.addText(text, row.last_article + 1)
for await (const piece of this.overviewPieces(group.name, row))
ix.addText(piece.text, piece.next)
ix.finish()
const batch = ix.take()
const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
@@ -160,50 +172,20 @@ export class PendingNamer {
})
}
/** The overview lines of the release's article span: the stored corpus chunks when its range is stored, else XOVER. */
private async overviewText(groupName: string, row: NamedRow): Promise<string> {
if (this.corpusDir) {
const stored = await db
.from('ranges')
.where('group_id', row.group_id)
.whereNotNull('stored_at')
.where('start_id', '<=', row.last_article)
.where('end_id', '>=', row.first_article)
.orderBy('start_id')
const parts: string[] = []
let covered = true
for (const r of stored) {
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
const size = coordinatorConfig.chunkArticles
for (
let from =
range.start +
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
from <= Math.min(row.last_article, range.end);
from += size
) {
const raw = await Corpus.read(
range,
{ from, to: Math.min(from + size - 1, range.end) },
this.corpusDir
)
if (!raw) {
covered = false
break
}
parts.push((await inflate(raw)).toString('latin1'))
}
if (!covered) break
}
if (
covered &&
parts.length > 0 &&
stored.length > 0 &&
Number(stored[0].start_id) <= row.first_article &&
Number(stored[stored.length - 1].end_id) >= row.last_article
) {
return parts.join('')
}
/**
* The overview lines of the release's article span, a chunk of articles at a time with the first article after
* it: the stored corpus chunks when its ranges are stored and contiguous, else XOVER in windows of a chunk.
*/
private async *overviewPieces(
groupName: string,
row: NamedRow
): AsyncGenerator<{ text: string; next: number }> {
const size = coordinatorConfig.chunkArticles
const stored = await this.storedChunks(groupName, row)
if (stored) {
for (const { raw, next } of stored)
yield { text: (await inflate(raw)).toString('latin1'), next }
return
}
this.client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
@@ -217,13 +199,55 @@ export class PendingNamer {
this.usedAt = Date.now()
try {
await this.client.group(groupName)
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
const raw = await this.client.xoverRaw(row.first_article, row.last_article)
return (await inflate(raw)).toString('latin1')
for (let from = row.first_article; from <= row.last_article; from += size) {
const to = Math.min(from + size - 1, row.last_article)
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
const raw = await this.client.xoverRaw(from, to)
yield { text: (await inflate(raw)).toString('latin1'), next: to + 1 }
}
} catch (error) {
this.client.close()
this.client = null
throw error
}
}
/** The span's stored corpus chunks (compressed, with the first article after each), or null when any is missing. */
private async storedChunks(
groupName: string,
row: NamedRow
): Promise<{ raw: Buffer; next: number }[] | null> {
if (!this.corpusDir) return null
const stored = await db
.from('ranges')
.where('group_id', row.group_id)
.whereNotNull('stored_at')
.where('start_id', '<=', row.last_article)
.where('end_id', '>=', row.first_article)
.orderBy('start_id')
if (
stored.length === 0 ||
Number(stored[0].start_id) > row.first_article ||
Number(stored[stored.length - 1].end_id) < row.last_article
)
return null
const size = coordinatorConfig.chunkArticles
const chunks: { raw: Buffer; next: number }[] = []
for (const r of stored) {
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
for (
let from =
range.start +
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
from <= Math.min(row.last_article, range.end);
from += size
) {
const to = Math.min(from + size - 1, range.end)
const raw = await Corpus.read(range, { from, to }, this.corpusDir)
if (!raw) return null
chunks.push({ raw, next: to + 1 })
}
}
return chunks
}
}
+2
View File
@@ -561,6 +561,8 @@ export class Prefetcher {
const groups = new Map<string, number>()
const candidates = await rows(db.query())
.where('r.capture', true)
// A group taken off the list (groups.active) isn't worth the connections.
.where('g.active', true)
.whereNull('r.stored_at')
.where((q: any) =>
q
+49 -8
View File
@@ -9,6 +9,7 @@ import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/i
import { NntpShare } from '#services/nntp_shares'
import { DiskSpace } from '#services/disk_space'
import { decidePar2Completeness } from '#services/par2_completeness'
import { twinKey, twinNames } from '#services/hash_twins'
import PendingLadder, { type PendingProbe } from '#services/pending_ladder'
import { PendingNamer } from '#services/pending_namer'
@@ -40,6 +41,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
const FLUSH_MS = 2000
/** Between sweeps that discard the waiting releases of dropped signatures. */
const SWEEP_MS = 5 * 60_000
/** A flush that deadlocks (against the batch store's pending upsert) keeps its outcomes for the next, this often. */
const DEADLOCK_RETRIES = 5
/**
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
@@ -81,6 +84,8 @@ export class Prober {
private refilling: Promise<void> | null = null
private stopped = false
private stats = Prober.freshStats()
/** Flushes deadlocked in a row, their outcomes kept. */
private deadlocks = 0
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
private lanes = new Set<Promise<void>>()
private running = 0
@@ -96,7 +101,6 @@ export class Prober {
}
/** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
private namer = new PendingNamer()
private sweptAt = 0
async run() {
// A previous run's claims: one prober at a time.
@@ -105,6 +109,17 @@ export class Prober {
const released = await PendingLadder.release()
if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
// The ladder's discard sweep beside the lanes: a dropped signature's rows take minutes on the HDD (LOG.md (133)).
const sweeper = (async () => {
while (!this.stopped) {
try {
this.stats.discarded += await PendingLadder.discard()
} catch (error: any) {
logger.error(`Probing: the discard sweep failed: ${error.message}`)
}
for (let waited = 0; waited < SWEEP_MS && !this.stopped; waited += 1000) await sleep(1000)
}
})()
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
const flusher = (async () => {
while (!this.stopped || this.outcomes.length > 0) {
@@ -135,6 +150,7 @@ export class Prober {
}
await Promise.all(this.lanes)
await namer
await sweeper
await this.share.leave().catch(() => {})
// What was claimed and not probed goes back.
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
@@ -265,10 +281,6 @@ export class Prober {
/** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
private async claimPending() {
if (Date.now() - this.sweptAt > SWEEP_MS) {
this.sweptAt = Date.now()
this.stats.discarded += await PendingLadder.discard()
}
const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
const probes = await PendingLadder.claim(want)
if (probes.length === 0) return
@@ -338,15 +350,34 @@ export class Prober {
}
}
/** The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. */
/**
* The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. A deadlock
* (the batch store's pending upsert locks the same rows; LOG.md (134)) keeps the outcomes for the next flush
* instead of losing the probes.
*/
private async flush() {
const taken = this.outcomes.splice(0)
if (taken.length === 0) return
try {
await this.store(taken)
} catch (error: any) {
if (error?.code === '40P01' && this.deadlocks < DEADLOCK_RETRIES) {
this.deadlocks++
this.outcomes.unshift(...taken)
logger.warn(`Probing: storing results deadlocked, kept for the next flush (${this.deadlocks}).`)
return
}
throw error
}
this.deadlocks = 0
for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
this.stats.pending += taken.filter((o) => o.candidate.pending && o.status !== 'waiting').length
}
private async store(taken: Outcome[]) {
// The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
const pending = taken.filter((o) => o.candidate.pending)
const outcomes = taken.filter((o) => !o.candidate.pending)
this.stats.pending += pending.filter((o) => o.status !== 'waiting').length
await db.transaction(async (trx) => {
if (outcomes.length > 0) {
await trx.rawQuery(
@@ -392,10 +423,20 @@ export class Prober {
entry.all.push({ kind: r.kind, result: r.result })
byRelease.set(id, entry)
}
// A twin by content names a release its own inspections didn't (hash_twins.ts; LOG.md (130)).
const twins = await twinNames(
done.filter((o) => o.inspection!.hash16k && o.inspection!.size).map((o) => ({ hash16k: o.inspection!.hash16k!, size: o.inspection!.size })),
trx
)
const twinOf = new Map<number, string>()
for (const o of done) {
const twin = o.inspection!.hash16k ? twins.get(twinKey(o.inspection!.hash16k, o.inspection!.size)) : undefined
if (twin && twin !== byRelease.get(o.candidate.releaseId)?.name) twinOf.set(o.candidate.releaseId, twin)
}
const updates = [...byRelease].map(([id, { name, all }]) => ({
id,
passworded: all.some((i) => i.result.passworded === true),
inner_name: innerName(all, name),
inner_name: innerName(all, name) ?? twinOf.get(id) ?? null,
}))
await trx.rawQuery(
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
+23
View File
@@ -1,6 +1,7 @@
import db from '@adonisjs/lucid/services/db'
import logger from '@adonisjs/core/services/logger'
import { SOURCES } from '#lib/release/source'
import { TINY_MIN_BYTES, NONSENSE_MIN_BYTES } from '#lib/rules/index'
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
export const JUDGE_VERSION = 1
@@ -28,6 +29,14 @@ const MEDIA =
const SCRAPED_POSTERS = `celeb[-_ ]{0,1}movie|celebwow|celebfx|iluvcelebs|nude[-_ ]{0,1}celebs|celeb-sex-tapes`
/** An executable's name. */
const EXECUTABLE = `\\.(exe|scr|com|bat|cmd|msi|lnk|vbs|js|jar|pif|cpl|hta|ps1|dll)$`
/**
* The tiny rule's name tests, lib/rules/rule_set.ts's RELEASE_LIKE_SPACED, EBOOK, TINY_CONTENT and
* STUB_EXEMPT_GROUPS in Postgres's dialect (and no "?", see MEDIA).
*/
const TINY_RELEASE_LIKE = `(\\y(19|20)[0-9][0-9]\\y|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-{0,1}dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\\.[0-9]+)+)\\y|-[a-z0-9]{2,}$|\\.[a-z0-9]+\\.[a-z0-9]+\\.)`
const TINY_EBOOK = `\\y(epub|mobi|azw3{0,1}|pdf|djvu|cbr|cbz|e-{0,1}books{0,1})\\y`
const TINY_CONTENT = `\\y(subs{0,1}|subtitles{0,1}|subpack|srt|untertitel|ondertitels{0,1}|ondertiteling|legendas{0,1}|napisy|sottotitoli|cracks{0,1}|cracked|keygen|keymaker|trainer|patch|patched|serials{0,1}|activator|loader|nocd|no-cd|fix|covers{0,1}|artwork|booklet|scans{0,1}|proof|sample|info|readme)\\y`
const TINY_EXEMPT_GROUPS = `(sound|mp3|karaoke|music|audio|flac|lossless|e-{0,1}book|epub|comic|guitar|font|magazine)`
export interface Rule {
name: string
@@ -100,6 +109,20 @@ export const RULES: Rule[] = [
// releases_passworded_idx: a scan of all of releases was 30 s a round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
},
{
name: 'tiny',
says: 'one file of one article, 2–100 kB, under a release-like name: the exe-stub flood in miniature',
// The catalog's side of the edge's tiny-archive rule (lib/rules/rule_set.ts isTinyArchive; LOG.md (131),
// (135)): the same name tests in Postgres's regex dialect ("\y" for "\b", "{0,1}" for "?"), the readable flag
// for its scrambled and random tests, the group set for its groups. The 2015–17 flood that the edge indexed
// before the rule: 1.5M rows, 98% of them with an archive candidate, the rest lone par2 files and scrambled
// names, so no join on the candidates. A scan of releases (9 GB) a full round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
WHERE r.files = 1 AND r.segments = 1 AND r.bytes >= ${TINY_MIN_BYTES} AND r.bytes < ${NONSENSE_MIN_BYTES}
AND r.readable AND NOT coalesce(r.sidecar_only, false)
AND r.name ~* '${TINY_RELEASE_LIKE}' AND r.name !~* '${TINY_EBOOK}' AND r.name !~* '${TINY_CONTENT}'
AND NOT EXISTS (SELECT 1 FROM group_sets gs WHERE gs.id = r.group_set_id AND gs.groups::text ~* '${TINY_EXEMPT_GROUPS}')`,
},
{
name: 'unusable',
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
+18 -3
View File
@@ -85,6 +85,19 @@ function contains(word: string): string {
return `%${word.replace(/[\\%_]/g, (c) => `\\${c}`)}%`
}
/**
* A search word's condition on release_texts: anywhere in a text (ILIKE), or with a leading "^" at a word's start, the
* text's or after a character that's no letter or digit ("^zed" finds "Zed.Dead", "The_Zed", not "Facialized"). Both
* use the trigram index.
*/
function wordMatch(word: string): { sql: string; binding: string } {
if (word.startsWith('^') && word.length > 1) {
const rest = word.slice(1).replace(/[\\^$.|?*+()[\]{}]/g, (c) => `\\${c}`)
return { sql: 't.text ~* ?', binding: `(^|[^[:alnum:]])${rest}` }
}
return { sql: 't.text ILIKE ?', binding: contains(word) }
}
/** Searching and browsing the catalog, for the web interface and Newznab. */
class SearchService {
async search(params: SearchParams) {
@@ -93,10 +106,12 @@ class SearchService {
if (!params.judged) where.push('NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = r.id)')
const bindings: unknown[] = []
// Every word in one of the release's texts (its name, the names the prober found inside; release_texts, by its
// trigram index, which yields ids: never an OR across the two tables); "The.Block.AU" and "the block au" alike.
// trigram index, which yields ids: never an OR across the two tables); "The.Block.AU" and "the block au" alike;
// "^zed" only at a word's start.
for (const word of (params.q ?? '').split(/[\s._-]+/).filter(Boolean).slice(0, 8)) {
where.push('r.id IN (SELECT t.release_id FROM release_texts t WHERE t.text ILIKE ?)')
bindings.push(contains(word))
const match = wordMatch(word)
where.push(`r.id IN (SELECT t.release_id FROM release_texts t WHERE ${match.sql})`)
bindings.push(match.binding)
}
if (params.readable) where.push('(r.readable OR r.inner_name IS NOT NULL)')
if (params.complete) where.push('r.complete')
+105
View File
@@ -0,0 +1,105 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import coordinatorConfig from '#config/coordinator'
import { describeClass } from '#lib/release/index'
/**
* The pending ladder's classes (LOG.md (130)) on the rows from before them: `klass` (the SQL pending_class) and
* `target_bytes` (the first target's bytes, when its candidates carry them). The finished rows (named, exhausted,
* naming) in one scan, seeding each class's record, which gets its verdict (dropped) as the ladder gives it; the
* waiting rows in windows of created_at by their index (a scan per window starved the prober on the HDD); then
* pending_classes' `waiting` counted afresh from the claim's index. --report shows the record.
*/
export default class PendingClasses extends BaseCommand {
static commandName = 'pending:classes'
static description =
'Classes the pending releases from before the classes and recounts pending_classes; --report shows the record'
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'Hours of created_at per window' })
declare hours: number
@flags.boolean({ description: 'Show the classes with the best and worst records, write nothing' })
declare report: boolean
async run() {
if (this.report) return this.showReport()
const hours = this.hours ?? 6
const started = Date.now()
// The finished rows (named, exhausted, naming; 6.5M on 2026-11-02) in one pass: a scan of the table, no index
// reaches them by status; they seed each class's record.
const { rows: seeded } = await db.rawQuery(
`WITH classed AS (
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
WHERE klass = 0 AND status IN (1, 3, 4)
RETURNING tool, group_id, klass, status),
upserted AS (
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed GROUP BY 1, 2, 3
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
RETURNING 1)
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM upserted) AS classes`
)
this.logger.info(
`Finished rows: ${seeded[0].classed} classed into ${seeded[0].classes} classes (${Math.round((Date.now() - started) / 1000)} s).`
)
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
await db.rawQuery(
`UPDATE pending_classes SET dropped = true, swept_at = NULL WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
[pendingMinProbes, pendingMaxNamed]
)
// The waiting rows by windows of created_at (pending_releases_waiting_idx reaches them).
const { rows: span } = await db.rawQuery(
`SELECT (SELECT min(created_at) FROM pending_releases WHERE status = 0) AS lo, (SELECT max(created_at) FROM pending_releases WHERE status = 0) AS hi`
)
if (span[0].lo) {
let from = new Date(span[0].lo)
const hi = new Date(span[0].hi)
let classed = 0
while (from <= hi) {
const to = new Date(from.getTime() + hours * 3600_000)
const result = await db.rawQuery(
`UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
WHERE status = 0 AND created_at >= ? AND created_at < ? AND klass = 0`,
[from.toISOString(), to.toISOString()]
)
classed += result.rowCount ?? 0
from = to
this.logger.info(
`Waiting rows up to ${to.toISOString().slice(0, 16)}: ${classed} classed (${Math.round((Date.now() - started) / 1000)} s).`
)
}
}
// The waiting counts from the rows themselves (the claim, the discard and the batch store keep them from here).
await db.rawQuery(`
INSERT INTO pending_classes (tool, group_id, klass, waiting)
SELECT tool, group_id, klass, count(*) FROM pending_releases WHERE status = 0 AND finished GROUP BY 1, 2, 3
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = excluded.waiting`)
await db.rawQuery(
`UPDATE pending_classes c SET waiting = 0 WHERE waiting > 0 AND NOT EXISTS (
SELECT 1 FROM pending_releases p WHERE p.status = 0 AND p.finished AND p.tool = c.tool AND p.group_id = c.group_id AND p.klass = c.klass)`
)
const { rows: totals } = await db.rawQuery(
`SELECT count(*) AS classes, coalesce(sum(waiting), 0) AS waiting, count(*) FILTER (WHERE dropped) AS dropped FROM pending_classes`
)
this.logger.info(
`Done: ${totals[0].classes} classes, ${totals[0].waiting} waiting, ${totals[0].dropped} dropped.`
)
}
private async showReport() {
const { rows } = await db.rawQuery(
`SELECT c.tool, g.name AS grp, c.klass, c.probed, c.named, c.waiting, c.dropped FROM pending_classes c JOIN groups g ON g.id = c.group_id
WHERE c.probed >= 20 ORDER BY c.named::float / c.probed DESC, c.probed DESC`
)
const line = (r: any) =>
` ${r.tool.padEnd(10)} ${r.grp.padEnd(32)} ${describeClass(Number(r.klass)).padEnd(26)} probed ${String(r.probed).padStart(7)} named ${String(Math.round((100 * r.named) / r.probed)).padStart(3)}% waiting ${String(r.waiting).padStart(8)}${r.dropped ? ' dropped' : ''}`
this.logger.info(`${rows.length} classes with 20+ climbs; the best:`)
for (const r of rows.slice(0, 25)) this.logger.info(line(r))
this.logger.info('The worst:')
for (const r of rows.slice(-25)) this.logger.info(line(r))
}
}
+90
View File
@@ -0,0 +1,90 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { isUnreadableName } from '#lib/release/index'
/** Its place in worker_cursors (the last release id done), for a run that was stopped. */
const CURSOR = 'readable'
/**
* The readable flag of the catalog's releases by today's classifier (isUnreadableName, LOG.md (129)): rows flagged
* readable whose name no one can read ("Zx5AjNaK-mh7CFg6Hf8uu", "Description - Lt5nHjlo3RiuBb3C") get readable =
* false, so the readable filter hides them (an inner name from the prober still shows them), and their probe
* candidates marked unreadable, the prober's first tiers. By the release name,
* which is the shared file stem for most rows; not by the chunks' file names, which the edge judges from now on.
* Batches by id, a dry run by default; --apply writes, and goes on from its cursor when run again.
*/
export default class ReleasesReadable extends BaseCommand {
static commandName = 'releases:readable'
static description =
'Clears the readable flag of catalog rows whose name is unreadable by the current classifier; counts only without --apply'
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'Release ids per batch' })
declare batch: number
@flags.number({ description: 'Stop after this id (a sample)' })
declare toId: number
@flags.boolean({ description: 'Write the flag; without it, count and show a sample' })
declare apply: boolean
@flags.boolean({ description: 'Start from the first id, not the cursor' })
declare restart: boolean
async run() {
const batch = this.batch ?? 20_000
const { rows: ends } = await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM releases`)
const hi = Math.min(Number(ends[0].hi), this.toId ?? Infinity)
const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [
CURSOR,
])
let after =
this.restart || !this.apply || !cursors[0] ? Number(ends[0].lo) - 1 : Number(cursors[0].key)
let looked = 0
let changed = 0
const sample: string[] = []
const started = Date.now()
while (after < hi) {
const to = Math.min(after + batch, hi)
const { rows } = await db.rawQuery(
`SELECT id, name FROM releases WHERE id > ? AND id <= ? AND readable`,
[after, to]
)
looked += rows.length
const ids: number[] = []
for (const r of rows) {
if (!isUnreadableName(r.name)) continue
ids.push(Number(r.id))
if (sample.length < 40) sample.push(r.name)
}
changed += ids.length
if (this.apply) {
if (ids.length > 0) {
await db.rawQuery(`UPDATE releases SET readable = false WHERE id = ANY(?::bigint[])`, [
ids,
])
// Their probe candidates to the unreadable tiers, which the prober takes first (LOG.md (130)).
await db.rawQuery(
`UPDATE probe_candidates SET unreadable = true WHERE release_id = ANY(?::bigint[]) AND NOT unreadable`,
[ids]
)
}
await db.rawQuery(
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
[CURSOR, String(to)]
)
}
after = to
if (Math.floor(after / 2_000_000) !== Math.floor((after - batch) / 2_000_000)) {
this.logger.info(
`Up to id ${after} of ${hi}: ${changed} unreadable of ${looked} readable (${Math.round((Date.now() - started) / 1000)} s).`
)
}
}
this.logger.info(
`${this.apply ? 'Cleared' : 'Would clear'} ${changed} of ${looked} readable rows.`
)
if (!this.apply) for (const n of sample) this.logger.info(` ${n}`)
}
}
+54
View File
@@ -0,0 +1,54 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import RuleSets from '#services/rule_sets'
import { classKey, describeClass } from '#lib/release/index'
/**
* Publishes a rule set whose dropped classes are the pending ladder's (pending_classes.dropped; LOG.md (130)): the
* edge keeps unnamed releases of those classes out of the pending store from then on, their explored sample
* excepted. Run again as the record grows; unchanged classes publish nothing.
*/
export default class RulesClasses extends BaseCommand {
static commandName = 'rules:classes'
static description = "Publishes a rule set with the pending ladder's dropped classes"
static options: CommandOptions = { startApp: true }
@flags.boolean({ description: 'Show what would be published, write nothing' })
declare dryRun: boolean
async run() {
const { rows } = await db.rawQuery(
`SELECT c.tool, g.name AS grp, c.klass, c.probed, c.named FROM pending_classes c JOIN groups g ON g.id = c.group_id WHERE c.dropped ORDER BY 1, 2, 3`
)
const next = new Set<string>(rows.map((r: any) => classKey(r.tool, r.grp, Number(r.klass))))
const { version, priorities: _p, ...current } = await RuleSets.current()
const before = new Set(current.droppedClasses ?? [])
const added = [...next].filter((c) => !before.has(c))
const removed = [...before].filter((c) => !next.has(c))
if (added.length === 0 && removed.length === 0) {
this.logger.info(
`Unchanged: rule set ${version} has the ${before.size} dropped classes already.`
)
return
}
for (const r of rows.slice(0, 40)) {
this.logger.info(
` ${r.tool.padEnd(10)} ${r.grp.padEnd(32)} ${describeClass(Number(r.klass)).padEnd(26)} ${r.named} of ${r.probed} named`
)
}
if (this.dryRun) {
this.logger.info(
`Would publish ${next.size} dropped classes (${added.length} added, ${removed.length} removed).`
)
return
}
const published = await RuleSets.publish(
{ ...current, droppedClasses: [...next].sort() },
`${next.size} dropped classes from the pending ladder`
)
this.logger.info(
`Rule set ${published}: ${next.size} dropped classes (${added.length} added, ${removed.length} removed).`
)
}
}
@@ -0,0 +1,57 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The pending ladder by class (REBUILD.md decision f; LOG.md (130)): a pending release's class (lib/release/classes.ts:
* size bucket × name shape × partial, 1..70; 0 for rows from before) and the bytes of its first target, so the
* claim takes the classes with the best record first and the cheapest article within one; pending_classes per
* (tool, group, class) how many climbed to the end and how many got a name, `waiting` how many wait (kept by the
* batch store, the claim and the discard; recounted by pending:classes), `dropped` as pending_signatures.
* The SQL pending_class() is the TypeScript's pendingClass, for the batch store's insert and the backfill.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`
CREATE OR REPLACE FUNCTION pending_class(name text, bytes bigint, partial boolean) RETURNS smallint LANGUAGE sql IMMUTABLE AS $$
SELECT ((CASE WHEN bytes < 1000000 THEN 0 WHEN bytes < 10000000 THEN 1 WHEN bytes < 100000000 THEN 2 WHEN bytes < 1000000000 THEN 3 ELSE 4 END) * 14
+ (CASE WHEN s ~ '^[0-9a-fA-F]{16,}$' THEN 1
WHEN s ~ '^[0-9]{6,}$' THEN 2
WHEN s ~ '^[a-z2-7]{20,}$' AND s ~ '[2-7]' THEN 3
WHEN s ~ '^[A-Za-z]+$' THEN 4
WHEN s ~ '^[A-Za-z0-9]+$' THEN 5
WHEN s ~ '^[A-Za-z0-9._-]+$' THEN 6
ELSE 7 END - 1) * 2
+ (CASE WHEN partial THEN 1 ELSE 0 END) + 1)::smallint
FROM (SELECT regexp_replace(name, '\\.[A-Za-z0-9]{1,4}$', '') AS s) x
$$`)
this.schema.raw(
`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS klass smallint NOT NULL DEFAULT 0`
)
this.schema.raw(`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS target_bytes bigint`)
this.schema.raw(`
CREATE TABLE IF NOT EXISTS pending_classes (
tool varchar(16) NOT NULL,
group_id integer NOT NULL,
klass smallint NOT NULL,
probed integer NOT NULL DEFAULT 0,
named integer NOT NULL DEFAULT 0,
waiting integer NOT NULL DEFAULT 0,
dropped boolean NOT NULL DEFAULT false,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (tool, group_id, klass)
)`)
// The claim by class: the cheapest target first within one, the oldest among equals.
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_class_idx ON pending_releases (tool, group_id, klass, target_bytes, created_at) WHERE status = 0`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_class_idx`)
this.schema.raw(`DROP TABLE IF EXISTS pending_classes`)
this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS target_bytes`)
this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS klass`)
this.schema.raw(`DROP FUNCTION IF EXISTS pending_class(text, bigint, boolean)`)
}
}
@@ -0,0 +1,20 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The prober's start puts a previous run's claims back (PendingLadder.release: status 9 → 0), which scanned the
* 58M-row pending table on the HDD for 20+ minutes after every deploy (LOG.md (130)): a partial index on the few
* taken rows makes it instant.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_taken_idx ON pending_releases (release_key) WHERE status = 9`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_taken_idx`)
}
}
@@ -0,0 +1,20 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The prober's start puts a previous run's claims back (probe_candidates status taken → waiting), which scanned
* the 7.8 GB candidates table on the HDD for 4 minutes after every deploy (LOG.md (133)): a partial index on the
* few taken rows makes it instant.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS probe_candidates_taken_idx ON probe_candidates (id) WHERE status = 'taken'`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS probe_candidates_taken_idx`)
}
}
@@ -0,0 +1,26 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The ladder's discard sweep (PendingLadder.discard) reads a dropped signature's or class's waiting rows once:
* `swept_at` marks it done (null again when the record drops it anew). Before, every sweep read every dropped
* signature's rows over again, minutes of HDD each, inside the claim (LOG.md (133)).
*/
export default class extends BaseSchema {
async up() {
this.schema.alterTable('pending_signatures', (table) => {
table.timestamp('swept_at', { useTz: true }).nullable()
})
this.schema.alterTable('pending_classes', (table) => {
table.timestamp('swept_at', { useTz: true }).nullable()
})
}
async down() {
this.schema.alterTable('pending_signatures', (table) => {
table.dropColumn('swept_at')
})
this.schema.alterTable('pending_classes', (table) => {
table.dropColumn('swept_at')
})
}
}
@@ -0,0 +1,23 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The discard sweep of a dropped signature (tool, group, poster) read the whole pending_releases table per
* signature: the planner took a parallel seq scan over the poster test, four minutes on the HDD that starved the
* lanes' claims (LOG.md (134)). The waiting rows by signature make it an index range. By the poster's md5: a
* poster of 2.7 kB broke the first build (a btree key is at most a third of a page), which left an invalid index
* behind that IF NOT EXISTS would have taken for the real one; the drop clears it.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_signature_idx ON pending_releases (group_id, tool, md5(poster)) WHERE status = 0`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
}
}
@@ -0,0 +1,34 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* probe_candidates, with its indexes, in the catalog's tablespace (default_tablespace: nvme on homelab), out of
* CHUNK_TABLESPACE where migration 6000 put it: the prober's catalog claim updates it from random points every
* minute, and on the HDD, beside the pending ladder's sweep and autovacuum, a claim took 2–3 minutes with all the
* lanes waiting (LOG.md 2026-11-03 (136)). Homelab moved it by hand (edge, node-api and the workers stopped; ~8 GB);
* a no-op where it's there already, or without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(
`SELECT current_setting('default_tablespace') AS target`
)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE (c.relname = 'probe_candidates' OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = 'probe_candidates'::regclass))
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
ORDER BY c.relkind DESC, c.relname`,
[target]
)
for (const { relname, relkind } of rows) {
await db.rawQuery(
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
)
}
})
}
async down() {}
}
@@ -0,0 +1,37 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Everything the prober reads or writes in the catalog's tablespace (default_tablespace: nvme on homelab), out of
* CHUNK_TABLESPACE (the HDD): pending_releases with its two HDD indexes (migration 6300), release_inspections
* (6000), probe_cache with its toast. The edge's batch upserts into pending_releases took minutes each on the HDD and
* the prober's flushes waited on them (LOG.md 2026-11-03 (137)); and with these on the NVMe the prober can run
* through the night without the HDD. Homelab moved them by hand (edge, node-api and the workers stopped; ~66 GB);
* a no-op where they're there already, or without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(
`SELECT current_setting('default_tablespace') AS target`
)
const target = setting[0].target
if (!target) return
for (const table of ['pending_releases', 'release_inspections', 'probe_cache']) {
const { rows } = await db.rawQuery(
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE (c.relname = ? OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = ?::regclass))
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
ORDER BY c.relkind DESC, c.relname`,
[table, table, target]
)
for (const { relname, relkind } of rows) {
await db.rawQuery(
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
)
}
}
})
}
async down() {}
}
+3 -2
View File
@@ -47,8 +47,9 @@ echo "== recreating app services"
docker compose stop edge
if [ -e "$CACHE/paused" ]; then
# Paused for the night (~/bin/usenet-rebuild-pause.sh): the workers stay stopped, the resume starts the new build.
echo "== paused: coordinator, prober, merger, judge, spotter, spot-inspector and edge left stopped"
docker compose up -d --no-deps node-api web
# The prober runs through the night (2026-11-03, LOG.md (137)), so it gets the new build now.
echo "== paused: coordinator, merger, judge, spotter, spot-inspector and edge left stopped"
docker compose up -d --no-deps node-api web prober
else
docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
if [ -e "$CACHE/no-edge" ]; then
+4 -3
View File
@@ -1,12 +1,13 @@
#!/bin/sh
# Installed as ~/bin/usenet-rebuild-pause.sh (by hand). Nightly (homelab crontab, 23:45): voyager 2's processing paused for the night; usenet-rebuild-resume.sh (07:45)
# starts it again. Postgres, web and node-api stay up. Takes the deploy lock, so it never runs during a CI deploy, and
# leaves .cache/paused, so a deploy in the night doesn't start the workers again.
# starts it again. Postgres, web, node-api and the prober stay up: since 2026-11-03 (LOG.md (137)) everything the
# prober touches is on the NVMe, so it probes through the night without the HDD. Takes the deploy lock, so it never
# runs during a CI deploy, and leaves .cache/paused, so a deploy in the night doesn't start the workers again.
cd /mnt/bulk/usenet-rebuild || exit 1
exec 9>.cache/deploy.lock
flock 9
touch .cache/paused
date "+%F %T pausing"
docker compose stop edge
docker compose stop coordinator prober merger judge spotter spot-inspector
docker compose stop coordinator merger judge spotter spot-inspector
docker compose ps --format "{{.Service}} {{.Status}}"
+41 -14
View File
@@ -5,7 +5,7 @@
use crate::codec::{encode_chunk, ChunkFile};
use crate::js::{self, sha1_hex};
use crate::names::{self, is_random_file, JudgedFile, Release, Rules, Verdict};
use crate::names::{self, is_random_file, is_unreadable_name, looks_scrambled, JudgedFile, Release, Rules, Verdict};
use rustc_hash::{FxHashMap, FxHashSet};
use serde::{Deserialize, Serialize};
use std::rc::Rc;
@@ -72,6 +72,8 @@ pub struct Candidate {
pub kind: &'static str,
pub message_id: String,
pub file: String,
/// The target article's bytes: the cheapest probe first within a class (LOG.md (130)).
pub bytes: i64,
}
#[derive(Default)]
@@ -1176,11 +1178,11 @@ impl RangeIndexer {
let t = std::time::Instant::now();
let done = self.aggregator.add(segments);
unsafe { PROF[3] += t.elapsed().as_nanos() as u64; }
self.files(done);
self.files(done, false);
self.frontier.update();
let t = std::time::Instant::now();
let gone = self.aggregator.evict(self.frontier.closed_before(), self.idle, false);
self.files(gone);
self.files(gone, false);
self.close(false);
unsafe { PROF[4] += t.elapsed().as_nanos() as u64; }
self.batch.next = next;
@@ -1191,7 +1193,7 @@ impl RangeIndexer {
pub fn finish(&mut self) {
self.frontier.settle();
let gone = self.aggregator.evict(None, 0, true);
self.files(gone);
self.files(gone, true);
self.close(true);
self.batch.next = self.end + 1;
}
@@ -1243,12 +1245,16 @@ impl RangeIndexer {
out
}
/// Finished files (complete, or the window closed on them) into the open release of their key and upload.
fn files(&mut self, files: Vec<Vec<Seg>>) {
/// Finished files (complete, or the window closed on them) into the open release of their key and upload; `at_end`
/// when the block ends on them.
fn files(&mut self, files: Vec<Vec<Seg>>, at_end: bool) {
let batch = self.aggregator.batch;
for file in files {
if file.len() == 1 && (file[0].number != 1 || file[0].file.total_parts > 1) && is_random_file(&file[0].file.name) {
// One article of a multi-part random file (its upload per article: ngPost's, nyuu's): not a release.
let lone = file.len() == 1 && (file[0].number != 1 || file[0].file.total_parts > 1);
// A scrambled name only before the block's end, where the rest of a file can still be in the next block.
if lone && (is_random_file(&file[0].file.name) || (!at_end && looks_scrambled(&file[0].file.name))) {
// One article of a multi-part random file (its upload per article: ngPost's, nyuu's, and a random poster's
// per article, 160k in alt.binaries.storage; LOG.md (122)): not a release.
let tool = names::upload_tool(&file[0].message_id);
self.count("per-article", tool.to_string(), 1, 0);
continue;
@@ -1419,9 +1425,21 @@ impl RangeIndexer {
fn write(&mut self, key: &str, upload: usize, list: Vec<Vec<Seg>>, finished: bool, edge: bool, album: Option<String>) {
let judged: Vec<JudgedFile> = list
.iter()
.map(|f| JudgedFile { name: &f[0].file.name, total_parts: f[0].file.total_parts, groups: &f[0].file.groups, first_message_id: &f[0].message_id, poster: &f[0].file.poster })
.map(|f| JudgedFile {
name: &f[0].file.name,
total_parts: f[0].file.total_parts,
groups: &f[0].file.groups,
first_message_id: &f[0].message_id,
poster: &f[0].file.poster,
segments: f.len() as u32,
bytes: f.iter().map(|s| s.bytes.max(0) as u64).sum(),
})
.collect();
match names::judge(key, &judged, &self.rules) {
// Settled for the single-archive rules: the window closed on it (a par2 can still join an open one). The
// start edge (posted within the window of the block's start) doesn't count: a par2 comes after its rar, so
// one in the previous block is rare (0 par2 siblings of 3,000 catalog hits, LOG.md (125)), and 7% of the
// catalog's releases are start edges, which would escape the rule.
match names::judge(key, &judged, finished, &self.rules) {
Verdict::Keep => {
let candidates = candidates_of(key, &list, upload);
let mut chunk = chunk_of(key, &list, upload);
@@ -1435,7 +1453,16 @@ impl RangeIndexer {
}
if !chunk.readable {
// Unnamed: no chunk (its segments are read again once a probe names it), the tool for its signature.
chunk.tool = Some(names::upload_tool(&list[0][0].message_id));
let tool = names::upload_tool(&list[0][0].message_id);
// A class the ladder found never names anything stays out of the pending store, its explored
// sample excepted (so the rate keeps being measured); counted as garbage (LOG.md (130)).
let klass = names::pending_class(&chunk.name, chunk.bytes.max(0) as u64, chunk.partial);
if self.rules.dropped_classes.contains(&names::class_key(tool, &self.group, klass)) && !names::explored(key, self.rules.sample_percent) {
let segments: usize = list.iter().map(|f| f.len()).sum();
self.count("garbage", format!("{tool} {} #{klass}", self.group), segments as u64, list.len() as u64);
return;
}
chunk.tool = Some(tool);
chunk.payload = encode_chunk(&[]);
}
if chunk.readable {
@@ -1544,7 +1571,7 @@ fn chunk_of(key: &str, files: &[Vec<Seg>], upload: usize) -> ReleaseChunk {
bytes,
first_article,
last_article,
readable: chunk_files.iter().any(|f| !is_random_file(f.name)),
readable: chunk_files.iter().any(|f| !is_unreadable_name(f.name)),
has_par2: chunk_files.iter().any(|f| names::is_par2(f.name)),
finished: true,
edge: false,
@@ -1558,7 +1585,7 @@ fn chunk_of(key: &str, files: &[Vec<Seg>], upload: usize) -> ReleaseChunk {
fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate> {
let mut out = Vec::new();
let mut tail: Option<(i64, &Vec<Seg>)> = None;
let c = |kind, s: &Seg| Candidate { release_key: key.to_string(), upload, kind, message_id: s.message_id.clone(), file: s.file.name.clone() };
let c = |kind, s: &Seg| Candidate { release_key: key.to_string(), upload, kind, message_id: s.message_id.clone(), file: s.file.name.clone(), bytes: s.bytes };
for f in files {
let name = &f[0].file.name;
let volume = names::seven_zip_volume(name);
@@ -1579,7 +1606,7 @@ fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate>
}
}
if let Some((_, f)) = tail {
out.push(Candidate { release_key: key.to_string(), upload, kind: "7z-tail", message_id: f.last().unwrap().message_id.clone(), file: f[0].file.name.clone() });
out.push(Candidate { release_key: key.to_string(), upload, kind: "7z-tail", message_id: f.last().unwrap().message_id.clone(), file: f[0].file.name.clone(), bytes: f.last().unwrap().bytes });
}
if out.is_empty() && files.len() == 1 && is_random_file(&files[0][0].file.name) {
out.push(c("lone", &files[0][0]));
+18
View File
@@ -62,6 +62,8 @@ pub struct RulesJson {
#[serde(default)]
pub dropped_signatures: Vec<String>,
#[serde(default)]
pub dropped_classes: Vec<String>,
#[serde(default)]
pub dropped_tools: Option<Vec<String>>,
#[serde(default)]
pub per_article_max_parts: Option<u32>,
@@ -71,6 +73,12 @@ pub struct RulesJson {
pub drop_executables: Option<bool>,
#[serde(default)]
pub dropped_posters: Option<Vec<String>>,
#[serde(default)]
pub drop_stub_archives: Option<bool>,
#[serde(default)]
pub drop_nonsense_pieces: Option<bool>,
#[serde(default)]
pub drop_tiny_archives: Option<bool>,
}
impl RulesJson {
@@ -79,6 +87,7 @@ impl RulesJson {
let mut rules = Rules::defaults(self.priorities.into_iter().collect());
rules.version = self.version;
rules.dropped_signatures = self.dropped_signatures.into_iter().collect();
rules.dropped_classes = self.dropped_classes.into_iter().collect();
if let Some(tools) = self.dropped_tools {
rules.dropped_tools = tools.into_iter().collect();
}
@@ -94,6 +103,15 @@ impl RulesJson {
if let Some(posters) = self.dropped_posters {
rules.dropped_posters = posters;
}
if let Some(d) = self.drop_stub_archives {
rules.drop_stub_archives = d;
}
if let Some(d) = self.drop_nonsense_pieces {
rules.drop_nonsense_pieces = d;
}
if let Some(d) = self.drop_tiny_archives {
rules.drop_tiny_archives = d;
}
rules
}
}
+565 -6
View File
@@ -449,6 +449,43 @@ fn finish_remainder(s: &str) -> String {
js::trim(&out).to_string()
}
/// displayName: a subject name as shown, brackets the counters left unmatched removed ("… Facialized 002/477] - …"
/// gave "… Facialized ]"). Only the name: the key keeps the remainder.
pub fn display_name(remainder: &str) -> String {
let b = remainder.as_bytes();
if !b.iter().any(|c| matches!(c, b'[' | b']' | b'(' | b')')) {
return remainder.to_string();
}
let mut drop = vec![false; b.len()];
for (open, close) in [(b'[', b']'), (b'(', b')')] {
let mut opens: Vec<usize> = Vec::new();
for (i, &c) in b.iter().enumerate() {
if c == open {
opens.push(i);
} else if c == close && opens.pop().is_none() {
drop[i] = true;
}
}
for i in opens {
drop[i] = true;
}
}
if !drop.contains(&true) {
return remainder.to_string();
}
let kept: Vec<u8> = b.iter().zip(&drop).filter(|(_, &d)| !d).map(|(&c, _)| c).collect();
let kept = String::from_utf8(kept).expect("only ASCII brackets removed");
let mut out = String::with_capacity(kept.len());
for c in kept.chars() {
if c == ' ' && out.ends_with(' ') {
continue;
}
out.push(c);
}
let name = js::trim(&out);
if name.is_empty() { remainder.to_string() } else { name.to_string() }
}
/// counterTotal: the subject's file counter total, first form found first.
pub fn counter_total(subject: &str) -> Option<String> {
let s = replace(&QUOTED, subject, " ", true);
@@ -841,11 +878,11 @@ pub fn release_identity(poster: &str, subject: &str, name: &str) -> Release {
if !image && test(&RANDOM_VOLUME, name) && stem.bytes().all(|c| c.is_ascii_alphanumeric() || c == b'_' || c == b'-') && looks_random(&stripped) {
parts.push(&stem);
}
return Release { key: hash(&parts), name: remainder, source: "subject", file_total };
return Release { key: hash(&parts), name: display_name(&remainder), source: "subject", file_total };
}
if let Some(total) = post_total.filter(|&n| n != 0 && !tag) {
if is_post_token(&remainder) {
return Release { key: hash(&["post", poster, &remainder, &total.to_string()]), name: remainder, source: "post", file_total };
return Release { key: hash(&["post", poster, &remainder, &total.to_string()]), name: display_name(&remainder), source: "post", file_total };
}
}
// A scene music post's index files ("00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg") under a bare counter go
@@ -1028,6 +1065,228 @@ pub fn is_random_file(name: &str) -> bool {
looks_random(&file_stem(name))
}
/// Readable or not (lib/release/readable.ts): whether a name says what a release is, read token by token, so
/// separators, spaces and random "extensions" don't hide a scramble that looks_random's single-token shapes miss
/// (LOG.md (129)). The readable flag; the drop rules keep looks_random, which is narrower.
/// Tokens that don't make a name readable on their own: packaging, volume numbers and the posting tools' fixed words.
static NOISE_TOKEN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)^(addon|samples?|proof|subs?|nfo|part[0-9]+|vol[0-9]+|description|repost|rep|extras?|bonus|cd[0-9]*|dvd[0-9]*|dis[ck][0-9]*|par2|rar|zip|7z|mkv|mp4|avi|vol)$"));
/// Mixed tokens that are codes, not scrambles: S02E10, 1080p, 10bit, x264, Win10x64, BDRip720p, 3DMark03, v2, 5th.
static CODE_TOKEN: LazyLock<Regex> = LazyLock::new(|| re(r"^([sS][0-9]{1,2}[eE][0-9]{1,3}|[xh]26[45]|[A-Za-z]+[0-9]+([a-z]{1,3}|x(64|86))?|[0-9]+[A-Za-z]{1,3}|[0-9]+[A-Za-z]+[0-9]+)$"));
/// The NZBSPLIT tools' hash between their markers is no part of the name.
static NZBSPLIT_HASH: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)__NZBS[P]LIT__[0-9a-f]{32}__NZBS[P]LIT__")); // "SP" is re()'s space class
/// An inner run of 2+ capitals with lowercase on both sides ("tBWTuz", "YDoaROv"; not "3DMark03").
static INNER_UPPER: LazyLock<Regex> = LazyLock::new(|| re(r"[a-z][A-Z]{2,}[a-z]"));
/// Scene tags write "i" for "I" between capitals (XCiTE, PRESCRiPTiON): read it as a capital there.
fn scene_case(t: &[u8]) -> Vec<u8> {
let mut out = t.to_vec();
for i in 1..t.len().saturating_sub(1) {
if t[i] == b'i' && t[i - 1].is_ascii_uppercase() && t[i + 1].is_ascii_uppercase() {
out[i] = b'I';
}
}
out
}
/// The signals of a scrambled token (readable.ts signals()).
struct TokenSignals {
/// Letter/digit and lower-to-upper switches.
switches: usize,
/// Case switches in both directions.
case_switches: usize,
/// Lone lowercase letters (not "i") between capitals ("NtpUgCmS").
lone: usize,
inner_upper: bool,
max_consonants: usize,
/// Vowels, a lowercase y too ("McCartney").
vowels: usize,
/// Runs of one character class.
runs: usize,
letters: usize,
/// The longest lowercase run.
max_lower: usize,
}
fn token_signals(t: &[u8]) -> TokenSignals {
let is_vowel = |c: u8| matches!(c.to_ascii_lowercase(), b'a' | b'e' | b'i' | b'o' | b'u') || c == b'y';
let class = |c: u8| if c.is_ascii_digit() { 0 } else if c.is_ascii_lowercase() { 1 } else { 2 };
let mut s = TokenSignals { switches: 0, case_switches: 0, lone: 0, inner_upper: false, max_consonants: 0, vowels: 0, runs: 0, letters: 0, max_lower: 0 };
let mut consonants = 0;
let mut lower = 0;
for i in 0..t.len() {
let c = t[i];
if c.is_ascii_alphabetic() && !is_vowel(c) && c.to_ascii_lowercase() != b'y' {
consonants += 1;
s.max_consonants = s.max_consonants.max(consonants);
} else {
consonants = 0;
}
if is_vowel(c) {
s.vowels += 1;
}
if c.is_ascii_alphabetic() {
s.letters += 1;
}
if c.is_ascii_lowercase() {
lower += 1;
s.max_lower = s.max_lower.max(lower);
} else {
lower = 0;
}
if i == 0 || class(t[i - 1]) != class(c) {
s.runs += 1;
}
if i == 0 {
continue;
}
let a = t[i - 1];
if a.is_ascii_digit() != c.is_ascii_digit() {
s.switches += 1;
} else if a.is_ascii_lowercase() && c.is_ascii_uppercase() {
s.switches += 1;
}
if (a.is_ascii_lowercase() && c.is_ascii_uppercase()) || (a.is_ascii_uppercase() && c.is_ascii_lowercase()) {
s.case_switches += 1;
}
if i < t.len() - 1 && c.is_ascii_lowercase() && c != b'i' && a.is_ascii_uppercase() && t[i + 1].is_ascii_uppercase() {
s.lone += 1;
}
}
s.inner_upper = INNER_UPPER.is_match(t);
s
}
#[derive(PartialEq, Eq, Debug, Clone, Copy)]
pub enum TokenKind {
Word,
Random,
Neutral,
}
/// What one alphanumeric token of a name is: a word (letters only, in one case or Capitalized, 3+ long, not a
/// packaging word), a scramble, or neither (digits, codes, CamelCase, short tokens). Kept in step with readable.ts.
pub fn token_kind(t: &[u8]) -> TokenKind {
let n = t.len();
if t.iter().all(u8::is_ascii_digit) || NOISE_TOKEN.is_match(t) || CODE_TOKEN.is_match(t) {
return TokenKind::Neutral;
}
if t.iter().all(u8::is_ascii_alphabetic) {
let s = token_signals(&scene_case(t));
let capitalized = t[0].is_ascii_uppercase() && t[1..].iter().all(u8::is_ascii_lowercase);
if s.case_switches == 0 || (s.case_switches == 1 && capitalized) {
// A base32 consonant soup ("mrrtmzrzmi4dgmrzmyytqmlbztczbx" without its digit) is no word.
if n >= 10 && s.vowels * 6 < n {
return TokenKind::Random;
}
return if n >= 3 { TokenKind::Word } else { TokenKind::Neutral };
}
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
// Two CamelCase words with few vowels (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone
// letter; a scramble has more runs, a lone letter or an inner capital run.
if n >= 6 && s.case_switches >= 2 && s.vowels * 4 < n && (s.runs >= 5 || s.lone >= 1 || s.inner_upper) {
return TokenKind::Random;
}
return TokenKind::Neutral;
}
if n >= 8 && t.iter().all(u8::is_ascii_hexdigit) && t.iter().filter(|c| c.is_ascii_digit()).count() >= 2 {
return TokenKind::Random;
}
if t.iter().all(|c| c.is_ascii_uppercase() || c.is_ascii_digit()) {
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
return if n >= 12 && token_signals(t).switches >= 6 { TokenKind::Random } else { TokenKind::Neutral };
}
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
if t.split(|c: &u8| !c.is_ascii_alphabetic()).filter(|w| !w.is_empty()).all(|w| {
let vowels = w.iter().filter(|c| matches!(c.to_ascii_lowercase(), b'a' | b'e' | b'i' | b'o' | b'u' | b'y')).count();
(w.len() >= 2 && w.iter().all(u8::is_ascii_uppercase))
|| (w.len() >= 2 && w[1..].iter().all(u8::is_ascii_lowercase) && (w.len() >= 3 || w[0].is_ascii_lowercase()) && vowels * 5 >= w.len())
}) {
return TokenKind::Neutral;
}
let s = token_signals(&scene_case(t));
if n >= 5 && s.switches >= 3 {
return TokenKind::Random;
}
if n >= 6 && s.switches >= 2 && (s.lone >= 1 || s.inner_upper || s.max_consonants >= 4) {
return TokenKind::Random;
}
// Few vowels and no lowercase run of 3+ ("Jk5Pud", "RKi9BCg"; not "HDR10Plus").
if n >= 6 && s.switches >= 2 && s.vowels * 4 < s.letters && s.max_lower <= 2 {
return TokenKind::Random;
}
TokenKind::Neutral
}
/// A file or release name no one can read (readable.ts isUnreadableName): looks_random's shapes, or more scrambled
/// tokens than words by letter count ("Zx5AjNaK-mh7CFg6Hf8uu", "Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ"; not
/// "Paul McCartney - Band on the Run", "ShesSoWild - Ester Ana 1080p").
pub fn is_unreadable_name(name: &str) -> bool {
let stem = file_stem(name);
let stem = NZBSPLIT_HASH.replace(stem.as_bytes(), &b" "[..]);
if looks_random(&String::from_utf8_lossy(&stem)) {
return true;
}
let mut words = 0;
let mut random = 0;
for t in stem.split(|c| !c.is_ascii_alphanumeric()).filter(|t| !t.is_empty()) {
match token_kind(t) {
TokenKind::Word => words += t.len(),
TokenKind::Random => random += t.len(),
TokenKind::Neutral => {}
}
}
random > 0 && random > words
}
/// The class of an unnamed release (lib/release/classes.ts pendingClass; LOG.md (130)): size bucket × name shape ×
/// partial, 1..70. The lease's dropped classes ("tool\tgroup\tklass") are skipped at the edge.
pub fn pending_class(name: &str, bytes: u64, partial: bool) -> u8 {
let size = [1_000_000u64, 10_000_000, 100_000_000, 1_000_000_000].iter().filter(|&&b| bytes >= b).count() as u8;
let s = match name.rfind('.') {
Some(i) if (1..=4).contains(&(name.len() - i - 1)) && name[i + 1..].bytes().all(|b| b.is_ascii_alphanumeric()) => &name[..i],
_ => name,
};
let b = s.as_bytes();
let all = |f: fn(&u8) -> bool| !b.is_empty() && b.iter().all(f);
let shape: u8 = if b.len() >= 16 && all(u8::is_ascii_hexdigit) {
1
} else if b.len() >= 6 && all(u8::is_ascii_digit) {
2
} else if b.len() >= 20 && all(|c| c.is_ascii_lowercase() || (b'2'..=b'7').contains(c)) && b.iter().any(|c| (b'2'..=b'7').contains(c)) {
3
} else if all(u8::is_ascii_alphabetic) {
4
} else if all(u8::is_ascii_alphanumeric) {
5
} else if all(|c| c.is_ascii_alphanumeric() || matches!(c, b'.' | b'_' | b'-')) {
6
} else {
7
};
size * 14 + (shape - 1) * 2 + u8::from(partial) + 1
}
/// A class as droppedClasses lists it.
pub fn class_key(tool: &str, group: &str, klass: u8) -> String {
format!("{tool}\t{group}\t{klass}")
}
/// looksScrambled: at least half the letters and digits of a file's stem in tokens of 5 or more that switch case
/// (lower to upper) or between letters and digits 3 times or more ("nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr", "bslXlwgPiqZadhRa").
/// The names a random-subject-per-article upload gives each article that looksRandom misses for their separators.
pub fn looks_scrambled(name: &str) -> bool {
let stem = file_stem(name);
let (mut total, mut scrambled) = (0, 0);
for token in stem.as_bytes().split(|c| !c.is_ascii_alphanumeric()).filter(|t| !t.is_empty()) {
total += token.len();
let switches = token.windows(2).filter(|w| (w[0].is_ascii_lowercase() && w[1].is_ascii_uppercase()) || w[0].is_ascii_digit() != w[1].is_ascii_digit()).count();
if token.len() >= 5 && switches >= 3 {
scrambled += token.len();
}
}
total >= 8 && scrambled * 2 >= total
}
static SEVEN_SPLIT: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.7z\.([0-9]{1,4})$"));
static SEVEN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.7z$"));
static PART1: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.part0*1\.rar$"));
@@ -1243,6 +1502,39 @@ pub fn is_sidecar(name: &str) -> bool {
test(&SIDECAR, name)
}
static EXECUTABLE: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(exe|scr|com|pif|bat|cmd|vbs|lnk)SP*$"));
/// An unsplit archive (the judge's ARCHIVE and VOLUME): one zip, rar or 7z that is not a volume of a set.
static ARCHIVE: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(zip|rar|7z)$"));
/// "cneg" is "part" in ROT13: the a.b.erotica posts named that way are volumes of passworded videos (LOG.md (128)).
static VOLUME: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.((part|cneg)[0-9]+\.rar|r[0-9][0-9]|z[0-9][0-9]|7z\.[0-9][0-9][0-9]|[0-9][0-9][0-9])$"));
/// A poster at a free mail provider (the judge's WEBMAIL): the stub floods' throwaway accounts.
static WEBMAIL: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)@(gmail|googlemail|yahoo|hotmail|outlook|live|msn|aol|gmx|web|t-online|freenet|mail|yandex|icloud|me|hushmail|protonmail|libero|orange|free|arcor|wanadoo|ymail|rocketmail|inbox|mail\.ru|bk|list)\.")
});
/// A name that reads as a release (the judge's RELEASE_LIKE): a year, a scene tag, a group suffix or dotted words.
static RELEASE_LIKE: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
});
/// RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way.
static RELEASE_LIKE_SPACED: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
});
/// The stub-archive rule (LOG.md (123)-(126), (128)): one unsplit archive, no par2, settled, posted to 2+ groups,
/// under STUB_MAX_BYTES (the fakes stop at ~90 MB; the hash-named uploads, real albums and passworded videos above
/// it don't), and its articles at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES (Power-Post's
/// 250 kB default; a one-article release always has small articles, so e-books and sidecars are exempt), or 4+
/// groups, or a webmail poster.
pub const STUB_ARTICLE_BYTES: u64 = 262_144;
pub const STUB_MIN_BYTES: u64 = 1_000_000;
pub const STUB_MAX_BYTES: u64 = 100_000_000;
pub const STUB_WIDE_GROUPS: usize = 4;
/// The nonsense-piece rule: one article holding one archive of NONSENSE_MIN_BYTES to NONSENSE_MAX_BYTES under a
/// release-like name that isn't scrambled (the newznzb.* pieces; under 100 kB are cue/log sidecars and adverts).
pub const NONSENSE_MIN_BYTES: u64 = 100_000;
pub const NONSENSE_MAX_BYTES: u64 = 1_000_000;
/// The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars.
pub const TINY_MIN_BYTES: u64 = 2_000;
pub struct JudgedFile<'a> {
pub name: &'a str,
@@ -1250,6 +1542,79 @@ pub struct JudgedFile<'a> {
pub groups: &'a [String],
pub first_message_id: &'a str,
pub poster: &'a str,
/// Articles seen and their bytes (the overview's :bytes).
pub segments: u32,
pub bytes: u64,
}
/// One unsplit archive and nothing else.
fn single_archive<'a, 'b>(files: &'b [JudgedFile<'a>]) -> Option<&'b JudgedFile<'a>> {
match files {
[f] if test(&ARCHIVE, f.name) && !test(&VOLUME, f.name) => Some(f),
_ => None,
}
}
/// Groups whose communities post real single small archives (karaoke zips, e-books, comics, tabs, fonts) the way
/// the stub floods do (Power-Post defaults, a crosspost or two): a release touching one is exempt from the
/// stub-archive rule. Costs 16 of 32,308 known fakes in the 217k sample (LOG.md (127)).
static STUB_EXEMPT_GROUPS: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)(sound|mp3|karaoke|music|audio|flac|lossless|e-?book|epub|comic|guitar|font|magazine)"));
/// The legs of the stub-archive rule a settled single archive under STUB_MAX_BYTES trips ("A" article size, "X" 4+
/// groups, "W" webmail), empty when none.
pub fn stub_archive_legs(f: &JudgedFile) -> String {
let mut legs = String::new();
if f.groups.len() < 2 || f.bytes >= STUB_MAX_BYTES || f.groups.iter().any(|g| test(&STUB_EXEMPT_GROUPS, g)) {
return legs;
}
let article = f.bytes / u64::from(f.segments.max(1));
if article <= STUB_ARTICLE_BYTES && f.bytes >= STUB_MIN_BYTES {
legs.push('A');
}
if f.groups.len() >= STUB_WIDE_GROUPS {
legs.push('X');
}
if test(&WEBMAIL, f.poster) {
legs.push('W');
}
legs
}
/// An e-book by its name ("Author - Title (2020) (epub).rar"): one small article is a book, not a nonsense piece.
static EBOOK: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\b(epub|mobi|azw3?|pdf|djvu|cbr|cbz|e-?books?)\b"));
/// A nonsense piece: one article, one small archive under a release-like, unscrambled name without spaces (the
/// newznzb.* pieces are scene-styled or random with a "-XX" suffix; books and albums are written with spaces).
pub fn is_nonsense_piece(f: &JudgedFile) -> bool {
f.total_parts == 1
&& f.segments == 1
&& (NONSENSE_MIN_BYTES..NONSENSE_MAX_BYTES).contains(&f.bytes)
&& !f.name.contains(' ')
&& test(&RELEASE_LIKE, &file_stem(f.name))
&& !test(&EBOOK, f.name)
&& !looks_scrambled(f.name)
&& !is_random_file(f.name)
}
/// Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
/// exe is the content), covers and sidecars.
static TINY_CONTENT: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b")
});
/// A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
/// unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
/// subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
pub fn is_tiny_archive(f: &JudgedFile) -> bool {
f.total_parts == 1
&& f.segments == 1
&& (TINY_MIN_BYTES..NONSENSE_MIN_BYTES).contains(&f.bytes)
&& !f.groups.iter().any(|g| test(&STUB_EXEMPT_GROUPS, g))
&& test(&RELEASE_LIKE_SPACED, &file_stem(f.name))
&& !test(&EBOOK, f.name)
&& !test(&TINY_CONTENT, f.name)
&& !looks_scrambled(f.name)
&& !is_random_file(f.name)
}
pub enum Verdict {
@@ -1265,12 +1630,19 @@ pub struct Rules {
pub priorities: rustc_hash::FxHashMap<String, u32>,
/// "tool\tgroup"
pub dropped_signatures: rustc_hash::FxHashSet<String>,
/// "tool\tgroup\tklass" (pending_class): unnamed releases of these classes don't go to the pending store.
pub dropped_classes: rustc_hash::FxHashSet<String>,
pub dropped_tools: rustc_hash::FxHashSet<String>,
pub per_article_max_parts: u32,
pub sample_percent: f64,
pub drop_executables: bool,
/// Fragments of the poster lowercased without punctuation (droppedPosters).
pub dropped_posters: Vec<String>,
/// The stub-archive rule (dropStubArchives), the nonsense-piece rule (dropNonsensePieces) and the tiny-archive
/// rule (dropTinyArchives).
pub drop_stub_archives: bool,
pub drop_nonsense_pieces: bool,
pub drop_tiny_archives: bool,
}
impl Rules {
@@ -1280,11 +1652,15 @@ impl Rules {
version: 1,
priorities,
dropped_signatures: Default::default(),
dropped_classes: Default::default(),
dropped_tools: ["ngPost", "hex32", "yuun", "JBinUp", "random"].iter().map(|t| t.to_string()).collect(),
per_article_max_parts: 3,
sample_percent: 1.0,
drop_executables: true,
dropped_posters: ["celebmovie", "celebwow", "celebfx", "iluvcelebs", "nudecelebs", "celebsextapes"].iter().map(|p| p.to_string()).collect(),
drop_stub_archives: true,
drop_nonsense_pieces: true,
drop_tiny_archives: true,
}
}
}
@@ -1294,8 +1670,9 @@ pub fn poster_words(poster: &str) -> String {
poster.chars().filter(|c| c.is_ascii_alphanumeric()).map(|c| c.to_ascii_lowercase()).collect()
}
/// RuleSet.judge.
pub fn judge(release_key: &str, files: &[JudgedFile], rules: &Rules) -> Verdict {
/// RuleSet.judge. `settled`: the window closed on the release (a par2 can still join an open one), which the
/// single-archive rules need.
pub fn judge(release_key: &str, files: &[JudgedFile], settled: bool, rules: &Rules) -> Verdict {
let sample = explored(release_key, rules.sample_percent);
if let (false, Some(first)) = (rules.dropped_posters.is_empty(), files.first()) {
let words = poster_words(first.poster);
@@ -1309,6 +1686,20 @@ pub fn judge(release_key: &str, files: &[JudgedFile], rules: &Rules) -> Verdict
let key = std::str::from_utf8(&c[1]).unwrap().to_lowercase();
return Verdict::Drop { rule: "executable", key, sample };
}
if let (true, Some(f)) = (settled, single_archive(files)) {
if rules.drop_stub_archives {
let legs = stub_archive_legs(f);
if !legs.is_empty() {
return Verdict::Drop { rule: "stub-archive", key: legs, sample };
}
}
if rules.drop_nonsense_pieces && is_nonsense_piece(f) {
return Verdict::Drop { rule: "nonsense-piece", key: upload_tool(f.first_message_id).to_string(), sample };
}
if rules.drop_tiny_archives && is_tiny_archive(f) {
return Verdict::Drop { rule: "tiny-archive", key: upload_tool(f.first_message_id).to_string(), sample };
}
}
if files.iter().any(|f| !is_random_file(f.name)) {
return Verdict::Keep;
}
@@ -1339,6 +1730,29 @@ mod tests {
assert_eq!(file_stem_re("Morning Lovers.wmv.058"), "Morning Lovers");
}
#[test]
fn scrambled_names() {
for n in ["nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr", "bslXlwgPiqZadhRa", "PNSVWSV-xLaNv6o5qS840e3lwSZyH", "Biq-YsXu8G33ktt"] {
assert!(looks_scrambled(n), "{n}");
}
// Short ones of the spam too (8% of alt.binaries.storage's): too few switches to tell.
for n in ["A1Monitor v6.2.2", "PDF2Word v1.3", "Some.Show.S01E01.1080p.WEB.part1.rar", "MacGyver.S01E01.mkv", "Jimmy and the Crawler (mobi)", "Q25m.PDUYtbA", "82WH.2iuTK"] {
assert!(!looks_scrambled(n), "{n}");
}
}
#[test]
fn display_names() {
let porn4you = "webmaster@hdbangvidz.com (Porn4You)";
let a = release_identity(porn4you, "Babe with Plump Bubble Butt Fucked and Facialized 002/477] - 0178.jpg", "0178.jpg");
let b = release_identity(porn4you, "Babe with Plump Bubble Butt Fucked and Facialized 003/477] - 0179.jpg", "0179.jpg");
assert_eq!(a.name, "Babe with Plump Bubble Butt Fucked and Facialized");
assert_eq!(a.key, b.key);
assert_eq!(display_name("[Group] Show (2020)"), "[Group] Show (2020)");
assert_eq!(display_name("( Show [x] name"), "Show [x] name");
assert_eq!(display_name("]"), "]");
}
#[test]
fn remainders() {
assert_eq!(subject_remainder(r#"Some.Release [01/45] - "x.rar" yEnc (1/50) 7168"#), "Some Release");
@@ -1451,6 +1865,46 @@ mod tests {
assert_eq!(id(r#"[2/6] - "x.jpg" yEnc"#, "x.jpg").key, id(r#"[1/6] - "x.part01.rar" yEnc"#, "x.part01.rar").key);
}
#[test]
fn pending_classes() {
use super::pending_class;
// Size bucket × shape × partial: a 5 MB hex name whole, the same partial, a 500 MB scramble with separators.
assert_eq!(pending_class("c04212a389cf69edc54767355ab5285f", 5_000_000, false), 15);
assert_eq!(pending_class("c04212a389cf69edc54767355ab5285f", 5_000_000, true), 16);
assert_eq!(pending_class("Zx5AjNaK-mh7CFg6Hf8uu", 500_000_000, false), 3 * 14 + 5 * 2 + 1);
assert_eq!(pending_class("1650777167", 100, false), 3);
assert_eq!(pending_class("mrrtmzrzmi4dgmrzmyytqmlbztczbx", 2_000_000_000, false), 4 * 14 + 2 * 2 + 1);
assert_eq!(pending_class("PsNdnYiugfZ.rar", 20_000_000, false), 2 * 14 + 3 * 2 + 1);
assert_eq!(pending_class("n5d1ueIf34X", 20_000_000, true), 2 * 14 + 4 * 2 + 2);
assert_eq!(pending_class("sgkra vU7u2yuw9LDLCjDu04pOez", 999, false), 13);
}
#[test]
fn unreadable_names() {
use super::is_unreadable_name;
// Scrambles with separators, spaces, a random "extension", a packaging word, a hex with an addon (LOG.md (129)).
for n in [
"Zx5AjNaK-mh7CFg6Hf8uu.part01.rar", "yWjfnT2ujG4wJj3fG46o0FLzsgkvhmrTsAfW2.E0WsOU", "PsNdnYiugfZ.rar", "phfplC9y07U2yrj", "n5d1ueIf34X",
"tJqr1tq7xK", "Description - Lt5nHjlo3RiuBb3C.par2", "bd5b4a59f3aa87114dcf75c0-addon.rar", "mrrtmzrzmi4dgmrzmyytqmlbztczbx",
"AA-NaR16302830434zxP32108309f3Ab", "_QBs01_MqZYnYwM_iAyRquhc-", "sgkra vU7u2yuw9LDLCjDu04pOez", "Jk5Pud.zQdNj", "7uKsD_NRgT-G_Uc_p8",
"NtpUgCmS_Xa1B2Pk", "{E11BF8E9-E2C9-4DAE-A1B2-8027BC465F3F}", "IkYovjrtZrItjQ.mkv", "ff75ce3f16c746c18b2d78cf.6c52f",
] {
assert!(is_unreadable_name(n), "{n}");
}
// Real names: scene tags, CamelCase, codes, leet, an acronym, a hash beside words.
for n in [
"Dark.Matter.2024.S02E10.No.Quiet.Life.1080p.ATVP.WEB-DL.DDP5.1.Atmos.H.264-Kitsune", "Paul McCartney - Band on the Run",
"ShesSoWild - Ester Ana 1080p", "MyDadsHotGirlfriend.13.02.01.Luna.Star.XXX.1080p.x264-SEXORS", "cFosSpeed 11.0 Build 2420 x86 x64 incl Patch.rar",
"Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing", "Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON",
"Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr", "Eclipse_IDE_2023-12_R_win32_x86_64.zip", "PartsHPGL 1.06c", "Go4Customer Fax Machine v1.2b",
"AdBeGone 1.2", "3DMark03 320", "COD4MW2REMASTERED", "Movie.2020.1080p-GRP [a3f9e2b1]", "EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps",
"iPhone", "Some.Show.S01E01.1080p.mkv", "FastFilms 2.04", "BestCrypt NP v4.0", "EasyHTML 2.35 German", "SchiffsModell - September 2026",
"CrossStitcher - Issue 440, October 2026", "Laura15Juli2026", "IHaveAWife - Isa Bella", "DickHDDaily - Santana Red", "1904-SmithsWork-Hasluck",
] {
assert!(!is_unreadable_name(n), "{n}");
}
}
#[test]
fn random_names() {
assert!(looks_random("prvjr7hapd0i1ny14pz4"));
@@ -1491,8 +1945,8 @@ mod poster_rule_tests {
fn verdict(poster: &str, rules: &Rules) -> Option<(&'static str, String)> {
let groups = vec!["alt.binaries.erotica".to_string()];
let files = [JudgedFile { name: "!index0001.jpg", total_parts: 1, groups: &groups, first_message_id: "<a@b>", poster }];
match judge("k", &files, rules) {
let files = [JudgedFile { name: "!index0001.jpg", total_parts: 1, groups: &groups, first_message_id: "<a@b>", poster, segments: 1, bytes: 200_000 }];
match judge("k", &files, true, rules) {
Verdict::Keep => None,
Verdict::Drop { rule, key, .. } => Some((rule, key)),
}
@@ -1512,6 +1966,111 @@ mod poster_rule_tests {
}
}
#[cfg(test)]
mod single_archive_rule_tests {
use super::{judge, JudgedFile, Rules, Verdict};
const NYUU: &str = "FmFwIaTpGgUiSfYcMwFlMeRo-1790637260478@nyuu";
fn verdict(name: &str, poster: &str, groups: usize, segments: u32, bytes: u64, settled: bool, rules: &Rules) -> Option<(&'static str, String)> {
let groups: Vec<String> = (0..groups).map(|i| format!("alt.binaries.g{i}")).collect();
let files = [JudgedFile { name, total_parts: segments, groups: &groups, first_message_id: NYUU, poster, segments, bytes }];
match judge("k", &files, settled, rules) {
Verdict::Keep => None,
Verdict::Drop { rule, key, .. } => Some((rule, key)),
}
}
#[test]
fn stub_archives_by_article_size_groups_or_webmail() {
let rules = Rules::defaults(Default::default());
// Power-Post's 250 kB articles, two groups, 12 MB: the 2016 wave.
assert_eq!(verdict("Chat Watch 1.2.1.zip", "jahnunsinos007@gmail.com (Jahnunsinos007)", 2, 48, 12_000_000, true, &rules), Some(("stub-archive", "AW".to_string())));
assert_eq!(verdict("Some.Movie.2014.720p.BluRay-GRP.rar", "Yenc@power-post.org (Yenc-PP-A&A)", 2, 48, 12_000_000, true, &rules), Some(("stub-archive", "A".to_string())));
// Five groups, 319 kB in two articles: the 2015 a.b.cracks wave (under 1 MB, so the groups leg alone).
assert_eq!(verdict("Big Hero 6 (2014) 720p BDRip [COMPRESSED].rar", "Yenc@power-post.org (Yenc-PP-A&A)", 5, 2, 319_000, true, &rules), Some(("stub-archive", "X".to_string())));
// A webmail poster with 650 kB articles: the 2014 "leaked Sex Video" wave.
assert_eq!(verdict("Leaked Sex Video.rar", "first.last88@gmail.com", 2, 400, 260_000_000, true, &rules), None, "over 100 MB without the article-size leg");
assert_eq!(verdict("Leaked Sex Video.rar", "first.last88@gmail.com", 2, 100, 65_000_000, true, &rules), Some(("stub-archive", "W".to_string())));
// Settled only: a par2 can still join an open release.
assert_eq!(verdict("Chat Watch 1.2.1.zip", "jahnunsinos007@gmail.com", 2, 48, 12_000_000, false, &rules), None);
}
#[test]
fn small_and_single_group_and_split_archives_are_kept() {
let rules = Rules::defaults(Default::default());
// An e-book: one or two small articles always have small articles (LOG.md (126)).
assert_eq!(verdict("Maeve Binchy - Tara Road (epub).rar", "kie@home.now (KiE)", 2, 2, 521_000, true, &rules), None);
// One group: most real single rars.
assert_eq!(verdict("Chat Watch 1.2.1.zip", "jahnunsinos007@gmail.com", 1, 48, 12_000_000, true, &rules), None);
// Modern 700 kB articles, no webmail, two groups.
assert_eq!(verdict("Some.Movie.2014.720p.BluRay-GRP.rar", "poster@example.org", 2, 100, 70_000_000, true, &rules), None);
// A volume, not a single archive; a hash-named 2 GB upload from a gmail-looking poster.
assert_eq!(verdict("Some.Movie.2014.part01.rar", "a@gmail.com", 2, 48, 12_000_000, true, &rules), None);
assert_eq!(verdict("cbcd32359460df1fa9b7d01d8720.rar", "b648de0d1c0f7a9a77be273d@gmail.com", 2, 3163, 2_340_000_000, true, &rules), None);
// ROT13-named volumes of passworded videos in a.b.erotica, 258 kB articles, two groups, 1-270 MB; and the
// 100 MB cap on every leg (LOG.md (128)).
assert_eq!(verdict("Ghful.17.12.07.Qnvvfl.Fgbar.cneg3.rar", "nospam@nospam.nl (EvH)", 2, 5, 1_147_000, true, &rules), None);
assert_eq!(verdict("Ghful.16.07.05.Wbwb.Xvff.cneg1.rar", "nospam@nospam.nl (EvH)", 2, 1051, 271_000_000, true, &rules), None);
assert_eq!(verdict("Tinasoft EasyCafe 2.2.14 by OmerSalaj.rar", "Seven7i@mail.com ({Seven7i})", 4, 463, 117_000_000, true, &rules), None);
assert_eq!(verdict("Tipard DVD Ripper - 10 0 12 x86 x64 incl Patch.rar", "Seven7i@mail.com ({Seven7i})", 4, 244, 62_000_000, true, &rules), Some(("stub-archive", "AXW".to_string())));
let off = Rules { drop_stub_archives: false, ..Rules::defaults(Default::default()) };
assert_eq!(verdict("Chat Watch 1.2.1.zip", "jahnunsinos007@gmail.com", 2, 48, 12_000_000, true, &off), None);
// A karaoke zip crossposted to a.b.test with Power-Post's articles: its group exempts it (LOG.md (127)).
let groups = vec!["alt.binaries.sounds.karaoke".to_string(), "alt.binaries.test".to_string()];
let files = [JudgedFile { name: "KV-1055806 - Walker Hayes - U Gurl.zip", total_parts: 13, groups: &groups, first_message_id: NYUU, poster: "home@home.com (TexasDj)", segments: 13, bytes: 3_300_000 }];
assert!(matches!(judge("k", &files, true, &rules), Verdict::Keep));
}
#[test]
fn nonsense_pieces() {
let rules = Rules::defaults(Default::default());
// One 790 kB article under a release-like name: the newznzb.* pieces.
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
// A 3 kB cue/log sidecar rar with a year in its name: under this rule's floor (the tiny-archive rule's outside the music groups).
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// A book (a year or a format in its name, written with spaces), a dotted one too: not a piece.
assert_eq!(verdict("Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar", "fake@address.com (Psychopath)", 2, 1, 303_000, true, &rules), None);
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 382_000, true, &rules), None);
assert_eq!(verdict("Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar", "saggir@domain.com", 1, 1, 113_000, true, &rules), None);
// Two articles, or no release-like name, or scrambled: not this rule.
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 2, 790_000, true, &rules), None);
assert_eq!(verdict("holiday snaps.rar", "poster@example.org", 1, 1, 790_000, true, &rules), None);
assert_eq!(verdict("nYDeXeF49dieZ1J7flK-JFzZ3j.rar", "poster@example.org", 1, 1, 790_000, true, &rules), None);
let off = Rules { drop_nonsense_pieces: false, ..Rules::defaults(Default::default()) };
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &off), None);
}
#[test]
fn tiny_archives() {
let rules = Rules::defaults(Default::default());
// One article of 2-100 kB under a release name, dotted or with spaces: a RAR holding "<title>.exe" (LOG.md (131)).
assert_eq!(verdict("Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar", "Yanc@power-post.org <Yanc@power-post.org>", 1, 1, 15_084, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
assert_eq!(verdict("Cicero Magazin fuer politische Kultur Mai 05 2016.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 17_478, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// A scene name without a year, a tag or a group suffix: not release-like, so kept (the flood's porn names mostly carry one).
assert_eq!(verdict("[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar", "Yanc@power-post.org", 2, 1, 23_449, true, &rules), None);
assert_eq!(verdict("Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 7_869, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
assert_eq!(verdict("The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar", "henk@example.org", 1, 1, 38_000, true, &rules), None);
assert_eq!(verdict("Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar", "poster@example.org", 1, 1, 60_000, true, &rules), None);
assert_eq!(verdict("Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
assert_eq!(verdict("Some Band - Album (2011) covers.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 38_000, true, &rules), None);
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's.
let groups = vec!["alt.binaries.sounds.karaoke".to_string()];
let files = [JudgedFile { name: "KV-1055806 - Walker Hayes - U Gurl (2017).zip", total_parts: 1, groups: &groups, first_message_id: NYUU, poster: "home@home.com", segments: 1, bytes: 60_000 }];
assert!(matches!(judge("k", &files, true, &rules), Verdict::Keep));
assert_eq!(verdict("holiday snaps.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
assert_eq!(verdict("nYDeXeF49dieZ1J7flK-JFzZ3j.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 2, 20_000, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 1_500, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 150_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, false, &rules), None);
let off = Rules { drop_tiny_archives: false, ..Rules::defaults(Default::default()) };
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, true, &off), None);
}
}
#[cfg(test)]
mod image_set_tests {
use super::{release_identity, sized_name, subject_remainder, subject_remainder_re};
+2 -1
View File
@@ -44,6 +44,7 @@ struct WireCandidate<'a> {
kind: &'a str,
message_id: &'a str,
file: &'a str,
bytes: i64,
}
#[derive(Serialize)]
@@ -97,7 +98,7 @@ fn release<'a>(r: &'a ReleaseChunk, rule: Option<(&'a str, &'a str)>) -> WireRel
}
fn candidate(c: &Candidate) -> WireCandidate<'_> {
WireCandidate { release_key: &c.release_key, upload: c.upload, kind: c.kind, message_id: &c.message_id, file: &c.file }
WireCandidate { release_key: &c.release_key, upload: c.upload, kind: c.kind, message_id: &c.message_id, file: &c.file, bytes: c.bytes }
}
/// toWire(batch) as JSON bytes.
+39 -11
View File
@@ -1,7 +1,9 @@
import { hash } from 'node:crypto'
import type { Overview } from '../nntp/index.js'
import { FileAggregator, type AggregatedFile, isArchiveStart, isLaterVolume, sevenZipVolume } from '../files/index.js'
import { RuleSet, isRandomFile, uploadTool, type Rules } from '../rules/index.js'
import { RuleSet, isRandomFile, isScrambledFile, uploadTool, type Rules } from '../rules/index.js'
import { classKey, isUnreadableName, pendingClass } from '../release/index.js'
import { explored } from '../rules/upload_tools.js'
import { SegmentScanner, scanOverviewText, type EdgeSegment } from './scan.js'
import { encodeChunk, type ChunkFile } from '../chunks/index.js'
@@ -58,11 +60,13 @@ export interface ProbeCandidate {
kind: CandidateKind
messageId: string
file: string
/** The target article's bytes: the cheapest probe first within a class (LOG.md (130)). */
bytes?: number
}
/** What a rule dropped in a batch (REBUILD.md, rules and recovery). */
export interface RuleCount {
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'incomplete'
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive' | 'incomplete'
key: string
articles: number
files: number
@@ -246,12 +250,34 @@ export class RangeIndexer {
this.completed = new Map()
for (const [key, all] of byRelease) {
uploadsOf(all).forEach((list, upload) => {
// Settled as far as this indexer knows: it judges a release's files as each batch completes them (the Rust
// window passes whether the window closed on the release and no block edge cut it).
const verdict = this.rules.judge(
key,
list.map((f) => ({ name: f.segments[0].file.name, totalParts: f.segments[0].totalParts, groups: f.segments[0].file.groups, firstMessageId: f.segments[0].messageId, poster: f.segments[0].file.poster }))
list.map((f) => ({
name: f.segments[0].file.name,
totalParts: f.segments[0].totalParts,
groups: f.segments[0].file.groups,
firstMessageId: f.segments[0].messageId,
poster: f.segments[0].file.poster,
segments: f.segments.length,
bytes: f.segments.reduce((n, s) => n + s.bytes, 0),
})),
true
)
if (verdict.rule === null) {
this.batch.releases.push(chunkOf(key, list, upload))
const chunk = chunkOf(key, list, upload)
if (!chunk.readable) {
// Unnamed: a class the ladder found never names anything stays out of the pending store, its explored
// sample excepted (so the rate keeps being measured); counted as garbage (LOG.md (130)).
const tool = uploadTool(list[0].segments[0].messageId)
const klass = pendingClass(chunk.name, chunk.bytes, chunk.partial === true)
if (this.rules.classes.has(classKey(tool, this.options.group, klass)) && !explored(key, this.options.rules.samplePercent)) {
this.count('garbage', `${tool} ${this.options.group} #${klass}`, list.reduce((n, f) => n + f.segments.length, 0), list.length)
return
}
}
this.batch.releases.push(chunk)
this.batch.candidates.push(...candidatesOf(key, list, upload))
return
}
@@ -280,8 +306,10 @@ export class RangeIndexer {
}
for (const list of byFile.values()) {
if (this.options.partialFiles) {
// As the Rust window's files(): one article of a multi-part random file is no release.
if (list.length === 1 && (list[0].number !== 1 || list[0].totalParts > 1) && isRandomFile(list[0].file.name)) {
// As the Rust window's files(): one article of a multi-part random file is no release; a scrambled name only
// before the end, where the rest of the file can still come.
const name = list[0].file.name
if (list.length === 1 && (list[0].number !== 1 || list[0].totalParts > 1) && (isRandomFile(name) || (!atEnd && isScrambledFile(name)))) {
this.count('per-article', uploadTool(list[0].messageId), 1, 0)
continue
}
@@ -445,7 +473,7 @@ export function chunkOf(key: string, files: AggregatedFile<EdgeSegment>[], uploa
bytes,
firstArticle,
lastArticle,
readable: chunkFiles.some((f) => !isRandomFile(f.name)),
readable: chunkFiles.some((f) => !isUnreadableName(f.name)),
hasPar2: chunkFiles.some((f) => /\.par2$/i.test(f.name)),
partial: files.some((f) => f.segments.length < f.segments[0].totalParts),
payload: encodeChunk(chunkFiles),
@@ -469,12 +497,12 @@ export function candidatesOf(key: string, files: AggregatedFile<EdgeSegment>[],
}
// An unsplit 7z of more than one segment: its listing is in its last one.
if (volume === 0 && f.segments.length > 1 && !tail) tail = { volume, file: f }
if (isArchiveStart(name)) out.push({ releaseKey: key, upload, kind: 'archive', messageId: f.segments[0].messageId, file: name })
else if (/\.par2$/i.test(name) && !isLaterVolume(name)) out.push({ releaseKey: key, upload, kind: 'par2', messageId: f.segments[0].messageId, file: name })
if (isArchiveStart(name)) out.push({ releaseKey: key, upload, kind: 'archive', messageId: f.segments[0].messageId, file: name, bytes: f.segments[0].bytes })
else if (/\.par2$/i.test(name) && !isLaterVolume(name)) out.push({ releaseKey: key, upload, kind: 'par2', messageId: f.segments[0].messageId, file: name, bytes: f.segments[0].bytes })
}
if (tail) out.push({ releaseKey: key, upload, kind: '7z-tail', messageId: tail.file.segments.at(-1)!.messageId, file: tail.file.segments[0].file.name })
if (tail) out.push({ releaseKey: key, upload, kind: '7z-tail', messageId: tail.file.segments.at(-1)!.messageId, file: tail.file.segments[0].file.name, bytes: tail.file.segments.at(-1)!.bytes })
if (out.length === 0 && files.length === 1 && isRandomFile(files[0].segments[0].file.name)) {
out.push({ releaseKey: key, upload, kind: 'lone', messageId: files[0].segments[0].messageId, file: files[0].segments[0].file.name })
out.push({ releaseKey: key, upload, kind: 'lone', messageId: files[0].segments[0].messageId, file: files[0].segments[0].file.name, bytes: files[0].segments[0].bytes })
}
return out.slice(0, MAX_CANDIDATES)
}
+2 -2
View File
@@ -5,7 +5,7 @@ import { parseRarHeaders } from '#lib/rar/index'
import { parseSevenZipTail } from '#lib/sevenzip/index'
import { parseZipStart } from '#lib/zip/index'
import { sniffContentType, type ContentType } from '#lib/content/index'
import { fileStem, looksRandom } from '#lib/release/index'
import { isUnreadableName } from '#lib/release/index'
/** What one probed article showed (REBUILD.md, part 6; stored per release and kind). */
export interface Inspection {
@@ -139,7 +139,7 @@ export function usableName(inspection: Inspection, kind: string): string | null
const usable = (n: string | undefined) => {
if (!n) return null
const name = n.replace(/^.*[\\/]/, '').slice(0, 255)
return name && !PACKAGING.test(name) && !looksRandom(fileStem(name)) ? name : null
return name && !PACKAGING.test(name) && !isUnreadableName(name) ? name : null
}
const named = (inspection.names ?? []).map(usable).find((n) => n)
if (named) return named
+55
View File
@@ -0,0 +1,55 @@
/**
* The class of an unnamed release (REBUILD.md decision f; LOG.md (130)): what the headers say about its chance of a
* name, from the pending ladder's own record. Size bucket × name shape × partial, 1..70, kept in step with
* names.rs pending_class and the SQL pending_class() (migration 6800): the batch store sets it with the SQL, the
* edge skips the classes a lease marks dropped.
*/
/** Size buckets: under 1 MB, 10 MB, 100 MB, 1 GB, and from 1 GB. */
export const SIZE_BUCKETS = [1_000_000, 10_000_000, 100_000_000, 1_000_000_000]
/** Name shapes, 1..7: hex, digits, base32, letters only, mixed letters and digits, with separators, other. */
export const SHAPES = [
'hex',
'digits',
'base32',
'letters',
'alnum',
'separators',
'other',
] as const
export function sizeBucket(bytes: number): number {
let i = 0
while (i < SIZE_BUCKETS.length && bytes >= SIZE_BUCKETS[i]) i++
return i
}
export function nameShape(name: string): number {
const s = name.replace(/\.[A-Za-z0-9]{1,4}$/, '')
if (/^[0-9a-fA-F]{16,}$/.test(s)) return 1
if (/^[0-9]{6,}$/.test(s)) return 2
if (/^[a-z2-7]{20,}$/.test(s) && /[2-7]/.test(s)) return 3
if (/^[A-Za-z]+$/.test(s)) return 4
if (/^[A-Za-z0-9]+$/.test(s)) return 5
if (/^[A-Za-z0-9._-]+$/.test(s)) return 6
return 7
}
/** 1..70: (size bucket × 7 + shape − 1) × 2 + partial + 1. */
export function pendingClass(name: string, bytes: number, partial: boolean): number {
return sizeBucket(bytes) * 14 + (nameShape(name) - 1) * 2 + (partial ? 1 : 0) + 1
}
/** A class as droppedClasses lists it: "tool\tgroup\tklass". */
export function classKey(tool: string, group: string, klass: number): string {
return `${tool}\t${group}\t${klass}`
}
/** A class in words ("10-100M hex partial"). */
export function describeClass(klass: number): string {
const k = klass - 1
const size = Math.floor(k / 14)
const shape = SHAPES[Math.floor((k % 14) / 2)]
const sizes = ['<1M', '1-10M', '10-100M', '100M-1G', '1G+']
return `${sizes[size]} ${shape}${k % 2 ? ' partial' : ''}`
}
+25 -2
View File
@@ -105,6 +105,29 @@ export function subjectRemainder(subject: string): string {
.trim()
}
/**
* A subject name as shown: brackets the counters left unmatched removed ("… Facialized 002/477] - 0177.jpg", the
* counter's "[" missing, gave "… Facialized ]"; LOG.md 2026-10-29 (121)). Only the name: the key keeps the remainder.
*/
export function displayName(remainder: string): string {
if (!/[[\]()]/.test(remainder)) return remainder
const drop = new Set<number>()
for (const [open, close] of [['[', ']'], ['(', ')']]) {
const opens: number[] = []
for (let i = 0; i < remainder.length; i++) {
if (remainder[i] === open) opens.push(i)
else if (remainder[i] === close) {
if (opens.length > 0) opens.pop()
else drop.add(i)
}
}
for (const i of opens) drop.add(i)
}
if (drop.size === 0) return remainder
const name = remainder.split('').filter((_, i) => !drop.has(i)).join('').replace(/ {2,}/g, ' ').trim()
return name || remainder
}
/**
* Posting tools put random per-file tokens and poster IDs in subjects (e.g.
* `elys-de81bcabef5faf6a - [Nyh5Q07Y]`); only text that still reads like a
@@ -448,7 +471,7 @@ export function releaseIdentity(input: ReleaseInput): ReleaseIdentity | null {
if (!image && RANDOM_VOLUME.test(input.name) && /^[A-Za-z0-9_-]+$/.test(stem) && looksRandom(stem.replace(/[-_]/g, ''))) parts.push(stem)
return {
key: hash(...parts),
name: remainder,
name: displayName(remainder),
fileTotal,
source: 'subject',
}
@@ -456,7 +479,7 @@ export function releaseIdentity(input: ReleaseInput): ReleaseIdentity | null {
// Not a name, but a post id (e.g. `83110629-n [10/18] - "random"`): with the
// [i/M] counter it groups the files of one post whose names are random.
if (!tag && postTotal && isPostToken(remainder)) {
return { key: hash('post', poster, remainder, postTotal), name: remainder, fileTotal, source: 'post' }
return { key: hash('post', poster, remainder, postTotal), name: displayName(remainder), fileTotal, source: 'post' }
}
// A scene music post's index files under a bare counter go with its tracks: the artist and the total ("[01/13] -
// "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with "[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the
+3 -1
View File
@@ -1,5 +1,7 @@
/**
* Grouping of files into releases (downloads), independent of AdonisJS.
*/
export { releaseIdentity, par2ReleaseIdentity, par2ReleaseName, rarReleaseIdentity, isGenericName, fileStem, sizeAfterName, runOn, subjectRemainder, namesRelease, isPostToken, type ReleaseIdentity, type ReleaseInput } from './identity.js'
export { releaseIdentity, displayName, par2ReleaseIdentity, par2ReleaseName, rarReleaseIdentity, isGenericName, fileStem, sizeAfterName, runOn, subjectRemainder, namesRelease, isPostToken, type ReleaseIdentity, type ReleaseInput } from './identity.js'
export { looksRandom } from './random.js'
export { isUnreadableName, tokenKind } from './readable.js'
export { pendingClass, classKey, describeClass, nameShape, sizeBucket } from './classes.js'
+148
View File
@@ -0,0 +1,148 @@
import { fileStem } from './identity.js'
import { looksRandom } from './random.js'
/**
* Readable or not: whether a name says what a release is (the catalog's readable flag, the prober's inner names).
* looksRandom (random.ts) knows the single-token shapes; this reads a name token by token, so separators, spaces
* and random "extensions" don't hide a scramble (LOG.md (129)).
*/
/**
* Tokens that don't make a name readable on their own: packaging, volume numbers and the posting tools' fixed
* words ("Description - Lt5nHjlo3RiuBb3C", "bd5b4a59f3aa87114dcf75c0-addon").
*/
const NOISE =
/^(addon|samples?|proof|subs?|nfo|part\d+|vol\d+|description|repost|rep|extras?|bonus|cd\d*|dvd\d*|dis[ck]\d*|par2|rar|zip|7z|mkv|mp4|avi|vol)$/i
/** Mixed tokens that are codes, not scrambles: S02E10, 1080p, 10bit, x264, Win10x64, BDRip720p, 3DMark03, v2, 5th. */
const CODE =
/^([sS]\d{1,2}[eE]\d{1,3}|[xh]26[45]|[A-Za-z]+\d+([a-z]{1,3}|x(64|86))?|\d+[A-Za-z]{1,3}|\d+[A-Za-z]+\d+)$/
/** The NZBSPLIT tools' hash between their markers is no part of the name. */
const NZBSPLIT = /__NZBSPLIT__[0-9a-f]{32}__NZBSPLIT__/i
/** Scene tags write "i" for "I" between capitals (XCiTE, PRESCRiPTiON): read it as a capital there. */
function sceneCase(t: string): string {
let out = ''
for (let i = 0; i < t.length; i++) {
out +=
t[i] === 'i' && i > 0 && i < t.length - 1 && /[A-Z]/.test(t[i - 1]) && /[A-Z]/.test(t[i + 1])
? 'I'
: t[i]
}
return out
}
/**
* The signals of a scrambled token: letter/digit and lower-to-upper switches; case switches in both directions; lone
* lowercase letters between capitals ("NtpUgCmS"); an inner run of 2+ capitals with lowercase on both sides
* ("tBWTuz", "YDoaROv"); the longest consonant run; vowels (a lowercase y too: "McCartney"); runs of one character class.
*/
function signals(t: string) {
let switches = 0
let caseSwitches = 0
let lone = 0
let consonants = 0
let maxConsonants = 0
for (let i = 0; i < t.length; i++) {
const c = t[i]
if (/[b-df-hj-np-tv-xz]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
else consonants = 0
if (i === 0) continue
const a = t[i - 1]
if (/\d/.test(a) !== /\d/.test(c)) switches++
else if (/[a-z]/.test(a) && /[A-Z]/.test(c)) switches++
if ((/[a-z]/.test(a) && /[A-Z]/.test(c)) || (/[A-Z]/.test(a) && /[a-z]/.test(c))) caseSwitches++
if (
i < t.length - 1 &&
/[a-z]/.test(c) &&
c !== 'i' &&
/[A-Z]/.test(a) &&
/[A-Z]/.test(t[i + 1])
)
lone++
}
const innerUpper = /[a-z][A-Z]{2,}[a-z]/.test(t)
const vowels = (t.match(/[aeiouAEIOU]|y/g) ?? []).length
const runs = (t.match(/[a-z]+|[A-Z]+|\d+/g) ?? []).length
const letters = (t.match(/[A-Za-z]/g) ?? []).length
const maxLower = Math.max(0, ...(t.match(/[a-z]+/g) ?? []).map((r) => r.length))
return {
switches,
caseSwitches,
lone,
innerUpper,
maxConsonants,
vowels,
runs,
letters,
maxLower,
}
}
/**
* What one alphanumeric token of a name is: a word (letters only, in one case or Capitalized, 3+ long, not a packaging
* word), a scramble, or neither (digits, codes, CamelCase, short tokens). Kept in step with names.rs token_kind.
*/
export function tokenKind(t: string): 'word' | 'random' | null {
if (/^\d+$/.test(t) || NOISE.test(t) || CODE.test(t)) return null
if (/^[A-Za-z]+$/.test(t)) {
const s = signals(sceneCase(t))
if (s.caseSwitches === 0 || (s.caseSwitches === 1 && /^[A-Z][a-z]+$/.test(t))) {
// A base32 consonant soup ("mrrtmzrzmi4dgmrzmyytqmlbztczbx" without its digit) is no word.
if (t.length >= 10 && s.vowels * 6 < t.length) return 'random'
return t.length >= 3 ? 'word' : null
}
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble. Two CamelCase words with few vowels
// (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone letter; a scramble has more runs, a lone
// letter or an inner capital run.
if (
t.length >= 6 &&
s.caseSwitches >= 2 &&
s.vowels * 4 < t.length &&
(s.runs >= 5 || s.lone >= 1 || s.innerUpper)
)
return 'random'
return null
}
if (t.length >= 8 && /^[0-9a-f]+$/i.test(t) && (t.match(/\d/g)?.length ?? 0) >= 2) return 'random'
if (/^[A-Z0-9]+$/.test(t)) {
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
return t.length >= 12 && signals(t).switches >= 6 ? 'random' : null
}
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
if (
(t.match(/[A-Za-z]+/g) ?? []).every(
(w) =>
/^[A-Z]{2,}$/.test(w) ||
(/^[A-Z]?[a-z]{2,}$/.test(w) && (w.match(/[aeiouy]/g) ?? []).length * 5 >= w.length)
)
)
return null
const s = signals(sceneCase(t))
if (t.length >= 5 && s.switches >= 3) return 'random'
if (t.length >= 6 && s.switches >= 2 && (s.lone >= 1 || s.innerUpper || s.maxConsonants >= 4))
return 'random'
// Few vowels and no lowercase run of 3+ ("Jk5Pud", "RKi9BCg"; not "HDR10Plus").
if (t.length >= 6 && s.switches >= 2 && s.vowels * 4 < s.letters && s.maxLower <= 2)
return 'random'
return null
}
/**
* A file or release name no one can read: looksRandom's shapes, or more scrambled tokens than words by letter count
* ("Zx5AjNaK-mh7CFg6Hf8uu", "Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ"; not "Paul McCartney - Band on the Run",
* "ShesSoWild - Ester Ana 1080p"). Separators and a random "extension" of any length don't hide a scramble, which
* looksRandom's single-token shapes miss (15% of the readable catalog on 2026-11-02; LOG.md (129)). The readable
* flag; the drop rules keep looksRandom, which is narrower.
*/
export function isUnreadableName(name: string): boolean {
const stem = fileStem(name).replace(NZBSPLIT, ' ')
if (looksRandom(stem)) return true
let words = 0
let random = 0
for (const t of stem.split(/[^A-Za-z0-9]+/)) {
if (!t) continue
const kind = tokenKind(t)
if (kind === 'word') words += t.length
else if (kind === 'random') random += t.length
}
return random > 0 && random > words
}
+1 -1
View File
@@ -1,2 +1,2 @@
export { uploadTool, signatureGroup, signatureKey, explored } from './upload_tools.js'
export { RuleSet, DEFAULT_RULES, isRandomFile, ownerOf, posterWords, type Rules, type Verdict } from './rule_set.js'
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, isTinyArchive, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, TINY_MIN_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
+174 -4
View File
@@ -12,6 +12,11 @@ export interface Rules {
priorities: Record<string, number>
/** Dropped signatures, as "tool\tgroup" (GarbageFilter's table). */
droppedSignatures: string[]
/**
* Dropped classes of unnamed releases, as "tool\tgroup\tklass" (lib/release/classes.ts; the pending ladder's record,
* LOG.md (130)): the edge keeps them out of the pending store, the explored sample excepted.
*/
droppedClasses: string[]
/** Tools whose random-named uploads are dropped in every group (named < 1%; LOG.md 2026-10-08 (5)). */
droppedTools: string[]
/** A random-named file of at most this many parts: a random subject per article, needs bodies. */
@@ -30,6 +35,24 @@ export interface Rules {
* again under other titles (119k releases on 2026-10-16; LOG.md 2026-10-16 (65)).
*/
droppedPosters: string[]
/**
* Drop settled releases of one unsplit archive without a par2, posted to 2+ groups, under STUB_MAX_BYTES, whose
* articles are at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES, or posted to 4+ groups, or by
* a webmail poster: the exe-stub and recycled-installer floods under film and software names (55% of the catalog
* on 2026-10-30; 99.3% of the known fakes, 0 known content; LOG.md (123)-(126), (128)).
*/
dropStubArchives: boolean
/**
* Drop one-article archives of NONSENSE_MIN_BYTES to NONSENSE_MAX_BYTES under a release-like, unscrambled name:
* the newznzb.* pieces (9% of the catalog; LOG.md (123), (125)).
*/
dropNonsensePieces: boolean
/**
* Drop one-article archives of TINY_MIN_BYTES to NONSENSE_MIN_BYTES under a release-like name (spaces allowed),
* outside the exempt groups, without an e-book, subtitle, crack or sidecar word: the 2015-17 exe-stub flood in
* miniature (a RAR holding "<title>.exe" at 2-60 kB; 1.65M catalog rows; LOG.md (131), (133)).
*/
dropTinyArchives: boolean
}
export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
@@ -39,9 +62,37 @@ export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
samplePercent: 1,
dropExecutables: true,
droppedPosters: ['celebmovie', 'celebwow', 'celebfx', 'iluvcelebs', 'nudecelebs', 'celebsextapes'],
dropStubArchives: true,
dropNonsensePieces: true,
dropTinyArchives: true,
droppedClasses: [],
}
export type Verdict = { rule: null } | { rule: 'poster' | 'executable' | 'garbage' | 'per-article'; key: string; sample: boolean }
/** The stub-archive rule's bounds (edge/src/names.rs): Power-Post's 250 kB default article, the fakes' size span. */
export const STUB_ARTICLE_BYTES = 262144
export const STUB_MIN_BYTES = 1_000_000
export const STUB_MAX_BYTES = 100_000_000
export const STUB_WIDE_GROUPS = 4
/** The nonsense-piece rule's bounds: under 100 kB are cue/log sidecars and adverts, 1 MB and up is rule 1's. */
export const NONSENSE_MIN_BYTES = 100_000
export const NONSENSE_MAX_BYTES = 1_000_000
/** The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars. */
export const TINY_MIN_BYTES = 2_000
export type Verdict =
| { rule: null }
| { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive'; key: string; sample: boolean }
/** A file as the judge sees it: its name, parts, groups, poster, and the articles seen with their bytes. */
export interface JudgedFile {
name: string
totalParts: number
groups: string[]
firstMessageId: string
poster: string
segments: number
bytes: number
}
/** A poster as droppedPosters matches it: lowercased, letters and digits only. */
export function posterWords(poster: string): string {
@@ -51,11 +102,117 @@ export function posterWords(poster: string): string {
/** Files that come along with a release's content: they don't make it one of executables or not. */
const SIDECAR = /\.(par2|nfo|sfv|txt|jpe?g|png|nzb)$/i
const EXECUTABLE = /\.(exe|scr|com|pif|bat|cmd|vbs|lnk)\s*$/i
/** One zip, rar or 7z that is not a volume of a set. */
const ARCHIVE = /\.(zip|rar|7z)$/i
/** "cneg" is "part" in ROT13: the a.b.erotica posts named that way are volumes of passworded videos (LOG.md (128)). */
const VOLUME = /\.((part|cneg)\d+\.rar|r\d\d|z\d\d|7z\.\d\d\d|\d\d\d)$/i
/** A poster at a free mail provider: the stub floods' throwaway accounts. */
const WEBMAIL = /@(gmail|googlemail|yahoo|hotmail|outlook|live|msn|aol|gmx|web|t-online|freenet|mail|yandex|icloud|me|hushmail|protonmail|libero|orange|free|arcor|wanadoo|ymail|rocketmail|inbox|mail\.ru|bk|list)\./i
/** A name that reads as a release: a year, a scene tag, a group suffix or dotted words. */
const RELEASE_LIKE = /(\b(19|20)\d\d\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s\d\de\d\d|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v\d+(\.\d+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)/i
/**
* Groups whose communities post real single small archives (karaoke zips, e-books, comics, tabs, fonts) the way the
* stub floods do (Power-Post defaults, a crosspost or two): a release touching one is exempt from the stub-archive
* rule. Costs 16 of 32,308 known fakes in the 217k sample (LOG.md (127)).
*/
const STUB_EXEMPT_GROUPS = /(sound|mp3|karaoke|music|audio|flac|lossless|e-?book|epub|comic|guitar|font|magazine)/i
/** The single unsplit archive a release consists of, or null. */
function singleArchive(files: JudgedFile[]): JudgedFile | null {
return files.length === 1 && ARCHIVE.test(files[0].name) && !VOLUME.test(files[0].name) ? files[0] : null
}
/**
* The legs of the stub-archive rule a settled single archive under STUB_MAX_BYTES (the fakes stop at ~90 MB;
* hash-named uploads, real albums and passworded videos above it don't) trips: "A" (articles <= STUB_ARTICLE_BYTES,
* release >= STUB_MIN_BYTES: a one-article release always has small articles, so e-books and sidecars are exempt),
* "X" (4+ groups) and "W" (webmail poster). Empty when none.
*/
export function stubArchiveLegs(f: JudgedFile): string {
if (f.groups.length < 2 || f.bytes >= STUB_MAX_BYTES || f.groups.some((g) => STUB_EXEMPT_GROUPS.test(g))) return ''
let legs = ''
if (f.bytes / Math.max(f.segments, 1) <= STUB_ARTICLE_BYTES && f.bytes >= STUB_MIN_BYTES) legs += 'A'
if (f.groups.length >= STUB_WIDE_GROUPS) legs += 'X'
if (WEBMAIL.test(f.poster)) legs += 'W'
return legs
}
/** An e-book by its name ("Author - Title (2020) (epub).rar"): one small article is a book, not a nonsense piece. */
const EBOOK = /\b(epub|mobi|azw3?|pdf|djvu|cbr|cbz|e-?books?)\b/i
/**
* A nonsense piece: one article, one small archive under a release-like, unscrambled name without spaces (the
* newznzb.* pieces are scene-styled or random with a "-XX" suffix; books and albums are written with spaces).
*/
export function isNonsensePiece(f: JudgedFile): boolean {
return (
f.totalParts === 1 &&
f.segments === 1 &&
f.bytes >= NONSENSE_MIN_BYTES &&
f.bytes < NONSENSE_MAX_BYTES &&
!f.name.includes(' ') &&
RELEASE_LIKE.test(fileStem(f.name)) &&
!EBOOK.test(f.name) &&
!isScrambledFile(f.name) &&
!isRandomFile(f.name)
)
}
/** RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way. */
const RELEASE_LIKE_SPACED = new RegExp(RELEASE_LIKE.source.replace('[._-]', '[._ -]'), 'i')
/**
* Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
* exe is the content), covers and sidecars.
*/
const TINY_CONTENT =
/\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b/i
/**
* A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
* unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
* subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
*/
export function isTinyArchive(f: JudgedFile): boolean {
return (
f.totalParts === 1 &&
f.segments === 1 &&
f.bytes >= TINY_MIN_BYTES &&
f.bytes < NONSENSE_MIN_BYTES &&
!f.groups.some((g) => STUB_EXEMPT_GROUPS.test(g)) &&
RELEASE_LIKE_SPACED.test(fileStem(f.name)) &&
!EBOOK.test(f.name) &&
!TINY_CONTENT.test(f.name) &&
!isScrambledFile(f.name) &&
!isRandomFile(f.name)
)
}
export function isRandomFile(name: string) {
return looksRandom(fileStem(name))
}
/**
* At least half the letters and digits of a file's stem in tokens of 5 or more that switch case (lower to upper) or
* between letters and digits 3 times or more ("nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr", "bslXlwgPiqZadhRa"): the names a
* random-subject-per-article upload gives each article that looksRandom misses for their separators.
*/
export function isScrambledFile(name: string) {
let total = 0
let scrambled = 0
for (const token of fileStem(name).split(/[^A-Za-z0-9]+/)) {
total += token.length
let switches = 0
for (let i = 1; i < token.length; i++) {
const [a, b] = [token[i - 1], token[i]]
if ((/[a-z]/.test(a) && /[A-Z]/.test(b)) || /\d/.test(a) !== /\d/.test(b)) switches++
}
if (token.length >= 5 && switches >= 3) scrambled += token.length
}
return total >= 8 && scrambled * 2 >= total
}
/** The owner of an article posted to `groups` (its Xref), or null when none of them is indexed. */
export function ownerOf(groups: string[], priorities: Record<string, number>): string | null {
let owner: string | null = null
@@ -68,20 +225,24 @@ export function ownerOf(groups: string[], priorities: Record<string, number>): s
export class RuleSet {
private signatures: Set<string>
readonly classes: Set<string>
private tools: Set<string>
private posters: string[]
constructor(readonly rules: Rules) {
this.posters = rules.droppedPosters ?? []
this.signatures = new Set(rules.droppedSignatures)
this.classes = new Set(rules.droppedClasses ?? [])
this.tools = new Set(rules.droppedTools)
}
/**
* A release's verdict from its files: dropped when its poster is; when its content is bare
* executables; else kept as a whole when any file has a readable name; else dropped as garbage when its signature or tool is
* dropped, as per-article random subjects when every file has few parts.
* executables; when it is a settled single archive that is a stub or a nonsense piece; else kept as a whole when
* any file has a readable name; else dropped as garbage when its signature or tool is dropped, as per-article
* random subjects when every file has few parts. `settled`: the window closed on the release (a par2 can still
* join an open one), which the single-archive rules need.
*/
judge(releaseKey: string, files: { name: string; totalParts: number; groups: string[]; firstMessageId: string; poster: string }[]): Verdict {
judge(releaseKey: string, files: JudgedFile[], settled = true): Verdict {
const sample = explored(releaseKey, this.rules.samplePercent)
if (this.posters.length > 0 && files.length > 0) {
const words = posterWords(files[0].poster)
@@ -94,6 +255,15 @@ export class RuleSet {
return { rule: 'executable', key: EXECUTABLE.exec(content[0].name)![1].toLowerCase(), sample }
}
}
const archive = settled ? singleArchive(files) : null
if (archive) {
if (this.rules.dropStubArchives) {
const legs = stubArchiveLegs(archive)
if (legs !== '') return { rule: 'stub-archive', key: legs, sample }
}
if (this.rules.dropNonsensePieces && isNonsensePiece(archive)) return { rule: 'nonsense-piece', key: uploadTool(archive.firstMessageId), sample }
if (this.rules.dropTinyArchives && isTinyArchive(archive)) return { rule: 'tiny-archive', key: uploadTool(archive.firstMessageId), sample }
}
if (files.some((f) => !isRandomFile(f.name))) return { rule: null }
const first = files[0]
const tool = uploadTool(first.firstMessageId)
+1 -1
View File
@@ -22,7 +22,7 @@
<label class="search-box">
<svg aria-hidden="true" viewBox="0 0 24 24"><circle cx="11" cy="11" r="7" /><path d="m20 20-3.5-3.5" /></svg>
<input id="q" type="search" autocomplete="off" spellcheck="false"
placeholder="Search release names…" aria-label="Search" />
placeholder="Search release names… (^word: at a word's start)" aria-label="Search" />
<kbd aria-hidden="true">/</kbd>
</label>
<label class="select">
+6 -6
View File
@@ -431,10 +431,10 @@ test.group('BatchStore', (group) => {
assert,
}) => {
await storeIn(GROUP, 1, [
...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
...upload(30, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'),
...upload(40, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'),
...upload(10, 'Some.Show.S01E04.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
...upload(20, 'Some.Show.S01E04.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
...upload(30, 'Some.Show.S01E04.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'),
...upload(40, 'Some.Show.S01E04.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'),
...upload(
50,
'Some.Show.S01E03.part01.rar',
@@ -455,8 +455,8 @@ test.group('BatchStore', (group) => {
candidates.map((c) => [c.kind, c.file]),
[
['archive', 'Some.Show.S01E03.part01.rar'],
['archive', 'TsIgNBewuFbuwWy.7z.001'],
['7z-tail', 'TsIgNBewuFbuwWy.7z.004'],
['archive', 'Some.Show.S01E04.7z.001'],
['7z-tail', 'Some.Show.S01E04.7z.004'],
]
)
})
+28
View File
@@ -0,0 +1,28 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import { pendingClass } from '#lib/release/index'
test.group('Pending classes / SQL pending_class()', () => {
test('the SQL function agrees with pendingClass', async ({ assert }) => {
const cases: [string, number, boolean][] = [
['c04212a389cf69edc54767355ab5285f', 5_000_000, false],
['c04212a389cf69edc54767355ab5285f', 5_000_000, true],
['Zx5AjNaK-mh7CFg6Hf8uu', 500_000_000, false],
['1650777167', 100, false],
['mrrtmzrzmi4dgmrzmyytqmlbztczbx', 2_000_000_000, false],
['PsNdnYiugfZ.rar', 20_000_000, false],
['n5d1ueIf34X', 20_000_000, true],
['sgkra vU7u2yuw9LDLCjDu04pOez', 999, false],
['Description - Lt5nHjlo3RiuBb3C', 40_000_000, false],
['ABCDEFGHIJKLMNOP', 0, false],
]
for (const [name, bytes, partial] of cases) {
const { rows } = await db.rawQuery(`SELECT pending_class(?, ?::bigint, ?) AS k`, [
name,
bytes,
partial,
])
assert.equal(Number(rows[0].k), pendingClass(name, bytes, partial), name)
}
})
})
+196 -1
View File
@@ -9,7 +9,7 @@ let n = 0
async function pending(
name: string,
candidates: { kind: string; messageId: string; file: string }[],
candidates: { kind: string; messageId: string; file: string; bytes?: number }[],
extra: Record<string, unknown> = {}
) {
n++
@@ -206,3 +206,198 @@ test.group('Pending ladder', (group) => {
assert.lengthOf(await db.from('pending_signatures'), 0)
})
})
test.group('Pending ladder / classes', (group) => {
group.each.setup(async () => {
PendingLadder.forget()
const rollback = await testUtils.db().withGlobalTransaction()
groupId = Number(
(
await db
.table('groups')
.insert({ name: 'alt.binaries.classes', priority: 1 })
.returning('id')
)[0].id
)
n = 0
return rollback
})
test('a claim takes the class with the best record first, the cheapest target within it, the rest oldest first', async ({
assert,
}) => {
// Class 15 (1-10 MB hex, whole) named 9 of 10; class 16 (partial) named 0 of 50; class 0: rows from before.
await db.table('pending_classes').multiInsert([
{ tool: 'other', group_id: groupId, klass: 15, probed: 10, named: 9, waiting: 2 },
{ tool: 'other', group_id: groupId, klass: 16, probed: 50, named: 0, waiting: 1 },
])
const old = await pending('00000000000000000000000000000031', [
{ kind: 'lone', messageId: '<o@x>', file: 'o' },
])
const poor = await pending(
'00000000000000000000000000000032',
[{ kind: 'lone', messageId: '<p@x>', file: 'p' }],
{ klass: 16 }
)
const bigTarget = await pending(
'00000000000000000000000000000033',
[{ kind: 'archive', messageId: '<b@x>', file: 'b.rar', bytes: 700_000 }],
{ klass: 15, target_bytes: 700_000 }
)
const smallTarget = await pending(
'00000000000000000000000000000034',
[{ kind: 'archive', messageId: '<s@x>', file: 's.rar', bytes: 50_000 }],
{ klass: 15, target_bytes: 50_000 }
)
const probes = await PendingLadder.claim(10)
assert.deepEqual(
probes.map((p) => [p.releaseKey, p.klass]),
[
[smallTarget, 15],
[bigTarget, 15],
[poor, 16],
[old, 0],
]
)
const waiting = await db
.from('pending_classes')
.where({ group_id: groupId })
.orderBy('klass')
.select('klass', 'waiting')
assert.deepEqual(
waiting.map((c) => [c.klass, c.waiting]),
[
[15, 0],
[16, 0],
]
)
// Outcomes count on the class; a probe given back waits again.
await PendingLadder.record([
{
probe: probes[0],
status: 'done',
inspection: inspection('s.rar', ['Some.Movie.2016.1080p.mkv']),
},
{ probe: probes[1], status: 'waiting' },
{ probe: probes[2], status: 'missing' },
])
const classes = await db.from('pending_classes').where({ group_id: groupId }).orderBy('klass')
assert.deepEqual(
classes.map((c) => [c.klass, c.probed, c.named, c.waiting, c.dropped]),
[
[15, 11, 10, 1, false],
[16, 51, 0, 0, false],
]
)
})
test('a class that names nothing is dropped and its waiting releases discarded, the other classes untouched', async ({
assert,
}) => {
await db
.table('pending_classes')
.insert({ tool: 'other', group_id: groupId, klass: 16, probed: 99, named: 0, waiting: 1 })
const last = await pending(
'00000000000000000000000000000041',
[{ kind: 'lone', messageId: '<z@x>', file: 'z' }],
{ klass: 16 }
)
const [probe] = await PendingLadder.claim(1)
assert.equal(probe.releaseKey, last)
await PendingLadder.record([{ probe, status: 'missing' }])
assert.include(
await db.from('pending_classes').where({ group_id: groupId, klass: 16 }).firstOrFail(),
{ probed: 100, named: 0, dropped: true }
)
const waiting = await pending(
'00000000000000000000000000000042',
[{ kind: 'lone', messageId: '<w@x>', file: 'w' }],
{ klass: 16 }
)
const other = await pending(
'00000000000000000000000000000043',
[{ kind: 'lone', messageId: '<o@x>', file: 'o' }],
{ klass: 15 }
)
await db.rawQuery(
`UPDATE pending_classes SET waiting = waiting + 1 WHERE group_id = ? AND klass = 16`,
[groupId]
)
await db.table('rule_sets').insert({
version: 98,
rules: JSON.stringify({ samplePercent: 0, droppedSignatures: [] }),
note: 'test',
})
assert.equal(await PendingLadder.discard(), 1)
const statusOf = async (key: string) =>
(
await db
.from('pending_releases')
.where('release_key', Buffer.from(key, 'hex'))
.firstOrFail()
).status
assert.equal(await statusOf(waiting), PENDING.discarded)
assert.equal(await statusOf(other), PENDING.waiting)
assert.equal(
(await db.from('pending_classes').where({ group_id: groupId, klass: 16 }).firstOrFail())
.waiting,
0
)
})
test('a twin by content names a release whose own probe did not', async ({ assert }) => {
// A catalog release with a readable name holds the same first 16 KiB and size.
const [poster] = await db.table('posters').insert({ poster: 'Twin <twin@x>' }).returning('id')
const [set] = await db
.table('group_sets')
.insert({ groups: JSON.stringify(['alt.binaries.classes']) })
.returning('id')
const [twin] = await db
.table('releases')
.insert({
release_key: Buffer.from('aa'.repeat(20), 'hex'),
identity_key: Buffer.from('bb'.repeat(20), 'hex'),
group_id: groupId,
group_set_id: set.id,
poster_id: poster.id,
name: 'Some.Movie.2016.1080p.BluRay-GRP',
source: 3,
posted_at: new Date('2026-10-01T00:00:00Z'),
first_article: 1,
last_article: 2,
files: 2,
segments: 2,
bytes: 1000,
readable: true,
})
.returning('id')
await db.table('file_hashes').insert({
candidate_id: 777_001,
release_id: twin.id,
hash16k: 'deadbeef'.repeat(4),
size: 1000,
})
const key = await pending('00000000000000000000000000000051', [
{ kind: 'lone', messageId: '<t@x>', file: '00000000000000000000000000000051' },
])
const [probe] = await PendingLadder.claim(1)
assert.equal(probe.releaseKey, key)
await PendingLadder.record([
{
probe,
status: 'done',
inspection: {
...inspection('00000000000000000000000000000051'),
hash16k: 'deadbeef'.repeat(4),
},
},
])
assert.include(
await db.from('pending_releases').where('release_key', Buffer.from(key, 'hex')).firstOrFail(),
{
status: PENDING.naming,
named: 'Some.Movie.2016.1080p.BluRay-GRP',
}
)
})
})
+49
View File
@@ -163,6 +163,55 @@ test.group('ReleaseJudge', (group) => {
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
})
test('judges tiny single-article files under release-like names', async ({ assert }) => {
let n = 0
const settled = new Date(Date.now() - 3 * 86400_000)
const release = async (name: string, extra: Record<string, unknown> = {}) => {
n++
const key = `jt${String(n).padStart(6, '0')}`.padEnd(40, '0')
return Number(
await insertRelease({
release_key: key,
identity_key: key,
name,
source: 'stem',
groups: ['alt.binaries.boneless'],
files: 1,
segments: 1,
bytes: 15_000,
posted_at: settled,
readable: true,
settled_at: settled,
...extra,
})
)
}
const dotted = await release('Velvet.S02E20.GERMAN.720p.HDTV.x264-ACED')
const spaced = await release('Pacific Rim (2013) BDRip DVDr-sailo1')
const subs = await release('Some.Film.2019.German.Subs-XYZ')
const ebook = await release('Some.Book.2019.epub-GRP')
const music = await release('Some.Album.2019.FLAC-GRP', { groups: ['alt.binaries.sounds.flac'] })
const big = await release('Big.Film.2019.1080p-GRP', { bytes: 500_000 })
const twoFiles = await release('Two.Films.2019.1080p-GRP', { files: 2, segments: 2 })
const plain = await release('Holiday photo')
const sidecar = await release('Side.Car.2019.1080p-GRP', { sidecar_only: true })
const unreadable = await release('Xk9v.2019.Qpz-ab', { readable: false })
const round = await new ReleaseJudge().full()
const verdicts = new Map(
(await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [
Number(v.release_id),
v.rule,
])
)
assert.equal(verdicts.get(dotted), 'tiny')
assert.equal(verdicts.get(spaced), 'tiny')
for (const id of [subs, ebook, music, big, twoFiles, plain, sidecar, unreadable]) {
assert.notEqual(verdicts.get(id), 'tiny')
}
assert.equal(round.rules.tiny.added, 2)
})
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
assert,
}) => {
+22
View File
@@ -138,4 +138,26 @@ test.group('Releases API', (group) => {
const list = (await client.get('/api/v1/releases?q=locked movie')).body() as { releases: { passwordKnown: boolean }[] }
assert.isTrue(list.releases[0].passwordKnown)
})
test('a word with a leading ^ matches at a word start only; without it anywhere', async ({ client, assert }) => {
const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
const [n] = await db.table('nodes').insert({ name: 'api-node' }).returning('id')
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: n.id }).returning('id')
const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
indexer.add([
...file('Zed.Dead.2019.1080p.mkv', 'Zed.Dead.2019 [1/1]', 2, 100),
...file('The_Zed_Files.S01E01.mkv', 'The_Zed_Files.S01E01 [1/1]', 2, 200),
...file('Babe.Facialized.2017.mkv', 'Babe.Facialized.2017 [1/1]', 2, 300),
], 1001)
indexer.finish()
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
const names = async (q: string) =>
((await client.get(`/api/v1/releases?q=${encodeURIComponent(q)}`)).body() as { releases: { name: string }[] }).releases.map((x) => x.name).sort()
assert.lengthOf(await names('zed'), 3)
assert.deepEqual(await names('^zed'), ['The_Zed_Files S01E01', 'Zed Dead 2019'])
assert.deepEqual(await names('^zed dead'), ['Zed Dead 2019'])
// Regex characters in the word are literal.
assert.deepEqual(await names('^ze+d'), [])
})
})
+14
View File
@@ -105,6 +105,20 @@ test.group('edge / RangeIndexer', () => {
assert.deepEqual(batch.fragments, [])
assert.deepEqual(batch.counts.map((c) => [c.rule, c.articles, c.files]), [['per-article', 1, 0]])
})
test('partialFiles: one article of a multi-part file under a scrambled name is per-article before the end, a partial release at it', ({ assert }) => {
const indexer = new RangeIndexer({ group: GROUP, start: 100, end: 1_000_000, rules, idleChunks: 1, edgeArticles: 0, partialFiles: true })
// Part 8 of 67, a random poster's: given up mid-range.
indexer.add(post(100, 'nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr', 67, { poster: 'guAym5TqSlp SCTLKQ0R <8nGc3Rk@OrjrYqFkp.z>' }).slice(7, 8), 200)
indexer.add([], 300)
indexer.add([], 400)
// The same shape still open when the block ends: the rest can be in the next one.
indexer.add(post(999_990, 'bslXlwgPiqZadhRa', 5).slice(1, 2), 1_000_001)
indexer.finish()
const batch = indexer.take()
assert.deepEqual(batch.releases.map((r) => [r.name, r.partial]), [['bslXlwgPiqZadhRa', true]])
assert.deepEqual(batch.counts.map((c) => [c.rule, c.articles, c.files]), [['per-article', 1, 0]])
})
})
test.group('edge / taking over a range', () => {
+22
View File
@@ -0,0 +1,22 @@
import { test } from '@japa/runner'
import { classKey, describeClass, pendingClass } from '#lib/release/index'
test.group('release / pendingClass', () => {
test('size bucket × name shape × partial, as names.rs pending_class', ({ assert }) => {
// The same cases as the Rust test.
assert.equal(pendingClass('c04212a389cf69edc54767355ab5285f', 5_000_000, false), 15)
assert.equal(pendingClass('c04212a389cf69edc54767355ab5285f', 5_000_000, true), 16)
assert.equal(pendingClass('Zx5AjNaK-mh7CFg6Hf8uu', 500_000_000, false), 3 * 14 + 5 * 2 + 1)
assert.equal(pendingClass('1650777167', 100, false), 3)
assert.equal(
pendingClass('mrrtmzrzmi4dgmrzmyytqmlbztczbx', 2_000_000_000, false),
4 * 14 + 2 * 2 + 1
)
assert.equal(pendingClass('PsNdnYiugfZ.rar', 20_000_000, false), 2 * 14 + 3 * 2 + 1)
assert.equal(pendingClass('n5d1ueIf34X', 20_000_000, true), 2 * 14 + 4 * 2 + 2)
assert.equal(pendingClass('sgkra vU7u2yuw9LDLCjDu04pOez', 999, false), 13)
assert.equal(describeClass(16), '1-10M hex partial')
assert.equal(describeClass(3 * 14 + 5 * 2 + 1), '100M-1G separators')
assert.equal(classKey('nyuu', 'alt.binaries.boneless', 16), 'nyuu\talt.binaries.boneless\t16')
})
})
+13 -1
View File
@@ -1,5 +1,5 @@
import { test } from '@japa/runner'
import { fileStem, releaseIdentity, runOn, sizeAfterName, subjectRemainder } from '#lib/release/index'
import { displayName, fileStem, releaseIdentity, runOn, sizeAfterName, subjectRemainder } from '#lib/release/index'
import { looksLikeFilename, weakFileKey } from '#lib/headers/index'
test.group('release / fileStem', () => {
@@ -604,3 +604,15 @@ test.group('release / par2 names', () => {
assert.equal(id('[2/6] - "x.jpg" yEnc', 'x.jpg').key, id('[1/6] - "x.part01.rar" yEnc', 'x.part01.rar').key)
})
})
test.group('release / displayName', () => {
test('brackets the counters left unmatched leave the name, the key keeps them', ({ assert }) => {
const subject = (i: number) => `Babe with Plump Bubble Butt Fucked and Facialized ${String(i).padStart(3, '0')}/477] - 0${176 + i}.jpg`
const id = (i: number) => releaseIdentity({ poster: 'webmaster@hdbangvidz.com (Porn4You)', weak: false, subject: subject(i), name: `0${176 + i}.jpg` })!
assert.equal(id(2).name, 'Babe with Plump Bubble Butt Fucked and Facialized')
assert.equal(id(2).key, id(3).key)
assert.equal(displayName('[Group] Show (2020)'), '[Group] Show (2020)')
assert.equal(displayName('( Show [x] name'), 'Show [x] name')
assert.equal(displayName(']'), ']')
})
})
+81
View File
@@ -0,0 +1,81 @@
import { test } from '@japa/runner'
import { isUnreadableName, tokenKind } from '#lib/release/index'
test.group('release / isUnreadableName', () => {
test('scrambles with separators, spaces, random extensions, packaging words and hex with an addon are unreadable', ({
assert,
}) => {
for (const n of [
'Zx5AjNaK-mh7CFg6Hf8uu.part01.rar',
'yWjfnT2ujG4wJj3fG46o0FLzsgkvhmrTsAfW2.E0WsOU',
'PsNdnYiugfZ.rar',
'phfplC9y07U2yrj',
'n5d1ueIf34X',
'tJqr1tq7xK',
'Description - Lt5nHjlo3RiuBb3C.par2',
'bd5b4a59f3aa87114dcf75c0-addon.rar',
'mrrtmzrzmi4dgmrzmyytqmlbztczbx',
'AA-NaR16302830434zxP32108309f3Ab',
'_QBs01_MqZYnYwM_iAyRquhc-',
'sgkra vU7u2yuw9LDLCjDu04pOez',
'Jk5Pud.zQdNj',
'7uKsD_NRgT-G_Uc_p8',
'NtpUgCmS_Xa1B2Pk',
'{E11BF8E9-E2C9-4DAE-A1B2-8027BC465F3F}',
'IkYovjrtZrItjQ.mkv',
'ff75ce3f16c746c18b2d78cf.6c52f',
'prvjr7hapd0i1ny14pz4',
]) {
assert.isTrue(isUnreadableName(n), n)
}
})
test('scene tags, CamelCase, codes, leet, acronyms and a hash beside words are readable', ({
assert,
}) => {
for (const n of [
'Dark.Matter.2024.S02E10.No.Quiet.Life.1080p.ATVP.WEB-DL.DDP5.1.Atmos.H.264-Kitsune',
'Paul McCartney - Band on the Run',
'ShesSoWild - Ester Ana 1080p',
'MyDadsHotGirlfriend.13.02.01.Luna.Star.XXX.1080p.x264-SEXORS',
'cFosSpeed 11.0 Build 2420 x86 x64 incl Patch.rar',
'Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing',
'Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON',
'Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr',
'Eclipse_IDE_2023-12_R_win32_x86_64.zip',
'PartsHPGL 1.06c',
'Go4Customer Fax Machine v1.2b',
'AdBeGone 1.2',
'3DMark03 320',
'COD4MW2REMASTERED',
'Movie.2020.1080p-GRP [a3f9e2b1]',
'EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps',
'iPhone',
'Some.Show.S01E01.1080p.mkv',
'FastFilms 2.04',
'BestCrypt NP v4.0',
'EasyHTML 2.35 German',
'SchiffsModell - September 2026',
'CrossStitcher - Issue 440, October 2026',
'Laura15Juli2026',
'IHaveAWife - Isa Bella',
'DickHDDaily - Santana Red',
'1904-SmithsWork-Hasluck',
]) {
assert.isFalse(isUnreadableName(n), n)
}
})
test('token kinds', ({ assert }) => {
assert.equal(tokenKind('Champions'), 'word')
assert.equal(tokenKind('PRESCRiPTiON'), 'word')
assert.equal(tokenKind('GarciaLive'), null)
assert.equal(tokenKind('S02E10'), null)
assert.equal(tokenKind('1080p'), null)
assert.equal(tokenKind('addon'), null)
assert.equal(tokenKind('NtpUgCmS'), 'random')
assert.equal(tokenKind('tBWTuz'), 'random')
assert.equal(tokenKind('Lt5nHjlo3RiuBb3C'), 'random')
assert.equal(tokenKind('a3f9e2b1'), 'random')
})
})
+85
View File
@@ -9,6 +9,18 @@ const file = (name: string, totalParts: number, firstMessageId: string, group =
groups: [group],
firstMessageId,
poster,
segments: totalParts,
bytes: totalParts * 700_000,
})
/** A single archive of `segments` articles and `bytes` in all, posted to `groups` groups. */
const archive = (name: string, poster: string, groups: number, segments: number, bytes: number, firstMessageId = NYUU) => ({
name,
totalParts: segments,
groups: Array.from({ length: groups }, (_, i) => `alt.binaries.g${i}`),
firstMessageId,
poster,
segments,
bytes,
})
const NGPOST = 'b94268bc702040f7b811d5318f727e4f@ngPost'
const NYUU = 'FmFwIaTpGgUiSfYcMwFlMeRo-1790637260478@nyuu'
@@ -50,6 +62,79 @@ test.group('rules / verdicts', () => {
assert.include(rules.judge('k', [file('Hs7dKq2LmZ9xWv3RtYp1Qa.part01.rar', 50, NYUU, 'alt.binaries.misc')]), { rule: 'garbage', key: 'nyuu alt.binaries.misc' })
assert.deepEqual(rules.judge('k', [file('Hs7dKq2LmZ9xWv3RtYp1Qa.part01.rar', 50, NYUU)]), { rule: null })
})
test('stub archives: a settled single archive in 2+ groups with 250 kB articles, or 4+ groups, or a webmail poster', ({ assert }) => {
// Power-Post's 250 kB articles, two groups, 12 MB: the 2016 wave; the article-size leg alone without webmail.
assert.include(rules.judge('k', [archive('Chat Watch 1.2.1.zip', 'jahnunsinos007@gmail.com (Jahnunsinos007)', 2, 48, 12_000_000)]), { rule: 'stub-archive', key: 'AW' })
assert.include(rules.judge('k', [archive('Some.Movie.2014.720p.BluRay-GRP.rar', 'Yenc@power-post.org (Yenc-PP-A&A)', 2, 48, 12_000_000)]), { rule: 'stub-archive', key: 'A' })
// Five groups, 319 kB in two articles: the 2015 a.b.cracks wave (under 1 MB, so the groups leg alone).
assert.include(rules.judge('k', [archive('Big Hero 6 (2014) 720p BDRip [COMPRESSED].rar', 'Yenc@power-post.org', 5, 2, 319_000)]), { rule: 'stub-archive', key: 'X' })
// A webmail poster with 650 kB articles: the 2014 "leaked Sex Video" wave, but only under 100 MB without the article-size leg.
assert.include(rules.judge('k', [archive('Leaked Sex Video.rar', 'first.last88@gmail.com', 2, 100, 65_000_000)]), { rule: 'stub-archive', key: 'W' })
assert.deepEqual(rules.judge('k', [archive('Leaked Sex Video.rar', 'first.last88@gmail.com', 2, 400, 260_000_000)]), { rule: null })
// Not settled: a par2 can still join.
assert.deepEqual(rules.judge('k', [archive('Chat Watch 1.2.1.zip', 'jahnunsinos007@gmail.com', 2, 48, 12_000_000)], false), { rule: null })
})
test('stub archives: small, single-group, split and large hash-named archives are kept', ({ assert }) => {
// An e-book: a one- or two-article release always has small articles (LOG.md (126)).
assert.deepEqual(rules.judge('k', [archive('Maeve Binchy - Tara Road (epub).rar', 'kie@home.now (KiE)', 2, 2, 521_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Chat Watch 1.2.1.zip', 'jahnunsinos007@gmail.com', 1, 48, 12_000_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Some.Movie.2014.720p.BluRay-GRP.rar', 'poster@example.org', 2, 100, 70_000_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Some.Movie.2014.part01.rar', 'a@gmail.com', 2, 48, 12_000_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('cbcd32359460df1fa9b7d01d8720.rar', 'b648de0d1c0f7a9a77be273d@gmail.com', 2, 3163, 2_340_000_000)]), { rule: null })
// ROT13-named volumes of passworded videos in a.b.erotica, 258 kB articles, two groups, 1-270 MB; and the 100 MB cap on every leg (LOG.md (128)).
assert.deepEqual(rules.judge('k', [archive('Ghful.17.12.07.Qnvvfl.Fgbar.cneg3.rar', 'nospam@nospam.nl (EvH)', 2, 5, 1_147_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Ghful.16.07.05.Wbwb.Xvff.cneg1.rar', 'nospam@nospam.nl (EvH)', 2, 1051, 271_000_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Tinasoft EasyCafe 2.2.14 by OmerSalaj.rar', 'Seven7i@mail.com ({Seven7i})', 4, 463, 117_000_000)]), { rule: null })
assert.include(rules.judge('k', [archive('Tipard DVD Ripper - 10 0 12 x86 x64 incl Patch.rar', 'Seven7i@mail.com ({Seven7i})', 4, 244, 62_000_000)]), { rule: 'stub-archive', key: 'AXW' })
// A karaoke zip crossposted to a.b.test with Power-Post's articles: its group exempts it (LOG.md (127)).
assert.deepEqual(rules.judge('k', [{ ...archive('KV-1055806 - Walker Hayes - U Gurl.zip', 'home@home.com (TexasDj)', 2, 13, 3_300_000), groups: ['alt.binaries.sounds.karaoke', 'alt.binaries.test'] }]), { rule: null })
// A rar with its par2 is no single archive.
assert.deepEqual(rules.judge('k', [archive('Chat Watch 1.2.1.zip', 'a@gmail.com', 2, 48, 12_000_000), file('Chat Watch 1.2.1.par2', 1, NYUU)]), { rule: null })
const off = new RuleSet({ ...DEFAULT_RULES, dropStubArchives: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
assert.deepEqual(off.judge('k', [archive('Chat Watch 1.2.1.zip', 'jahnunsinos007@gmail.com', 2, 48, 12_000_000)]), { rule: null })
})
test('nonsense pieces: one article of 100 kB to 1 MB under a release-like name', ({ assert }) => {
assert.include(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: 'nonsense-piece', key: 'nyuu' })
// A 3 kB cue/log sidecar rar with a year in its name: under the floor (the tiny-archive rule's outside the music groups). A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
assert.deepEqual(rules.judge('k', [{ ...archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000), groups: ['alt.binaries.sounds.lossless'] }]), { rule: null })
assert.include(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: 'tiny-archive' })
assert.deepEqual(rules.judge('k', [archive('Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar', 'fake@address.com (Psychopath)', 2, 1, 303_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 382_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar', 'saggir@domain.com', 1, 1, 113_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 2, 790_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('holiday snaps.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('nYDeXeF49dieZ1J7flK-JFzZ3j.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
const off = new RuleSet({ ...DEFAULT_RULES, dropNonsensePieces: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
assert.deepEqual(off.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
})
test('tiny archives: one article of 2-100 kB under a release name, spaces allowed', ({ assert }) => {
// A RAR holding "<title>.exe": the 2015-17 flood in miniature (LOG.md (131)).
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar', 'Yanc@power-post.org <Yanc@power-post.org>', 1, 1, 15_084)]), { rule: 'tiny-archive', key: 'nyuu' })
assert.include(rules.judge('k', [archive('Cicero Magazin fuer politische Kultur Mai 05 2016.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 17_478)]), { rule: 'tiny-archive', key: 'nyuu' })
// A scene name without a year, a tag or a group suffix: not release-like, so kept.
assert.deepEqual(rules.judge('k', [archive('[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar', 'Yanc@power-post.org', 2, 1, 23_449)]), { rule: null })
assert.include(rules.judge('k', [archive('Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 7_869)]), { rule: 'tiny-archive', key: 'nyuu' })
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
assert.deepEqual(rules.judge('k', [archive('The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar', 'henk@example.org', 1, 1, 38_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar', 'poster@example.org', 1, 1, 60_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Some Band - Album (2011) covers.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 38_000)]), { rule: null })
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's; not settled.
assert.deepEqual(rules.judge('k', [{ ...archive('KV-1055806 - Walker Hayes - U Gurl (2017).zip', 'home@home.com', 1, 1, 60_000), groups: ['alt.binaries.sounds.karaoke'] }]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('holiday snaps.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('nYDeXeF49dieZ1J7flK-JFzZ3j.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 2, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 1_500)]), { rule: null })
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 150_000)]), { rule: 'nonsense-piece' })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)], false), { rule: null })
const off = new RuleSet({ ...DEFAULT_RULES, dropTinyArchives: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
assert.deepEqual(off.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
})
})
test.group('rules / upload tools', () => {