15 Commits
Author SHA1 Message Date
Daan MeijerandClaude Fable 5.1 c6a095e740 REBUILD.md: a pending release without a target is exhausted at once
Test and deploy / test (push) Successful in 2m54s
Test and deploy / deploy (push) Successful in 43s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 18:06:40 +01:00
Daan MeijerandClaude Fable 5.1 294f2d73ba Pending ladder: a hash-named release of several pieces gets a lone target
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Successful in 3m1s
The edge gave an unnamed release a "lone" probe target only when it had
exactly one file, so a release of several random-named pieces (Bob's,
1.55M rows in five groups) reached the pending store with no target at
all: never claimed, never named by a twin, its class never learned.
candidates_of now takes the first random-named file when there is no
par2, archive or 7z volume. The batch store writes a finished release
without a target as exhausted instead of waiting, and leaves it out of
its class's waiting count. LOG.md (140).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 17:49:04 +01:00
Daan MeijerandClaude Fable 5.1 66980829dc Twins by content: at most 500 twins read per pair
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m16s
With the hash typed as the column (f537add) the planner walks every twin of a
pair by index; 55 pairs have over 500 twins, up to 131k, and a flush with such
a pair ran the query for 37 s inside its transaction, the other flushes and
node-api's class counters waiting on it. A LATERAL per pair now reads its
lowest 500 release ids and takes the first named one: 49 ms for 305 pairs with
the five hottest among them. LOG.md (138).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 09:49:40 +01:00
Daan MeijerandClaude Fable 5.1 f537add573 Twins by content: type the hash as the column
Test and deploy / test (push) Successful in 3m49s
Test and deploy / deploy (push) Successful in 1m5s
jsonb_to_recordset declared hash16k as text; against the char(32) column the join
cast the column, so Postgres scanned all of file_hashes for every call (4 s, 20
calls a minute: 44% of the database's time). Declared char(32), the join walks
file_hashes_hash_size_release_idx: 24 ms for 300 pairs. LOG.md (138).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 09:31:06 +01:00
Daan MeijerandClaude Fable 5.1 d7171a5ca4 Pending ladder: the sweep and the record lock their rows in key order
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m4s
discardKeys updated with `release_key = ANY(subquery)`, which locks rows in
the plan's scan order, and deadlocked against a batch upsert that takes its
rows by key (LOG.md (137)). It now updates from a subquery that selects the
rows ORDER BY release_key FOR UPDATE, so the locks are taken sorted (the
plan: LockRows above Sort). record() runs the same SELECT ... FOR UPDATE over
its taken rows before its UPDATE ... FROM jsonb_to_recordset, whose join
order the planner chooses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 00:32:09 +01:00
Daan MeijerandClaude Fable 5.1 eadae82b0c The prober's tables on the NVMe; the prober runs through the night
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m30s
Migration 7400 moves pending_releases (with its two HDD indexes),
release_inspections and probe_cache to the default tablespace (nvme on
homelab): the edge's batch upserts into pending_releases took minutes each
on the HDD and the prober's flushes waited on them (LOG.md (137)). Homelab
moved them by hand with the services stopped: 66 GB in 10.5 minutes.

With everything the prober touches on the NVMe, the nightly pause leaves the
prober running (usenet-rebuild-pause.sh), and a deploy while paused
recreates it with node-api and web (deploy.sh).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 23:57:59 +01:00
Daan MeijerandClaude Fable 5.1 e02a7c89b4 REBUILD.md: probe_candidates back on the NVMe (migration 7300)
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:52 +01:00
Daan MeijerandClaude Fable 5.1 6ef7ddca92 Batch store: one lock order for the pending upsert; probe_candidates on the NVMe
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m33s
writePending upserted pending_releases in chunks of 1,000 rows, each chunk
followed by its class upsert, so one transaction locked releases, classes,
releases, classes, and a later chunk's keys and classes could sort before an
earlier chunk's: the opposite of the ladder's record(), which holds all its
rows and then takes all its classes. Four deadlocks between the two class
upserts in six hours (LOG.md (136)). Now all rows are sorted once and the
classes upserted once, after the last chunk.

Migration 7300 moves probe_candidates and its indexes to the default
tablespace (nvme on homelab): the catalog claim's UPDATE from random points
took 2-3 minutes on the HDD beside the sweep and the autovacuum, with all
lanes waiting. Homelab moved it by hand (83 s, ~8 GB, services stopped); the
migration is a no-op there, and without a default tablespace.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:09 +01:00
Daan MeijerandClaude Fable 5.1 a71de56a7c Judge: the tiny rule, the catalog's side of the edge's tiny-archive rule
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 4m23s
One file of one article, 2–100 kB, under a readable release-like name outside the music, e-book and comic
groups, without an e-book, subtitle, crack or sidecar word: the 2015–17 flood's 1.5M rows that the edge
indexed before its rule. The same tests as isTinyArchive in Postgres's regex dialect; no join on the
candidates (98% of a sample have an archive candidate, the rest lone par2 files and scrambled names).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 13:15:03 +01:00
Daan MeijerandClaude Fable 5.1 411c36eec0 Prober: the signature index by the poster's md5
Test and deploy / test (push) Successful in 3m5s
Test and deploy / deploy (push) Successful in 1h2m47s
A poster of 2.7 kB broke the index build (a btree key is at most a third of a page) and left an invalid index
that IF NOT EXISTS would have taken for the real one: the migration drops it first and indexes md5(poster), and
the sweep asks by it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 12:08:53 +01:00
Daan MeijerandClaude Fable 5.1 8e008da364 Prober: the sweep by a signature index, pending rows locked in one order, deadlocked flushes kept
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Failing after 21m32s
The discard sweep of a dropped signature read the whole pending_releases table per signature (a parallel seq
scan over the poster test, four minutes on the HDD that starved the lanes' claims): a partial index on
(group_id, tool, poster) WHERE status = 0 makes it a range, and the sweep queries a poster's rows by it and a
tool's and group's by the ladder index.

The batch store's pending upsert, the record, and the sweep deadlocked on pending_releases and pending_classes
rows locked in different orders, and a deadlocked flush dropped its outcomes: lockOrder() sorts every
multi-row statement's set the same way, and a flush that deadlocks anyway keeps its outcomes for the next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 10:36:19 +01:00
Daan MeijerandClaude Fable 5.1 5d3b578322 The discard sweep beside the lanes, each dropped signature swept once
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 38m14s
After aa40222 the prober still idled: a dropped signature's range of the
ladder index is minutes of HDD (309 dropped, 242 with a poster tested on
the rows) and the sweep read the same ranges again every five minutes
inside the claim. Now each dropped class and signature is swept once
(swept_at, migration 7100; null again when the record drops it anew),
20 of each per sweep in rounds of 10k rows, in a loop of its own in the
prober beside the lanes and the namer (LOG.md (133)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 21:12:51 +01:00
Daan MeijerandClaude Fable 5.1 aa4022295a Tiny-archive rule; the ladder's discard by index; the namer in chunks
Test and deploy / test (push) Successful in 3m1s
Test and deploy / deploy (push) Successful in 32m21s
The tiny-archive rule (dropTinyArchives, both indexers): a settled single
archive of one article and 2-100 kB under a release-like name, spaces
allowed, outside the exempt groups and without an e-book, subtitle,
crack, cover or sidecar word: the 2015-17 exe-stub flood in miniature,
1.66M catalog rows (LOG.md (131), (133)).

PendingLadder.discard scanned every waiting row against the dropped
signatures and classes, 9+ minutes a sweep inside the claim, stalling
every lane: now driven from the dropped classes and signatures into the
indexes. The namer feeds the indexer the span's overview text chunk by
chunk (one string of it topped V8's limit) and gives up spans over 3M
articles. Migration 7000: a partial index on the taken probe candidates
for the prober's start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 20:23:16 +01:00
Daan MeijerandClaude Fable 5.1 4d46db291e pending:classes without table scans; an index for the prober's startup reset
Test and deploy / test (push) Successful in 2m55s
Test and deploy / deploy (push) Successful in 25m6s
The first run scanned the 58M-row pending table on the HDD once per
6-hour window and starved the prober and the batch store. Now one scan
classes the finished rows and seeds the record, and the waiting rows go
by windows of created_at through their index. The prober's start puts a
previous run's claims back with a status scan of the same table, 20+
minutes after every deploy: a partial index on the taken rows (migration
6900) makes it instant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:33:48 +01:00
Daan MeijerandClaude Fable 5.1 3612bf3e43 Readable names: CamelCase product names and words with numbers stay readable
Test and deploy / test (push) Successful in 2m49s
Test and deploy / deploy (push) Successful in 3m24s
The production dry run of releases:readable flagged 5.37M of 19.8M rows,
and its first sample held real names: "CrossStitcher - Issue 440",
"SchiffsModell", "FastFilms 2.04", "BestCrypt NP v4.0", "SmithsWork",
"Laura15Juli2026". Two CamelCase words with few vowels have four runs and
no lone letter: a scramble now needs five runs, a lone letter or an inner
capital run besides its few vowels; the consonant-run clause is gone
(it counted y); a token whose letter runs are all words or acronyms with
numbers between them is neutral. On a 49k-row production sample the
flagged share went from 27.4% to 26.4% with the letters-only flags all
scrambles; the evaluation sets are unchanged (LOG.md (129)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:22:54 +01:00
27 changed files with 841 additions and 170 deletions

No files matched your search

+45 -14
View File
@@ -341,8 +341,13 @@ or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
*locked* (passworded, no password from a spot, its NZB or the title),
*unusable* (random names, no inner name, no par2 or inner-name merge, no
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
*tiny* (one file of one article, 2–100 kB, under a readable release-like
name outside the music, e-book and comic groups, without an e-book,
subtitle, crack or sidecar word: the catalog's side of the edge's
tiny-archive rule, 1.5M rows of the 2015–17 flood that the edge indexed
before the rule; LOG.md (131), (135)), *unusable* (random names, no inner
name, no par2 or inner-name merge, no spot, a day after settling).
Verdicts (`release_verdicts`, with the rule
set's version) hide releases from search (`judged` shows them); a verdict
whose rule no longer matches is taken back. Nothing is dropped yet.
@@ -712,10 +717,13 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
`settled_at IS NOT NULL`), the settler, the fragment merger,
`releases_category_newest_idx`.
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
rule read all of `release_inspections` every round, 17 min on the HDD;
the judge must read it by release or keep what it needs on the NVMe.
- `release_inspections` and `probe_candidates` were on `CHUNK_TABLESPACE`
(the HDD) with their indexes; both are back on the NVMe (migrations 4800
and 7300). Note LOG.md 2026-10-19 (81): the judge's fake rule read all of
`release_inspections` every round, 17 min on the HDD; and 2026-11-03
(136): the prober's catalog claim updates `probe_candidates` from random
points every minute, 2–3 min on the HDD beside the pending ladder's sweep.
What a worker reads or updates by random key every minute stays on the NVMe.
### Batch store (built 2026-10-25)
@@ -806,9 +814,17 @@ unscrambled name without spaces and without an e-book format in it is
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
with spaces are books and albums. Both rules are switched by the
rule set (`dropStubArchives`, `dropNonsensePieces`), sampled like every
drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
with spaces are books and albums. Below that, a settled single archive of
one article and 2–100 kB under a release-like name (spaces allowed), outside
the exempt groups and without an e-book, subtitle, crack/keygen/trainer,
cover or sidecar word, is dropped as a **tiny archive** (`tiny-archive`,
key: the upload tool; LOG.md (131), (133)): the 2015–17 flood's RARs
holding "<title>.exe" at 2–60 kB, 1.66M catalog rows from a dozen posters
(12 of 12 fetched fake); the word exemptions keep the real tiny archives
(subtitles, cracks, trainers) at the price of ~50k fakes named "Incl
Keygen" that the prober judges later. The rules are switched by the
rule set (`dropStubArchives`, `dropNonsensePieces`, `dropTinyArchives`),
sampled like every drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
judges as each batch completes files, so it passes settled without the
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
`check.ts` and the fixture comparison (the parity with the TypeScript
@@ -916,7 +932,10 @@ f. **Nothing enters the catalog without a usable name** (amended
11% of the segments and nothing anyone can find; the full history has
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
edge sends an unnamed release with an empty payload and its tool, the
batch store writes `pending_releases` (LOG.md (114)); the ladder in the
batch store writes `pending_releases` (LOG.md (114)); a finished release
the edge found no target for (the edge's `candidates_of`: par2, archive
start, 7z tail, else the first random-named file as the lone target;
LOG.md (140)) is written exhausted at once; the ladder in the
prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
the oldest waiting releases' next target (par2, archive, lone, 7z-tail
in that order), at least PENDING_SHARE (25%) of each claim next to the
@@ -927,12 +946,24 @@ f. **Nothing enters the catalog without a usable name** (amended
target or status 3 (exhausted); each finished climb counts on
`pending_signatures` for (tool, group, poster) and (tool, group), and a
signature with PENDING_MIN_PROBES (100) climbs and at most
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
waiting releases of dropped signatures are discarded (status 2), the
rules' `samplePercent` of them left to climb. The namer
PENDING_MAX_NAMED (0.5%) named is dropped: a sweep beside the lanes
(every five minutes, in its own loop) discards the waiting releases
of each dropped signature and class once (`swept_at`; the edge drops
their new releases itself), the rules' `samplePercent` of them left to
climb, each by an index (`pending_releases_signature_idx` for a
poster's rows: tested on the rows, a signature was a scan of the whole
table; LOG.md (133), (134)). The batch store, the record, the claim
and the sweep lock pending rows in one order, by key and the counters
after (`lockOrder` for the multi-row statements; the record and the
sweep lock their rows with `SELECT … ORDER BY release_key FOR UPDATE`
first, since an UPDATE's join or `= ANY(subquery)` locks in plan
order), and a flush that deadlocks anyway keeps its outcomes for the
next (LOG.md (134), (137)). The namer
(`app/services/pending_namer.ts`, in the prober's process): a named
release's article span read again from the corpus (the stored range's
chunks) or by XOVER on a connection of its own, through the TypeScript
chunks) or by XOVER on a connection of its own, a chunk of articles
at a time (one string of a wide span topped V8's limit; spans over 3M
articles are given up), through the TypeScript
indexer with `partialFiles` (a file given up short of parts stays a
partial file of its release, as the Rust window writes it; without it
the range path's fragment rule left nothing to find: by 2026-10-29
+36 -21
View File
@@ -9,7 +9,7 @@ import { SIDECAR } from '#services/sidecar_folder'
import { identityKey, releaseKey } from '#lib/release/keys'
import { sourceCode } from '#lib/release/source'
import { uploadTool } from '#lib/rules/index'
import { byEffort } from '#services/pending_ladder'
import { PENDING, byEffort, lockOrder } from '#services/pending_ladder'
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
/**
@@ -190,9 +190,11 @@ class BatchStore {
list.push({ kind: c.kind, messageId: c.messageId, file: c.file, bytes: c.bytes })
targets.set(key, list)
}
let written = 0
for (let i = 0; i < releases.length; i += 1000) {
const rows = releases.slice(i, i + 1000).map((r) => {
// In the record's lock order (LOG.md (136)): all rows sorted, then all classes sorted, in one transaction. The
// upsert locks a finished row too, before its WHERE passes it by; a chunk's classes before the next chunk's rows
// crossed the ladder's record(), which holds its rows and then takes its classes.
const rows = lockOrder(
releases.map((r) => {
const probes = targets.get(slot(r.key, r.upload)) ?? []
return {
release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
@@ -211,46 +213,59 @@ class BatchStore {
file_total: r.fileTotal,
tool: r.tool ?? (probes[0] ? uploadTool(probes[0].messageId) : 'unknown'),
candidates: probes,
// A finished release without a target is through the ladder before it starts: exhausted, not waiting
// (1.57M rows waited forever, never claimed, before the edge gave the hash-named pieces one; LOG.md (140)).
status: r.finished !== false && probes.length === 0 ? PENDING.exhausted : PENDING.waiting,
// The first target's article (by effort): the cheapest probe first within a class (LOG.md (130)).
target_bytes: byEffort(probes)[0]?.bytes ?? null,
}
})
}),
(r) => r.release_key
)
let written = 0
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
for (let i = 0; i < rows.length; i += 1000) {
const { rows: done } = await trx.rawQuery(
`INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes)
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes, status)
SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.name, v.poster, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0),
v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates,
pending_class(v.name, v.bytes, v.partial), v.target_bytes
pending_class(v.name, v.bytes, v.partial), v.target_bytes, v.status
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, identity_key text, name text, poster text, first bigint, last bigint, first_article bigint, last_article bigint,
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb, target_bytes bigint)
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb, target_bytes bigint, status int)
ON CONFLICT (release_key) DO UPDATE SET
files = pending_releases.files + excluded.files, segments = pending_releases.segments + excluded.segments, bytes = pending_releases.bytes + excluded.bytes,
last_posted_at = greatest(pending_releases.last_posted_at, excluded.last_posted_at), last_article = greatest(pending_releases.last_article, excluded.last_article),
partial = excluded.partial, finished = excluded.finished, file_total = coalesce(excluded.file_total, pending_releases.file_total),
candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END,
klass = pending_class(pending_releases.name, pending_releases.bytes + excluded.bytes, excluded.partial),
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes)
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes),
status = CASE WHEN pending_releases.status = ? AND excluded.finished
AND jsonb_array_length(pending_releases.candidates) = 0 AND jsonb_array_length(excluded.candidates) = 0
THEN ? ELSE pending_releases.status END
WHERE NOT pending_releases.finished
RETURNING release_key, tool, klass, (xmax = 0) AS inserted`,
[groupId, JSON.stringify(rows)]
RETURNING release_key, tool, klass, (xmax = 0) AS inserted, status`,
[groupId, JSON.stringify(rows.slice(i, i + 1000)), PENDING.waiting, PENDING.exhausted]
)
written += done.length
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
for (const d of done) {
const key = `${d.tool}\t${d.klass}`
const entry = classes.get(key) ?? { tool: d.tool, klass: Number(d.klass), waiting: 0 }
if (d.inserted) entry.waiting++
if (d.inserted && Number(d.status) === PENDING.waiting) entry.waiting++
classes.set(key, entry)
}
if (classes.size > 0) {
await trx.rawQuery(
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
[groupId, JSON.stringify([...classes.values()])]
)
}
}
if (classes.size > 0) {
await trx.rawQuery(
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
[
groupId,
JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${groupId}\t${c.klass}`)),
]
)
}
return written
}
+16 -7
View File
@@ -5,20 +5,29 @@ import db from '@adonisjs/lucid/services/db'
* repeated verdict match) in a catalog release that has a name, readable or found inside. The same film posted
* under scrambles by other tools gets that name without another fetch. Releases with a junk verdict don't count.
*/
/** Twins read per pair: the hottest pairs have 131k. */
const TWIN_SCAN = 500
export async function twinNames(
pairs: { hash16k: string; size: number }[],
trx: { rawQuery: typeof db.rawQuery } = db
): Promise<Map<string, string>> {
const out = new Map<string, string>()
if (pairs.length === 0) return out
// Per pair the first named twin among its lowest TWIN_SCAN release ids. hash16k is typed as the column
// (char(32)): as text the join cast the column and scanned all of file_hashes, 4 s a call; and with the index
// the hottest pairs (55 with over 500 twins, up to 131k, all readable) made a call walk every twin, 37 s inside
// the flush's transaction (LOG.md (138)). One name is all a pair needs.
const { rows } = await trx.rawQuery(
`SELECT DISTINCT ON (h.hash16k, h.size) h.hash16k, h.size, coalesce(r.inner_name, r.name) AS name
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k text, size bigint)
JOIN file_hashes h ON h.hash16k = v.hash16k AND h.size = v.size
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
ORDER BY h.hash16k, h.size, r.readable DESC, r.id`,
[JSON.stringify(pairs)]
`SELECT v.hash16k, v.size, t.name
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k char(32), size bigint)
CROSS JOIN LATERAL (
SELECT coalesce(r.inner_name, r.name) AS name
FROM (SELECT release_id FROM file_hashes h WHERE h.hash16k = v.hash16k AND h.size = v.size ORDER BY release_id LIMIT ?) h
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
ORDER BY r.readable DESC, r.id LIMIT 1) t`,
[JSON.stringify(pairs), TWIN_SCAN]
)
for (const r of rows) out.set(`${r.hash16k}\t${r.size}`, r.name)
return out
+111 -20
View File
@@ -72,6 +72,19 @@ interface ClassRow {
const CLASSES_MS = 60_000
/** A class found short of a claim's share is left alone this long. */
const EMPTY_MS = 5 * 60_000
/**
* Rows in one order for a statement that locks many of them (pending_releases by key, pending_classes and
* pending_signatures by theirs): the batch store, the record, the claim and the sweep all lock the same way, so two
* of them can't deadlock on each other (LOG.md (134)).
*/
export function lockOrder<T>(rows: Iterable<T>, key: (row: T) => string): T[] {
return [...rows].sort((a, b) => {
const ka = key(a)
const kb = key(b)
return ka < kb ? -1 : ka > kb ? 1 : 0
})
}
/** The prior rate of an unexplored class and its weight in probes: the ladder's overall rate on 2026-11-02 (LOG.md (130)). */
const PRIOR_RATE = 0.15
const PRIOR_WEIGHT = 10
@@ -276,13 +289,26 @@ class PendingLadder {
final: null,
}
})
// The rows locked in key order first: the UPDATE's join below takes them in whatever order the plan does.
await trx.rawQuery(
`SELECT 1 FROM pending_releases WHERE release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
ORDER BY release_key FOR UPDATE`,
[rows.map((r) => r.release_key).sort()]
)
await trx.rawQuery(
`UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
WHERE p.release_key = decode(v.release_key, 'hex')`,
[JSON.stringify(rows.map(({ final: _final, ...row }) => row))]
[
JSON.stringify(
lockOrder(
rows.map(({ final: _final, ...row }) => row),
(r) => r.release_key
)
),
]
)
// The signatures: the poster's own and the tool's and group's together.
const counts = new Map<
@@ -312,17 +338,20 @@ class PendingLadder {
`INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
[JSON.stringify([...counts.values()])]
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`))]
)
await trx.rawQuery(
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float)
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float),
swept_at = CASE WHEN s.dropped THEN s.swept_at END
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
[
pendingMinProbes,
pendingMaxNamed,
JSON.stringify(
[...counts.values()].map(({ tool, group_id, poster }) => ({ tool, group_id, poster }))
lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`).map(
({ tool, group_id, poster }) => ({ tool, group_id, poster })
)
),
]
)
@@ -359,41 +388,103 @@ class PendingLadder {
SELECT v.tool, v.group_id, v.klass, v.probed, v.named, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, probed int, named int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named,
waiting = pending_classes.waiting + excluded.waiting, updated_at = now()`,
[JSON.stringify([...classes.values()])]
[JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
)
await trx.rawQuery(
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float)
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float),
swept_at = CASE WHEN c.dropped THEN c.swept_at END
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int)
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
[
pendingMinProbes,
pendingMaxNamed,
JSON.stringify(
[...classes.values()].map(({ tool, group_id, klass }) => ({ tool, group_id, klass }))
lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`).map(
({ tool, group_id, klass }) => ({ tool, group_id, klass })
)
),
]
)
}
/** Waiting releases of a dropped signature discarded without a probe, the explored sample excepted; how many. */
async discard(limit = 10_000): Promise<number> {
/**
* Waiting releases of a dropped class or signature discarded without a probe, the explored sample excepted; how
* many. Each dropped class or signature is swept once (swept_at; null again when the record drops it anew),
* `perSweep` of them at most, each by its index (the claim's for a class, the signature index for a poster's
* rows, the ladder's for a tool's and group's) in rounds of `limit`. The prober runs this beside the lanes, not
* in the claim (LOG.md (133)); the edge drops new releases of dropped signatures and classes itself, so one
* sweep is all a drop needs.
*/
async discard(limit = 10_000, perSweep = 20): Promise<number> {
const { samplePercent } = await RuleSets.current()
const { rows } = await db.rawQuery(
`SELECT encode(p.release_key, 'hex') AS release_key FROM pending_releases p
WHERE p.status = ? AND (EXISTS (
SELECT 1 FROM pending_signatures s
WHERE s.dropped AND s.tool = p.tool AND s.group_id = p.group_id AND (s.poster = '' OR s.poster = coalesce(p.poster, '')))
OR EXISTS (SELECT 1 FROM pending_classes c WHERE c.dropped AND c.tool = p.tool AND c.group_id = p.group_id AND c.klass = p.klass))
LIMIT ?`,
[PENDING.waiting, limit]
let discarded = 0
const { rows: classes } = await db.rawQuery(
`SELECT tool, group_id, klass FROM pending_classes WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, klass LIMIT ?`,
[perSweep]
)
for (const c of classes) {
let rows: any[]
do {
;({ rows } = await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND tool = ? AND group_id = ? AND klass = ? LIMIT ?`,
[PENDING.waiting, c.tool, c.group_id, c.klass, limit]
))
discarded += await this.discardKeys(rows, samplePercent)
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
await db.rawQuery(
`UPDATE pending_classes SET swept_at = now() WHERE tool = ? AND group_id = ? AND klass = ?`,
[c.tool, c.group_id, c.klass]
)
}
const { rows: signatures } = await db.rawQuery(
`SELECT tool, group_id, poster FROM pending_signatures WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, poster LIMIT ?`,
[perSweep]
)
for (const s of signatures) {
let rows: any[]
do {
// The poster's rows by the signature index (on the poster's md5: a poster can be longer than a btree
// key), a tool's and group's by the ladder's: never a test on the rows, which was a scan of the whole
// table per signature on the HDD (LOG.md (134)).
;({ rows } =
s.poster === ''
? await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND group_id = ? AND tool = ? LIMIT ?`,
[PENDING.waiting, s.group_id, s.tool, limit]
)
: await db.rawQuery(
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
WHERE status = ? AND group_id = ? AND tool = ? AND md5(poster) = md5(?::text) AND poster = ? LIMIT ?`,
[PENDING.waiting, s.group_id, s.tool, s.poster, s.poster, limit]
))
discarded += await this.discardKeys(rows, samplePercent)
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
await db.rawQuery(
`UPDATE pending_signatures SET swept_at = now() WHERE tool = ? AND group_id = ? AND poster = ?`,
[s.tool, s.group_id, s.poster]
)
}
return discarded
}
/** The rows discarded (status 2), the explored sample excepted, their classes' waiting counts lowered; how many. */
private async discardKeys(rows: any[], samplePercent: number): Promise<number> {
const keys = rows
.map((r: any) => r.release_key as string)
.filter((key: string) => !explored(key, samplePercent))
.sort()
if (keys.length === 0) return 0
// Locked in key order (the record's lock order, LOG.md (137)): a plain `key = ANY(subquery)` locks in scan
// order and deadlocked against a batch upsert; ORDER BY sorts before FOR UPDATE locks.
const { rows: gone } = await db.rawQuery(
`UPDATE pending_releases SET status = ? WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
RETURNING tool, group_id, klass`,
`UPDATE pending_releases p SET status = ?
FROM (SELECT release_key FROM pending_releases
WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
ORDER BY release_key FOR UPDATE) k
WHERE p.release_key = k.release_key
RETURNING p.tool, p.group_id, p.klass`,
[PENDING.discarded, PENDING.waiting, keys]
)
const counts = new Map<string, { tool: string; group_id: number; klass: number; n: number }>()
@@ -413,7 +504,7 @@ class PendingLadder {
`UPDATE pending_classes c SET waiting = greatest(0, c.waiting - v.n)
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, n int)
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
[JSON.stringify([...counts.values()])]
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
)
}
return gone.length
+74 -50
View File
@@ -30,6 +30,13 @@ interface NamedRow {
/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
const MAX_MOVES = 3
/**
* The widest article span the namer reads again (a busy group posts 50M articles a day, a large upload among
* them spans a million or two): one wider is a name the edge saw again much later ("v-lgpc.3ds"), and its span
* of segments would not fit the indexer's memory (LOG.md (133)).
*/
const MAX_SPAN = 3_000_000
/**
* The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
* read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
@@ -94,6 +101,10 @@ export class PendingNamer {
/** One named release into the catalog. */
async move(row: NamedRow): Promise<void> {
if (row.last_article - row.first_article > MAX_SPAN)
throw new Error(
`its articles ${row.first_article}–${row.last_article} span more than ${MAX_SPAN}`
)
const group = await db.from('groups').where('id', row.group_id).firstOrFail()
const range = await db
.from('ranges')
@@ -103,19 +114,20 @@ export class PendingNamer {
.orderBy('id')
.first()
if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
const text = await this.overviewText(group.name, row)
const rules = await RuleSets.current()
const ix = new RangeIndexer({
group: group.name,
start: row.first_article,
end: row.last_article,
rules: { ...rules, samplePercent: 0 },
idleChunks: 0,
// No file given up before the end: the span is read chunk by chunk (one text of it topped V8's string).
idleChunks: Number.MAX_SAFE_INTEGER,
edgeArticles: 0,
// The edge wrote it partial when its file is short of parts on the server: so must the re-read.
partialFiles: true,
})
ix.addText(text, row.last_article + 1)
for await (const piece of this.overviewPieces(group.name, row))
ix.addText(piece.text, piece.next)
ix.finish()
const batch = ix.take()
const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
@@ -160,50 +172,20 @@ export class PendingNamer {
})
}
/** The overview lines of the release's article span: the stored corpus chunks when its range is stored, else XOVER. */
private async overviewText(groupName: string, row: NamedRow): Promise<string> {
if (this.corpusDir) {
const stored = await db
.from('ranges')
.where('group_id', row.group_id)
.whereNotNull('stored_at')
.where('start_id', '<=', row.last_article)
.where('end_id', '>=', row.first_article)
.orderBy('start_id')
const parts: string[] = []
let covered = true
for (const r of stored) {
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
const size = coordinatorConfig.chunkArticles
for (
let from =
range.start +
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
from <= Math.min(row.last_article, range.end);
from += size
) {
const raw = await Corpus.read(
range,
{ from, to: Math.min(from + size - 1, range.end) },
this.corpusDir
)
if (!raw) {
covered = false
break
}
parts.push((await inflate(raw)).toString('latin1'))
}
if (!covered) break
}
if (
covered &&
parts.length > 0 &&
stored.length > 0 &&
Number(stored[0].start_id) <= row.first_article &&
Number(stored[stored.length - 1].end_id) >= row.last_article
) {
return parts.join('')
}
/**
* The overview lines of the release's article span, a chunk of articles at a time with the first article after
* it: the stored corpus chunks when its ranges are stored and contiguous, else XOVER in windows of a chunk.
*/
private async *overviewPieces(
groupName: string,
row: NamedRow
): AsyncGenerator<{ text: string; next: number }> {
const size = coordinatorConfig.chunkArticles
const stored = await this.storedChunks(groupName, row)
if (stored) {
for (const { raw, next } of stored)
yield { text: (await inflate(raw)).toString('latin1'), next }
return
}
this.client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
@@ -217,13 +199,55 @@ export class PendingNamer {
this.usedAt = Date.now()
try {
await this.client.group(groupName)
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
const raw = await this.client.xoverRaw(row.first_article, row.last_article)
return (await inflate(raw)).toString('latin1')
for (let from = row.first_article; from <= row.last_article; from += size) {
const to = Math.min(from + size - 1, row.last_article)
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
const raw = await this.client.xoverRaw(from, to)
yield { text: (await inflate(raw)).toString('latin1'), next: to + 1 }
}
} catch (error) {
this.client.close()
this.client = null
throw error
}
}
/** The span's stored corpus chunks (compressed, with the first article after each), or null when any is missing. */
private async storedChunks(
groupName: string,
row: NamedRow
): Promise<{ raw: Buffer; next: number }[] | null> {
if (!this.corpusDir) return null
const stored = await db
.from('ranges')
.where('group_id', row.group_id)
.whereNotNull('stored_at')
.where('start_id', '<=', row.last_article)
.where('end_id', '>=', row.first_article)
.orderBy('start_id')
if (
stored.length === 0 ||
Number(stored[0].start_id) > row.first_article ||
Number(stored[stored.length - 1].end_id) < row.last_article
)
return null
const size = coordinatorConfig.chunkArticles
const chunks: { raw: Buffer; next: number }[] = []
for (const r of stored) {
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
for (
let from =
range.start +
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
from <= Math.min(row.last_article, range.end);
from += size
) {
const to = Math.min(from + size - 1, range.end)
const raw = await Corpus.read(range, { from, to }, this.corpusDir)
if (!raw) return null
chunks.push({ raw, next: to + 1 })
}
}
return chunks
}
}
+37 -7
View File
@@ -41,6 +41,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
const FLUSH_MS = 2000
/** Between sweeps that discard the waiting releases of dropped signatures. */
const SWEEP_MS = 5 * 60_000
/** A flush that deadlocks (against the batch store's pending upsert) keeps its outcomes for the next, this often. */
const DEADLOCK_RETRIES = 5
/**
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
@@ -82,6 +84,8 @@ export class Prober {
private refilling: Promise<void> | null = null
private stopped = false
private stats = Prober.freshStats()
/** Flushes deadlocked in a row, their outcomes kept. */
private deadlocks = 0
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
private lanes = new Set<Promise<void>>()
private running = 0
@@ -97,7 +101,6 @@ export class Prober {
}
/** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
private namer = new PendingNamer()
private sweptAt = 0
async run() {
// A previous run's claims: one prober at a time.
@@ -106,6 +109,17 @@ export class Prober {
const released = await PendingLadder.release()
if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
// The ladder's discard sweep beside the lanes: a dropped signature's rows take minutes on the HDD (LOG.md (133)).
const sweeper = (async () => {
while (!this.stopped) {
try {
this.stats.discarded += await PendingLadder.discard()
} catch (error: any) {
logger.error(`Probing: the discard sweep failed: ${error.message}`)
}
for (let waited = 0; waited < SWEEP_MS && !this.stopped; waited += 1000) await sleep(1000)
}
})()
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
const flusher = (async () => {
while (!this.stopped || this.outcomes.length > 0) {
@@ -136,6 +150,7 @@ export class Prober {
}
await Promise.all(this.lanes)
await namer
await sweeper
await this.share.leave().catch(() => {})
// What was claimed and not probed goes back.
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
@@ -266,10 +281,6 @@ export class Prober {
/** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
private async claimPending() {
if (Date.now() - this.sweptAt > SWEEP_MS) {
this.sweptAt = Date.now()
this.stats.discarded += await PendingLadder.discard()
}
const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
const probes = await PendingLadder.claim(want)
if (probes.length === 0) return
@@ -339,15 +350,34 @@ export class Prober {
}
}
/** The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. */
/**
* The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. A deadlock
* (the batch store's pending upsert locks the same rows; LOG.md (134)) keeps the outcomes for the next flush
* instead of losing the probes.
*/
private async flush() {
const taken = this.outcomes.splice(0)
if (taken.length === 0) return
try {
await this.store(taken)
} catch (error: any) {
if (error?.code === '40P01' && this.deadlocks < DEADLOCK_RETRIES) {
this.deadlocks++
this.outcomes.unshift(...taken)
logger.warn(`Probing: storing results deadlocked, kept for the next flush (${this.deadlocks}).`)
return
}
throw error
}
this.deadlocks = 0
for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
this.stats.pending += taken.filter((o) => o.candidate.pending && o.status !== 'waiting').length
}
private async store(taken: Outcome[]) {
// The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
const pending = taken.filter((o) => o.candidate.pending)
const outcomes = taken.filter((o) => !o.candidate.pending)
this.stats.pending += pending.filter((o) => o.status !== 'waiting').length
await db.transaction(async (trx) => {
if (outcomes.length > 0) {
await trx.rawQuery(
+23
View File
@@ -1,6 +1,7 @@
import db from '@adonisjs/lucid/services/db'
import logger from '@adonisjs/core/services/logger'
import { SOURCES } from '#lib/release/source'
import { TINY_MIN_BYTES, NONSENSE_MIN_BYTES } from '#lib/rules/index'
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
export const JUDGE_VERSION = 1
@@ -28,6 +29,14 @@ const MEDIA =
const SCRAPED_POSTERS = `celeb[-_ ]{0,1}movie|celebwow|celebfx|iluvcelebs|nude[-_ ]{0,1}celebs|celeb-sex-tapes`
/** An executable's name. */
const EXECUTABLE = `\\.(exe|scr|com|bat|cmd|msi|lnk|vbs|js|jar|pif|cpl|hta|ps1|dll)$`
/**
* The tiny rule's name tests, lib/rules/rule_set.ts's RELEASE_LIKE_SPACED, EBOOK, TINY_CONTENT and
* STUB_EXEMPT_GROUPS in Postgres's dialect (and no "?", see MEDIA).
*/
const TINY_RELEASE_LIKE = `(\\y(19|20)[0-9][0-9]\\y|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-{0,1}dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\\.[0-9]+)+)\\y|-[a-z0-9]{2,}$|\\.[a-z0-9]+\\.[a-z0-9]+\\.)`
const TINY_EBOOK = `\\y(epub|mobi|azw3{0,1}|pdf|djvu|cbr|cbz|e-{0,1}books{0,1})\\y`
const TINY_CONTENT = `\\y(subs{0,1}|subtitles{0,1}|subpack|srt|untertitel|ondertitels{0,1}|ondertiteling|legendas{0,1}|napisy|sottotitoli|cracks{0,1}|cracked|keygen|keymaker|trainer|patch|patched|serials{0,1}|activator|loader|nocd|no-cd|fix|covers{0,1}|artwork|booklet|scans{0,1}|proof|sample|info|readme)\\y`
const TINY_EXEMPT_GROUPS = `(sound|mp3|karaoke|music|audio|flac|lossless|e-{0,1}book|epub|comic|guitar|font|magazine)`
export interface Rule {
name: string
@@ -100,6 +109,20 @@ export const RULES: Rule[] = [
// releases_passworded_idx: a scan of all of releases was 30 s a round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
},
{
name: 'tiny',
says: 'one file of one article, 2–100 kB, under a release-like name: the exe-stub flood in miniature',
// The catalog's side of the edge's tiny-archive rule (lib/rules/rule_set.ts isTinyArchive; LOG.md (131),
// (135)): the same name tests in Postgres's regex dialect ("\y" for "\b", "{0,1}" for "?"), the readable flag
// for its scrambled and random tests, the group set for its groups. The 2015–17 flood that the edge indexed
// before the rule: 1.5M rows, 98% of them with an archive candidate, the rest lone par2 files and scrambled
// names, so no join on the candidates. A scan of releases (9 GB) a full round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
WHERE r.files = 1 AND r.segments = 1 AND r.bytes >= ${TINY_MIN_BYTES} AND r.bytes < ${NONSENSE_MIN_BYTES}
AND r.readable AND NOT coalesce(r.sidecar_only, false)
AND r.name ~* '${TINY_RELEASE_LIKE}' AND r.name !~* '${TINY_EBOOK}' AND r.name !~* '${TINY_CONTENT}'
AND NOT EXISTS (SELECT 1 FROM group_sets gs WHERE gs.id = r.group_set_id AND gs.groups::text ~* '${TINY_EXEMPT_GROUPS}')`,
},
{
name: 'unusable',
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
+38 -30
View File
@@ -6,9 +6,10 @@ import { describeClass } from '#lib/release/index'
/**
* The pending ladder's classes (LOG.md (130)) on the rows from before them: `klass` (the SQL pending_class) and
* `target_bytes` (the first target's bytes, when its candidates carry them) set in windows of created_at; the
* finished rows among them (named, exhausted, naming) seed each class's record, which gets its verdict (dropped) as
* the ladder gives it; then pending_classes' `waiting` counted afresh from the rows. --report shows the record.
* `target_bytes` (the first target's bytes, when its candidates carry them). The finished rows (named, exhausted,
* naming) in one scan, seeding each class's record, which gets its verdict (dropped) as the ladder gives it; the
* waiting rows in windows of created_at by their index (a scan per window starved the prober on the HDD); then
* pending_classes' `waiting` counted afresh from the claim's index. --report shows the record.
*/
export default class PendingClasses extends BaseCommand {
static commandName = 'pending:classes'
@@ -25,45 +26,52 @@ export default class PendingClasses extends BaseCommand {
async run() {
if (this.report) return this.showReport()
const hours = this.hours ?? 6
const { rows: span } = await db.rawQuery(
`SELECT min(created_at) AS lo, max(created_at) AS hi FROM pending_releases WHERE klass = 0`
const started = Date.now()
// The finished rows (named, exhausted, naming; 6.5M on 2026-11-02) in one pass: a scan of the table, no index
// reaches them by status; they seed each class's record.
const { rows: seeded } = await db.rawQuery(
`WITH classed AS (
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
WHERE klass = 0 AND status IN (1, 3, 4)
RETURNING tool, group_id, klass, status),
upserted AS (
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed GROUP BY 1, 2, 3
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
RETURNING 1)
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM upserted) AS classes`
)
if (!span[0].lo) {
this.logger.info('No rows without a class.')
} else {
this.logger.info(
`Finished rows: ${seeded[0].classed} classed into ${seeded[0].classes} classes (${Math.round((Date.now() - started) / 1000)} s).`
)
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
await db.rawQuery(
`UPDATE pending_classes SET dropped = true, swept_at = NULL WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
[pendingMinProbes, pendingMaxNamed]
)
// The waiting rows by windows of created_at (pending_releases_waiting_idx reaches them).
const { rows: span } = await db.rawQuery(
`SELECT (SELECT min(created_at) FROM pending_releases WHERE status = 0) AS lo, (SELECT max(created_at) FROM pending_releases WHERE status = 0) AS hi`
)
if (span[0].lo) {
let from = new Date(span[0].lo)
const hi = new Date(span[0].hi)
let classed = 0
const started = Date.now()
while (from <= hi) {
const to = new Date(from.getTime() + hours * 3600_000)
// Rows of the window classed; the finished ones among them (named, exhausted, naming) seed the record.
const { rows } = await db.rawQuery(
`WITH classed AS (
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
WHERE klass = 0 AND created_at >= ? AND created_at < ?
RETURNING tool, group_id, klass, status),
seeded AS (
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed WHERE status IN (1, 3, 4) GROUP BY 1, 2, 3
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
RETURNING 1)
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM seeded) AS seeded`,
const result = await db.rawQuery(
`UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
WHERE status = 0 AND created_at >= ? AND created_at < ? AND klass = 0`,
[from.toISOString(), to.toISOString()]
)
classed += Number(rows[0].classed)
classed += result.rowCount ?? 0
from = to
this.logger.info(
`Up to ${to.toISOString().slice(0, 16)}: ${classed} rows classed (${Math.round((Date.now() - started) / 1000)} s).`
`Waiting rows up to ${to.toISOString().slice(0, 16)}: ${classed} classed (${Math.round((Date.now() - started) / 1000)} s).`
)
}
// The record's verdicts on the seeded classes.
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
await db.rawQuery(
`UPDATE pending_classes SET dropped = true WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
[pendingMinProbes, pendingMaxNamed]
)
}
// The waiting counts from the rows themselves (the claim, the discard and the batch store keep them from here).
await db.rawQuery(`
@@ -0,0 +1,20 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The prober's start puts a previous run's claims back (PendingLadder.release: status 9 → 0), which scanned the
* 58M-row pending table on the HDD for 20+ minutes after every deploy (LOG.md (130)): a partial index on the few
* taken rows makes it instant.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_taken_idx ON pending_releases (release_key) WHERE status = 9`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_taken_idx`)
}
}
@@ -0,0 +1,20 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The prober's start puts a previous run's claims back (probe_candidates status taken → waiting), which scanned
* the 7.8 GB candidates table on the HDD for 4 minutes after every deploy (LOG.md (133)): a partial index on the
* few taken rows makes it instant.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS probe_candidates_taken_idx ON probe_candidates (id) WHERE status = 'taken'`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS probe_candidates_taken_idx`)
}
}
@@ -0,0 +1,26 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The ladder's discard sweep (PendingLadder.discard) reads a dropped signature's or class's waiting rows once:
* `swept_at` marks it done (null again when the record drops it anew). Before, every sweep read every dropped
* signature's rows over again, minutes of HDD each, inside the claim (LOG.md (133)).
*/
export default class extends BaseSchema {
async up() {
this.schema.alterTable('pending_signatures', (table) => {
table.timestamp('swept_at', { useTz: true }).nullable()
})
this.schema.alterTable('pending_classes', (table) => {
table.timestamp('swept_at', { useTz: true }).nullable()
})
}
async down() {
this.schema.alterTable('pending_signatures', (table) => {
table.dropColumn('swept_at')
})
this.schema.alterTable('pending_classes', (table) => {
table.dropColumn('swept_at')
})
}
}
@@ -0,0 +1,23 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The discard sweep of a dropped signature (tool, group, poster) read the whole pending_releases table per
* signature: the planner took a parallel seq scan over the poster test, four minutes on the HDD that starved the
* lanes' claims (LOG.md (134)). The waiting rows by signature make it an index range. By the poster's md5: a
* poster of 2.7 kB broke the first build (a btree key is at most a third of a page), which left an invalid index
* behind that IF NOT EXISTS would have taken for the real one; the drop clears it.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
this.schema.raw(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_signature_idx ON pending_releases (group_id, tool, md5(poster)) WHERE status = 0`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
}
}
@@ -0,0 +1,34 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* probe_candidates, with its indexes, in the catalog's tablespace (default_tablespace: nvme on homelab), out of
* CHUNK_TABLESPACE where migration 6000 put it: the prober's catalog claim updates it from random points every
* minute, and on the HDD, beside the pending ladder's sweep and autovacuum, a claim took 2–3 minutes with all the
* lanes waiting (LOG.md 2026-11-03 (136)). Homelab moved it by hand (edge, node-api and the workers stopped; ~8 GB);
* a no-op where it's there already, or without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(
`SELECT current_setting('default_tablespace') AS target`
)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE (c.relname = 'probe_candidates' OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = 'probe_candidates'::regclass))
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
ORDER BY c.relkind DESC, c.relname`,
[target]
)
for (const { relname, relkind } of rows) {
await db.rawQuery(
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
)
}
})
}
async down() {}
}
@@ -0,0 +1,37 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Everything the prober reads or writes in the catalog's tablespace (default_tablespace: nvme on homelab), out of
* CHUNK_TABLESPACE (the HDD): pending_releases with its two HDD indexes (migration 6300), release_inspections
* (6000), probe_cache with its toast. The edge's batch upserts into pending_releases took minutes each on the HDD and
* the prober's flushes waited on them (LOG.md 2026-11-03 (137)); and with these on the NVMe the prober can run
* through the night without the HDD. Homelab moved them by hand (edge, node-api and the workers stopped; ~66 GB);
* a no-op where they're there already, or without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(
`SELECT current_setting('default_tablespace') AS target`
)
const target = setting[0].target
if (!target) return
for (const table of ['pending_releases', 'release_inspections', 'probe_cache']) {
const { rows } = await db.rawQuery(
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE (c.relname = ? OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = ?::regclass))
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
ORDER BY c.relkind DESC, c.relname`,
[table, table, target]
)
for (const { relname, relkind } of rows) {
await db.rawQuery(
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
)
}
}
})
}
async down() {}
}
+3 -2
View File
@@ -47,8 +47,9 @@ echo "== recreating app services"
docker compose stop edge
if [ -e "$CACHE/paused" ]; then
# Paused for the night (~/bin/usenet-rebuild-pause.sh): the workers stay stopped, the resume starts the new build.
echo "== paused: coordinator, prober, merger, judge, spotter, spot-inspector and edge left stopped"
docker compose up -d --no-deps node-api web
# The prober runs through the night (2026-11-03, LOG.md (137)), so it gets the new build now.
echo "== paused: coordinator, merger, judge, spotter, spot-inspector and edge left stopped"
docker compose up -d --no-deps node-api web prober
else
docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
if [ -e "$CACHE/no-edge" ]; then
+4 -3
View File
@@ -1,12 +1,13 @@
#!/bin/sh
# Installed as ~/bin/usenet-rebuild-pause.sh (by hand). Nightly (homelab crontab, 23:45): voyager 2's processing paused for the night; usenet-rebuild-resume.sh (07:45)
# starts it again. Postgres, web and node-api stay up. Takes the deploy lock, so it never runs during a CI deploy, and
# leaves .cache/paused, so a deploy in the night doesn't start the workers again.
# starts it again. Postgres, web, node-api and the prober stay up: since 2026-11-03 (LOG.md (137)) everything the
# prober touches is on the NVMe, so it probes through the night without the HDD. Takes the deploy lock, so it never
# runs during a CI deploy, and leaves .cache/paused, so a deploy in the night doesn't start the workers again.
cd /mnt/bulk/usenet-rebuild || exit 1
exec 9>.cache/deploy.lock
flock 9
touch .cache/paused
date "+%F %T pausing"
docker compose stop edge
docker compose stop coordinator prober merger judge spotter spot-inspector
docker compose stop coordinator merger judge spotter spot-inspector
docker compose ps --format "{{.Service}} {{.Status}}"
+31 -2
View File
@@ -1608,8 +1608,13 @@ fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate>
if let Some((_, f)) = tail {
out.push(Candidate { release_key: key.to_string(), upload, kind: "7z-tail", message_id: f.last().unwrap().message_id.clone(), file: f[0].file.name.clone(), bytes: f.last().unwrap().bytes });
}
if out.is_empty() && files.len() == 1 && is_random_file(&files[0][0].file.name) {
out.push(c("lone", &files[0][0]));
// No set and no archive: the first random-named file's first segment, for its yEnc name or a twin by content
// (hash_twins.ts). Any count of files: a hash-named release of several pieces (Bob's, LOG.md (140)) had no
// target at all, so the ladder never probed it, never named it and never learned its class.
if out.is_empty() {
if let Some(f) = files.iter().find(|f| is_random_file(&f[0].file.name)) {
out.push(c("lone", &f[0]));
}
}
out.truncate(4);
out
@@ -1738,6 +1743,30 @@ mod tests {
assert_eq!(ix.pending(), 0);
}
#[test]
fn a_multi_file_random_release_gets_a_lone_target() {
let mut priorities = FxHashMap::default();
priorities.insert(GROUP.to_string(), 1);
let mut ix = RangeIndexer::new(GROUP, 1, 10_000, 1, Rules::defaults(priorities), 5);
// Three hash-named pieces of one hash-named upload, five segments each (more than a per-article post): no
// archive, no par2, no 7z volume.
let mut lines = Vec::new();
for (i, name) in ["9e7fc2741cdd450287aff2733ca4c193", "c5ebdd5487f5b132c76e8d5c2572e560", "8114c42c9694380c8b25857e8b3331c3"].iter().enumerate() {
for part in 0..5 {
lines.push(line(100 + (i * 5 + part) as i64, &format!(r#"b3a75d234aa73891e78a70fc1f5496b1 [{}/3] - "{name}" yEnc ({}/5)"#, i + 1, part + 1), (i * 5 + part) as i64));
}
}
ix.add_text(&lines.join("\n"), 10_001);
ix.finish();
let batch = ix.take();
assert_eq!(batch.releases.len(), 1, "counts {:?}", batch.counts);
let r = &batch.releases[0];
assert!(!r.readable && r.files == 3 && r.segments == 15, "readable {} files {} segments {}", r.readable, r.files, r.segments);
let kinds: Vec<_> = batch.candidates.iter().map(|c| (c.kind, c.file.as_str())).collect();
assert_eq!(kinds, vec![("lone", "9e7fc2741cdd450287aff2733ca4c193")]);
assert_eq!(batch.candidates[0].message_id, "u100@test");
}
#[test]
fn album_templates_and_names() {
assert_eq!(album_template("1481439079025_014.jpg.data").as_deref(), Some("#_#.jpg.data"));
+5
View File
@@ -77,6 +77,8 @@ pub struct RulesJson {
pub drop_stub_archives: Option<bool>,
#[serde(default)]
pub drop_nonsense_pieces: Option<bool>,
#[serde(default)]
pub drop_tiny_archives: Option<bool>,
}
impl RulesJson {
@@ -107,6 +109,9 @@ impl RulesJson {
if let Some(d) = self.drop_nonsense_pieces {
rules.drop_nonsense_pieces = d;
}
if let Some(d) = self.drop_tiny_archives {
rules.drop_tiny_archives = d;
}
rules
}
}
+80 -6
View File
@@ -1115,7 +1115,7 @@ fn token_signals(t: &[u8]) -> TokenSignals {
let mut lower = 0;
for i in 0..t.len() {
let c = t[i];
if c.is_ascii_alphabetic() && !is_vowel(c) {
if c.is_ascii_alphabetic() && !is_vowel(c) && c.to_ascii_lowercase() != b'y' {
consonants += 1;
s.max_consonants = s.max_consonants.max(consonants);
} else {
@@ -1181,7 +1181,9 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
return if n >= 3 { TokenKind::Word } else { TokenKind::Neutral };
}
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
if n >= 6 && s.case_switches >= 2 && (s.inner_upper || (s.vowels * 4 < n && (s.runs >= 4 || s.lone >= 1))) {
// Two CamelCase words with few vowels (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone
// letter; a scramble has more runs, a lone letter or an inner capital run.
if n >= 6 && s.case_switches >= 2 && s.vowels * 4 < n && (s.runs >= 5 || s.lone >= 1 || s.inner_upper) {
return TokenKind::Random;
}
return TokenKind::Neutral;
@@ -1193,6 +1195,14 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
return if n >= 12 && token_signals(t).switches >= 6 { TokenKind::Random } else { TokenKind::Neutral };
}
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
if t.split(|c: &u8| !c.is_ascii_alphabetic()).filter(|w| !w.is_empty()).all(|w| {
let vowels = w.iter().filter(|c| matches!(c.to_ascii_lowercase(), b'a' | b'e' | b'i' | b'o' | b'u' | b'y')).count();
(w.len() >= 2 && w.iter().all(u8::is_ascii_uppercase))
|| (w.len() >= 2 && w[1..].iter().all(u8::is_ascii_lowercase) && (w.len() >= 3 || w[0].is_ascii_lowercase()) && vowels * 5 >= w.len())
}) {
return TokenKind::Neutral;
}
let s = token_signals(&scene_case(t));
if n >= 5 && s.switches >= 3 {
return TokenKind::Random;
@@ -1505,6 +1515,11 @@ static RELEASE_LIKE: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
});
/// RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way.
static RELEASE_LIKE_SPACED: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
});
/// The stub-archive rule (LOG.md (123)-(126), (128)): one unsplit archive, no par2, settled, posted to 2+ groups,
/// under STUB_MAX_BYTES (the fakes stop at ~90 MB; the hash-named uploads, real albums and passworded videos above
/// it don't), and its articles at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES (Power-Post's
@@ -1518,6 +1533,8 @@ pub const STUB_WIDE_GROUPS: usize = 4;
/// release-like name that isn't scrambled (the newznzb.* pieces; under 100 kB are cue/log sidecars and adverts).
pub const NONSENSE_MIN_BYTES: u64 = 100_000;
pub const NONSENSE_MAX_BYTES: u64 = 1_000_000;
/// The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars.
pub const TINY_MIN_BYTES: u64 = 2_000;
pub struct JudgedFile<'a> {
pub name: &'a str,
@@ -1579,6 +1596,27 @@ pub fn is_nonsense_piece(f: &JudgedFile) -> bool {
&& !is_random_file(f.name)
}
/// Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
/// exe is the content), covers and sidecars.
static TINY_CONTENT: LazyLock<Regex> = LazyLock::new(|| {
re(r"(?i)\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b")
});
/// A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
/// unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
/// subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
pub fn is_tiny_archive(f: &JudgedFile) -> bool {
f.total_parts == 1
&& f.segments == 1
&& (TINY_MIN_BYTES..NONSENSE_MIN_BYTES).contains(&f.bytes)
&& !f.groups.iter().any(|g| test(&STUB_EXEMPT_GROUPS, g))
&& test(&RELEASE_LIKE_SPACED, &file_stem(f.name))
&& !test(&EBOOK, f.name)
&& !test(&TINY_CONTENT, f.name)
&& !looks_scrambled(f.name)
&& !is_random_file(f.name)
}
pub enum Verdict {
Keep,
Drop { rule: &'static str, key: String, sample: bool },
@@ -1600,9 +1638,11 @@ pub struct Rules {
pub drop_executables: bool,
/// Fragments of the poster lowercased without punctuation (droppedPosters).
pub dropped_posters: Vec<String>,
/// The stub-archive rule (dropStubArchives) and the nonsense-piece rule (dropNonsensePieces).
/// The stub-archive rule (dropStubArchives), the nonsense-piece rule (dropNonsensePieces) and the tiny-archive
/// rule (dropTinyArchives).
pub drop_stub_archives: bool,
pub drop_nonsense_pieces: bool,
pub drop_tiny_archives: bool,
}
impl Rules {
@@ -1620,6 +1660,7 @@ impl Rules {
dropped_posters: ["celebmovie", "celebwow", "celebfx", "iluvcelebs", "nudecelebs", "celebsextapes"].iter().map(|p| p.to_string()).collect(),
drop_stub_archives: true,
drop_nonsense_pieces: true,
drop_tiny_archives: true,
}
}
}
@@ -1655,6 +1696,9 @@ pub fn judge(release_key: &str, files: &[JudgedFile], settled: bool, rules: &Rul
if rules.drop_nonsense_pieces && is_nonsense_piece(f) {
return Verdict::Drop { rule: "nonsense-piece", key: upload_tool(f.first_message_id).to_string(), sample };
}
if rules.drop_tiny_archives && is_tiny_archive(f) {
return Verdict::Drop { rule: "tiny-archive", key: upload_tool(f.first_message_id).to_string(), sample };
}
}
if files.iter().any(|f| !is_random_file(f.name)) {
return Verdict::Keep;
@@ -1854,7 +1898,8 @@ mod tests {
"Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing", "Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON",
"Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr", "Eclipse_IDE_2023-12_R_win32_x86_64.zip", "PartsHPGL 1.06c", "Go4Customer Fax Machine v1.2b",
"AdBeGone 1.2", "3DMark03 320", "COD4MW2REMASTERED", "Movie.2020.1080p-GRP [a3f9e2b1]", "EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps",
"iPhone", "Some.Show.S01E01.1080p.mkv",
"iPhone", "Some.Show.S01E01.1080p.mkv", "FastFilms 2.04", "BestCrypt NP v4.0", "EasyHTML 2.35 German", "SchiffsModell - September 2026",
"CrossStitcher - Issue 440, October 2026", "Laura15Juli2026", "IHaveAWife - Isa Bella", "DickHDDaily - Santana Red", "1904-SmithsWork-Hasluck",
] {
assert!(!is_unreadable_name(n), "{n}");
}
@@ -1982,8 +2027,8 @@ mod single_archive_rule_tests {
let rules = Rules::defaults(Default::default());
// One 790 kB article under a release-like name: the newznzb.* pieces.
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
// A 3 kB cue/log sidecar rar with a year in its name, and a diet advert: under the floor.
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), None);
// A 3 kB cue/log sidecar rar with a year in its name: under this rule's floor (the tiny-archive rule's outside the music groups).
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// A book (a year or a format in its name, written with spaces), a dotted one too: not a piece.
assert_eq!(verdict("Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar", "fake@address.com (Psychopath)", 2, 1, 303_000, true, &rules), None);
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 382_000, true, &rules), None);
@@ -1995,6 +2040,35 @@ mod single_archive_rule_tests {
let off = Rules { drop_nonsense_pieces: false, ..Rules::defaults(Default::default()) };
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &off), None);
}
#[test]
fn tiny_archives() {
let rules = Rules::defaults(Default::default());
// One article of 2-100 kB under a release name, dotted or with spaces: a RAR holding "<title>.exe" (LOG.md (131)).
assert_eq!(verdict("Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar", "Yanc@power-post.org <Yanc@power-post.org>", 1, 1, 15_084, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
assert_eq!(verdict("Cicero Magazin fuer politische Kultur Mai 05 2016.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 17_478, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// A scene name without a year, a tag or a group suffix: not release-like, so kept (the flood's porn names mostly carry one).
assert_eq!(verdict("[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar", "Yanc@power-post.org", 2, 1, 23_449, true, &rules), None);
assert_eq!(verdict("Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 7_869, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
assert_eq!(verdict("The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar", "henk@example.org", 1, 1, 38_000, true, &rules), None);
assert_eq!(verdict("Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar", "poster@example.org", 1, 1, 60_000, true, &rules), None);
assert_eq!(verdict("Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
assert_eq!(verdict("Some Band - Album (2011) covers.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 38_000, true, &rules), None);
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's.
let groups = vec!["alt.binaries.sounds.karaoke".to_string()];
let files = [JudgedFile { name: "KV-1055806 - Walker Hayes - U Gurl (2017).zip", total_parts: 1, groups: &groups, first_message_id: NYUU, poster: "home@home.com", segments: 1, bytes: 60_000 }];
assert!(matches!(judge("k", &files, true, &rules), Verdict::Keep));
assert_eq!(verdict("holiday snaps.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
assert_eq!(verdict("nYDeXeF49dieZ1J7flK-JFzZ3j.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 2, 20_000, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 1_500, true, &rules), None);
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 150_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, false, &rules), None);
let off = Rules { drop_tiny_archives: false, ..Rules::defaults(Default::default()) };
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, true, &off), None);
}
}
#[cfg(test)]
+1 -1
View File
@@ -66,7 +66,7 @@ export interface ProbeCandidate {
/** What a rule dropped in a batch (REBUILD.md, rules and recovery). */
export interface RuleCount {
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'incomplete'
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive' | 'incomplete'
key: string
articles: number
files: number
+15 -3
View File
@@ -43,7 +43,7 @@ function signals(t: string) {
let maxConsonants = 0
for (let i = 0; i < t.length; i++) {
const c = t[i]
if (/[b-df-hj-np-tv-z]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
if (/[b-df-hj-np-tv-xz]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
else consonants = 0
if (i === 0) continue
const a = t[i - 1]
@@ -90,11 +90,14 @@ export function tokenKind(t: string): 'word' | 'random' | null {
if (t.length >= 10 && s.vowels * 6 < t.length) return 'random'
return t.length >= 3 ? 'word' : null
}
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble. Two CamelCase words with few vowels
// (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone letter; a scramble has more runs, a lone
// letter or an inner capital run.
if (
t.length >= 6 &&
s.caseSwitches >= 2 &&
(s.innerUpper || (s.vowels * 4 < t.length && (s.runs >= 4 || s.lone >= 1)))
s.vowels * 4 < t.length &&
(s.runs >= 5 || s.lone >= 1 || s.innerUpper)
)
return 'random'
return null
@@ -104,6 +107,15 @@ export function tokenKind(t: string): 'word' | 'random' | null {
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
return t.length >= 12 && signals(t).switches >= 6 ? 'random' : null
}
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
if (
(t.match(/[A-Za-z]+/g) ?? []).every(
(w) =>
/^[A-Z]{2,}$/.test(w) ||
(/^[A-Z]?[a-z]{2,}$/.test(w) && (w.match(/[aeiouy]/g) ?? []).length * 5 >= w.length)
)
)
return null
const s = signals(sceneCase(t))
if (t.length >= 5 && s.switches >= 3) return 'random'
if (t.length >= 6 && s.switches >= 2 && (s.lone >= 1 || s.innerUpper || s.maxConsonants >= 4))
+1 -1
View File
@@ -1,2 +1,2 @@
export { uploadTool, signatureGroup, signatureKey, explored } from './upload_tools.js'
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, isTinyArchive, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, TINY_MIN_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
+43 -1
View File
@@ -47,6 +47,12 @@ export interface Rules {
* the newznzb.* pieces (9% of the catalog; LOG.md (123), (125)).
*/
dropNonsensePieces: boolean
/**
* Drop one-article archives of TINY_MIN_BYTES to NONSENSE_MIN_BYTES under a release-like name (spaces allowed),
* outside the exempt groups, without an e-book, subtitle, crack or sidecar word: the 2015-17 exe-stub flood in
* miniature (a RAR holding "<title>.exe" at 2-60 kB; 1.65M catalog rows; LOG.md (131), (133)).
*/
dropTinyArchives: boolean
}
export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
@@ -58,6 +64,7 @@ export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
droppedPosters: ['celebmovie', 'celebwow', 'celebfx', 'iluvcelebs', 'nudecelebs', 'celebsextapes'],
dropStubArchives: true,
dropNonsensePieces: true,
dropTinyArchives: true,
droppedClasses: [],
}
@@ -69,8 +76,12 @@ export const STUB_WIDE_GROUPS = 4
/** The nonsense-piece rule's bounds: under 100 kB are cue/log sidecars and adverts, 1 MB and up is rule 1's. */
export const NONSENSE_MIN_BYTES = 100_000
export const NONSENSE_MAX_BYTES = 1_000_000
/** The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars. */
export const TINY_MIN_BYTES = 2_000
export type Verdict = { rule: null } | { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece'; key: string; sample: boolean }
export type Verdict =
| { rule: null }
| { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive'; key: string; sample: boolean }
/** A file as the judge sees it: its name, parts, groups, poster, and the articles seen with their bytes. */
export interface JudgedFile {
@@ -148,6 +159,36 @@ export function isNonsensePiece(f: JudgedFile): boolean {
)
}
/** RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way. */
const RELEASE_LIKE_SPACED = new RegExp(RELEASE_LIKE.source.replace('[._-]', '[._ -]'), 'i')
/**
* Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
* exe is the content), covers and sidecars.
*/
const TINY_CONTENT =
/\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b/i
/**
* A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
* unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
* subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
*/
export function isTinyArchive(f: JudgedFile): boolean {
return (
f.totalParts === 1 &&
f.segments === 1 &&
f.bytes >= TINY_MIN_BYTES &&
f.bytes < NONSENSE_MIN_BYTES &&
!f.groups.some((g) => STUB_EXEMPT_GROUPS.test(g)) &&
RELEASE_LIKE_SPACED.test(fileStem(f.name)) &&
!EBOOK.test(f.name) &&
!TINY_CONTENT.test(f.name) &&
!isScrambledFile(f.name) &&
!isRandomFile(f.name)
)
}
export function isRandomFile(name: string) {
return looksRandom(fileStem(name))
}
@@ -221,6 +262,7 @@ export class RuleSet {
if (legs !== '') return { rule: 'stub-archive', key: legs, sample }
}
if (this.rules.dropNonsensePieces && isNonsensePiece(archive)) return { rule: 'nonsense-piece', key: uploadTool(archive.firstMessageId), sample }
if (this.rules.dropTinyArchives && isTinyArchive(archive)) return { rule: 'tiny-archive', key: uploadTool(archive.firstMessageId), sample }
}
if (files.some((f) => !isRandomFile(f.name))) return { rule: null }
const first = files[0]
+32
View File
@@ -538,6 +538,38 @@ test.group('BatchStore', (group) => {
assert.include(await db.from('pending_releases').firstOrFail(), { files: 2, segments: '5' })
})
test('a finished unnamed release without a probe target is exhausted at once, not waiting', async ({
assert,
}) => {
const { rangeId, nodeId } = await rangeIn(GROUP, 1)
// Two hash-named pieces: the edge gives the first a lone target; without one the ladder could never claim it.
const batch = batchOf(GROUP, 1, [
...upload(
10,
'9e7fc2741cdd450287aff2733ca4c193',
4,
'b3a75d234aa73891e78a70fc1f5496b1 [1/2]',
'2026-10-05T10:00:00Z'
),
...upload(
20,
'c5ebdd5487f5b132c76e8d5c2572e560',
4,
'b3a75d234aa73891e78a70fc1f5496b1 [2/2]',
'2026-10-05T10:01:00Z'
),
])
assert.lengthOf(batch.releases, 1)
assert.isFalse(batch.releases[0].readable)
const stored = await BatchStore.store(rangeId, nodeId, { ...batch, candidates: [] })
assert.include(stored, { releases: 1, newReleases: 0, candidates: 0, pending: 1 })
const pending = await db.from('pending_releases').firstOrFail()
assert.include(pending, { files: 2, finished: true, status: 3 })
assert.deepEqual(pending.candidates, [])
const klass = await db.from('pending_classes').firstOrFail()
assert.include(klass, { waiting: 0 })
})
test("an album's members stay findable by their own names (release_texts kind member)", async ({
assert,
}) => {
+49
View File
@@ -163,6 +163,55 @@ test.group('ReleaseJudge', (group) => {
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
})
test('judges tiny single-article files under release-like names', async ({ assert }) => {
let n = 0
const settled = new Date(Date.now() - 3 * 86400_000)
const release = async (name: string, extra: Record<string, unknown> = {}) => {
n++
const key = `jt${String(n).padStart(6, '0')}`.padEnd(40, '0')
return Number(
await insertRelease({
release_key: key,
identity_key: key,
name,
source: 'stem',
groups: ['alt.binaries.boneless'],
files: 1,
segments: 1,
bytes: 15_000,
posted_at: settled,
readable: true,
settled_at: settled,
...extra,
})
)
}
const dotted = await release('Velvet.S02E20.GERMAN.720p.HDTV.x264-ACED')
const spaced = await release('Pacific Rim (2013) BDRip DVDr-sailo1')
const subs = await release('Some.Film.2019.German.Subs-XYZ')
const ebook = await release('Some.Book.2019.epub-GRP')
const music = await release('Some.Album.2019.FLAC-GRP', { groups: ['alt.binaries.sounds.flac'] })
const big = await release('Big.Film.2019.1080p-GRP', { bytes: 500_000 })
const twoFiles = await release('Two.Films.2019.1080p-GRP', { files: 2, segments: 2 })
const plain = await release('Holiday photo')
const sidecar = await release('Side.Car.2019.1080p-GRP', { sidecar_only: true })
const unreadable = await release('Xk9v.2019.Qpz-ab', { readable: false })
const round = await new ReleaseJudge().full()
const verdicts = new Map(
(await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [
Number(v.release_id),
v.rule,
])
)
assert.equal(verdicts.get(dotted), 'tiny')
assert.equal(verdicts.get(spaced), 'tiny')
for (const id of [subs, ebook, music, big, twoFiles, plain, sidecar, unreadable]) {
assert.notEqual(verdicts.get(id), 'tiny')
}
assert.equal(round.rules.tiny.added, 2)
})
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
assert,
}) => {
+9
View File
@@ -52,6 +52,15 @@ test.group('release / isUnreadableName', () => {
'EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps',
'iPhone',
'Some.Show.S01E01.1080p.mkv',
'FastFilms 2.04',
'BestCrypt NP v4.0',
'EasyHTML 2.35 German',
'SchiffsModell - September 2026',
'CrossStitcher - Issue 440, October 2026',
'Laura15Juli2026',
'IHaveAWife - Isa Bella',
'DickHDDaily - Santana Red',
'1904-SmithsWork-Hasluck',
]) {
assert.isFalse(isUnreadableName(n), n)
}
+28 -2
View File
@@ -98,8 +98,9 @@ test.group('rules / verdicts', () => {
test('nonsense pieces: one article of 100 kB to 1 MB under a release-like name', ({ assert }) => {
assert.include(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: 'nonsense-piece', key: 'nyuu' })
// A 3 kB cue/log sidecar rar with a year in its name: under the floor. A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
assert.deepEqual(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: null })
// A 3 kB cue/log sidecar rar with a year in its name: under the floor (the tiny-archive rule's outside the music groups). A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
assert.deepEqual(rules.judge('k', [{ ...archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000), groups: ['alt.binaries.sounds.lossless'] }]), { rule: null })
assert.include(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: 'tiny-archive' })
assert.deepEqual(rules.judge('k', [archive('Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar', 'fake@address.com (Psychopath)', 2, 1, 303_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 382_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar', 'saggir@domain.com', 1, 1, 113_000)]), { rule: null })
@@ -109,6 +110,31 @@ test.group('rules / verdicts', () => {
const off = new RuleSet({ ...DEFAULT_RULES, dropNonsensePieces: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
assert.deepEqual(off.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
})
test('tiny archives: one article of 2-100 kB under a release name, spaces allowed', ({ assert }) => {
// A RAR holding "<title>.exe": the 2015-17 flood in miniature (LOG.md (131)).
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar', 'Yanc@power-post.org <Yanc@power-post.org>', 1, 1, 15_084)]), { rule: 'tiny-archive', key: 'nyuu' })
assert.include(rules.judge('k', [archive('Cicero Magazin fuer politische Kultur Mai 05 2016.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 17_478)]), { rule: 'tiny-archive', key: 'nyuu' })
// A scene name without a year, a tag or a group suffix: not release-like, so kept.
assert.deepEqual(rules.judge('k', [archive('[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar', 'Yanc@power-post.org', 2, 1, 23_449)]), { rule: null })
assert.include(rules.judge('k', [archive('Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 7_869)]), { rule: 'tiny-archive', key: 'nyuu' })
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
assert.deepEqual(rules.judge('k', [archive('The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar', 'henk@example.org', 1, 1, 38_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar', 'poster@example.org', 1, 1, 60_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Some Band - Album (2011) covers.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 38_000)]), { rule: null })
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's; not settled.
assert.deepEqual(rules.judge('k', [{ ...archive('KV-1055806 - Walker Hayes - U Gurl (2017).zip', 'home@home.com', 1, 1, 60_000), groups: ['alt.binaries.sounds.karaoke'] }]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('holiday snaps.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('nYDeXeF49dieZ1J7flK-JFzZ3j.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 2, 20_000)]), { rule: null })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 1_500)]), { rule: null })
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 150_000)]), { rule: 'nonsense-piece' })
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)], false), { rule: null })
const off = new RuleSet({ ...DEFAULT_RULES, dropTinyArchives: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
assert.deepEqual(off.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
})
})
test.group('rules / upload tools', () => {