Compare commits
15
Commits
44c0c73b05
...
rebuild
No files matched your search
+45
-14
@@ -341,8 +341,13 @@ or music name whose content is a program: archive entries all
|
|||||||
executables, or the file itself one by name or by its first bytes, sniffed
|
executables, or the file itself one by name or by its first bytes, sniffed
|
||||||
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
|
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
|
||||||
*locked* (passworded, no password from a spot, its NZB or the title),
|
*locked* (passworded, no password from a spot, its NZB or the title),
|
||||||
*unusable* (random names, no inner name, no par2 or inner-name merge, no
|
*tiny* (one file of one article, 2–100 kB, under a readable release-like
|
||||||
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
|
name outside the music, e-book and comic groups, without an e-book,
|
||||||
|
subtitle, crack or sidecar word: the catalog's side of the edge's
|
||||||
|
tiny-archive rule, 1.5M rows of the 2015–17 flood that the edge indexed
|
||||||
|
before the rule; LOG.md (131), (135)), *unusable* (random names, no inner
|
||||||
|
name, no par2 or inner-name merge, no spot, a day after settling).
|
||||||
|
Verdicts (`release_verdicts`, with the rule
|
||||||
set's version) hide releases from search (`judged` shows them); a verdict
|
set's version) hide releases from search (`judged` shows them); a verdict
|
||||||
whose rule no longer matches is taken back. Nothing is dropped yet.
|
whose rule no longer matches is taken back. Nothing is dropped yet.
|
||||||
|
|
||||||
@@ -712,10 +717,13 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
|||||||
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
|
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
|
||||||
`settled_at IS NOT NULL`), the settler, the fragment merger,
|
`settled_at IS NOT NULL`), the settler, the fragment merger,
|
||||||
`releases_category_newest_idx`.
|
`releases_category_newest_idx`.
|
||||||
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
|
- `release_inspections` and `probe_candidates` were on `CHUNK_TABLESPACE`
|
||||||
HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
|
(the HDD) with their indexes; both are back on the NVMe (migrations 4800
|
||||||
rule read all of `release_inspections` every round, 17 min on the HDD;
|
and 7300). Note LOG.md 2026-10-19 (81): the judge's fake rule read all of
|
||||||
the judge must read it by release or keep what it needs on the NVMe.
|
`release_inspections` every round, 17 min on the HDD; and 2026-11-03
|
||||||
|
(136): the prober's catalog claim updates `probe_candidates` from random
|
||||||
|
points every minute, 2–3 min on the HDD beside the pending ladder's sweep.
|
||||||
|
What a worker reads or updates by random key every minute stays on the NVMe.
|
||||||
|
|
||||||
### Batch store (built 2026-10-25)
|
### Batch store (built 2026-10-25)
|
||||||
|
|
||||||
@@ -806,9 +814,17 @@ unscrambled name without spaces and without an e-book format in it is
|
|||||||
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
|
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
|
||||||
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
|
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
|
||||||
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
|
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
|
||||||
with spaces are books and albums. Both rules are switched by the
|
with spaces are books and albums. Below that, a settled single archive of
|
||||||
rule set (`dropStubArchives`, `dropNonsensePieces`), sampled like every
|
one article and 2–100 kB under a release-like name (spaces allowed), outside
|
||||||
drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
|
the exempt groups and without an e-book, subtitle, crack/keygen/trainer,
|
||||||
|
cover or sidecar word, is dropped as a **tiny archive** (`tiny-archive`,
|
||||||
|
key: the upload tool; LOG.md (131), (133)): the 2015–17 flood's RARs
|
||||||
|
holding "<title>.exe" at 2–60 kB, 1.66M catalog rows from a dozen posters
|
||||||
|
(12 of 12 fetched fake); the word exemptions keep the real tiny archives
|
||||||
|
(subtitles, cracks, trainers) at the price of ~50k fakes named "Incl
|
||||||
|
Keygen" that the prober judges later. The rules are switched by the
|
||||||
|
rule set (`dropStubArchives`, `dropNonsensePieces`, `dropTinyArchives`),
|
||||||
|
sampled like every drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
|
||||||
judges as each batch completes files, so it passes settled without the
|
judges as each batch completes files, so it passes settled without the
|
||||||
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
|
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
|
||||||
`check.ts` and the fixture comparison (the parity with the TypeScript
|
`check.ts` and the fixture comparison (the parity with the TypeScript
|
||||||
@@ -916,7 +932,10 @@ f. **Nothing enters the catalog without a usable name** (amended
|
|||||||
11% of the segments and nothing anyone can find; the full history has
|
11% of the segments and nothing anyone can find; the full history has
|
||||||
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
|
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
|
||||||
edge sends an unnamed release with an empty payload and its tool, the
|
edge sends an unnamed release with an empty payload and its tool, the
|
||||||
batch store writes `pending_releases` (LOG.md (114)); the ladder in the
|
batch store writes `pending_releases` (LOG.md (114)); a finished release
|
||||||
|
the edge found no target for (the edge's `candidates_of`: par2, archive
|
||||||
|
start, 7z tail, else the first random-named file as the lone target;
|
||||||
|
LOG.md (140)) is written exhausted at once; the ladder in the
|
||||||
prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
|
prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
|
||||||
the oldest waiting releases' next target (par2, archive, lone, 7z-tail
|
the oldest waiting releases' next target (par2, archive, lone, 7z-tail
|
||||||
in that order), at least PENDING_SHARE (25%) of each claim next to the
|
in that order), at least PENDING_SHARE (25%) of each claim next to the
|
||||||
@@ -927,12 +946,24 @@ f. **Nothing enters the catalog without a usable name** (amended
|
|||||||
target or status 3 (exhausted); each finished climb counts on
|
target or status 3 (exhausted); each finished climb counts on
|
||||||
`pending_signatures` for (tool, group, poster) and (tool, group), and a
|
`pending_signatures` for (tool, group, poster) and (tool, group), and a
|
||||||
signature with PENDING_MIN_PROBES (100) climbs and at most
|
signature with PENDING_MIN_PROBES (100) climbs and at most
|
||||||
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
|
PENDING_MAX_NAMED (0.5%) named is dropped: a sweep beside the lanes
|
||||||
waiting releases of dropped signatures are discarded (status 2), the
|
(every five minutes, in its own loop) discards the waiting releases
|
||||||
rules' `samplePercent` of them left to climb. The namer
|
of each dropped signature and class once (`swept_at`; the edge drops
|
||||||
|
their new releases itself), the rules' `samplePercent` of them left to
|
||||||
|
climb, each by an index (`pending_releases_signature_idx` for a
|
||||||
|
poster's rows: tested on the rows, a signature was a scan of the whole
|
||||||
|
table; LOG.md (133), (134)). The batch store, the record, the claim
|
||||||
|
and the sweep lock pending rows in one order, by key and the counters
|
||||||
|
after (`lockOrder` for the multi-row statements; the record and the
|
||||||
|
sweep lock their rows with `SELECT … ORDER BY release_key FOR UPDATE`
|
||||||
|
first, since an UPDATE's join or `= ANY(subquery)` locks in plan
|
||||||
|
order), and a flush that deadlocks anyway keeps its outcomes for the
|
||||||
|
next (LOG.md (134), (137)). The namer
|
||||||
(`app/services/pending_namer.ts`, in the prober's process): a named
|
(`app/services/pending_namer.ts`, in the prober's process): a named
|
||||||
release's article span read again from the corpus (the stored range's
|
release's article span read again from the corpus (the stored range's
|
||||||
chunks) or by XOVER on a connection of its own, through the TypeScript
|
chunks) or by XOVER on a connection of its own, a chunk of articles
|
||||||
|
at a time (one string of a wide span topped V8's limit; spans over 3M
|
||||||
|
articles are given up), through the TypeScript
|
||||||
indexer with `partialFiles` (a file given up short of parts stays a
|
indexer with `partialFiles` (a file given up short of parts stays a
|
||||||
partial file of its release, as the Rust window writes it; without it
|
partial file of its release, as the Rust window writes it; without it
|
||||||
the range path's fragment rule left nothing to find: by 2026-10-29
|
the range path's fragment rule left nothing to find: by 2026-10-29
|
||||||
|
|||||||
+36
-21
@@ -9,7 +9,7 @@ import { SIDECAR } from '#services/sidecar_folder'
|
|||||||
import { identityKey, releaseKey } from '#lib/release/keys'
|
import { identityKey, releaseKey } from '#lib/release/keys'
|
||||||
import { sourceCode } from '#lib/release/source'
|
import { sourceCode } from '#lib/release/source'
|
||||||
import { uploadTool } from '#lib/rules/index'
|
import { uploadTool } from '#lib/rules/index'
|
||||||
import { byEffort } from '#services/pending_ladder'
|
import { PENDING, byEffort, lockOrder } from '#services/pending_ladder'
|
||||||
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
|
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -190,9 +190,11 @@ class BatchStore {
|
|||||||
list.push({ kind: c.kind, messageId: c.messageId, file: c.file, bytes: c.bytes })
|
list.push({ kind: c.kind, messageId: c.messageId, file: c.file, bytes: c.bytes })
|
||||||
targets.set(key, list)
|
targets.set(key, list)
|
||||||
}
|
}
|
||||||
let written = 0
|
// In the record's lock order (LOG.md (136)): all rows sorted, then all classes sorted, in one transaction. The
|
||||||
for (let i = 0; i < releases.length; i += 1000) {
|
// upsert locks a finished row too, before its WHERE passes it by; a chunk's classes before the next chunk's rows
|
||||||
const rows = releases.slice(i, i + 1000).map((r) => {
|
// crossed the ladder's record(), which holds its rows and then takes its classes.
|
||||||
|
const rows = lockOrder(
|
||||||
|
releases.map((r) => {
|
||||||
const probes = targets.get(slot(r.key, r.upload)) ?? []
|
const probes = targets.get(slot(r.key, r.upload)) ?? []
|
||||||
return {
|
return {
|
||||||
release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
|
release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
|
||||||
@@ -211,46 +213,59 @@ class BatchStore {
|
|||||||
file_total: r.fileTotal,
|
file_total: r.fileTotal,
|
||||||
tool: r.tool ?? (probes[0] ? uploadTool(probes[0].messageId) : 'unknown'),
|
tool: r.tool ?? (probes[0] ? uploadTool(probes[0].messageId) : 'unknown'),
|
||||||
candidates: probes,
|
candidates: probes,
|
||||||
|
// A finished release without a target is through the ladder before it starts: exhausted, not waiting
|
||||||
|
// (1.57M rows waited forever, never claimed, before the edge gave the hash-named pieces one; LOG.md (140)).
|
||||||
|
status: r.finished !== false && probes.length === 0 ? PENDING.exhausted : PENDING.waiting,
|
||||||
// The first target's article (by effort): the cheapest probe first within a class (LOG.md (130)).
|
// The first target's article (by effort): the cheapest probe first within a class (LOG.md (130)).
|
||||||
target_bytes: byEffort(probes)[0]?.bytes ?? null,
|
target_bytes: byEffort(probes)[0]?.bytes ?? null,
|
||||||
}
|
}
|
||||||
})
|
}),
|
||||||
|
(r) => r.release_key
|
||||||
|
)
|
||||||
|
let written = 0
|
||||||
|
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
|
||||||
|
for (let i = 0; i < rows.length; i += 1000) {
|
||||||
const { rows: done } = await trx.rawQuery(
|
const { rows: done } = await trx.rawQuery(
|
||||||
`INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
|
`INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
|
||||||
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes)
|
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes, status)
|
||||||
SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.name, v.poster, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0),
|
SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.name, v.poster, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0),
|
||||||
v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates,
|
v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates,
|
||||||
pending_class(v.name, v.bytes, v.partial), v.target_bytes
|
pending_class(v.name, v.bytes, v.partial), v.target_bytes, v.status
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, identity_key text, name text, poster text, first bigint, last bigint, first_article bigint, last_article bigint,
|
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, identity_key text, name text, poster text, first bigint, last bigint, first_article bigint, last_article bigint,
|
||||||
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb, target_bytes bigint)
|
files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb, target_bytes bigint, status int)
|
||||||
ON CONFLICT (release_key) DO UPDATE SET
|
ON CONFLICT (release_key) DO UPDATE SET
|
||||||
files = pending_releases.files + excluded.files, segments = pending_releases.segments + excluded.segments, bytes = pending_releases.bytes + excluded.bytes,
|
files = pending_releases.files + excluded.files, segments = pending_releases.segments + excluded.segments, bytes = pending_releases.bytes + excluded.bytes,
|
||||||
last_posted_at = greatest(pending_releases.last_posted_at, excluded.last_posted_at), last_article = greatest(pending_releases.last_article, excluded.last_article),
|
last_posted_at = greatest(pending_releases.last_posted_at, excluded.last_posted_at), last_article = greatest(pending_releases.last_article, excluded.last_article),
|
||||||
partial = excluded.partial, finished = excluded.finished, file_total = coalesce(excluded.file_total, pending_releases.file_total),
|
partial = excluded.partial, finished = excluded.finished, file_total = coalesce(excluded.file_total, pending_releases.file_total),
|
||||||
candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END,
|
candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END,
|
||||||
klass = pending_class(pending_releases.name, pending_releases.bytes + excluded.bytes, excluded.partial),
|
klass = pending_class(pending_releases.name, pending_releases.bytes + excluded.bytes, excluded.partial),
|
||||||
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes)
|
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes),
|
||||||
|
status = CASE WHEN pending_releases.status = ? AND excluded.finished
|
||||||
|
AND jsonb_array_length(pending_releases.candidates) = 0 AND jsonb_array_length(excluded.candidates) = 0
|
||||||
|
THEN ? ELSE pending_releases.status END
|
||||||
WHERE NOT pending_releases.finished
|
WHERE NOT pending_releases.finished
|
||||||
RETURNING release_key, tool, klass, (xmax = 0) AS inserted`,
|
RETURNING release_key, tool, klass, (xmax = 0) AS inserted, status`,
|
||||||
[groupId, JSON.stringify(rows)]
|
[groupId, JSON.stringify(rows.slice(i, i + 1000)), PENDING.waiting, PENDING.exhausted]
|
||||||
)
|
)
|
||||||
written += done.length
|
written += done.length
|
||||||
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
|
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
|
||||||
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
|
|
||||||
for (const d of done) {
|
for (const d of done) {
|
||||||
const key = `${d.tool}\t${d.klass}`
|
const key = `${d.tool}\t${d.klass}`
|
||||||
const entry = classes.get(key) ?? { tool: d.tool, klass: Number(d.klass), waiting: 0 }
|
const entry = classes.get(key) ?? { tool: d.tool, klass: Number(d.klass), waiting: 0 }
|
||||||
if (d.inserted) entry.waiting++
|
if (d.inserted && Number(d.status) === PENDING.waiting) entry.waiting++
|
||||||
classes.set(key, entry)
|
classes.set(key, entry)
|
||||||
}
|
}
|
||||||
if (classes.size > 0) {
|
}
|
||||||
await trx.rawQuery(
|
if (classes.size > 0) {
|
||||||
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
|
await trx.rawQuery(
|
||||||
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
|
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
|
||||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
|
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
|
||||||
[groupId, JSON.stringify([...classes.values()])]
|
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
|
||||||
)
|
[
|
||||||
}
|
groupId,
|
||||||
|
JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${groupId}\t${c.klass}`)),
|
||||||
|
]
|
||||||
|
)
|
||||||
}
|
}
|
||||||
return written
|
return written
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,20 +5,29 @@ import db from '@adonisjs/lucid/services/db'
|
|||||||
* repeated verdict match) in a catalog release that has a name, readable or found inside. The same film posted
|
* repeated verdict match) in a catalog release that has a name, readable or found inside. The same film posted
|
||||||
* under scrambles by other tools gets that name without another fetch. Releases with a junk verdict don't count.
|
* under scrambles by other tools gets that name without another fetch. Releases with a junk verdict don't count.
|
||||||
*/
|
*/
|
||||||
|
/** Twins read per pair: the hottest pairs have 131k. */
|
||||||
|
const TWIN_SCAN = 500
|
||||||
|
|
||||||
export async function twinNames(
|
export async function twinNames(
|
||||||
pairs: { hash16k: string; size: number }[],
|
pairs: { hash16k: string; size: number }[],
|
||||||
trx: { rawQuery: typeof db.rawQuery } = db
|
trx: { rawQuery: typeof db.rawQuery } = db
|
||||||
): Promise<Map<string, string>> {
|
): Promise<Map<string, string>> {
|
||||||
const out = new Map<string, string>()
|
const out = new Map<string, string>()
|
||||||
if (pairs.length === 0) return out
|
if (pairs.length === 0) return out
|
||||||
|
// Per pair the first named twin among its lowest TWIN_SCAN release ids. hash16k is typed as the column
|
||||||
|
// (char(32)): as text the join cast the column and scanned all of file_hashes, 4 s a call; and with the index
|
||||||
|
// the hottest pairs (55 with over 500 twins, up to 131k, all readable) made a call walk every twin, 37 s inside
|
||||||
|
// the flush's transaction (LOG.md (138)). One name is all a pair needs.
|
||||||
const { rows } = await trx.rawQuery(
|
const { rows } = await trx.rawQuery(
|
||||||
`SELECT DISTINCT ON (h.hash16k, h.size) h.hash16k, h.size, coalesce(r.inner_name, r.name) AS name
|
`SELECT v.hash16k, v.size, t.name
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k text, size bigint)
|
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k char(32), size bigint)
|
||||||
JOIN file_hashes h ON h.hash16k = v.hash16k AND h.size = v.size
|
CROSS JOIN LATERAL (
|
||||||
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
|
SELECT coalesce(r.inner_name, r.name) AS name
|
||||||
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
|
FROM (SELECT release_id FROM file_hashes h WHERE h.hash16k = v.hash16k AND h.size = v.size ORDER BY release_id LIMIT ?) h
|
||||||
ORDER BY h.hash16k, h.size, r.readable DESC, r.id`,
|
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
|
||||||
[JSON.stringify(pairs)]
|
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
|
||||||
|
ORDER BY r.readable DESC, r.id LIMIT 1) t`,
|
||||||
|
[JSON.stringify(pairs), TWIN_SCAN]
|
||||||
)
|
)
|
||||||
for (const r of rows) out.set(`${r.hash16k}\t${r.size}`, r.name)
|
for (const r of rows) out.set(`${r.hash16k}\t${r.size}`, r.name)
|
||||||
return out
|
return out
|
||||||
|
|||||||
+111
-20
@@ -72,6 +72,19 @@ interface ClassRow {
|
|||||||
const CLASSES_MS = 60_000
|
const CLASSES_MS = 60_000
|
||||||
/** A class found short of a claim's share is left alone this long. */
|
/** A class found short of a claim's share is left alone this long. */
|
||||||
const EMPTY_MS = 5 * 60_000
|
const EMPTY_MS = 5 * 60_000
|
||||||
|
/**
|
||||||
|
* Rows in one order for a statement that locks many of them (pending_releases by key, pending_classes and
|
||||||
|
* pending_signatures by theirs): the batch store, the record, the claim and the sweep all lock the same way, so two
|
||||||
|
* of them can't deadlock on each other (LOG.md (134)).
|
||||||
|
*/
|
||||||
|
export function lockOrder<T>(rows: Iterable<T>, key: (row: T) => string): T[] {
|
||||||
|
return [...rows].sort((a, b) => {
|
||||||
|
const ka = key(a)
|
||||||
|
const kb = key(b)
|
||||||
|
return ka < kb ? -1 : ka > kb ? 1 : 0
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/** The prior rate of an unexplored class and its weight in probes: the ladder's overall rate on 2026-11-02 (LOG.md (130)). */
|
/** The prior rate of an unexplored class and its weight in probes: the ladder's overall rate on 2026-11-02 (LOG.md (130)). */
|
||||||
const PRIOR_RATE = 0.15
|
const PRIOR_RATE = 0.15
|
||||||
const PRIOR_WEIGHT = 10
|
const PRIOR_WEIGHT = 10
|
||||||
@@ -276,13 +289,26 @@ class PendingLadder {
|
|||||||
final: null,
|
final: null,
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
// The rows locked in key order first: the UPDATE's join below takes them in whatever order the plan does.
|
||||||
|
await trx.rawQuery(
|
||||||
|
`SELECT 1 FROM pending_releases WHERE release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
||||||
|
ORDER BY release_key FOR UPDATE`,
|
||||||
|
[rows.map((r) => r.release_key).sort()]
|
||||||
|
)
|
||||||
await trx.rawQuery(
|
await trx.rawQuery(
|
||||||
`UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
|
`UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
|
||||||
inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
|
inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
|
||||||
probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
|
probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
|
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
|
||||||
WHERE p.release_key = decode(v.release_key, 'hex')`,
|
WHERE p.release_key = decode(v.release_key, 'hex')`,
|
||||||
[JSON.stringify(rows.map(({ final: _final, ...row }) => row))]
|
[
|
||||||
|
JSON.stringify(
|
||||||
|
lockOrder(
|
||||||
|
rows.map(({ final: _final, ...row }) => row),
|
||||||
|
(r) => r.release_key
|
||||||
|
)
|
||||||
|
),
|
||||||
|
]
|
||||||
)
|
)
|
||||||
// The signatures: the poster's own and the tool's and group's together.
|
// The signatures: the poster's own and the tool's and group's together.
|
||||||
const counts = new Map<
|
const counts = new Map<
|
||||||
@@ -312,17 +338,20 @@ class PendingLadder {
|
|||||||
`INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
|
`INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
|
||||||
SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
|
SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
|
||||||
ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
|
ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
|
||||||
[JSON.stringify([...counts.values()])]
|
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`))]
|
||||||
)
|
)
|
||||||
await trx.rawQuery(
|
await trx.rawQuery(
|
||||||
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float)
|
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float),
|
||||||
|
swept_at = CASE WHEN s.dropped THEN s.swept_at END
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
|
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
|
||||||
WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
|
WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
|
||||||
[
|
[
|
||||||
pendingMinProbes,
|
pendingMinProbes,
|
||||||
pendingMaxNamed,
|
pendingMaxNamed,
|
||||||
JSON.stringify(
|
JSON.stringify(
|
||||||
[...counts.values()].map(({ tool, group_id, poster }) => ({ tool, group_id, poster }))
|
lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`).map(
|
||||||
|
({ tool, group_id, poster }) => ({ tool, group_id, poster })
|
||||||
|
)
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
@@ -359,41 +388,103 @@ class PendingLadder {
|
|||||||
SELECT v.tool, v.group_id, v.klass, v.probed, v.named, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, probed int, named int, waiting int)
|
SELECT v.tool, v.group_id, v.klass, v.probed, v.named, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, probed int, named int, waiting int)
|
||||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named,
|
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named,
|
||||||
waiting = pending_classes.waiting + excluded.waiting, updated_at = now()`,
|
waiting = pending_classes.waiting + excluded.waiting, updated_at = now()`,
|
||||||
[JSON.stringify([...classes.values()])]
|
[JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
|
||||||
)
|
)
|
||||||
await trx.rawQuery(
|
await trx.rawQuery(
|
||||||
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float)
|
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float),
|
||||||
|
swept_at = CASE WHEN c.dropped THEN c.swept_at END
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int)
|
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int)
|
||||||
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
||||||
[
|
[
|
||||||
pendingMinProbes,
|
pendingMinProbes,
|
||||||
pendingMaxNamed,
|
pendingMaxNamed,
|
||||||
JSON.stringify(
|
JSON.stringify(
|
||||||
[...classes.values()].map(({ tool, group_id, klass }) => ({ tool, group_id, klass }))
|
lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`).map(
|
||||||
|
({ tool, group_id, klass }) => ({ tool, group_id, klass })
|
||||||
|
)
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Waiting releases of a dropped signature discarded without a probe, the explored sample excepted; how many. */
|
/**
|
||||||
async discard(limit = 10_000): Promise<number> {
|
* Waiting releases of a dropped class or signature discarded without a probe, the explored sample excepted; how
|
||||||
|
* many. Each dropped class or signature is swept once (swept_at; null again when the record drops it anew),
|
||||||
|
* `perSweep` of them at most, each by its index (the claim's for a class, the signature index for a poster's
|
||||||
|
* rows, the ladder's for a tool's and group's) in rounds of `limit`. The prober runs this beside the lanes, not
|
||||||
|
* in the claim (LOG.md (133)); the edge drops new releases of dropped signatures and classes itself, so one
|
||||||
|
* sweep is all a drop needs.
|
||||||
|
*/
|
||||||
|
async discard(limit = 10_000, perSweep = 20): Promise<number> {
|
||||||
const { samplePercent } = await RuleSets.current()
|
const { samplePercent } = await RuleSets.current()
|
||||||
const { rows } = await db.rawQuery(
|
let discarded = 0
|
||||||
`SELECT encode(p.release_key, 'hex') AS release_key FROM pending_releases p
|
const { rows: classes } = await db.rawQuery(
|
||||||
WHERE p.status = ? AND (EXISTS (
|
`SELECT tool, group_id, klass FROM pending_classes WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, klass LIMIT ?`,
|
||||||
SELECT 1 FROM pending_signatures s
|
[perSweep]
|
||||||
WHERE s.dropped AND s.tool = p.tool AND s.group_id = p.group_id AND (s.poster = '' OR s.poster = coalesce(p.poster, '')))
|
|
||||||
OR EXISTS (SELECT 1 FROM pending_classes c WHERE c.dropped AND c.tool = p.tool AND c.group_id = p.group_id AND c.klass = p.klass))
|
|
||||||
LIMIT ?`,
|
|
||||||
[PENDING.waiting, limit]
|
|
||||||
)
|
)
|
||||||
|
for (const c of classes) {
|
||||||
|
let rows: any[]
|
||||||
|
do {
|
||||||
|
;({ rows } = await db.rawQuery(
|
||||||
|
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||||
|
WHERE status = ? AND tool = ? AND group_id = ? AND klass = ? LIMIT ?`,
|
||||||
|
[PENDING.waiting, c.tool, c.group_id, c.klass, limit]
|
||||||
|
))
|
||||||
|
discarded += await this.discardKeys(rows, samplePercent)
|
||||||
|
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
|
||||||
|
await db.rawQuery(
|
||||||
|
`UPDATE pending_classes SET swept_at = now() WHERE tool = ? AND group_id = ? AND klass = ?`,
|
||||||
|
[c.tool, c.group_id, c.klass]
|
||||||
|
)
|
||||||
|
}
|
||||||
|
const { rows: signatures } = await db.rawQuery(
|
||||||
|
`SELECT tool, group_id, poster FROM pending_signatures WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, poster LIMIT ?`,
|
||||||
|
[perSweep]
|
||||||
|
)
|
||||||
|
for (const s of signatures) {
|
||||||
|
let rows: any[]
|
||||||
|
do {
|
||||||
|
// The poster's rows by the signature index (on the poster's md5: a poster can be longer than a btree
|
||||||
|
// key), a tool's and group's by the ladder's: never a test on the rows, which was a scan of the whole
|
||||||
|
// table per signature on the HDD (LOG.md (134)).
|
||||||
|
;({ rows } =
|
||||||
|
s.poster === ''
|
||||||
|
? await db.rawQuery(
|
||||||
|
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||||
|
WHERE status = ? AND group_id = ? AND tool = ? LIMIT ?`,
|
||||||
|
[PENDING.waiting, s.group_id, s.tool, limit]
|
||||||
|
)
|
||||||
|
: await db.rawQuery(
|
||||||
|
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||||
|
WHERE status = ? AND group_id = ? AND tool = ? AND md5(poster) = md5(?::text) AND poster = ? LIMIT ?`,
|
||||||
|
[PENDING.waiting, s.group_id, s.tool, s.poster, s.poster, limit]
|
||||||
|
))
|
||||||
|
discarded += await this.discardKeys(rows, samplePercent)
|
||||||
|
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
|
||||||
|
await db.rawQuery(
|
||||||
|
`UPDATE pending_signatures SET swept_at = now() WHERE tool = ? AND group_id = ? AND poster = ?`,
|
||||||
|
[s.tool, s.group_id, s.poster]
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return discarded
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The rows discarded (status 2), the explored sample excepted, their classes' waiting counts lowered; how many. */
|
||||||
|
private async discardKeys(rows: any[], samplePercent: number): Promise<number> {
|
||||||
const keys = rows
|
const keys = rows
|
||||||
.map((r: any) => r.release_key as string)
|
.map((r: any) => r.release_key as string)
|
||||||
.filter((key: string) => !explored(key, samplePercent))
|
.filter((key: string) => !explored(key, samplePercent))
|
||||||
|
.sort()
|
||||||
if (keys.length === 0) return 0
|
if (keys.length === 0) return 0
|
||||||
|
// Locked in key order (the record's lock order, LOG.md (137)): a plain `key = ANY(subquery)` locks in scan
|
||||||
|
// order and deadlocked against a batch upsert; ORDER BY sorts before FOR UPDATE locks.
|
||||||
const { rows: gone } = await db.rawQuery(
|
const { rows: gone } = await db.rawQuery(
|
||||||
`UPDATE pending_releases SET status = ? WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
`UPDATE pending_releases p SET status = ?
|
||||||
RETURNING tool, group_id, klass`,
|
FROM (SELECT release_key FROM pending_releases
|
||||||
|
WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
||||||
|
ORDER BY release_key FOR UPDATE) k
|
||||||
|
WHERE p.release_key = k.release_key
|
||||||
|
RETURNING p.tool, p.group_id, p.klass`,
|
||||||
[PENDING.discarded, PENDING.waiting, keys]
|
[PENDING.discarded, PENDING.waiting, keys]
|
||||||
)
|
)
|
||||||
const counts = new Map<string, { tool: string; group_id: number; klass: number; n: number }>()
|
const counts = new Map<string, { tool: string; group_id: number; klass: number; n: number }>()
|
||||||
@@ -413,7 +504,7 @@ class PendingLadder {
|
|||||||
`UPDATE pending_classes c SET waiting = greatest(0, c.waiting - v.n)
|
`UPDATE pending_classes c SET waiting = greatest(0, c.waiting - v.n)
|
||||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, n int)
|
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, n int)
|
||||||
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
||||||
[JSON.stringify([...counts.values()])]
|
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
return gone.length
|
return gone.length
|
||||||
|
|||||||
@@ -30,6 +30,13 @@ interface NamedRow {
|
|||||||
/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
|
/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
|
||||||
const MAX_MOVES = 3
|
const MAX_MOVES = 3
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The widest article span the namer reads again (a busy group posts 50M articles a day, a large upload among
|
||||||
|
* them spans a million or two): one wider is a name the edge saw again much later ("v-lgpc.3ds"), and its span
|
||||||
|
* of segments would not fit the indexer's memory (LOG.md (133)).
|
||||||
|
*/
|
||||||
|
const MAX_SPAN = 3_000_000
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
|
* The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
|
||||||
* read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
|
* read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
|
||||||
@@ -94,6 +101,10 @@ export class PendingNamer {
|
|||||||
|
|
||||||
/** One named release into the catalog. */
|
/** One named release into the catalog. */
|
||||||
async move(row: NamedRow): Promise<void> {
|
async move(row: NamedRow): Promise<void> {
|
||||||
|
if (row.last_article - row.first_article > MAX_SPAN)
|
||||||
|
throw new Error(
|
||||||
|
`its articles ${row.first_article}–${row.last_article} span more than ${MAX_SPAN}`
|
||||||
|
)
|
||||||
const group = await db.from('groups').where('id', row.group_id).firstOrFail()
|
const group = await db.from('groups').where('id', row.group_id).firstOrFail()
|
||||||
const range = await db
|
const range = await db
|
||||||
.from('ranges')
|
.from('ranges')
|
||||||
@@ -103,19 +114,20 @@ export class PendingNamer {
|
|||||||
.orderBy('id')
|
.orderBy('id')
|
||||||
.first()
|
.first()
|
||||||
if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
|
if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
|
||||||
const text = await this.overviewText(group.name, row)
|
|
||||||
const rules = await RuleSets.current()
|
const rules = await RuleSets.current()
|
||||||
const ix = new RangeIndexer({
|
const ix = new RangeIndexer({
|
||||||
group: group.name,
|
group: group.name,
|
||||||
start: row.first_article,
|
start: row.first_article,
|
||||||
end: row.last_article,
|
end: row.last_article,
|
||||||
rules: { ...rules, samplePercent: 0 },
|
rules: { ...rules, samplePercent: 0 },
|
||||||
idleChunks: 0,
|
// No file given up before the end: the span is read chunk by chunk (one text of it topped V8's string).
|
||||||
|
idleChunks: Number.MAX_SAFE_INTEGER,
|
||||||
edgeArticles: 0,
|
edgeArticles: 0,
|
||||||
// The edge wrote it partial when its file is short of parts on the server: so must the re-read.
|
// The edge wrote it partial when its file is short of parts on the server: so must the re-read.
|
||||||
partialFiles: true,
|
partialFiles: true,
|
||||||
})
|
})
|
||||||
ix.addText(text, row.last_article + 1)
|
for await (const piece of this.overviewPieces(group.name, row))
|
||||||
|
ix.addText(piece.text, piece.next)
|
||||||
ix.finish()
|
ix.finish()
|
||||||
const batch = ix.take()
|
const batch = ix.take()
|
||||||
const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
|
const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
|
||||||
@@ -160,50 +172,20 @@ export class PendingNamer {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The overview lines of the release's article span: the stored corpus chunks when its range is stored, else XOVER. */
|
/**
|
||||||
private async overviewText(groupName: string, row: NamedRow): Promise<string> {
|
* The overview lines of the release's article span, a chunk of articles at a time with the first article after
|
||||||
if (this.corpusDir) {
|
* it: the stored corpus chunks when its ranges are stored and contiguous, else XOVER in windows of a chunk.
|
||||||
const stored = await db
|
*/
|
||||||
.from('ranges')
|
private async *overviewPieces(
|
||||||
.where('group_id', row.group_id)
|
groupName: string,
|
||||||
.whereNotNull('stored_at')
|
row: NamedRow
|
||||||
.where('start_id', '<=', row.last_article)
|
): AsyncGenerator<{ text: string; next: number }> {
|
||||||
.where('end_id', '>=', row.first_article)
|
const size = coordinatorConfig.chunkArticles
|
||||||
.orderBy('start_id')
|
const stored = await this.storedChunks(groupName, row)
|
||||||
const parts: string[] = []
|
if (stored) {
|
||||||
let covered = true
|
for (const { raw, next } of stored)
|
||||||
for (const r of stored) {
|
yield { text: (await inflate(raw)).toString('latin1'), next }
|
||||||
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
|
return
|
||||||
const size = coordinatorConfig.chunkArticles
|
|
||||||
for (
|
|
||||||
let from =
|
|
||||||
range.start +
|
|
||||||
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
|
|
||||||
from <= Math.min(row.last_article, range.end);
|
|
||||||
from += size
|
|
||||||
) {
|
|
||||||
const raw = await Corpus.read(
|
|
||||||
range,
|
|
||||||
{ from, to: Math.min(from + size - 1, range.end) },
|
|
||||||
this.corpusDir
|
|
||||||
)
|
|
||||||
if (!raw) {
|
|
||||||
covered = false
|
|
||||||
break
|
|
||||||
}
|
|
||||||
parts.push((await inflate(raw)).toString('latin1'))
|
|
||||||
}
|
|
||||||
if (!covered) break
|
|
||||||
}
|
|
||||||
if (
|
|
||||||
covered &&
|
|
||||||
parts.length > 0 &&
|
|
||||||
stored.length > 0 &&
|
|
||||||
Number(stored[0].start_id) <= row.first_article &&
|
|
||||||
Number(stored[stored.length - 1].end_id) >= row.last_article
|
|
||||||
) {
|
|
||||||
return parts.join('')
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
this.client ??= await XoverClient.connect({
|
this.client ??= await XoverClient.connect({
|
||||||
host: env.get('NNTP_HOST')!,
|
host: env.get('NNTP_HOST')!,
|
||||||
@@ -217,13 +199,55 @@ export class PendingNamer {
|
|||||||
this.usedAt = Date.now()
|
this.usedAt = Date.now()
|
||||||
try {
|
try {
|
||||||
await this.client.group(groupName)
|
await this.client.group(groupName)
|
||||||
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
|
for (let from = row.first_article; from <= row.last_article; from += size) {
|
||||||
const raw = await this.client.xoverRaw(row.first_article, row.last_article)
|
const to = Math.min(from + size - 1, row.last_article)
|
||||||
return (await inflate(raw)).toString('latin1')
|
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
|
||||||
|
const raw = await this.client.xoverRaw(from, to)
|
||||||
|
yield { text: (await inflate(raw)).toString('latin1'), next: to + 1 }
|
||||||
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.client.close()
|
this.client.close()
|
||||||
this.client = null
|
this.client = null
|
||||||
throw error
|
throw error
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** The span's stored corpus chunks (compressed, with the first article after each), or null when any is missing. */
|
||||||
|
private async storedChunks(
|
||||||
|
groupName: string,
|
||||||
|
row: NamedRow
|
||||||
|
): Promise<{ raw: Buffer; next: number }[] | null> {
|
||||||
|
if (!this.corpusDir) return null
|
||||||
|
const stored = await db
|
||||||
|
.from('ranges')
|
||||||
|
.where('group_id', row.group_id)
|
||||||
|
.whereNotNull('stored_at')
|
||||||
|
.where('start_id', '<=', row.last_article)
|
||||||
|
.where('end_id', '>=', row.first_article)
|
||||||
|
.orderBy('start_id')
|
||||||
|
if (
|
||||||
|
stored.length === 0 ||
|
||||||
|
Number(stored[0].start_id) > row.first_article ||
|
||||||
|
Number(stored[stored.length - 1].end_id) < row.last_article
|
||||||
|
)
|
||||||
|
return null
|
||||||
|
const size = coordinatorConfig.chunkArticles
|
||||||
|
const chunks: { raw: Buffer; next: number }[] = []
|
||||||
|
for (const r of stored) {
|
||||||
|
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
|
||||||
|
for (
|
||||||
|
let from =
|
||||||
|
range.start +
|
||||||
|
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
|
||||||
|
from <= Math.min(row.last_article, range.end);
|
||||||
|
from += size
|
||||||
|
) {
|
||||||
|
const to = Math.min(from + size - 1, range.end)
|
||||||
|
const raw = await Corpus.read(range, { from, to }, this.corpusDir)
|
||||||
|
if (!raw) return null
|
||||||
|
chunks.push({ raw, next: to + 1 })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return chunks
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+37
-7
@@ -41,6 +41,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
|
|||||||
const FLUSH_MS = 2000
|
const FLUSH_MS = 2000
|
||||||
/** Between sweeps that discard the waiting releases of dropped signatures. */
|
/** Between sweeps that discard the waiting releases of dropped signatures. */
|
||||||
const SWEEP_MS = 5 * 60_000
|
const SWEEP_MS = 5 * 60_000
|
||||||
|
/** A flush that deadlocks (against the batch store's pending upsert) keeps its outcomes for the next, this often. */
|
||||||
|
const DEADLOCK_RETRIES = 5
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
|
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
|
||||||
@@ -82,6 +84,8 @@ export class Prober {
|
|||||||
private refilling: Promise<void> | null = null
|
private refilling: Promise<void> | null = null
|
||||||
private stopped = false
|
private stopped = false
|
||||||
private stats = Prober.freshStats()
|
private stats = Prober.freshStats()
|
||||||
|
/** Flushes deadlocked in a row, their outcomes kept. */
|
||||||
|
private deadlocks = 0
|
||||||
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
|
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
|
||||||
private lanes = new Set<Promise<void>>()
|
private lanes = new Set<Promise<void>>()
|
||||||
private running = 0
|
private running = 0
|
||||||
@@ -97,7 +101,6 @@ export class Prober {
|
|||||||
}
|
}
|
||||||
/** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
|
/** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
|
||||||
private namer = new PendingNamer()
|
private namer = new PendingNamer()
|
||||||
private sweptAt = 0
|
|
||||||
|
|
||||||
async run() {
|
async run() {
|
||||||
// A previous run's claims: one prober at a time.
|
// A previous run's claims: one prober at a time.
|
||||||
@@ -106,6 +109,17 @@ export class Prober {
|
|||||||
const released = await PendingLadder.release()
|
const released = await PendingLadder.release()
|
||||||
if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
|
if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
|
||||||
const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
|
const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
|
||||||
|
// The ladder's discard sweep beside the lanes: a dropped signature's rows take minutes on the HDD (LOG.md (133)).
|
||||||
|
const sweeper = (async () => {
|
||||||
|
while (!this.stopped) {
|
||||||
|
try {
|
||||||
|
this.stats.discarded += await PendingLadder.discard()
|
||||||
|
} catch (error: any) {
|
||||||
|
logger.error(`Probing: the discard sweep failed: ${error.message}`)
|
||||||
|
}
|
||||||
|
for (let waited = 0; waited < SWEEP_MS && !this.stopped; waited += 1000) await sleep(1000)
|
||||||
|
}
|
||||||
|
})()
|
||||||
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
|
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
|
||||||
const flusher = (async () => {
|
const flusher = (async () => {
|
||||||
while (!this.stopped || this.outcomes.length > 0) {
|
while (!this.stopped || this.outcomes.length > 0) {
|
||||||
@@ -136,6 +150,7 @@ export class Prober {
|
|||||||
}
|
}
|
||||||
await Promise.all(this.lanes)
|
await Promise.all(this.lanes)
|
||||||
await namer
|
await namer
|
||||||
|
await sweeper
|
||||||
await this.share.leave().catch(() => {})
|
await this.share.leave().catch(() => {})
|
||||||
// What was claimed and not probed goes back.
|
// What was claimed and not probed goes back.
|
||||||
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
|
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
|
||||||
@@ -266,10 +281,6 @@ export class Prober {
|
|||||||
|
|
||||||
/** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
|
/** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
|
||||||
private async claimPending() {
|
private async claimPending() {
|
||||||
if (Date.now() - this.sweptAt > SWEEP_MS) {
|
|
||||||
this.sweptAt = Date.now()
|
|
||||||
this.stats.discarded += await PendingLadder.discard()
|
|
||||||
}
|
|
||||||
const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
|
const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
|
||||||
const probes = await PendingLadder.claim(want)
|
const probes = await PendingLadder.claim(want)
|
||||||
if (probes.length === 0) return
|
if (probes.length === 0) return
|
||||||
@@ -339,15 +350,34 @@ export class Prober {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. */
|
/**
|
||||||
|
* The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. A deadlock
|
||||||
|
* (the batch store's pending upsert locks the same rows; LOG.md (134)) keeps the outcomes for the next flush
|
||||||
|
* instead of losing the probes.
|
||||||
|
*/
|
||||||
private async flush() {
|
private async flush() {
|
||||||
const taken = this.outcomes.splice(0)
|
const taken = this.outcomes.splice(0)
|
||||||
if (taken.length === 0) return
|
if (taken.length === 0) return
|
||||||
|
try {
|
||||||
|
await this.store(taken)
|
||||||
|
} catch (error: any) {
|
||||||
|
if (error?.code === '40P01' && this.deadlocks < DEADLOCK_RETRIES) {
|
||||||
|
this.deadlocks++
|
||||||
|
this.outcomes.unshift(...taken)
|
||||||
|
logger.warn(`Probing: storing results deadlocked, kept for the next flush (${this.deadlocks}).`)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
this.deadlocks = 0
|
||||||
for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
|
for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
|
||||||
|
this.stats.pending += taken.filter((o) => o.candidate.pending && o.status !== 'waiting').length
|
||||||
|
}
|
||||||
|
|
||||||
|
private async store(taken: Outcome[]) {
|
||||||
// The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
|
// The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
|
||||||
const pending = taken.filter((o) => o.candidate.pending)
|
const pending = taken.filter((o) => o.candidate.pending)
|
||||||
const outcomes = taken.filter((o) => !o.candidate.pending)
|
const outcomes = taken.filter((o) => !o.candidate.pending)
|
||||||
this.stats.pending += pending.filter((o) => o.status !== 'waiting').length
|
|
||||||
await db.transaction(async (trx) => {
|
await db.transaction(async (trx) => {
|
||||||
if (outcomes.length > 0) {
|
if (outcomes.length > 0) {
|
||||||
await trx.rawQuery(
|
await trx.rawQuery(
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import db from '@adonisjs/lucid/services/db'
|
import db from '@adonisjs/lucid/services/db'
|
||||||
import logger from '@adonisjs/core/services/logger'
|
import logger from '@adonisjs/core/services/logger'
|
||||||
import { SOURCES } from '#lib/release/source'
|
import { SOURCES } from '#lib/release/source'
|
||||||
|
import { TINY_MIN_BYTES, NONSENSE_MIN_BYTES } from '#lib/rules/index'
|
||||||
|
|
||||||
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
|
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
|
||||||
export const JUDGE_VERSION = 1
|
export const JUDGE_VERSION = 1
|
||||||
@@ -28,6 +29,14 @@ const MEDIA =
|
|||||||
const SCRAPED_POSTERS = `celeb[-_ ]{0,1}movie|celebwow|celebfx|iluvcelebs|nude[-_ ]{0,1}celebs|celeb-sex-tapes`
|
const SCRAPED_POSTERS = `celeb[-_ ]{0,1}movie|celebwow|celebfx|iluvcelebs|nude[-_ ]{0,1}celebs|celeb-sex-tapes`
|
||||||
/** An executable's name. */
|
/** An executable's name. */
|
||||||
const EXECUTABLE = `\\.(exe|scr|com|bat|cmd|msi|lnk|vbs|js|jar|pif|cpl|hta|ps1|dll)$`
|
const EXECUTABLE = `\\.(exe|scr|com|bat|cmd|msi|lnk|vbs|js|jar|pif|cpl|hta|ps1|dll)$`
|
||||||
|
/**
|
||||||
|
* The tiny rule's name tests, lib/rules/rule_set.ts's RELEASE_LIKE_SPACED, EBOOK, TINY_CONTENT and
|
||||||
|
* STUB_EXEMPT_GROUPS in Postgres's dialect (and no "?", see MEDIA).
|
||||||
|
*/
|
||||||
|
const TINY_RELEASE_LIKE = `(\\y(19|20)[0-9][0-9]\\y|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-{0,1}dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\\.[0-9]+)+)\\y|-[a-z0-9]{2,}$|\\.[a-z0-9]+\\.[a-z0-9]+\\.)`
|
||||||
|
const TINY_EBOOK = `\\y(epub|mobi|azw3{0,1}|pdf|djvu|cbr|cbz|e-{0,1}books{0,1})\\y`
|
||||||
|
const TINY_CONTENT = `\\y(subs{0,1}|subtitles{0,1}|subpack|srt|untertitel|ondertitels{0,1}|ondertiteling|legendas{0,1}|napisy|sottotitoli|cracks{0,1}|cracked|keygen|keymaker|trainer|patch|patched|serials{0,1}|activator|loader|nocd|no-cd|fix|covers{0,1}|artwork|booklet|scans{0,1}|proof|sample|info|readme)\\y`
|
||||||
|
const TINY_EXEMPT_GROUPS = `(sound|mp3|karaoke|music|audio|flac|lossless|e-{0,1}book|epub|comic|guitar|font|magazine)`
|
||||||
|
|
||||||
export interface Rule {
|
export interface Rule {
|
||||||
name: string
|
name: string
|
||||||
@@ -100,6 +109,20 @@ export const RULES: Rule[] = [
|
|||||||
// releases_passworded_idx: a scan of all of releases was 30 s a round.
|
// releases_passworded_idx: a scan of all of releases was 30 s a round.
|
||||||
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
|
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: 'tiny',
|
||||||
|
says: 'one file of one article, 2–100 kB, under a release-like name: the exe-stub flood in miniature',
|
||||||
|
// The catalog's side of the edge's tiny-archive rule (lib/rules/rule_set.ts isTinyArchive; LOG.md (131),
|
||||||
|
// (135)): the same name tests in Postgres's regex dialect ("\y" for "\b", "{0,1}" for "?"), the readable flag
|
||||||
|
// for its scrambled and random tests, the group set for its groups. The 2015–17 flood that the edge indexed
|
||||||
|
// before the rule: 1.5M rows, 98% of them with an archive candidate, the rest lone par2 files and scrambled
|
||||||
|
// names, so no join on the candidates. A scan of releases (9 GB) a full round.
|
||||||
|
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
|
||||||
|
WHERE r.files = 1 AND r.segments = 1 AND r.bytes >= ${TINY_MIN_BYTES} AND r.bytes < ${NONSENSE_MIN_BYTES}
|
||||||
|
AND r.readable AND NOT coalesce(r.sidecar_only, false)
|
||||||
|
AND r.name ~* '${TINY_RELEASE_LIKE}' AND r.name !~* '${TINY_EBOOK}' AND r.name !~* '${TINY_CONTENT}'
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM group_sets gs WHERE gs.id = r.group_set_id AND gs.groups::text ~* '${TINY_EXEMPT_GROUPS}')`,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: 'unusable',
|
name: 'unusable',
|
||||||
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
|
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
|
||||||
|
|||||||
+38
-30
@@ -6,9 +6,10 @@ import { describeClass } from '#lib/release/index'
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* The pending ladder's classes (LOG.md (130)) on the rows from before them: `klass` (the SQL pending_class) and
|
* The pending ladder's classes (LOG.md (130)) on the rows from before them: `klass` (the SQL pending_class) and
|
||||||
* `target_bytes` (the first target's bytes, when its candidates carry them) set in windows of created_at; the
|
* `target_bytes` (the first target's bytes, when its candidates carry them). The finished rows (named, exhausted,
|
||||||
* finished rows among them (named, exhausted, naming) seed each class's record, which gets its verdict (dropped) as
|
* naming) in one scan, seeding each class's record, which gets its verdict (dropped) as the ladder gives it; the
|
||||||
* the ladder gives it; then pending_classes' `waiting` counted afresh from the rows. --report shows the record.
|
* waiting rows in windows of created_at by their index (a scan per window starved the prober on the HDD); then
|
||||||
|
* pending_classes' `waiting` counted afresh from the claim's index. --report shows the record.
|
||||||
*/
|
*/
|
||||||
export default class PendingClasses extends BaseCommand {
|
export default class PendingClasses extends BaseCommand {
|
||||||
static commandName = 'pending:classes'
|
static commandName = 'pending:classes'
|
||||||
@@ -25,45 +26,52 @@ export default class PendingClasses extends BaseCommand {
|
|||||||
async run() {
|
async run() {
|
||||||
if (this.report) return this.showReport()
|
if (this.report) return this.showReport()
|
||||||
const hours = this.hours ?? 6
|
const hours = this.hours ?? 6
|
||||||
const { rows: span } = await db.rawQuery(
|
const started = Date.now()
|
||||||
`SELECT min(created_at) AS lo, max(created_at) AS hi FROM pending_releases WHERE klass = 0`
|
// The finished rows (named, exhausted, naming; 6.5M on 2026-11-02) in one pass: a scan of the table, no index
|
||||||
|
// reaches them by status; they seed each class's record.
|
||||||
|
const { rows: seeded } = await db.rawQuery(
|
||||||
|
`WITH classed AS (
|
||||||
|
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
||||||
|
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
||||||
|
WHERE klass = 0 AND status IN (1, 3, 4)
|
||||||
|
RETURNING tool, group_id, klass, status),
|
||||||
|
upserted AS (
|
||||||
|
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
|
||||||
|
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed GROUP BY 1, 2, 3
|
||||||
|
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
|
||||||
|
RETURNING 1)
|
||||||
|
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM upserted) AS classes`
|
||||||
)
|
)
|
||||||
if (!span[0].lo) {
|
this.logger.info(
|
||||||
this.logger.info('No rows without a class.')
|
`Finished rows: ${seeded[0].classed} classed into ${seeded[0].classes} classes (${Math.round((Date.now() - started) / 1000)} s).`
|
||||||
} else {
|
)
|
||||||
|
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
|
||||||
|
await db.rawQuery(
|
||||||
|
`UPDATE pending_classes SET dropped = true, swept_at = NULL WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
|
||||||
|
[pendingMinProbes, pendingMaxNamed]
|
||||||
|
)
|
||||||
|
// The waiting rows by windows of created_at (pending_releases_waiting_idx reaches them).
|
||||||
|
const { rows: span } = await db.rawQuery(
|
||||||
|
`SELECT (SELECT min(created_at) FROM pending_releases WHERE status = 0) AS lo, (SELECT max(created_at) FROM pending_releases WHERE status = 0) AS hi`
|
||||||
|
)
|
||||||
|
if (span[0].lo) {
|
||||||
let from = new Date(span[0].lo)
|
let from = new Date(span[0].lo)
|
||||||
const hi = new Date(span[0].hi)
|
const hi = new Date(span[0].hi)
|
||||||
let classed = 0
|
let classed = 0
|
||||||
const started = Date.now()
|
|
||||||
while (from <= hi) {
|
while (from <= hi) {
|
||||||
const to = new Date(from.getTime() + hours * 3600_000)
|
const to = new Date(from.getTime() + hours * 3600_000)
|
||||||
// Rows of the window classed; the finished ones among them (named, exhausted, naming) seed the record.
|
const result = await db.rawQuery(
|
||||||
const { rows } = await db.rawQuery(
|
`UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
||||||
`WITH classed AS (
|
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
||||||
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
WHERE status = 0 AND created_at >= ? AND created_at < ? AND klass = 0`,
|
||||||
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
|
||||||
WHERE klass = 0 AND created_at >= ? AND created_at < ?
|
|
||||||
RETURNING tool, group_id, klass, status),
|
|
||||||
seeded AS (
|
|
||||||
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
|
|
||||||
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed WHERE status IN (1, 3, 4) GROUP BY 1, 2, 3
|
|
||||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
|
|
||||||
RETURNING 1)
|
|
||||||
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM seeded) AS seeded`,
|
|
||||||
[from.toISOString(), to.toISOString()]
|
[from.toISOString(), to.toISOString()]
|
||||||
)
|
)
|
||||||
classed += Number(rows[0].classed)
|
classed += result.rowCount ?? 0
|
||||||
from = to
|
from = to
|
||||||
this.logger.info(
|
this.logger.info(
|
||||||
`Up to ${to.toISOString().slice(0, 16)}: ${classed} rows classed (${Math.round((Date.now() - started) / 1000)} s).`
|
`Waiting rows up to ${to.toISOString().slice(0, 16)}: ${classed} classed (${Math.round((Date.now() - started) / 1000)} s).`
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
// The record's verdicts on the seeded classes.
|
|
||||||
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
|
|
||||||
await db.rawQuery(
|
|
||||||
`UPDATE pending_classes SET dropped = true WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
|
|
||||||
[pendingMinProbes, pendingMaxNamed]
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
// The waiting counts from the rows themselves (the claim, the discard and the batch store keep them from here).
|
// The waiting counts from the rows themselves (the claim, the discard and the batch store keep them from here).
|
||||||
await db.rawQuery(`
|
await db.rawQuery(`
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The prober's start puts a previous run's claims back (PendingLadder.release: status 9 → 0), which scanned the
|
||||||
|
* 58M-row pending table on the HDD for 20+ minutes after every deploy (LOG.md (130)): a partial index on the few
|
||||||
|
* taken rows makes it instant.
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
static disableTransactions = true
|
||||||
|
|
||||||
|
async up() {
|
||||||
|
this.schema.raw(
|
||||||
|
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_taken_idx ON pending_releases (release_key) WHERE status = 9`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {
|
||||||
|
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_taken_idx`)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The prober's start puts a previous run's claims back (probe_candidates status taken → waiting), which scanned
|
||||||
|
* the 7.8 GB candidates table on the HDD for 4 minutes after every deploy (LOG.md (133)): a partial index on the
|
||||||
|
* few taken rows makes it instant.
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
static disableTransactions = true
|
||||||
|
|
||||||
|
async up() {
|
||||||
|
this.schema.raw(
|
||||||
|
`CREATE INDEX CONCURRENTLY IF NOT EXISTS probe_candidates_taken_idx ON probe_candidates (id) WHERE status = 'taken'`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {
|
||||||
|
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS probe_candidates_taken_idx`)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The ladder's discard sweep (PendingLadder.discard) reads a dropped signature's or class's waiting rows once:
|
||||||
|
* `swept_at` marks it done (null again when the record drops it anew). Before, every sweep read every dropped
|
||||||
|
* signature's rows over again, minutes of HDD each, inside the claim (LOG.md (133)).
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
async up() {
|
||||||
|
this.schema.alterTable('pending_signatures', (table) => {
|
||||||
|
table.timestamp('swept_at', { useTz: true }).nullable()
|
||||||
|
})
|
||||||
|
this.schema.alterTable('pending_classes', (table) => {
|
||||||
|
table.timestamp('swept_at', { useTz: true }).nullable()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {
|
||||||
|
this.schema.alterTable('pending_signatures', (table) => {
|
||||||
|
table.dropColumn('swept_at')
|
||||||
|
})
|
||||||
|
this.schema.alterTable('pending_classes', (table) => {
|
||||||
|
table.dropColumn('swept_at')
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The discard sweep of a dropped signature (tool, group, poster) read the whole pending_releases table per
|
||||||
|
* signature: the planner took a parallel seq scan over the poster test, four minutes on the HDD that starved the
|
||||||
|
* lanes' claims (LOG.md (134)). The waiting rows by signature make it an index range. By the poster's md5: a
|
||||||
|
* poster of 2.7 kB broke the first build (a btree key is at most a third of a page), which left an invalid index
|
||||||
|
* behind that IF NOT EXISTS would have taken for the real one; the drop clears it.
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
static disableTransactions = true
|
||||||
|
|
||||||
|
async up() {
|
||||||
|
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
|
||||||
|
this.schema.raw(
|
||||||
|
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_signature_idx ON pending_releases (group_id, tool, md5(poster)) WHERE status = 0`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {
|
||||||
|
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* probe_candidates, with its indexes, in the catalog's tablespace (default_tablespace: nvme on homelab), out of
|
||||||
|
* CHUNK_TABLESPACE where migration 6000 put it: the prober's catalog claim updates it from random points every
|
||||||
|
* minute, and on the HDD, beside the pending ladder's sweep and autovacuum, a claim took 2–3 minutes with all the
|
||||||
|
* lanes waiting (LOG.md 2026-11-03 (136)). Homelab moved it by hand (edge, node-api and the workers stopped; ~8 GB);
|
||||||
|
* a no-op where it's there already, or without a default tablespace (tests).
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
async up() {
|
||||||
|
this.defer(async (db) => {
|
||||||
|
const { rows: setting } = await db.rawQuery(
|
||||||
|
`SELECT current_setting('default_tablespace') AS target`
|
||||||
|
)
|
||||||
|
const target = setting[0].target
|
||||||
|
if (!target) return
|
||||||
|
const { rows } = await db.rawQuery(
|
||||||
|
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||||
|
WHERE (c.relname = 'probe_candidates' OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = 'probe_candidates'::regclass))
|
||||||
|
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
|
||||||
|
ORDER BY c.relkind DESC, c.relname`,
|
||||||
|
[target]
|
||||||
|
)
|
||||||
|
for (const { relname, relkind } of rows) {
|
||||||
|
await db.rawQuery(
|
||||||
|
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Everything the prober reads or writes in the catalog's tablespace (default_tablespace: nvme on homelab), out of
|
||||||
|
* CHUNK_TABLESPACE (the HDD): pending_releases with its two HDD indexes (migration 6300), release_inspections
|
||||||
|
* (6000), probe_cache with its toast. The edge's batch upserts into pending_releases took minutes each on the HDD and
|
||||||
|
* the prober's flushes waited on them (LOG.md 2026-11-03 (137)); and with these on the NVMe the prober can run
|
||||||
|
* through the night without the HDD. Homelab moved them by hand (edge, node-api and the workers stopped; ~66 GB);
|
||||||
|
* a no-op where they're there already, or without a default tablespace (tests).
|
||||||
|
*/
|
||||||
|
export default class extends BaseSchema {
|
||||||
|
async up() {
|
||||||
|
this.defer(async (db) => {
|
||||||
|
const { rows: setting } = await db.rawQuery(
|
||||||
|
`SELECT current_setting('default_tablespace') AS target`
|
||||||
|
)
|
||||||
|
const target = setting[0].target
|
||||||
|
if (!target) return
|
||||||
|
for (const table of ['pending_releases', 'release_inspections', 'probe_cache']) {
|
||||||
|
const { rows } = await db.rawQuery(
|
||||||
|
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||||
|
WHERE (c.relname = ? OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = ?::regclass))
|
||||||
|
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
|
||||||
|
ORDER BY c.relkind DESC, c.relname`,
|
||||||
|
[table, table, target]
|
||||||
|
)
|
||||||
|
for (const { relname, relkind } of rows) {
|
||||||
|
await db.rawQuery(
|
||||||
|
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async down() {}
|
||||||
|
}
|
||||||
+3
-2
@@ -47,8 +47,9 @@ echo "== recreating app services"
|
|||||||
docker compose stop edge
|
docker compose stop edge
|
||||||
if [ -e "$CACHE/paused" ]; then
|
if [ -e "$CACHE/paused" ]; then
|
||||||
# Paused for the night (~/bin/usenet-rebuild-pause.sh): the workers stay stopped, the resume starts the new build.
|
# Paused for the night (~/bin/usenet-rebuild-pause.sh): the workers stay stopped, the resume starts the new build.
|
||||||
echo "== paused: coordinator, prober, merger, judge, spotter, spot-inspector and edge left stopped"
|
# The prober runs through the night (2026-11-03, LOG.md (137)), so it gets the new build now.
|
||||||
docker compose up -d --no-deps node-api web
|
echo "== paused: coordinator, merger, judge, spotter, spot-inspector and edge left stopped"
|
||||||
|
docker compose up -d --no-deps node-api web prober
|
||||||
else
|
else
|
||||||
docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
|
docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
|
||||||
if [ -e "$CACHE/no-edge" ]; then
|
if [ -e "$CACHE/no-edge" ]; then
|
||||||
|
|||||||
@@ -1,12 +1,13 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# Installed as ~/bin/usenet-rebuild-pause.sh (by hand). Nightly (homelab crontab, 23:45): voyager 2's processing paused for the night; usenet-rebuild-resume.sh (07:45)
|
# Installed as ~/bin/usenet-rebuild-pause.sh (by hand). Nightly (homelab crontab, 23:45): voyager 2's processing paused for the night; usenet-rebuild-resume.sh (07:45)
|
||||||
# starts it again. Postgres, web and node-api stay up. Takes the deploy lock, so it never runs during a CI deploy, and
|
# starts it again. Postgres, web, node-api and the prober stay up: since 2026-11-03 (LOG.md (137)) everything the
|
||||||
# leaves .cache/paused, so a deploy in the night doesn't start the workers again.
|
# prober touches is on the NVMe, so it probes through the night without the HDD. Takes the deploy lock, so it never
|
||||||
|
# runs during a CI deploy, and leaves .cache/paused, so a deploy in the night doesn't start the workers again.
|
||||||
cd /mnt/bulk/usenet-rebuild || exit 1
|
cd /mnt/bulk/usenet-rebuild || exit 1
|
||||||
exec 9>.cache/deploy.lock
|
exec 9>.cache/deploy.lock
|
||||||
flock 9
|
flock 9
|
||||||
touch .cache/paused
|
touch .cache/paused
|
||||||
date "+%F %T pausing"
|
date "+%F %T pausing"
|
||||||
docker compose stop edge
|
docker compose stop edge
|
||||||
docker compose stop coordinator prober merger judge spotter spot-inspector
|
docker compose stop coordinator merger judge spotter spot-inspector
|
||||||
docker compose ps --format "{{.Service}} {{.Status}}"
|
docker compose ps --format "{{.Service}} {{.Status}}"
|
||||||
+31
-2
@@ -1608,8 +1608,13 @@ fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate>
|
|||||||
if let Some((_, f)) = tail {
|
if let Some((_, f)) = tail {
|
||||||
out.push(Candidate { release_key: key.to_string(), upload, kind: "7z-tail", message_id: f.last().unwrap().message_id.clone(), file: f[0].file.name.clone(), bytes: f.last().unwrap().bytes });
|
out.push(Candidate { release_key: key.to_string(), upload, kind: "7z-tail", message_id: f.last().unwrap().message_id.clone(), file: f[0].file.name.clone(), bytes: f.last().unwrap().bytes });
|
||||||
}
|
}
|
||||||
if out.is_empty() && files.len() == 1 && is_random_file(&files[0][0].file.name) {
|
// No set and no archive: the first random-named file's first segment, for its yEnc name or a twin by content
|
||||||
out.push(c("lone", &files[0][0]));
|
// (hash_twins.ts). Any count of files: a hash-named release of several pieces (Bob's, LOG.md (140)) had no
|
||||||
|
// target at all, so the ladder never probed it, never named it and never learned its class.
|
||||||
|
if out.is_empty() {
|
||||||
|
if let Some(f) = files.iter().find(|f| is_random_file(&f[0].file.name)) {
|
||||||
|
out.push(c("lone", &f[0]));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
out.truncate(4);
|
out.truncate(4);
|
||||||
out
|
out
|
||||||
@@ -1738,6 +1743,30 @@ mod tests {
|
|||||||
assert_eq!(ix.pending(), 0);
|
assert_eq!(ix.pending(), 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_multi_file_random_release_gets_a_lone_target() {
|
||||||
|
let mut priorities = FxHashMap::default();
|
||||||
|
priorities.insert(GROUP.to_string(), 1);
|
||||||
|
let mut ix = RangeIndexer::new(GROUP, 1, 10_000, 1, Rules::defaults(priorities), 5);
|
||||||
|
// Three hash-named pieces of one hash-named upload, five segments each (more than a per-article post): no
|
||||||
|
// archive, no par2, no 7z volume.
|
||||||
|
let mut lines = Vec::new();
|
||||||
|
for (i, name) in ["9e7fc2741cdd450287aff2733ca4c193", "c5ebdd5487f5b132c76e8d5c2572e560", "8114c42c9694380c8b25857e8b3331c3"].iter().enumerate() {
|
||||||
|
for part in 0..5 {
|
||||||
|
lines.push(line(100 + (i * 5 + part) as i64, &format!(r#"b3a75d234aa73891e78a70fc1f5496b1 [{}/3] - "{name}" yEnc ({}/5)"#, i + 1, part + 1), (i * 5 + part) as i64));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ix.add_text(&lines.join("\n"), 10_001);
|
||||||
|
ix.finish();
|
||||||
|
let batch = ix.take();
|
||||||
|
assert_eq!(batch.releases.len(), 1, "counts {:?}", batch.counts);
|
||||||
|
let r = &batch.releases[0];
|
||||||
|
assert!(!r.readable && r.files == 3 && r.segments == 15, "readable {} files {} segments {}", r.readable, r.files, r.segments);
|
||||||
|
let kinds: Vec<_> = batch.candidates.iter().map(|c| (c.kind, c.file.as_str())).collect();
|
||||||
|
assert_eq!(kinds, vec![("lone", "9e7fc2741cdd450287aff2733ca4c193")]);
|
||||||
|
assert_eq!(batch.candidates[0].message_id, "u100@test");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn album_templates_and_names() {
|
fn album_templates_and_names() {
|
||||||
assert_eq!(album_template("1481439079025_014.jpg.data").as_deref(), Some("#_#.jpg.data"));
|
assert_eq!(album_template("1481439079025_014.jpg.data").as_deref(), Some("#_#.jpg.data"));
|
||||||
|
|||||||
@@ -77,6 +77,8 @@ pub struct RulesJson {
|
|||||||
pub drop_stub_archives: Option<bool>,
|
pub drop_stub_archives: Option<bool>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub drop_nonsense_pieces: Option<bool>,
|
pub drop_nonsense_pieces: Option<bool>,
|
||||||
|
#[serde(default)]
|
||||||
|
pub drop_tiny_archives: Option<bool>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RulesJson {
|
impl RulesJson {
|
||||||
@@ -107,6 +109,9 @@ impl RulesJson {
|
|||||||
if let Some(d) = self.drop_nonsense_pieces {
|
if let Some(d) = self.drop_nonsense_pieces {
|
||||||
rules.drop_nonsense_pieces = d;
|
rules.drop_nonsense_pieces = d;
|
||||||
}
|
}
|
||||||
|
if let Some(d) = self.drop_tiny_archives {
|
||||||
|
rules.drop_tiny_archives = d;
|
||||||
|
}
|
||||||
rules
|
rules
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
+80
-6
@@ -1115,7 +1115,7 @@ fn token_signals(t: &[u8]) -> TokenSignals {
|
|||||||
let mut lower = 0;
|
let mut lower = 0;
|
||||||
for i in 0..t.len() {
|
for i in 0..t.len() {
|
||||||
let c = t[i];
|
let c = t[i];
|
||||||
if c.is_ascii_alphabetic() && !is_vowel(c) {
|
if c.is_ascii_alphabetic() && !is_vowel(c) && c.to_ascii_lowercase() != b'y' {
|
||||||
consonants += 1;
|
consonants += 1;
|
||||||
s.max_consonants = s.max_consonants.max(consonants);
|
s.max_consonants = s.max_consonants.max(consonants);
|
||||||
} else {
|
} else {
|
||||||
@@ -1181,7 +1181,9 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
|
|||||||
return if n >= 3 { TokenKind::Word } else { TokenKind::Neutral };
|
return if n >= 3 { TokenKind::Word } else { TokenKind::Neutral };
|
||||||
}
|
}
|
||||||
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
|
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
|
||||||
if n >= 6 && s.case_switches >= 2 && (s.inner_upper || (s.vowels * 4 < n && (s.runs >= 4 || s.lone >= 1))) {
|
// Two CamelCase words with few vowels (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone
|
||||||
|
// letter; a scramble has more runs, a lone letter or an inner capital run.
|
||||||
|
if n >= 6 && s.case_switches >= 2 && s.vowels * 4 < n && (s.runs >= 5 || s.lone >= 1 || s.inner_upper) {
|
||||||
return TokenKind::Random;
|
return TokenKind::Random;
|
||||||
}
|
}
|
||||||
return TokenKind::Neutral;
|
return TokenKind::Neutral;
|
||||||
@@ -1193,6 +1195,14 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
|
|||||||
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
||||||
return if n >= 12 && token_signals(t).switches >= 6 { TokenKind::Random } else { TokenKind::Neutral };
|
return if n >= 12 && token_signals(t).switches >= 6 { TokenKind::Random } else { TokenKind::Neutral };
|
||||||
}
|
}
|
||||||
|
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
|
||||||
|
if t.split(|c: &u8| !c.is_ascii_alphabetic()).filter(|w| !w.is_empty()).all(|w| {
|
||||||
|
let vowels = w.iter().filter(|c| matches!(c.to_ascii_lowercase(), b'a' | b'e' | b'i' | b'o' | b'u' | b'y')).count();
|
||||||
|
(w.len() >= 2 && w.iter().all(u8::is_ascii_uppercase))
|
||||||
|
|| (w.len() >= 2 && w[1..].iter().all(u8::is_ascii_lowercase) && (w.len() >= 3 || w[0].is_ascii_lowercase()) && vowels * 5 >= w.len())
|
||||||
|
}) {
|
||||||
|
return TokenKind::Neutral;
|
||||||
|
}
|
||||||
let s = token_signals(&scene_case(t));
|
let s = token_signals(&scene_case(t));
|
||||||
if n >= 5 && s.switches >= 3 {
|
if n >= 5 && s.switches >= 3 {
|
||||||
return TokenKind::Random;
|
return TokenKind::Random;
|
||||||
@@ -1505,6 +1515,11 @@ static RELEASE_LIKE: LazyLock<Regex> = LazyLock::new(|| {
|
|||||||
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
|
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/// RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way.
|
||||||
|
static RELEASE_LIKE_SPACED: LazyLock<Regex> = LazyLock::new(|| {
|
||||||
|
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
|
||||||
|
});
|
||||||
|
|
||||||
/// The stub-archive rule (LOG.md (123)-(126), (128)): one unsplit archive, no par2, settled, posted to 2+ groups,
|
/// The stub-archive rule (LOG.md (123)-(126), (128)): one unsplit archive, no par2, settled, posted to 2+ groups,
|
||||||
/// under STUB_MAX_BYTES (the fakes stop at ~90 MB; the hash-named uploads, real albums and passworded videos above
|
/// under STUB_MAX_BYTES (the fakes stop at ~90 MB; the hash-named uploads, real albums and passworded videos above
|
||||||
/// it don't), and its articles at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES (Power-Post's
|
/// it don't), and its articles at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES (Power-Post's
|
||||||
@@ -1518,6 +1533,8 @@ pub const STUB_WIDE_GROUPS: usize = 4;
|
|||||||
/// release-like name that isn't scrambled (the newznzb.* pieces; under 100 kB are cue/log sidecars and adverts).
|
/// release-like name that isn't scrambled (the newznzb.* pieces; under 100 kB are cue/log sidecars and adverts).
|
||||||
pub const NONSENSE_MIN_BYTES: u64 = 100_000;
|
pub const NONSENSE_MIN_BYTES: u64 = 100_000;
|
||||||
pub const NONSENSE_MAX_BYTES: u64 = 1_000_000;
|
pub const NONSENSE_MAX_BYTES: u64 = 1_000_000;
|
||||||
|
/// The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars.
|
||||||
|
pub const TINY_MIN_BYTES: u64 = 2_000;
|
||||||
|
|
||||||
pub struct JudgedFile<'a> {
|
pub struct JudgedFile<'a> {
|
||||||
pub name: &'a str,
|
pub name: &'a str,
|
||||||
@@ -1579,6 +1596,27 @@ pub fn is_nonsense_piece(f: &JudgedFile) -> bool {
|
|||||||
&& !is_random_file(f.name)
|
&& !is_random_file(f.name)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
|
||||||
|
/// exe is the content), covers and sidecars.
|
||||||
|
static TINY_CONTENT: LazyLock<Regex> = LazyLock::new(|| {
|
||||||
|
re(r"(?i)\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b")
|
||||||
|
});
|
||||||
|
|
||||||
|
/// A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
|
||||||
|
/// unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
|
||||||
|
/// subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
|
||||||
|
pub fn is_tiny_archive(f: &JudgedFile) -> bool {
|
||||||
|
f.total_parts == 1
|
||||||
|
&& f.segments == 1
|
||||||
|
&& (TINY_MIN_BYTES..NONSENSE_MIN_BYTES).contains(&f.bytes)
|
||||||
|
&& !f.groups.iter().any(|g| test(&STUB_EXEMPT_GROUPS, g))
|
||||||
|
&& test(&RELEASE_LIKE_SPACED, &file_stem(f.name))
|
||||||
|
&& !test(&EBOOK, f.name)
|
||||||
|
&& !test(&TINY_CONTENT, f.name)
|
||||||
|
&& !looks_scrambled(f.name)
|
||||||
|
&& !is_random_file(f.name)
|
||||||
|
}
|
||||||
|
|
||||||
pub enum Verdict {
|
pub enum Verdict {
|
||||||
Keep,
|
Keep,
|
||||||
Drop { rule: &'static str, key: String, sample: bool },
|
Drop { rule: &'static str, key: String, sample: bool },
|
||||||
@@ -1600,9 +1638,11 @@ pub struct Rules {
|
|||||||
pub drop_executables: bool,
|
pub drop_executables: bool,
|
||||||
/// Fragments of the poster lowercased without punctuation (droppedPosters).
|
/// Fragments of the poster lowercased without punctuation (droppedPosters).
|
||||||
pub dropped_posters: Vec<String>,
|
pub dropped_posters: Vec<String>,
|
||||||
/// The stub-archive rule (dropStubArchives) and the nonsense-piece rule (dropNonsensePieces).
|
/// The stub-archive rule (dropStubArchives), the nonsense-piece rule (dropNonsensePieces) and the tiny-archive
|
||||||
|
/// rule (dropTinyArchives).
|
||||||
pub drop_stub_archives: bool,
|
pub drop_stub_archives: bool,
|
||||||
pub drop_nonsense_pieces: bool,
|
pub drop_nonsense_pieces: bool,
|
||||||
|
pub drop_tiny_archives: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Rules {
|
impl Rules {
|
||||||
@@ -1620,6 +1660,7 @@ impl Rules {
|
|||||||
dropped_posters: ["celebmovie", "celebwow", "celebfx", "iluvcelebs", "nudecelebs", "celebsextapes"].iter().map(|p| p.to_string()).collect(),
|
dropped_posters: ["celebmovie", "celebwow", "celebfx", "iluvcelebs", "nudecelebs", "celebsextapes"].iter().map(|p| p.to_string()).collect(),
|
||||||
drop_stub_archives: true,
|
drop_stub_archives: true,
|
||||||
drop_nonsense_pieces: true,
|
drop_nonsense_pieces: true,
|
||||||
|
drop_tiny_archives: true,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1655,6 +1696,9 @@ pub fn judge(release_key: &str, files: &[JudgedFile], settled: bool, rules: &Rul
|
|||||||
if rules.drop_nonsense_pieces && is_nonsense_piece(f) {
|
if rules.drop_nonsense_pieces && is_nonsense_piece(f) {
|
||||||
return Verdict::Drop { rule: "nonsense-piece", key: upload_tool(f.first_message_id).to_string(), sample };
|
return Verdict::Drop { rule: "nonsense-piece", key: upload_tool(f.first_message_id).to_string(), sample };
|
||||||
}
|
}
|
||||||
|
if rules.drop_tiny_archives && is_tiny_archive(f) {
|
||||||
|
return Verdict::Drop { rule: "tiny-archive", key: upload_tool(f.first_message_id).to_string(), sample };
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if files.iter().any(|f| !is_random_file(f.name)) {
|
if files.iter().any(|f| !is_random_file(f.name)) {
|
||||||
return Verdict::Keep;
|
return Verdict::Keep;
|
||||||
@@ -1854,7 +1898,8 @@ mod tests {
|
|||||||
"Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing", "Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON",
|
"Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing", "Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON",
|
||||||
"Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr", "Eclipse_IDE_2023-12_R_win32_x86_64.zip", "PartsHPGL 1.06c", "Go4Customer Fax Machine v1.2b",
|
"Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr", "Eclipse_IDE_2023-12_R_win32_x86_64.zip", "PartsHPGL 1.06c", "Go4Customer Fax Machine v1.2b",
|
||||||
"AdBeGone 1.2", "3DMark03 320", "COD4MW2REMASTERED", "Movie.2020.1080p-GRP [a3f9e2b1]", "EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps",
|
"AdBeGone 1.2", "3DMark03 320", "COD4MW2REMASTERED", "Movie.2020.1080p-GRP [a3f9e2b1]", "EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps",
|
||||||
"iPhone", "Some.Show.S01E01.1080p.mkv",
|
"iPhone", "Some.Show.S01E01.1080p.mkv", "FastFilms 2.04", "BestCrypt NP v4.0", "EasyHTML 2.35 German", "SchiffsModell - September 2026",
|
||||||
|
"CrossStitcher - Issue 440, October 2026", "Laura15Juli2026", "IHaveAWife - Isa Bella", "DickHDDaily - Santana Red", "1904-SmithsWork-Hasluck",
|
||||||
] {
|
] {
|
||||||
assert!(!is_unreadable_name(n), "{n}");
|
assert!(!is_unreadable_name(n), "{n}");
|
||||||
}
|
}
|
||||||
@@ -1982,8 +2027,8 @@ mod single_archive_rule_tests {
|
|||||||
let rules = Rules::defaults(Default::default());
|
let rules = Rules::defaults(Default::default());
|
||||||
// One 790 kB article under a release-like name: the newznzb.* pieces.
|
// One 790 kB article under a release-like name: the newznzb.* pieces.
|
||||||
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
|
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
|
||||||
// A 3 kB cue/log sidecar rar with a year in its name, and a diet advert: under the floor.
|
// A 3 kB cue/log sidecar rar with a year in its name: under this rule's floor (the tiny-archive rule's outside the music groups).
|
||||||
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), None);
|
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||||
// A book (a year or a format in its name, written with spaces), a dotted one too: not a piece.
|
// A book (a year or a format in its name, written with spaces), a dotted one too: not a piece.
|
||||||
assert_eq!(verdict("Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar", "fake@address.com (Psychopath)", 2, 1, 303_000, true, &rules), None);
|
assert_eq!(verdict("Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar", "fake@address.com (Psychopath)", 2, 1, 303_000, true, &rules), None);
|
||||||
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 382_000, true, &rules), None);
|
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 382_000, true, &rules), None);
|
||||||
@@ -1995,6 +2040,35 @@ mod single_archive_rule_tests {
|
|||||||
let off = Rules { drop_nonsense_pieces: false, ..Rules::defaults(Default::default()) };
|
let off = Rules { drop_nonsense_pieces: false, ..Rules::defaults(Default::default()) };
|
||||||
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &off), None);
|
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &off), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn tiny_archives() {
|
||||||
|
let rules = Rules::defaults(Default::default());
|
||||||
|
// One article of 2-100 kB under a release name, dotted or with spaces: a RAR holding "<title>.exe" (LOG.md (131)).
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar", "Yanc@power-post.org <Yanc@power-post.org>", 1, 1, 15_084, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||||
|
assert_eq!(verdict("Cicero Magazin fuer politische Kultur Mai 05 2016.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 17_478, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||||
|
// A scene name without a year, a tag or a group suffix: not release-like, so kept (the flood's porn names mostly carry one).
|
||||||
|
assert_eq!(verdict("[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar", "Yanc@power-post.org", 2, 1, 23_449, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 7_869, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||||
|
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
|
||||||
|
assert_eq!(verdict("The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar", "henk@example.org", 1, 1, 38_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar", "poster@example.org", 1, 1, 60_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Some Band - Album (2011) covers.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 38_000, true, &rules), None);
|
||||||
|
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's.
|
||||||
|
let groups = vec!["alt.binaries.sounds.karaoke".to_string()];
|
||||||
|
let files = [JudgedFile { name: "KV-1055806 - Walker Hayes - U Gurl (2017).zip", total_parts: 1, groups: &groups, first_message_id: NYUU, poster: "home@home.com", segments: 1, bytes: 60_000 }];
|
||||||
|
assert!(matches!(judge("k", &files, true, &rules), Verdict::Keep));
|
||||||
|
assert_eq!(verdict("holiday snaps.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("nYDeXeF49dieZ1J7flK-JFzZ3j.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 2, 20_000, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 1_500, true, &rules), None);
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 150_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, false, &rules), None);
|
||||||
|
let off = Rules { drop_tiny_archives: false, ..Rules::defaults(Default::default()) };
|
||||||
|
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, true, &off), None);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ export interface ProbeCandidate {
|
|||||||
|
|
||||||
/** What a rule dropped in a batch (REBUILD.md, rules and recovery). */
|
/** What a rule dropped in a batch (REBUILD.md, rules and recovery). */
|
||||||
export interface RuleCount {
|
export interface RuleCount {
|
||||||
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'incomplete'
|
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive' | 'incomplete'
|
||||||
key: string
|
key: string
|
||||||
articles: number
|
articles: number
|
||||||
files: number
|
files: number
|
||||||
|
|||||||
+15
-3
@@ -43,7 +43,7 @@ function signals(t: string) {
|
|||||||
let maxConsonants = 0
|
let maxConsonants = 0
|
||||||
for (let i = 0; i < t.length; i++) {
|
for (let i = 0; i < t.length; i++) {
|
||||||
const c = t[i]
|
const c = t[i]
|
||||||
if (/[b-df-hj-np-tv-z]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
|
if (/[b-df-hj-np-tv-xz]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
|
||||||
else consonants = 0
|
else consonants = 0
|
||||||
if (i === 0) continue
|
if (i === 0) continue
|
||||||
const a = t[i - 1]
|
const a = t[i - 1]
|
||||||
@@ -90,11 +90,14 @@ export function tokenKind(t: string): 'word' | 'random' | null {
|
|||||||
if (t.length >= 10 && s.vowels * 6 < t.length) return 'random'
|
if (t.length >= 10 && s.vowels * 6 < t.length) return 'random'
|
||||||
return t.length >= 3 ? 'word' : null
|
return t.length >= 3 ? 'word' : null
|
||||||
}
|
}
|
||||||
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
|
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble. Two CamelCase words with few vowels
|
||||||
|
// (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone letter; a scramble has more runs, a lone
|
||||||
|
// letter or an inner capital run.
|
||||||
if (
|
if (
|
||||||
t.length >= 6 &&
|
t.length >= 6 &&
|
||||||
s.caseSwitches >= 2 &&
|
s.caseSwitches >= 2 &&
|
||||||
(s.innerUpper || (s.vowels * 4 < t.length && (s.runs >= 4 || s.lone >= 1)))
|
s.vowels * 4 < t.length &&
|
||||||
|
(s.runs >= 5 || s.lone >= 1 || s.innerUpper)
|
||||||
)
|
)
|
||||||
return 'random'
|
return 'random'
|
||||||
return null
|
return null
|
||||||
@@ -104,6 +107,15 @@ export function tokenKind(t: string): 'word' | 'random' | null {
|
|||||||
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
||||||
return t.length >= 12 && signals(t).switches >= 6 ? 'random' : null
|
return t.length >= 12 && signals(t).switches >= 6 ? 'random' : null
|
||||||
}
|
}
|
||||||
|
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
|
||||||
|
if (
|
||||||
|
(t.match(/[A-Za-z]+/g) ?? []).every(
|
||||||
|
(w) =>
|
||||||
|
/^[A-Z]{2,}$/.test(w) ||
|
||||||
|
(/^[A-Z]?[a-z]{2,}$/.test(w) && (w.match(/[aeiouy]/g) ?? []).length * 5 >= w.length)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return null
|
||||||
const s = signals(sceneCase(t))
|
const s = signals(sceneCase(t))
|
||||||
if (t.length >= 5 && s.switches >= 3) return 'random'
|
if (t.length >= 5 && s.switches >= 3) return 'random'
|
||||||
if (t.length >= 6 && s.switches >= 2 && (s.lone >= 1 || s.innerUpper || s.maxConsonants >= 4))
|
if (t.length >= 6 && s.switches >= 2 && (s.lone >= 1 || s.innerUpper || s.maxConsonants >= 4))
|
||||||
|
|||||||
+1
-1
@@ -1,2 +1,2 @@
|
|||||||
export { uploadTool, signatureGroup, signatureKey, explored } from './upload_tools.js'
|
export { uploadTool, signatureGroup, signatureKey, explored } from './upload_tools.js'
|
||||||
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
|
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, isTinyArchive, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, TINY_MIN_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
|
||||||
+43
-1
@@ -47,6 +47,12 @@ export interface Rules {
|
|||||||
* the newznzb.* pieces (9% of the catalog; LOG.md (123), (125)).
|
* the newznzb.* pieces (9% of the catalog; LOG.md (123), (125)).
|
||||||
*/
|
*/
|
||||||
dropNonsensePieces: boolean
|
dropNonsensePieces: boolean
|
||||||
|
/**
|
||||||
|
* Drop one-article archives of TINY_MIN_BYTES to NONSENSE_MIN_BYTES under a release-like name (spaces allowed),
|
||||||
|
* outside the exempt groups, without an e-book, subtitle, crack or sidecar word: the 2015-17 exe-stub flood in
|
||||||
|
* miniature (a RAR holding "<title>.exe" at 2-60 kB; 1.65M catalog rows; LOG.md (131), (133)).
|
||||||
|
*/
|
||||||
|
dropTinyArchives: boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
|
export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
|
||||||
@@ -58,6 +64,7 @@ export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
|
|||||||
droppedPosters: ['celebmovie', 'celebwow', 'celebfx', 'iluvcelebs', 'nudecelebs', 'celebsextapes'],
|
droppedPosters: ['celebmovie', 'celebwow', 'celebfx', 'iluvcelebs', 'nudecelebs', 'celebsextapes'],
|
||||||
dropStubArchives: true,
|
dropStubArchives: true,
|
||||||
dropNonsensePieces: true,
|
dropNonsensePieces: true,
|
||||||
|
dropTinyArchives: true,
|
||||||
droppedClasses: [],
|
droppedClasses: [],
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,8 +76,12 @@ export const STUB_WIDE_GROUPS = 4
|
|||||||
/** The nonsense-piece rule's bounds: under 100 kB are cue/log sidecars and adverts, 1 MB and up is rule 1's. */
|
/** The nonsense-piece rule's bounds: under 100 kB are cue/log sidecars and adverts, 1 MB and up is rule 1's. */
|
||||||
export const NONSENSE_MIN_BYTES = 100_000
|
export const NONSENSE_MIN_BYTES = 100_000
|
||||||
export const NONSENSE_MAX_BYTES = 1_000_000
|
export const NONSENSE_MAX_BYTES = 1_000_000
|
||||||
|
/** The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars. */
|
||||||
|
export const TINY_MIN_BYTES = 2_000
|
||||||
|
|
||||||
export type Verdict = { rule: null } | { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece'; key: string; sample: boolean }
|
export type Verdict =
|
||||||
|
| { rule: null }
|
||||||
|
| { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive'; key: string; sample: boolean }
|
||||||
|
|
||||||
/** A file as the judge sees it: its name, parts, groups, poster, and the articles seen with their bytes. */
|
/** A file as the judge sees it: its name, parts, groups, poster, and the articles seen with their bytes. */
|
||||||
export interface JudgedFile {
|
export interface JudgedFile {
|
||||||
@@ -148,6 +159,36 @@ export function isNonsensePiece(f: JudgedFile): boolean {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way. */
|
||||||
|
const RELEASE_LIKE_SPACED = new RegExp(RELEASE_LIKE.source.replace('[._-]', '[._ -]'), 'i')
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
|
||||||
|
* exe is the content), covers and sidecars.
|
||||||
|
*/
|
||||||
|
const TINY_CONTENT =
|
||||||
|
/\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b/i
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
|
||||||
|
* unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
|
||||||
|
* subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
|
||||||
|
*/
|
||||||
|
export function isTinyArchive(f: JudgedFile): boolean {
|
||||||
|
return (
|
||||||
|
f.totalParts === 1 &&
|
||||||
|
f.segments === 1 &&
|
||||||
|
f.bytes >= TINY_MIN_BYTES &&
|
||||||
|
f.bytes < NONSENSE_MIN_BYTES &&
|
||||||
|
!f.groups.some((g) => STUB_EXEMPT_GROUPS.test(g)) &&
|
||||||
|
RELEASE_LIKE_SPACED.test(fileStem(f.name)) &&
|
||||||
|
!EBOOK.test(f.name) &&
|
||||||
|
!TINY_CONTENT.test(f.name) &&
|
||||||
|
!isScrambledFile(f.name) &&
|
||||||
|
!isRandomFile(f.name)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
export function isRandomFile(name: string) {
|
export function isRandomFile(name: string) {
|
||||||
return looksRandom(fileStem(name))
|
return looksRandom(fileStem(name))
|
||||||
}
|
}
|
||||||
@@ -221,6 +262,7 @@ export class RuleSet {
|
|||||||
if (legs !== '') return { rule: 'stub-archive', key: legs, sample }
|
if (legs !== '') return { rule: 'stub-archive', key: legs, sample }
|
||||||
}
|
}
|
||||||
if (this.rules.dropNonsensePieces && isNonsensePiece(archive)) return { rule: 'nonsense-piece', key: uploadTool(archive.firstMessageId), sample }
|
if (this.rules.dropNonsensePieces && isNonsensePiece(archive)) return { rule: 'nonsense-piece', key: uploadTool(archive.firstMessageId), sample }
|
||||||
|
if (this.rules.dropTinyArchives && isTinyArchive(archive)) return { rule: 'tiny-archive', key: uploadTool(archive.firstMessageId), sample }
|
||||||
}
|
}
|
||||||
if (files.some((f) => !isRandomFile(f.name))) return { rule: null }
|
if (files.some((f) => !isRandomFile(f.name))) return { rule: null }
|
||||||
const first = files[0]
|
const first = files[0]
|
||||||
|
|||||||
@@ -538,6 +538,38 @@ test.group('BatchStore', (group) => {
|
|||||||
assert.include(await db.from('pending_releases').firstOrFail(), { files: 2, segments: '5' })
|
assert.include(await db.from('pending_releases').firstOrFail(), { files: 2, segments: '5' })
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('a finished unnamed release without a probe target is exhausted at once, not waiting', async ({
|
||||||
|
assert,
|
||||||
|
}) => {
|
||||||
|
const { rangeId, nodeId } = await rangeIn(GROUP, 1)
|
||||||
|
// Two hash-named pieces: the edge gives the first a lone target; without one the ladder could never claim it.
|
||||||
|
const batch = batchOf(GROUP, 1, [
|
||||||
|
...upload(
|
||||||
|
10,
|
||||||
|
'9e7fc2741cdd450287aff2733ca4c193',
|
||||||
|
4,
|
||||||
|
'b3a75d234aa73891e78a70fc1f5496b1 [1/2]',
|
||||||
|
'2026-10-05T10:00:00Z'
|
||||||
|
),
|
||||||
|
...upload(
|
||||||
|
20,
|
||||||
|
'c5ebdd5487f5b132c76e8d5c2572e560',
|
||||||
|
4,
|
||||||
|
'b3a75d234aa73891e78a70fc1f5496b1 [2/2]',
|
||||||
|
'2026-10-05T10:01:00Z'
|
||||||
|
),
|
||||||
|
])
|
||||||
|
assert.lengthOf(batch.releases, 1)
|
||||||
|
assert.isFalse(batch.releases[0].readable)
|
||||||
|
const stored = await BatchStore.store(rangeId, nodeId, { ...batch, candidates: [] })
|
||||||
|
assert.include(stored, { releases: 1, newReleases: 0, candidates: 0, pending: 1 })
|
||||||
|
const pending = await db.from('pending_releases').firstOrFail()
|
||||||
|
assert.include(pending, { files: 2, finished: true, status: 3 })
|
||||||
|
assert.deepEqual(pending.candidates, [])
|
||||||
|
const klass = await db.from('pending_classes').firstOrFail()
|
||||||
|
assert.include(klass, { waiting: 0 })
|
||||||
|
})
|
||||||
|
|
||||||
test("an album's members stay findable by their own names (release_texts kind member)", async ({
|
test("an album's members stay findable by their own names (release_texts kind member)", async ({
|
||||||
assert,
|
assert,
|
||||||
}) => {
|
}) => {
|
||||||
|
|||||||
@@ -163,6 +163,55 @@ test.group('ReleaseJudge', (group) => {
|
|||||||
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
|
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('judges tiny single-article files under release-like names', async ({ assert }) => {
|
||||||
|
let n = 0
|
||||||
|
const settled = new Date(Date.now() - 3 * 86400_000)
|
||||||
|
const release = async (name: string, extra: Record<string, unknown> = {}) => {
|
||||||
|
n++
|
||||||
|
const key = `jt${String(n).padStart(6, '0')}`.padEnd(40, '0')
|
||||||
|
return Number(
|
||||||
|
await insertRelease({
|
||||||
|
release_key: key,
|
||||||
|
identity_key: key,
|
||||||
|
name,
|
||||||
|
source: 'stem',
|
||||||
|
groups: ['alt.binaries.boneless'],
|
||||||
|
files: 1,
|
||||||
|
segments: 1,
|
||||||
|
bytes: 15_000,
|
||||||
|
posted_at: settled,
|
||||||
|
readable: true,
|
||||||
|
settled_at: settled,
|
||||||
|
...extra,
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
const dotted = await release('Velvet.S02E20.GERMAN.720p.HDTV.x264-ACED')
|
||||||
|
const spaced = await release('Pacific Rim (2013) BDRip DVDr-sailo1')
|
||||||
|
const subs = await release('Some.Film.2019.German.Subs-XYZ')
|
||||||
|
const ebook = await release('Some.Book.2019.epub-GRP')
|
||||||
|
const music = await release('Some.Album.2019.FLAC-GRP', { groups: ['alt.binaries.sounds.flac'] })
|
||||||
|
const big = await release('Big.Film.2019.1080p-GRP', { bytes: 500_000 })
|
||||||
|
const twoFiles = await release('Two.Films.2019.1080p-GRP', { files: 2, segments: 2 })
|
||||||
|
const plain = await release('Holiday photo')
|
||||||
|
const sidecar = await release('Side.Car.2019.1080p-GRP', { sidecar_only: true })
|
||||||
|
const unreadable = await release('Xk9v.2019.Qpz-ab', { readable: false })
|
||||||
|
|
||||||
|
const round = await new ReleaseJudge().full()
|
||||||
|
const verdicts = new Map(
|
||||||
|
(await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [
|
||||||
|
Number(v.release_id),
|
||||||
|
v.rule,
|
||||||
|
])
|
||||||
|
)
|
||||||
|
assert.equal(verdicts.get(dotted), 'tiny')
|
||||||
|
assert.equal(verdicts.get(spaced), 'tiny')
|
||||||
|
for (const id of [subs, ebook, music, big, twoFiles, plain, sidecar, unreadable]) {
|
||||||
|
assert.notEqual(verdicts.get(id), 'tiny')
|
||||||
|
}
|
||||||
|
assert.equal(round.rules.tiny.added, 2)
|
||||||
|
})
|
||||||
|
|
||||||
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
|
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
|
||||||
assert,
|
assert,
|
||||||
}) => {
|
}) => {
|
||||||
|
|||||||
@@ -52,6 +52,15 @@ test.group('release / isUnreadableName', () => {
|
|||||||
'EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps',
|
'EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps',
|
||||||
'iPhone',
|
'iPhone',
|
||||||
'Some.Show.S01E01.1080p.mkv',
|
'Some.Show.S01E01.1080p.mkv',
|
||||||
|
'FastFilms 2.04',
|
||||||
|
'BestCrypt NP v4.0',
|
||||||
|
'EasyHTML 2.35 German',
|
||||||
|
'SchiffsModell - September 2026',
|
||||||
|
'CrossStitcher - Issue 440, October 2026',
|
||||||
|
'Laura15Juli2026',
|
||||||
|
'IHaveAWife - Isa Bella',
|
||||||
|
'DickHDDaily - Santana Red',
|
||||||
|
'1904-SmithsWork-Hasluck',
|
||||||
]) {
|
]) {
|
||||||
assert.isFalse(isUnreadableName(n), n)
|
assert.isFalse(isUnreadableName(n), n)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -98,8 +98,9 @@ test.group('rules / verdicts', () => {
|
|||||||
|
|
||||||
test('nonsense pieces: one article of 100 kB to 1 MB under a release-like name', ({ assert }) => {
|
test('nonsense pieces: one article of 100 kB to 1 MB under a release-like name', ({ assert }) => {
|
||||||
assert.include(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: 'nonsense-piece', key: 'nyuu' })
|
assert.include(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: 'nonsense-piece', key: 'nyuu' })
|
||||||
// A 3 kB cue/log sidecar rar with a year in its name: under the floor. A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
|
// A 3 kB cue/log sidecar rar with a year in its name: under the floor (the tiny-archive rule's outside the music groups). A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
|
||||||
assert.deepEqual(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: null })
|
assert.deepEqual(rules.judge('k', [{ ...archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000), groups: ['alt.binaries.sounds.lossless'] }]), { rule: null })
|
||||||
|
assert.include(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: 'tiny-archive' })
|
||||||
assert.deepEqual(rules.judge('k', [archive('Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar', 'fake@address.com (Psychopath)', 2, 1, 303_000)]), { rule: null })
|
assert.deepEqual(rules.judge('k', [archive('Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar', 'fake@address.com (Psychopath)', 2, 1, 303_000)]), { rule: null })
|
||||||
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 382_000)]), { rule: null })
|
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 382_000)]), { rule: null })
|
||||||
assert.deepEqual(rules.judge('k', [archive('Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar', 'saggir@domain.com', 1, 1, 113_000)]), { rule: null })
|
assert.deepEqual(rules.judge('k', [archive('Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar', 'saggir@domain.com', 1, 1, 113_000)]), { rule: null })
|
||||||
@@ -109,6 +110,31 @@ test.group('rules / verdicts', () => {
|
|||||||
const off = new RuleSet({ ...DEFAULT_RULES, dropNonsensePieces: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
|
const off = new RuleSet({ ...DEFAULT_RULES, dropNonsensePieces: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
|
||||||
assert.deepEqual(off.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
|
assert.deepEqual(off.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('tiny archives: one article of 2-100 kB under a release name, spaces allowed', ({ assert }) => {
|
||||||
|
// A RAR holding "<title>.exe": the 2015-17 flood in miniature (LOG.md (131)).
|
||||||
|
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar', 'Yanc@power-post.org <Yanc@power-post.org>', 1, 1, 15_084)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||||
|
assert.include(rules.judge('k', [archive('Cicero Magazin fuer politische Kultur Mai 05 2016.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 17_478)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||||
|
// A scene name without a year, a tag or a group suffix: not release-like, so kept.
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar', 'Yanc@power-post.org', 2, 1, 23_449)]), { rule: null })
|
||||||
|
assert.include(rules.judge('k', [archive('Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 7_869)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||||
|
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar', 'henk@example.org', 1, 1, 38_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar', 'poster@example.org', 1, 1, 60_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Some Band - Album (2011) covers.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 38_000)]), { rule: null })
|
||||||
|
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's; not settled.
|
||||||
|
assert.deepEqual(rules.judge('k', [{ ...archive('KV-1055806 - Walker Hayes - U Gurl (2017).zip', 'home@home.com', 1, 1, 60_000), groups: ['alt.binaries.sounds.karaoke'] }]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('holiday snaps.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('nYDeXeF49dieZ1J7flK-JFzZ3j.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 2, 20_000)]), { rule: null })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 1_500)]), { rule: null })
|
||||||
|
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 150_000)]), { rule: 'nonsense-piece' })
|
||||||
|
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)], false), { rule: null })
|
||||||
|
const off = new RuleSet({ ...DEFAULT_RULES, dropTinyArchives: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
|
||||||
|
assert.deepEqual(off.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
test.group('rules / upload tools', () => {
|
test.group('rules / upload tools', () => {
|
||||||
|
|||||||
Reference in new issue
Block a user