Compare commits
13
Commits
44c0c73b05
...
rebuild
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
66980829dc | ||
|
|
f537add573 | ||
|
|
d7171a5ca4 | ||
|
|
eadae82b0c | ||
|
|
e02a7c89b4 | ||
|
|
6ef7ddca92 | ||
|
|
a71de56a7c | ||
|
|
411c36eec0 | ||
|
|
8e008da364 | ||
|
|
5d3b578322 | ||
|
|
aa4022295a | ||
|
|
4d46db291e | ||
|
|
3612bf3e43 |
No files matched your search
+41
-13
@@ -341,8 +341,13 @@ or music name whose content is a program: archive entries all
|
||||
executables, or the file itself one by name or by its first bytes, sniffed
|
||||
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
|
||||
*locked* (passworded, no password from a spot, its NZB or the title),
|
||||
*unusable* (random names, no inner name, no par2 or inner-name merge, no
|
||||
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
|
||||
*tiny* (one file of one article, 2–100 kB, under a readable release-like
|
||||
name outside the music, e-book and comic groups, without an e-book,
|
||||
subtitle, crack or sidecar word: the catalog's side of the edge's
|
||||
tiny-archive rule, 1.5M rows of the 2015–17 flood that the edge indexed
|
||||
before the rule; LOG.md (131), (135)), *unusable* (random names, no inner
|
||||
name, no par2 or inner-name merge, no spot, a day after settling).
|
||||
Verdicts (`release_verdicts`, with the rule
|
||||
set's version) hide releases from search (`judged` shows them); a verdict
|
||||
whose rule no longer matches is taken back. Nothing is dropped yet.
|
||||
|
||||
@@ -712,10 +717,13 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
||||
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
|
||||
`settled_at IS NOT NULL`), the settler, the fragment merger,
|
||||
`releases_category_newest_idx`.
|
||||
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
|
||||
HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
|
||||
rule read all of `release_inspections` every round, 17 min on the HDD;
|
||||
the judge must read it by release or keep what it needs on the NVMe.
|
||||
- `release_inspections` and `probe_candidates` were on `CHUNK_TABLESPACE`
|
||||
(the HDD) with their indexes; both are back on the NVMe (migrations 4800
|
||||
and 7300). Note LOG.md 2026-10-19 (81): the judge's fake rule read all of
|
||||
`release_inspections` every round, 17 min on the HDD; and 2026-11-03
|
||||
(136): the prober's catalog claim updates `probe_candidates` from random
|
||||
points every minute, 2–3 min on the HDD beside the pending ladder's sweep.
|
||||
What a worker reads or updates by random key every minute stays on the NVMe.
|
||||
|
||||
### Batch store (built 2026-10-25)
|
||||
|
||||
@@ -806,9 +814,17 @@ unscrambled name without spaces and without an e-book format in it is
|
||||
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
|
||||
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
|
||||
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
|
||||
with spaces are books and albums. Both rules are switched by the
|
||||
rule set (`dropStubArchives`, `dropNonsensePieces`), sampled like every
|
||||
drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
|
||||
with spaces are books and albums. Below that, a settled single archive of
|
||||
one article and 2–100 kB under a release-like name (spaces allowed), outside
|
||||
the exempt groups and without an e-book, subtitle, crack/keygen/trainer,
|
||||
cover or sidecar word, is dropped as a **tiny archive** (`tiny-archive`,
|
||||
key: the upload tool; LOG.md (131), (133)): the 2015–17 flood's RARs
|
||||
holding "<title>.exe" at 2–60 kB, 1.66M catalog rows from a dozen posters
|
||||
(12 of 12 fetched fake); the word exemptions keep the real tiny archives
|
||||
(subtitles, cracks, trainers) at the price of ~50k fakes named "Incl
|
||||
Keygen" that the prober judges later. The rules are switched by the
|
||||
rule set (`dropStubArchives`, `dropNonsensePieces`, `dropTinyArchives`),
|
||||
sampled like every drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
|
||||
judges as each batch completes files, so it passes settled without the
|
||||
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
|
||||
`check.ts` and the fixture comparison (the parity with the TypeScript
|
||||
@@ -927,12 +943,24 @@ f. **Nothing enters the catalog without a usable name** (amended
|
||||
target or status 3 (exhausted); each finished climb counts on
|
||||
`pending_signatures` for (tool, group, poster) and (tool, group), and a
|
||||
signature with PENDING_MIN_PROBES (100) climbs and at most
|
||||
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
|
||||
waiting releases of dropped signatures are discarded (status 2), the
|
||||
rules' `samplePercent` of them left to climb. The namer
|
||||
PENDING_MAX_NAMED (0.5%) named is dropped: a sweep beside the lanes
|
||||
(every five minutes, in its own loop) discards the waiting releases
|
||||
of each dropped signature and class once (`swept_at`; the edge drops
|
||||
their new releases itself), the rules' `samplePercent` of them left to
|
||||
climb, each by an index (`pending_releases_signature_idx` for a
|
||||
poster's rows: tested on the rows, a signature was a scan of the whole
|
||||
table; LOG.md (133), (134)). The batch store, the record, the claim
|
||||
and the sweep lock pending rows in one order, by key and the counters
|
||||
after (`lockOrder` for the multi-row statements; the record and the
|
||||
sweep lock their rows with `SELECT … ORDER BY release_key FOR UPDATE`
|
||||
first, since an UPDATE's join or `= ANY(subquery)` locks in plan
|
||||
order), and a flush that deadlocks anyway keeps its outcomes for the
|
||||
next (LOG.md (134), (137)). The namer
|
||||
(`app/services/pending_namer.ts`, in the prober's process): a named
|
||||
release's article span read again from the corpus (the stored range's
|
||||
chunks) or by XOVER on a connection of its own, through the TypeScript
|
||||
chunks) or by XOVER on a connection of its own, a chunk of articles
|
||||
at a time (one string of a wide span topped V8's limit; spans over 3M
|
||||
articles are given up), through the TypeScript
|
||||
indexer with `partialFiles` (a file given up short of parts stays a
|
||||
partial file of its release, as the Rust window writes it; without it
|
||||
the range path's fragment rule left nothing to find: by 2026-10-29
|
||||
|
||||
+24
-15
@@ -9,7 +9,7 @@ import { SIDECAR } from '#services/sidecar_folder'
|
||||
import { identityKey, releaseKey } from '#lib/release/keys'
|
||||
import { sourceCode } from '#lib/release/source'
|
||||
import { uploadTool } from '#lib/rules/index'
|
||||
import { byEffort } from '#services/pending_ladder'
|
||||
import { byEffort, lockOrder } from '#services/pending_ladder'
|
||||
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
|
||||
|
||||
/**
|
||||
@@ -190,9 +190,11 @@ class BatchStore {
|
||||
list.push({ kind: c.kind, messageId: c.messageId, file: c.file, bytes: c.bytes })
|
||||
targets.set(key, list)
|
||||
}
|
||||
let written = 0
|
||||
for (let i = 0; i < releases.length; i += 1000) {
|
||||
const rows = releases.slice(i, i + 1000).map((r) => {
|
||||
// In the record's lock order (LOG.md (136)): all rows sorted, then all classes sorted, in one transaction. The
|
||||
// upsert locks a finished row too, before its WHERE passes it by; a chunk's classes before the next chunk's rows
|
||||
// crossed the ladder's record(), which holds its rows and then takes its classes.
|
||||
const rows = lockOrder(
|
||||
releases.map((r) => {
|
||||
const probes = targets.get(slot(r.key, r.upload)) ?? []
|
||||
return {
|
||||
release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
|
||||
@@ -214,7 +216,12 @@ class BatchStore {
|
||||
// The first target's article (by effort): the cheapest probe first within a class (LOG.md (130)).
|
||||
target_bytes: byEffort(probes)[0]?.bytes ?? null,
|
||||
}
|
||||
})
|
||||
}),
|
||||
(r) => r.release_key
|
||||
)
|
||||
let written = 0
|
||||
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
|
||||
for (let i = 0; i < rows.length; i += 1000) {
|
||||
const { rows: done } = await trx.rawQuery(
|
||||
`INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
|
||||
files, segments, bytes, partial, finished, file_total, tool, candidates, klass, target_bytes)
|
||||
@@ -232,25 +239,27 @@ class BatchStore {
|
||||
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes)
|
||||
WHERE NOT pending_releases.finished
|
||||
RETURNING release_key, tool, klass, (xmax = 0) AS inserted`,
|
||||
[groupId, JSON.stringify(rows)]
|
||||
[groupId, JSON.stringify(rows.slice(i, i + 1000))]
|
||||
)
|
||||
written += done.length
|
||||
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
|
||||
const classes = new Map<string, { tool: string; klass: number; waiting: number }>()
|
||||
for (const d of done) {
|
||||
const key = `${d.tool}\t${d.klass}`
|
||||
const entry = classes.get(key) ?? { tool: d.tool, klass: Number(d.klass), waiting: 0 }
|
||||
if (d.inserted) entry.waiting++
|
||||
classes.set(key, entry)
|
||||
}
|
||||
if (classes.size > 0) {
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
|
||||
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
|
||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
|
||||
[groupId, JSON.stringify([...classes.values()])]
|
||||
)
|
||||
}
|
||||
}
|
||||
if (classes.size > 0) {
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
|
||||
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
|
||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
|
||||
[
|
||||
groupId,
|
||||
JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${groupId}\t${c.klass}`)),
|
||||
]
|
||||
)
|
||||
}
|
||||
return written
|
||||
}
|
||||
|
||||
@@ -5,20 +5,29 @@ import db from '@adonisjs/lucid/services/db'
|
||||
* repeated verdict match) in a catalog release that has a name, readable or found inside. The same film posted
|
||||
* under scrambles by other tools gets that name without another fetch. Releases with a junk verdict don't count.
|
||||
*/
|
||||
/** Twins read per pair: the hottest pairs have 131k. */
|
||||
const TWIN_SCAN = 500
|
||||
|
||||
export async function twinNames(
|
||||
pairs: { hash16k: string; size: number }[],
|
||||
trx: { rawQuery: typeof db.rawQuery } = db
|
||||
): Promise<Map<string, string>> {
|
||||
const out = new Map<string, string>()
|
||||
if (pairs.length === 0) return out
|
||||
// Per pair the first named twin among its lowest TWIN_SCAN release ids. hash16k is typed as the column
|
||||
// (char(32)): as text the join cast the column and scanned all of file_hashes, 4 s a call; and with the index
|
||||
// the hottest pairs (55 with over 500 twins, up to 131k, all readable) made a call walk every twin, 37 s inside
|
||||
// the flush's transaction (LOG.md (138)). One name is all a pair needs.
|
||||
const { rows } = await trx.rawQuery(
|
||||
`SELECT DISTINCT ON (h.hash16k, h.size) h.hash16k, h.size, coalesce(r.inner_name, r.name) AS name
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k text, size bigint)
|
||||
JOIN file_hashes h ON h.hash16k = v.hash16k AND h.size = v.size
|
||||
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
|
||||
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
|
||||
ORDER BY h.hash16k, h.size, r.readable DESC, r.id`,
|
||||
[JSON.stringify(pairs)]
|
||||
`SELECT v.hash16k, v.size, t.name
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(hash16k char(32), size bigint)
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT coalesce(r.inner_name, r.name) AS name
|
||||
FROM (SELECT release_id FROM file_hashes h WHERE h.hash16k = v.hash16k AND h.size = v.size ORDER BY release_id LIMIT ?) h
|
||||
JOIN releases r ON r.id = h.release_id AND (r.readable OR r.inner_name IS NOT NULL)
|
||||
WHERE NOT EXISTS (SELECT 1 FROM release_verdicts x WHERE x.release_id = r.id)
|
||||
ORDER BY r.readable DESC, r.id LIMIT 1) t`,
|
||||
[JSON.stringify(pairs), TWIN_SCAN]
|
||||
)
|
||||
for (const r of rows) out.set(`${r.hash16k}\t${r.size}`, r.name)
|
||||
return out
|
||||
|
||||
+111
-20
@@ -72,6 +72,19 @@ interface ClassRow {
|
||||
const CLASSES_MS = 60_000
|
||||
/** A class found short of a claim's share is left alone this long. */
|
||||
const EMPTY_MS = 5 * 60_000
|
||||
/**
|
||||
* Rows in one order for a statement that locks many of them (pending_releases by key, pending_classes and
|
||||
* pending_signatures by theirs): the batch store, the record, the claim and the sweep all lock the same way, so two
|
||||
* of them can't deadlock on each other (LOG.md (134)).
|
||||
*/
|
||||
export function lockOrder<T>(rows: Iterable<T>, key: (row: T) => string): T[] {
|
||||
return [...rows].sort((a, b) => {
|
||||
const ka = key(a)
|
||||
const kb = key(b)
|
||||
return ka < kb ? -1 : ka > kb ? 1 : 0
|
||||
})
|
||||
}
|
||||
|
||||
/** The prior rate of an unexplored class and its weight in probes: the ladder's overall rate on 2026-11-02 (LOG.md (130)). */
|
||||
const PRIOR_RATE = 0.15
|
||||
const PRIOR_WEIGHT = 10
|
||||
@@ -276,13 +289,26 @@ class PendingLadder {
|
||||
final: null,
|
||||
}
|
||||
})
|
||||
// The rows locked in key order first: the UPDATE's join below takes them in whatever order the plan does.
|
||||
await trx.rawQuery(
|
||||
`SELECT 1 FROM pending_releases WHERE release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
||||
ORDER BY release_key FOR UPDATE`,
|
||||
[rows.map((r) => r.release_key).sort()]
|
||||
)
|
||||
await trx.rawQuery(
|
||||
`UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
|
||||
inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
|
||||
probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
|
||||
WHERE p.release_key = decode(v.release_key, 'hex')`,
|
||||
[JSON.stringify(rows.map(({ final: _final, ...row }) => row))]
|
||||
[
|
||||
JSON.stringify(
|
||||
lockOrder(
|
||||
rows.map(({ final: _final, ...row }) => row),
|
||||
(r) => r.release_key
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
// The signatures: the poster's own and the tool's and group's together.
|
||||
const counts = new Map<
|
||||
@@ -312,17 +338,20 @@ class PendingLadder {
|
||||
`INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
|
||||
SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
|
||||
ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
|
||||
[JSON.stringify([...counts.values()])]
|
||||
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`))]
|
||||
)
|
||||
await trx.rawQuery(
|
||||
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float)
|
||||
`UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float),
|
||||
swept_at = CASE WHEN s.dropped THEN s.swept_at END
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
|
||||
WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
|
||||
[
|
||||
pendingMinProbes,
|
||||
pendingMaxNamed,
|
||||
JSON.stringify(
|
||||
[...counts.values()].map(({ tool, group_id, poster }) => ({ tool, group_id, poster }))
|
||||
lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.poster}`).map(
|
||||
({ tool, group_id, poster }) => ({ tool, group_id, poster })
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
@@ -359,41 +388,103 @@ class PendingLadder {
|
||||
SELECT v.tool, v.group_id, v.klass, v.probed, v.named, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, probed int, named int, waiting int)
|
||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named,
|
||||
waiting = pending_classes.waiting + excluded.waiting, updated_at = now()`,
|
||||
[JSON.stringify([...classes.values()])]
|
||||
[JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
|
||||
)
|
||||
await trx.rawQuery(
|
||||
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float)
|
||||
`UPDATE pending_classes c SET dropped = (c.probed >= ?::int AND c.named::float <= c.probed * ?::float),
|
||||
swept_at = CASE WHEN c.dropped THEN c.swept_at END
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int)
|
||||
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
||||
[
|
||||
pendingMinProbes,
|
||||
pendingMaxNamed,
|
||||
JSON.stringify(
|
||||
[...classes.values()].map(({ tool, group_id, klass }) => ({ tool, group_id, klass }))
|
||||
lockOrder(classes.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`).map(
|
||||
({ tool, group_id, klass }) => ({ tool, group_id, klass })
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
}
|
||||
|
||||
/** Waiting releases of a dropped signature discarded without a probe, the explored sample excepted; how many. */
|
||||
async discard(limit = 10_000): Promise<number> {
|
||||
/**
|
||||
* Waiting releases of a dropped class or signature discarded without a probe, the explored sample excepted; how
|
||||
* many. Each dropped class or signature is swept once (swept_at; null again when the record drops it anew),
|
||||
* `perSweep` of them at most, each by its index (the claim's for a class, the signature index for a poster's
|
||||
* rows, the ladder's for a tool's and group's) in rounds of `limit`. The prober runs this beside the lanes, not
|
||||
* in the claim (LOG.md (133)); the edge drops new releases of dropped signatures and classes itself, so one
|
||||
* sweep is all a drop needs.
|
||||
*/
|
||||
async discard(limit = 10_000, perSweep = 20): Promise<number> {
|
||||
const { samplePercent } = await RuleSets.current()
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT encode(p.release_key, 'hex') AS release_key FROM pending_releases p
|
||||
WHERE p.status = ? AND (EXISTS (
|
||||
SELECT 1 FROM pending_signatures s
|
||||
WHERE s.dropped AND s.tool = p.tool AND s.group_id = p.group_id AND (s.poster = '' OR s.poster = coalesce(p.poster, '')))
|
||||
OR EXISTS (SELECT 1 FROM pending_classes c WHERE c.dropped AND c.tool = p.tool AND c.group_id = p.group_id AND c.klass = p.klass))
|
||||
LIMIT ?`,
|
||||
[PENDING.waiting, limit]
|
||||
let discarded = 0
|
||||
const { rows: classes } = await db.rawQuery(
|
||||
`SELECT tool, group_id, klass FROM pending_classes WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, klass LIMIT ?`,
|
||||
[perSweep]
|
||||
)
|
||||
for (const c of classes) {
|
||||
let rows: any[]
|
||||
do {
|
||||
;({ rows } = await db.rawQuery(
|
||||
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||
WHERE status = ? AND tool = ? AND group_id = ? AND klass = ? LIMIT ?`,
|
||||
[PENDING.waiting, c.tool, c.group_id, c.klass, limit]
|
||||
))
|
||||
discarded += await this.discardKeys(rows, samplePercent)
|
||||
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
|
||||
await db.rawQuery(
|
||||
`UPDATE pending_classes SET swept_at = now() WHERE tool = ? AND group_id = ? AND klass = ?`,
|
||||
[c.tool, c.group_id, c.klass]
|
||||
)
|
||||
}
|
||||
const { rows: signatures } = await db.rawQuery(
|
||||
`SELECT tool, group_id, poster FROM pending_signatures WHERE dropped AND swept_at IS NULL ORDER BY group_id, tool, poster LIMIT ?`,
|
||||
[perSweep]
|
||||
)
|
||||
for (const s of signatures) {
|
||||
let rows: any[]
|
||||
do {
|
||||
// The poster's rows by the signature index (on the poster's md5: a poster can be longer than a btree
|
||||
// key), a tool's and group's by the ladder's: never a test on the rows, which was a scan of the whole
|
||||
// table per signature on the HDD (LOG.md (134)).
|
||||
;({ rows } =
|
||||
s.poster === ''
|
||||
? await db.rawQuery(
|
||||
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||
WHERE status = ? AND group_id = ? AND tool = ? LIMIT ?`,
|
||||
[PENDING.waiting, s.group_id, s.tool, limit]
|
||||
)
|
||||
: await db.rawQuery(
|
||||
`SELECT encode(release_key, 'hex') AS release_key FROM pending_releases
|
||||
WHERE status = ? AND group_id = ? AND tool = ? AND md5(poster) = md5(?::text) AND poster = ? LIMIT ?`,
|
||||
[PENDING.waiting, s.group_id, s.tool, s.poster, s.poster, limit]
|
||||
))
|
||||
discarded += await this.discardKeys(rows, samplePercent)
|
||||
} while (rows.length === limit && rows.some((r) => !explored(r.release_key, samplePercent)))
|
||||
await db.rawQuery(
|
||||
`UPDATE pending_signatures SET swept_at = now() WHERE tool = ? AND group_id = ? AND poster = ?`,
|
||||
[s.tool, s.group_id, s.poster]
|
||||
)
|
||||
}
|
||||
return discarded
|
||||
}
|
||||
|
||||
/** The rows discarded (status 2), the explored sample excepted, their classes' waiting counts lowered; how many. */
|
||||
private async discardKeys(rows: any[], samplePercent: number): Promise<number> {
|
||||
const keys = rows
|
||||
.map((r: any) => r.release_key as string)
|
||||
.filter((key: string) => !explored(key, samplePercent))
|
||||
.sort()
|
||||
if (keys.length === 0) return 0
|
||||
// Locked in key order (the record's lock order, LOG.md (137)): a plain `key = ANY(subquery)` locks in scan
|
||||
// order and deadlocked against a batch upsert; ORDER BY sorts before FOR UPDATE locks.
|
||||
const { rows: gone } = await db.rawQuery(
|
||||
`UPDATE pending_releases SET status = ? WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
||||
RETURNING tool, group_id, klass`,
|
||||
`UPDATE pending_releases p SET status = ?
|
||||
FROM (SELECT release_key FROM pending_releases
|
||||
WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)
|
||||
ORDER BY release_key FOR UPDATE) k
|
||||
WHERE p.release_key = k.release_key
|
||||
RETURNING p.tool, p.group_id, p.klass`,
|
||||
[PENDING.discarded, PENDING.waiting, keys]
|
||||
)
|
||||
const counts = new Map<string, { tool: string; group_id: number; klass: number; n: number }>()
|
||||
@@ -413,7 +504,7 @@ class PendingLadder {
|
||||
`UPDATE pending_classes c SET waiting = greatest(0, c.waiting - v.n)
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, klass int, n int)
|
||||
WHERE c.tool = v.tool AND c.group_id = v.group_id AND c.klass = v.klass`,
|
||||
[JSON.stringify([...counts.values()])]
|
||||
[JSON.stringify(lockOrder(counts.values(), (c) => `${c.tool}\t${c.group_id}\t${c.klass}`))]
|
||||
)
|
||||
}
|
||||
return gone.length
|
||||
|
||||
@@ -30,6 +30,13 @@ interface NamedRow {
|
||||
/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
|
||||
const MAX_MOVES = 3
|
||||
|
||||
/**
|
||||
* The widest article span the namer reads again (a busy group posts 50M articles a day, a large upload among
|
||||
* them spans a million or two): one wider is a name the edge saw again much later ("v-lgpc.3ds"), and its span
|
||||
* of segments would not fit the indexer's memory (LOG.md (133)).
|
||||
*/
|
||||
const MAX_SPAN = 3_000_000
|
||||
|
||||
/**
|
||||
* The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
|
||||
* read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
|
||||
@@ -94,6 +101,10 @@ export class PendingNamer {
|
||||
|
||||
/** One named release into the catalog. */
|
||||
async move(row: NamedRow): Promise<void> {
|
||||
if (row.last_article - row.first_article > MAX_SPAN)
|
||||
throw new Error(
|
||||
`its articles ${row.first_article}–${row.last_article} span more than ${MAX_SPAN}`
|
||||
)
|
||||
const group = await db.from('groups').where('id', row.group_id).firstOrFail()
|
||||
const range = await db
|
||||
.from('ranges')
|
||||
@@ -103,19 +114,20 @@ export class PendingNamer {
|
||||
.orderBy('id')
|
||||
.first()
|
||||
if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
|
||||
const text = await this.overviewText(group.name, row)
|
||||
const rules = await RuleSets.current()
|
||||
const ix = new RangeIndexer({
|
||||
group: group.name,
|
||||
start: row.first_article,
|
||||
end: row.last_article,
|
||||
rules: { ...rules, samplePercent: 0 },
|
||||
idleChunks: 0,
|
||||
// No file given up before the end: the span is read chunk by chunk (one text of it topped V8's string).
|
||||
idleChunks: Number.MAX_SAFE_INTEGER,
|
||||
edgeArticles: 0,
|
||||
// The edge wrote it partial when its file is short of parts on the server: so must the re-read.
|
||||
partialFiles: true,
|
||||
})
|
||||
ix.addText(text, row.last_article + 1)
|
||||
for await (const piece of this.overviewPieces(group.name, row))
|
||||
ix.addText(piece.text, piece.next)
|
||||
ix.finish()
|
||||
const batch = ix.take()
|
||||
const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
|
||||
@@ -160,50 +172,20 @@ export class PendingNamer {
|
||||
})
|
||||
}
|
||||
|
||||
/** The overview lines of the release's article span: the stored corpus chunks when its range is stored, else XOVER. */
|
||||
private async overviewText(groupName: string, row: NamedRow): Promise<string> {
|
||||
if (this.corpusDir) {
|
||||
const stored = await db
|
||||
.from('ranges')
|
||||
.where('group_id', row.group_id)
|
||||
.whereNotNull('stored_at')
|
||||
.where('start_id', '<=', row.last_article)
|
||||
.where('end_id', '>=', row.first_article)
|
||||
.orderBy('start_id')
|
||||
const parts: string[] = []
|
||||
let covered = true
|
||||
for (const r of stored) {
|
||||
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
|
||||
const size = coordinatorConfig.chunkArticles
|
||||
for (
|
||||
let from =
|
||||
range.start +
|
||||
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
|
||||
from <= Math.min(row.last_article, range.end);
|
||||
from += size
|
||||
) {
|
||||
const raw = await Corpus.read(
|
||||
range,
|
||||
{ from, to: Math.min(from + size - 1, range.end) },
|
||||
this.corpusDir
|
||||
)
|
||||
if (!raw) {
|
||||
covered = false
|
||||
break
|
||||
}
|
||||
parts.push((await inflate(raw)).toString('latin1'))
|
||||
}
|
||||
if (!covered) break
|
||||
}
|
||||
if (
|
||||
covered &&
|
||||
parts.length > 0 &&
|
||||
stored.length > 0 &&
|
||||
Number(stored[0].start_id) <= row.first_article &&
|
||||
Number(stored[stored.length - 1].end_id) >= row.last_article
|
||||
) {
|
||||
return parts.join('')
|
||||
}
|
||||
/**
|
||||
* The overview lines of the release's article span, a chunk of articles at a time with the first article after
|
||||
* it: the stored corpus chunks when its ranges are stored and contiguous, else XOVER in windows of a chunk.
|
||||
*/
|
||||
private async *overviewPieces(
|
||||
groupName: string,
|
||||
row: NamedRow
|
||||
): AsyncGenerator<{ text: string; next: number }> {
|
||||
const size = coordinatorConfig.chunkArticles
|
||||
const stored = await this.storedChunks(groupName, row)
|
||||
if (stored) {
|
||||
for (const { raw, next } of stored)
|
||||
yield { text: (await inflate(raw)).toString('latin1'), next }
|
||||
return
|
||||
}
|
||||
this.client ??= await XoverClient.connect({
|
||||
host: env.get('NNTP_HOST')!,
|
||||
@@ -217,13 +199,55 @@ export class PendingNamer {
|
||||
this.usedAt = Date.now()
|
||||
try {
|
||||
await this.client.group(groupName)
|
||||
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
|
||||
const raw = await this.client.xoverRaw(row.first_article, row.last_article)
|
||||
return (await inflate(raw)).toString('latin1')
|
||||
for (let from = row.first_article; from <= row.last_article; from += size) {
|
||||
const to = Math.min(from + size - 1, row.last_article)
|
||||
// One zlib/gzip stream, as the server sent it or deflated by the client (the corpus stores it so).
|
||||
const raw = await this.client.xoverRaw(from, to)
|
||||
yield { text: (await inflate(raw)).toString('latin1'), next: to + 1 }
|
||||
}
|
||||
} catch (error) {
|
||||
this.client.close()
|
||||
this.client = null
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** The span's stored corpus chunks (compressed, with the first article after each), or null when any is missing. */
|
||||
private async storedChunks(
|
||||
groupName: string,
|
||||
row: NamedRow
|
||||
): Promise<{ raw: Buffer; next: number }[] | null> {
|
||||
if (!this.corpusDir) return null
|
||||
const stored = await db
|
||||
.from('ranges')
|
||||
.where('group_id', row.group_id)
|
||||
.whereNotNull('stored_at')
|
||||
.where('start_id', '<=', row.last_article)
|
||||
.where('end_id', '>=', row.first_article)
|
||||
.orderBy('start_id')
|
||||
if (
|
||||
stored.length === 0 ||
|
||||
Number(stored[0].start_id) > row.first_article ||
|
||||
Number(stored[stored.length - 1].end_id) < row.last_article
|
||||
)
|
||||
return null
|
||||
const size = coordinatorConfig.chunkArticles
|
||||
const chunks: { raw: Buffer; next: number }[] = []
|
||||
for (const r of stored) {
|
||||
const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
|
||||
for (
|
||||
let from =
|
||||
range.start +
|
||||
Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
|
||||
from <= Math.min(row.last_article, range.end);
|
||||
from += size
|
||||
) {
|
||||
const to = Math.min(from + size - 1, range.end)
|
||||
const raw = await Corpus.read(range, { from, to }, this.corpusDir)
|
||||
if (!raw) return null
|
||||
chunks.push({ raw, next: to + 1 })
|
||||
}
|
||||
}
|
||||
return chunks
|
||||
}
|
||||
}
|
||||
+37
-7
@@ -41,6 +41,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
|
||||
const FLUSH_MS = 2000
|
||||
/** Between sweeps that discard the waiting releases of dropped signatures. */
|
||||
const SWEEP_MS = 5 * 60_000
|
||||
/** A flush that deadlocks (against the batch store's pending upsert) keeps its outcomes for the next, this often. */
|
||||
const DEADLOCK_RETRIES = 5
|
||||
|
||||
/**
|
||||
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
|
||||
@@ -82,6 +84,8 @@ export class Prober {
|
||||
private refilling: Promise<void> | null = null
|
||||
private stopped = false
|
||||
private stats = Prober.freshStats()
|
||||
/** Flushes deadlocked in a row, their outcomes kept. */
|
||||
private deadlocks = 0
|
||||
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
|
||||
private lanes = new Set<Promise<void>>()
|
||||
private running = 0
|
||||
@@ -97,7 +101,6 @@ export class Prober {
|
||||
}
|
||||
/** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
|
||||
private namer = new PendingNamer()
|
||||
private sweptAt = 0
|
||||
|
||||
async run() {
|
||||
// A previous run's claims: one prober at a time.
|
||||
@@ -106,6 +109,17 @@ export class Prober {
|
||||
const released = await PendingLadder.release()
|
||||
if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
|
||||
const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
|
||||
// The ladder's discard sweep beside the lanes: a dropped signature's rows take minutes on the HDD (LOG.md (133)).
|
||||
const sweeper = (async () => {
|
||||
while (!this.stopped) {
|
||||
try {
|
||||
this.stats.discarded += await PendingLadder.discard()
|
||||
} catch (error: any) {
|
||||
logger.error(`Probing: the discard sweep failed: ${error.message}`)
|
||||
}
|
||||
for (let waited = 0; waited < SWEEP_MS && !this.stopped; waited += 1000) await sleep(1000)
|
||||
}
|
||||
})()
|
||||
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
|
||||
const flusher = (async () => {
|
||||
while (!this.stopped || this.outcomes.length > 0) {
|
||||
@@ -136,6 +150,7 @@ export class Prober {
|
||||
}
|
||||
await Promise.all(this.lanes)
|
||||
await namer
|
||||
await sweeper
|
||||
await this.share.leave().catch(() => {})
|
||||
// What was claimed and not probed goes back.
|
||||
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
|
||||
@@ -266,10 +281,6 @@ export class Prober {
|
||||
|
||||
/** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
|
||||
private async claimPending() {
|
||||
if (Date.now() - this.sweptAt > SWEEP_MS) {
|
||||
this.sweptAt = Date.now()
|
||||
this.stats.discarded += await PendingLadder.discard()
|
||||
}
|
||||
const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
|
||||
const probes = await PendingLadder.claim(want)
|
||||
if (probes.length === 0) return
|
||||
@@ -339,15 +350,34 @@ export class Prober {
|
||||
}
|
||||
}
|
||||
|
||||
/** The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. */
|
||||
/**
|
||||
* The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. A deadlock
|
||||
* (the batch store's pending upsert locks the same rows; LOG.md (134)) keeps the outcomes for the next flush
|
||||
* instead of losing the probes.
|
||||
*/
|
||||
private async flush() {
|
||||
const taken = this.outcomes.splice(0)
|
||||
if (taken.length === 0) return
|
||||
try {
|
||||
await this.store(taken)
|
||||
} catch (error: any) {
|
||||
if (error?.code === '40P01' && this.deadlocks < DEADLOCK_RETRIES) {
|
||||
this.deadlocks++
|
||||
this.outcomes.unshift(...taken)
|
||||
logger.warn(`Probing: storing results deadlocked, kept for the next flush (${this.deadlocks}).`)
|
||||
return
|
||||
}
|
||||
throw error
|
||||
}
|
||||
this.deadlocks = 0
|
||||
for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
|
||||
this.stats.pending += taken.filter((o) => o.candidate.pending && o.status !== 'waiting').length
|
||||
}
|
||||
|
||||
private async store(taken: Outcome[]) {
|
||||
// The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
|
||||
const pending = taken.filter((o) => o.candidate.pending)
|
||||
const outcomes = taken.filter((o) => !o.candidate.pending)
|
||||
this.stats.pending += pending.filter((o) => o.status !== 'waiting').length
|
||||
await db.transaction(async (trx) => {
|
||||
if (outcomes.length > 0) {
|
||||
await trx.rawQuery(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import logger from '@adonisjs/core/services/logger'
|
||||
import { SOURCES } from '#lib/release/source'
|
||||
import { TINY_MIN_BYTES, NONSENSE_MIN_BYTES } from '#lib/rules/index'
|
||||
|
||||
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
|
||||
export const JUDGE_VERSION = 1
|
||||
@@ -28,6 +29,14 @@ const MEDIA =
|
||||
const SCRAPED_POSTERS = `celeb[-_ ]{0,1}movie|celebwow|celebfx|iluvcelebs|nude[-_ ]{0,1}celebs|celeb-sex-tapes`
|
||||
/** An executable's name. */
|
||||
const EXECUTABLE = `\\.(exe|scr|com|bat|cmd|msi|lnk|vbs|js|jar|pif|cpl|hta|ps1|dll)$`
|
||||
/**
|
||||
* The tiny rule's name tests, lib/rules/rule_set.ts's RELEASE_LIKE_SPACED, EBOOK, TINY_CONTENT and
|
||||
* STUB_EXEMPT_GROUPS in Postgres's dialect (and no "?", see MEDIA).
|
||||
*/
|
||||
const TINY_RELEASE_LIKE = `(\\y(19|20)[0-9][0-9]\\y|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-{0,1}dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\\.[0-9]+)+)\\y|-[a-z0-9]{2,}$|\\.[a-z0-9]+\\.[a-z0-9]+\\.)`
|
||||
const TINY_EBOOK = `\\y(epub|mobi|azw3{0,1}|pdf|djvu|cbr|cbz|e-{0,1}books{0,1})\\y`
|
||||
const TINY_CONTENT = `\\y(subs{0,1}|subtitles{0,1}|subpack|srt|untertitel|ondertitels{0,1}|ondertiteling|legendas{0,1}|napisy|sottotitoli|cracks{0,1}|cracked|keygen|keymaker|trainer|patch|patched|serials{0,1}|activator|loader|nocd|no-cd|fix|covers{0,1}|artwork|booklet|scans{0,1}|proof|sample|info|readme)\\y`
|
||||
const TINY_EXEMPT_GROUPS = `(sound|mp3|karaoke|music|audio|flac|lossless|e-{0,1}book|epub|comic|guitar|font|magazine)`
|
||||
|
||||
export interface Rule {
|
||||
name: string
|
||||
@@ -100,6 +109,20 @@ export const RULES: Rule[] = [
|
||||
// releases_passworded_idx: a scan of all of releases was 30 s a round.
|
||||
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
|
||||
},
|
||||
{
|
||||
name: 'tiny',
|
||||
says: 'one file of one article, 2–100 kB, under a release-like name: the exe-stub flood in miniature',
|
||||
// The catalog's side of the edge's tiny-archive rule (lib/rules/rule_set.ts isTinyArchive; LOG.md (131),
|
||||
// (135)): the same name tests in Postgres's regex dialect ("\y" for "\b", "{0,1}" for "?"), the readable flag
|
||||
// for its scrambled and random tests, the group set for its groups. The 2015–17 flood that the edge indexed
|
||||
// before the rule: 1.5M rows, 98% of them with an archive candidate, the rest lone par2 files and scrambled
|
||||
// names, so no join on the candidates. A scan of releases (9 GB) a full round.
|
||||
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
|
||||
WHERE r.files = 1 AND r.segments = 1 AND r.bytes >= ${TINY_MIN_BYTES} AND r.bytes < ${NONSENSE_MIN_BYTES}
|
||||
AND r.readable AND NOT coalesce(r.sidecar_only, false)
|
||||
AND r.name ~* '${TINY_RELEASE_LIKE}' AND r.name !~* '${TINY_EBOOK}' AND r.name !~* '${TINY_CONTENT}'
|
||||
AND NOT EXISTS (SELECT 1 FROM group_sets gs WHERE gs.id = r.group_set_id AND gs.groups::text ~* '${TINY_EXEMPT_GROUPS}')`,
|
||||
},
|
||||
{
|
||||
name: 'unusable',
|
||||
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
|
||||
|
||||
+38
-30
@@ -6,9 +6,10 @@ import { describeClass } from '#lib/release/index'
|
||||
|
||||
/**
|
||||
* The pending ladder's classes (LOG.md (130)) on the rows from before them: `klass` (the SQL pending_class) and
|
||||
* `target_bytes` (the first target's bytes, when its candidates carry them) set in windows of created_at; the
|
||||
* finished rows among them (named, exhausted, naming) seed each class's record, which gets its verdict (dropped) as
|
||||
* the ladder gives it; then pending_classes' `waiting` counted afresh from the rows. --report shows the record.
|
||||
* `target_bytes` (the first target's bytes, when its candidates carry them). The finished rows (named, exhausted,
|
||||
* naming) in one scan, seeding each class's record, which gets its verdict (dropped) as the ladder gives it; the
|
||||
* waiting rows in windows of created_at by their index (a scan per window starved the prober on the HDD); then
|
||||
* pending_classes' `waiting` counted afresh from the claim's index. --report shows the record.
|
||||
*/
|
||||
export default class PendingClasses extends BaseCommand {
|
||||
static commandName = 'pending:classes'
|
||||
@@ -25,45 +26,52 @@ export default class PendingClasses extends BaseCommand {
|
||||
async run() {
|
||||
if (this.report) return this.showReport()
|
||||
const hours = this.hours ?? 6
|
||||
const { rows: span } = await db.rawQuery(
|
||||
`SELECT min(created_at) AS lo, max(created_at) AS hi FROM pending_releases WHERE klass = 0`
|
||||
const started = Date.now()
|
||||
// The finished rows (named, exhausted, naming; 6.5M on 2026-11-02) in one pass: a scan of the table, no index
|
||||
// reaches them by status; they seed each class's record.
|
||||
const { rows: seeded } = await db.rawQuery(
|
||||
`WITH classed AS (
|
||||
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
||||
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
||||
WHERE klass = 0 AND status IN (1, 3, 4)
|
||||
RETURNING tool, group_id, klass, status),
|
||||
upserted AS (
|
||||
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
|
||||
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed GROUP BY 1, 2, 3
|
||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
|
||||
RETURNING 1)
|
||||
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM upserted) AS classes`
|
||||
)
|
||||
if (!span[0].lo) {
|
||||
this.logger.info('No rows without a class.')
|
||||
} else {
|
||||
this.logger.info(
|
||||
`Finished rows: ${seeded[0].classed} classed into ${seeded[0].classes} classes (${Math.round((Date.now() - started) / 1000)} s).`
|
||||
)
|
||||
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
|
||||
await db.rawQuery(
|
||||
`UPDATE pending_classes SET dropped = true, swept_at = NULL WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
|
||||
[pendingMinProbes, pendingMaxNamed]
|
||||
)
|
||||
// The waiting rows by windows of created_at (pending_releases_waiting_idx reaches them).
|
||||
const { rows: span } = await db.rawQuery(
|
||||
`SELECT (SELECT min(created_at) FROM pending_releases WHERE status = 0) AS lo, (SELECT max(created_at) FROM pending_releases WHERE status = 0) AS hi`
|
||||
)
|
||||
if (span[0].lo) {
|
||||
let from = new Date(span[0].lo)
|
||||
const hi = new Date(span[0].hi)
|
||||
let classed = 0
|
||||
const started = Date.now()
|
||||
while (from <= hi) {
|
||||
const to = new Date(from.getTime() + hours * 3600_000)
|
||||
// Rows of the window classed; the finished ones among them (named, exhausted, naming) seed the record.
|
||||
const { rows } = await db.rawQuery(
|
||||
`WITH classed AS (
|
||||
UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
||||
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
||||
WHERE klass = 0 AND created_at >= ? AND created_at < ?
|
||||
RETURNING tool, group_id, klass, status),
|
||||
seeded AS (
|
||||
INSERT INTO pending_classes (tool, group_id, klass, probed, named)
|
||||
SELECT tool, group_id, klass, count(*), count(*) FILTER (WHERE status IN (1, 4)) FROM classed WHERE status IN (1, 3, 4) GROUP BY 1, 2, 3
|
||||
ON CONFLICT (tool, group_id, klass) DO UPDATE SET probed = pending_classes.probed + excluded.probed, named = pending_classes.named + excluded.named
|
||||
RETURNING 1)
|
||||
SELECT (SELECT count(*) FROM classed) AS classed, (SELECT count(*) FROM seeded) AS seeded`,
|
||||
const result = await db.rawQuery(
|
||||
`UPDATE pending_releases SET klass = pending_class(name, bytes, partial),
|
||||
target_bytes = (SELECT min((c->>'bytes')::bigint) FROM jsonb_array_elements(candidates) c WHERE c->>'bytes' IS NOT NULL)
|
||||
WHERE status = 0 AND created_at >= ? AND created_at < ? AND klass = 0`,
|
||||
[from.toISOString(), to.toISOString()]
|
||||
)
|
||||
classed += Number(rows[0].classed)
|
||||
classed += result.rowCount ?? 0
|
||||
from = to
|
||||
this.logger.info(
|
||||
`Up to ${to.toISOString().slice(0, 16)}: ${classed} rows classed (${Math.round((Date.now() - started) / 1000)} s).`
|
||||
`Waiting rows up to ${to.toISOString().slice(0, 16)}: ${classed} classed (${Math.round((Date.now() - started) / 1000)} s).`
|
||||
)
|
||||
}
|
||||
// The record's verdicts on the seeded classes.
|
||||
const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
|
||||
await db.rawQuery(
|
||||
`UPDATE pending_classes SET dropped = true WHERE NOT dropped AND probed >= ?::int AND named::float <= probed * ?::float`,
|
||||
[pendingMinProbes, pendingMaxNamed]
|
||||
)
|
||||
}
|
||||
// The waiting counts from the rows themselves (the claim, the discard and the batch store keep them from here).
|
||||
await db.rawQuery(`
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The prober's start puts a previous run's claims back (PendingLadder.release: status 9 → 0), which scanned the
|
||||
* 58M-row pending table on the HDD for 20+ minutes after every deploy (LOG.md (130)): a partial index on the few
|
||||
* taken rows makes it instant.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
static disableTransactions = true
|
||||
|
||||
async up() {
|
||||
this.schema.raw(
|
||||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_taken_idx ON pending_releases (release_key) WHERE status = 9`
|
||||
)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_taken_idx`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The prober's start puts a previous run's claims back (probe_candidates status taken → waiting), which scanned
|
||||
* the 7.8 GB candidates table on the HDD for 4 minutes after every deploy (LOG.md (133)): a partial index on the
|
||||
* few taken rows makes it instant.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
static disableTransactions = true
|
||||
|
||||
async up() {
|
||||
this.schema.raw(
|
||||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS probe_candidates_taken_idx ON probe_candidates (id) WHERE status = 'taken'`
|
||||
)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS probe_candidates_taken_idx`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The ladder's discard sweep (PendingLadder.discard) reads a dropped signature's or class's waiting rows once:
|
||||
* `swept_at` marks it done (null again when the record drops it anew). Before, every sweep read every dropped
|
||||
* signature's rows over again, minutes of HDD each, inside the claim (LOG.md (133)).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.alterTable('pending_signatures', (table) => {
|
||||
table.timestamp('swept_at', { useTz: true }).nullable()
|
||||
})
|
||||
this.schema.alterTable('pending_classes', (table) => {
|
||||
table.timestamp('swept_at', { useTz: true }).nullable()
|
||||
})
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.alterTable('pending_signatures', (table) => {
|
||||
table.dropColumn('swept_at')
|
||||
})
|
||||
this.schema.alterTable('pending_classes', (table) => {
|
||||
table.dropColumn('swept_at')
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The discard sweep of a dropped signature (tool, group, poster) read the whole pending_releases table per
|
||||
* signature: the planner took a parallel seq scan over the poster test, four minutes on the HDD that starved the
|
||||
* lanes' claims (LOG.md (134)). The waiting rows by signature make it an index range. By the poster's md5: a
|
||||
* poster of 2.7 kB broke the first build (a btree key is at most a third of a page), which left an invalid index
|
||||
* behind that IF NOT EXISTS would have taken for the real one; the drop clears it.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
static disableTransactions = true
|
||||
|
||||
async up() {
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
|
||||
this.schema.raw(
|
||||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_signature_idx ON pending_releases (group_id, tool, md5(poster)) WHERE status = 0`
|
||||
)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_signature_idx`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* probe_candidates, with its indexes, in the catalog's tablespace (default_tablespace: nvme on homelab), out of
|
||||
* CHUNK_TABLESPACE where migration 6000 put it: the prober's catalog claim updates it from random points every
|
||||
* minute, and on the HDD, beside the pending ladder's sweep and autovacuum, a claim took 2–3 minutes with all the
|
||||
* lanes waiting (LOG.md 2026-11-03 (136)). Homelab moved it by hand (edge, node-api and the workers stopped; ~8 GB);
|
||||
* a no-op where it's there already, or without a default tablespace (tests).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.defer(async (db) => {
|
||||
const { rows: setting } = await db.rawQuery(
|
||||
`SELECT current_setting('default_tablespace') AS target`
|
||||
)
|
||||
const target = setting[0].target
|
||||
if (!target) return
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||
WHERE (c.relname = 'probe_candidates' OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = 'probe_candidates'::regclass))
|
||||
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
|
||||
ORDER BY c.relkind DESC, c.relname`,
|
||||
[target]
|
||||
)
|
||||
for (const { relname, relkind } of rows) {
|
||||
await db.rawQuery(
|
||||
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Everything the prober reads or writes in the catalog's tablespace (default_tablespace: nvme on homelab), out of
|
||||
* CHUNK_TABLESPACE (the HDD): pending_releases with its two HDD indexes (migration 6300), release_inspections
|
||||
* (6000), probe_cache with its toast. The edge's batch upserts into pending_releases took minutes each on the HDD and
|
||||
* the prober's flushes waited on them (LOG.md 2026-11-03 (137)); and with these on the NVMe the prober can run
|
||||
* through the night without the HDD. Homelab moved them by hand (edge, node-api and the workers stopped; ~66 GB);
|
||||
* a no-op where they're there already, or without a default tablespace (tests).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.defer(async (db) => {
|
||||
const { rows: setting } = await db.rawQuery(
|
||||
`SELECT current_setting('default_tablespace') AS target`
|
||||
)
|
||||
const target = setting[0].target
|
||||
if (!target) return
|
||||
for (const table of ['pending_releases', 'release_inspections', 'probe_cache']) {
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT c.relname, c.relkind FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||
WHERE (c.relname = ? OR c.oid IN (SELECT indexrelid FROM pg_index WHERE indrelid = ?::regclass))
|
||||
AND c.relkind IN ('r', 'i') AND coalesce(t.spcname, 'pg_default') <> ?
|
||||
ORDER BY c.relkind DESC, c.relname`,
|
||||
[table, table, target]
|
||||
)
|
||||
for (const { relname, relkind } of rows) {
|
||||
await db.rawQuery(
|
||||
`ALTER ${relkind === 'i' ? 'INDEX' : 'TABLE'} ${relname} SET TABLESPACE ${target}`
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
+3
-2
@@ -47,8 +47,9 @@ echo "== recreating app services"
|
||||
docker compose stop edge
|
||||
if [ -e "$CACHE/paused" ]; then
|
||||
# Paused for the night (~/bin/usenet-rebuild-pause.sh): the workers stay stopped, the resume starts the new build.
|
||||
echo "== paused: coordinator, prober, merger, judge, spotter, spot-inspector and edge left stopped"
|
||||
docker compose up -d --no-deps node-api web
|
||||
# The prober runs through the night (2026-11-03, LOG.md (137)), so it gets the new build now.
|
||||
echo "== paused: coordinator, merger, judge, spotter, spot-inspector and edge left stopped"
|
||||
docker compose up -d --no-deps node-api web prober
|
||||
else
|
||||
docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
|
||||
if [ -e "$CACHE/no-edge" ]; then
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
#!/bin/sh
|
||||
# Installed as ~/bin/usenet-rebuild-pause.sh (by hand). Nightly (homelab crontab, 23:45): voyager 2's processing paused for the night; usenet-rebuild-resume.sh (07:45)
|
||||
# starts it again. Postgres, web and node-api stay up. Takes the deploy lock, so it never runs during a CI deploy, and
|
||||
# leaves .cache/paused, so a deploy in the night doesn't start the workers again.
|
||||
# starts it again. Postgres, web, node-api and the prober stay up: since 2026-11-03 (LOG.md (137)) everything the
|
||||
# prober touches is on the NVMe, so it probes through the night without the HDD. Takes the deploy lock, so it never
|
||||
# runs during a CI deploy, and leaves .cache/paused, so a deploy in the night doesn't start the workers again.
|
||||
cd /mnt/bulk/usenet-rebuild || exit 1
|
||||
exec 9>.cache/deploy.lock
|
||||
flock 9
|
||||
touch .cache/paused
|
||||
date "+%F %T pausing"
|
||||
docker compose stop edge
|
||||
docker compose stop coordinator prober merger judge spotter spot-inspector
|
||||
docker compose stop coordinator merger judge spotter spot-inspector
|
||||
docker compose ps --format "{{.Service}} {{.Status}}"
|
||||
@@ -77,6 +77,8 @@ pub struct RulesJson {
|
||||
pub drop_stub_archives: Option<bool>,
|
||||
#[serde(default)]
|
||||
pub drop_nonsense_pieces: Option<bool>,
|
||||
#[serde(default)]
|
||||
pub drop_tiny_archives: Option<bool>,
|
||||
}
|
||||
|
||||
impl RulesJson {
|
||||
@@ -107,6 +109,9 @@ impl RulesJson {
|
||||
if let Some(d) = self.drop_nonsense_pieces {
|
||||
rules.drop_nonsense_pieces = d;
|
||||
}
|
||||
if let Some(d) = self.drop_tiny_archives {
|
||||
rules.drop_tiny_archives = d;
|
||||
}
|
||||
rules
|
||||
}
|
||||
}
|
||||
+80
-6
@@ -1115,7 +1115,7 @@ fn token_signals(t: &[u8]) -> TokenSignals {
|
||||
let mut lower = 0;
|
||||
for i in 0..t.len() {
|
||||
let c = t[i];
|
||||
if c.is_ascii_alphabetic() && !is_vowel(c) {
|
||||
if c.is_ascii_alphabetic() && !is_vowel(c) && c.to_ascii_lowercase() != b'y' {
|
||||
consonants += 1;
|
||||
s.max_consonants = s.max_consonants.max(consonants);
|
||||
} else {
|
||||
@@ -1181,7 +1181,9 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
|
||||
return if n >= 3 { TokenKind::Word } else { TokenKind::Neutral };
|
||||
}
|
||||
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
|
||||
if n >= 6 && s.case_switches >= 2 && (s.inner_upper || (s.vowels * 4 < n && (s.runs >= 4 || s.lone >= 1))) {
|
||||
// Two CamelCase words with few vowels (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone
|
||||
// letter; a scramble has more runs, a lone letter or an inner capital run.
|
||||
if n >= 6 && s.case_switches >= 2 && s.vowels * 4 < n && (s.runs >= 5 || s.lone >= 1 || s.inner_upper) {
|
||||
return TokenKind::Random;
|
||||
}
|
||||
return TokenKind::Neutral;
|
||||
@@ -1193,6 +1195,14 @@ pub fn token_kind(t: &[u8]) -> TokenKind {
|
||||
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
||||
return if n >= 12 && token_signals(t).switches >= 6 { TokenKind::Random } else { TokenKind::Neutral };
|
||||
}
|
||||
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
|
||||
if t.split(|c: &u8| !c.is_ascii_alphabetic()).filter(|w| !w.is_empty()).all(|w| {
|
||||
let vowels = w.iter().filter(|c| matches!(c.to_ascii_lowercase(), b'a' | b'e' | b'i' | b'o' | b'u' | b'y')).count();
|
||||
(w.len() >= 2 && w.iter().all(u8::is_ascii_uppercase))
|
||||
|| (w.len() >= 2 && w[1..].iter().all(u8::is_ascii_lowercase) && (w.len() >= 3 || w[0].is_ascii_lowercase()) && vowels * 5 >= w.len())
|
||||
}) {
|
||||
return TokenKind::Neutral;
|
||||
}
|
||||
let s = token_signals(&scene_case(t));
|
||||
if n >= 5 && s.switches >= 3 {
|
||||
return TokenKind::Random;
|
||||
@@ -1505,6 +1515,11 @@ static RELEASE_LIKE: LazyLock<Regex> = LazyLock::new(|| {
|
||||
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._-](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
|
||||
});
|
||||
|
||||
/// RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way.
|
||||
static RELEASE_LIKE_SPACED: LazyLock<Regex> = LazyLock::new(|| {
|
||||
re(r"(?i)(\b(19|20)[0-9][0-9]\b|[._ -](720p|1080p|2160p|x264|x265|hevc|bluray|web-?dl|webrip|hdtv|xvid|s[0-9][0-9]e[0-9][0-9]|german|multi|crack|repack|proper|flac|mp3|xxx|mp4|retail|keygen|portable|incl|v[0-9]+(\.[0-9]+)+)\b|-[A-Za-z0-9]{2,}$|\.[A-Za-z0-9]+\.[A-Za-z0-9]+\.)")
|
||||
});
|
||||
|
||||
/// The stub-archive rule (LOG.md (123)-(126), (128)): one unsplit archive, no par2, settled, posted to 2+ groups,
|
||||
/// under STUB_MAX_BYTES (the fakes stop at ~90 MB; the hash-named uploads, real albums and passworded videos above
|
||||
/// it don't), and its articles at most STUB_ARTICLE_BYTES with the release at least STUB_MIN_BYTES (Power-Post's
|
||||
@@ -1518,6 +1533,8 @@ pub const STUB_WIDE_GROUPS: usize = 4;
|
||||
/// release-like name that isn't scrambled (the newznzb.* pieces; under 100 kB are cue/log sidecars and adverts).
|
||||
pub const NONSENSE_MIN_BYTES: u64 = 100_000;
|
||||
pub const NONSENSE_MAX_BYTES: u64 = 1_000_000;
|
||||
/// The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars.
|
||||
pub const TINY_MIN_BYTES: u64 = 2_000;
|
||||
|
||||
pub struct JudgedFile<'a> {
|
||||
pub name: &'a str,
|
||||
@@ -1579,6 +1596,27 @@ pub fn is_nonsense_piece(f: &JudgedFile) -> bool {
|
||||
&& !is_random_file(f.name)
|
||||
}
|
||||
|
||||
/// Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
|
||||
/// exe is the content), covers and sidecars.
|
||||
static TINY_CONTENT: LazyLock<Regex> = LazyLock::new(|| {
|
||||
re(r"(?i)\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b")
|
||||
});
|
||||
|
||||
/// A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
|
||||
/// unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
|
||||
/// subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
|
||||
pub fn is_tiny_archive(f: &JudgedFile) -> bool {
|
||||
f.total_parts == 1
|
||||
&& f.segments == 1
|
||||
&& (TINY_MIN_BYTES..NONSENSE_MIN_BYTES).contains(&f.bytes)
|
||||
&& !f.groups.iter().any(|g| test(&STUB_EXEMPT_GROUPS, g))
|
||||
&& test(&RELEASE_LIKE_SPACED, &file_stem(f.name))
|
||||
&& !test(&EBOOK, f.name)
|
||||
&& !test(&TINY_CONTENT, f.name)
|
||||
&& !looks_scrambled(f.name)
|
||||
&& !is_random_file(f.name)
|
||||
}
|
||||
|
||||
pub enum Verdict {
|
||||
Keep,
|
||||
Drop { rule: &'static str, key: String, sample: bool },
|
||||
@@ -1600,9 +1638,11 @@ pub struct Rules {
|
||||
pub drop_executables: bool,
|
||||
/// Fragments of the poster lowercased without punctuation (droppedPosters).
|
||||
pub dropped_posters: Vec<String>,
|
||||
/// The stub-archive rule (dropStubArchives) and the nonsense-piece rule (dropNonsensePieces).
|
||||
/// The stub-archive rule (dropStubArchives), the nonsense-piece rule (dropNonsensePieces) and the tiny-archive
|
||||
/// rule (dropTinyArchives).
|
||||
pub drop_stub_archives: bool,
|
||||
pub drop_nonsense_pieces: bool,
|
||||
pub drop_tiny_archives: bool,
|
||||
}
|
||||
|
||||
impl Rules {
|
||||
@@ -1620,6 +1660,7 @@ impl Rules {
|
||||
dropped_posters: ["celebmovie", "celebwow", "celebfx", "iluvcelebs", "nudecelebs", "celebsextapes"].iter().map(|p| p.to_string()).collect(),
|
||||
drop_stub_archives: true,
|
||||
drop_nonsense_pieces: true,
|
||||
drop_tiny_archives: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1655,6 +1696,9 @@ pub fn judge(release_key: &str, files: &[JudgedFile], settled: bool, rules: &Rul
|
||||
if rules.drop_nonsense_pieces && is_nonsense_piece(f) {
|
||||
return Verdict::Drop { rule: "nonsense-piece", key: upload_tool(f.first_message_id).to_string(), sample };
|
||||
}
|
||||
if rules.drop_tiny_archives && is_tiny_archive(f) {
|
||||
return Verdict::Drop { rule: "tiny-archive", key: upload_tool(f.first_message_id).to_string(), sample };
|
||||
}
|
||||
}
|
||||
if files.iter().any(|f| !is_random_file(f.name)) {
|
||||
return Verdict::Keep;
|
||||
@@ -1854,7 +1898,8 @@ mod tests {
|
||||
"Rocketship-Hurt_Feelings-WEB-2026-PreMaxXxing", "Drum-and-BassDJ_Samurai-Sound_With_Fire_Artemis-(FQY026)-WEB-2006-PRESCRiPTiON",
|
||||
"Kleverig-H4rd-N12uk3b18-luc16037-720p-smpwr", "Eclipse_IDE_2023-12_R_win32_x86_64.zip", "PartsHPGL 1.06c", "Go4Customer Fax Machine v1.2b",
|
||||
"AdBeGone 1.2", "3DMark03 320", "COD4MW2REMASTERED", "Movie.2020.1080p-GRP [a3f9e2b1]", "EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps",
|
||||
"iPhone", "Some.Show.S01E01.1080p.mkv",
|
||||
"iPhone", "Some.Show.S01E01.1080p.mkv", "FastFilms 2.04", "BestCrypt NP v4.0", "EasyHTML 2.35 German", "SchiffsModell - September 2026",
|
||||
"CrossStitcher - Issue 440, October 2026", "Laura15Juli2026", "IHaveAWife - Isa Bella", "DickHDDaily - Santana Red", "1904-SmithsWork-Hasluck",
|
||||
] {
|
||||
assert!(!is_unreadable_name(n), "{n}");
|
||||
}
|
||||
@@ -1982,8 +2027,8 @@ mod single_archive_rule_tests {
|
||||
let rules = Rules::defaults(Default::default());
|
||||
// One 790 kB article under a release-like name: the newznzb.* pieces.
|
||||
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
|
||||
// A 3 kB cue/log sidecar rar with a year in its name, and a diet advert: under the floor.
|
||||
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), None);
|
||||
// A 3 kB cue/log sidecar rar with a year in its name: under this rule's floor (the tiny-archive rule's outside the music groups).
|
||||
assert_eq!(verdict("The Smiths-the queen is dead(Remaster)(2011).rar", "Yenc@power-post.org (Sir Cedrik)", 1, 1, 3_000, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||
// A book (a year or a format in its name, written with spaces), a dotted one too: not a piece.
|
||||
assert_eq!(verdict("Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar", "fake@address.com (Psychopath)", 2, 1, 303_000, true, &rules), None);
|
||||
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 382_000, true, &rules), None);
|
||||
@@ -1995,6 +2040,35 @@ mod single_archive_rule_tests {
|
||||
let off = Rules { drop_nonsense_pieces: false, ..Rules::defaults(Default::default()) };
|
||||
assert_eq!(verdict("Some.Show.S01E01.1080p.WEB-DL.rar", "poster@example.org", 1, 1, 790_000, true, &off), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tiny_archives() {
|
||||
let rules = Rules::defaults(Default::default());
|
||||
// One article of 2-100 kB under a release name, dotted or with spaces: a RAR holding "<title>.exe" (LOG.md (131)).
|
||||
assert_eq!(verdict("Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar", "Yanc@power-post.org <Yanc@power-post.org>", 1, 1, 15_084, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||
assert_eq!(verdict("Cicero Magazin fuer politische Kultur Mai 05 2016.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 17_478, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||
// A scene name without a year, a tag or a group suffix: not release-like, so kept (the flood's porn names mostly carry one).
|
||||
assert_eq!(verdict("[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar", "Yanc@power-post.org", 2, 1, 23_449, true, &rules), None);
|
||||
assert_eq!(verdict("Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar", "WarezGeek@WarezGeek.org (WarezGeek)", 1, 1, 7_869, true, &rules), Some(("tiny-archive", "nyuu".to_string())));
|
||||
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
|
||||
assert_eq!(verdict("The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar", "henk@example.org", 1, 1, 38_000, true, &rules), None);
|
||||
assert_eq!(verdict("Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar", "poster@example.org", 1, 1, 60_000, true, &rules), None);
|
||||
assert_eq!(verdict("Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
|
||||
assert_eq!(verdict("Some Band - Album (2011) covers.rar", "poster@example.org", 1, 1, 45_000, true, &rules), None);
|
||||
assert_eq!(verdict("One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar", "fake@address.com (Psychopath)", 1, 1, 38_000, true, &rules), None);
|
||||
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's.
|
||||
let groups = vec!["alt.binaries.sounds.karaoke".to_string()];
|
||||
let files = [JudgedFile { name: "KV-1055806 - Walker Hayes - U Gurl (2017).zip", total_parts: 1, groups: &groups, first_message_id: NYUU, poster: "home@home.com", segments: 1, bytes: 60_000 }];
|
||||
assert!(matches!(judge("k", &files, true, &rules), Verdict::Keep));
|
||||
assert_eq!(verdict("holiday snaps.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
|
||||
assert_eq!(verdict("nYDeXeF49dieZ1J7flK-JFzZ3j.rar", "poster@example.org", 1, 1, 20_000, true, &rules), None);
|
||||
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 2, 20_000, true, &rules), None);
|
||||
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 1_500, true, &rules), None);
|
||||
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 150_000, true, &rules), Some(("nonsense-piece", "nyuu".to_string())));
|
||||
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, false, &rules), None);
|
||||
let off = Rules { drop_tiny_archives: false, ..Rules::defaults(Default::default()) };
|
||||
assert_eq!(verdict("Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar", "poster@example.org", 1, 1, 20_000, true, &off), None);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -66,7 +66,7 @@ export interface ProbeCandidate {
|
||||
|
||||
/** What a rule dropped in a batch (REBUILD.md, rules and recovery). */
|
||||
export interface RuleCount {
|
||||
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'incomplete'
|
||||
rule: 'crosspost' | 'obfuscated' | 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive' | 'incomplete'
|
||||
key: string
|
||||
articles: number
|
||||
files: number
|
||||
|
||||
+15
-3
@@ -43,7 +43,7 @@ function signals(t: string) {
|
||||
let maxConsonants = 0
|
||||
for (let i = 0; i < t.length; i++) {
|
||||
const c = t[i]
|
||||
if (/[b-df-hj-np-tv-z]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
|
||||
if (/[b-df-hj-np-tv-xz]/i.test(c)) maxConsonants = Math.max(maxConsonants, ++consonants)
|
||||
else consonants = 0
|
||||
if (i === 0) continue
|
||||
const a = t[i - 1]
|
||||
@@ -90,11 +90,14 @@ export function tokenKind(t: string): 'word' | 'random' | null {
|
||||
if (t.length >= 10 && s.vowels * 6 < t.length) return 'random'
|
||||
return t.length >= 3 ? 'word' : null
|
||||
}
|
||||
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble, which has few vowels and short runs.
|
||||
// Mixed case beyond Capitalization: CamelCase (neutral) or a scramble. Two CamelCase words with few vowels
|
||||
// (FastFilms, BestCrypt, SchiffsModell) have four runs and no lone letter; a scramble has more runs, a lone
|
||||
// letter or an inner capital run.
|
||||
if (
|
||||
t.length >= 6 &&
|
||||
s.caseSwitches >= 2 &&
|
||||
(s.innerUpper || (s.vowels * 4 < t.length && (s.runs >= 4 || s.lone >= 1)))
|
||||
s.vowels * 4 < t.length &&
|
||||
(s.runs >= 5 || s.lone >= 1 || s.innerUpper)
|
||||
)
|
||||
return 'random'
|
||||
return null
|
||||
@@ -104,6 +107,15 @@ export function tokenKind(t: string): 'word' | 'random' | null {
|
||||
// 51294P1CUKR2R35X; COD4MW2REMASTERED (4 switches) and BEATLES1964REMASTER (2) stay.
|
||||
return t.length >= 12 && signals(t).switches >= 6 ? 'random' : null
|
||||
}
|
||||
// Words with numbers between them (Laura15Juli2026, Back4WinXP): every letter run a word or an acronym.
|
||||
if (
|
||||
(t.match(/[A-Za-z]+/g) ?? []).every(
|
||||
(w) =>
|
||||
/^[A-Z]{2,}$/.test(w) ||
|
||||
(/^[A-Z]?[a-z]{2,}$/.test(w) && (w.match(/[aeiouy]/g) ?? []).length * 5 >= w.length)
|
||||
)
|
||||
)
|
||||
return null
|
||||
const s = signals(sceneCase(t))
|
||||
if (t.length >= 5 && s.switches >= 3) return 'random'
|
||||
if (t.length >= 6 && s.switches >= 2 && (s.lone >= 1 || s.innerUpper || s.maxConsonants >= 4))
|
||||
|
||||
+1
-1
@@ -1,2 +1,2 @@
|
||||
export { uploadTool, signatureGroup, signatureKey, explored } from './upload_tools.js'
|
||||
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
|
||||
export { RuleSet, DEFAULT_RULES, isRandomFile, isScrambledFile, ownerOf, posterWords, stubArchiveLegs, isNonsensePiece, isTinyArchive, STUB_ARTICLE_BYTES, STUB_MIN_BYTES, STUB_MAX_BYTES, STUB_WIDE_GROUPS, NONSENSE_MIN_BYTES, NONSENSE_MAX_BYTES, TINY_MIN_BYTES, type Rules, type Verdict, type JudgedFile } from './rule_set.js'
|
||||
+43
-1
@@ -47,6 +47,12 @@ export interface Rules {
|
||||
* the newznzb.* pieces (9% of the catalog; LOG.md (123), (125)).
|
||||
*/
|
||||
dropNonsensePieces: boolean
|
||||
/**
|
||||
* Drop one-article archives of TINY_MIN_BYTES to NONSENSE_MIN_BYTES under a release-like name (spaces allowed),
|
||||
* outside the exempt groups, without an e-book, subtitle, crack or sidecar word: the 2015-17 exe-stub flood in
|
||||
* miniature (a RAR holding "<title>.exe" at 2-60 kB; 1.65M catalog rows; LOG.md (131), (133)).
|
||||
*/
|
||||
dropTinyArchives: boolean
|
||||
}
|
||||
|
||||
export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
|
||||
@@ -58,6 +64,7 @@ export const DEFAULT_RULES: Omit<Rules, 'priorities' | 'droppedSignatures'> = {
|
||||
droppedPosters: ['celebmovie', 'celebwow', 'celebfx', 'iluvcelebs', 'nudecelebs', 'celebsextapes'],
|
||||
dropStubArchives: true,
|
||||
dropNonsensePieces: true,
|
||||
dropTinyArchives: true,
|
||||
droppedClasses: [],
|
||||
}
|
||||
|
||||
@@ -69,8 +76,12 @@ export const STUB_WIDE_GROUPS = 4
|
||||
/** The nonsense-piece rule's bounds: under 100 kB are cue/log sidecars and adverts, 1 MB and up is rule 1's. */
|
||||
export const NONSENSE_MIN_BYTES = 100_000
|
||||
export const NONSENSE_MAX_BYTES = 1_000_000
|
||||
/** The tiny-archive rule's floor: a RAR holding one exe stub is 2 kB and up; below are empty archives and nfo rars. */
|
||||
export const TINY_MIN_BYTES = 2_000
|
||||
|
||||
export type Verdict = { rule: null } | { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece'; key: string; sample: boolean }
|
||||
export type Verdict =
|
||||
| { rule: null }
|
||||
| { rule: 'poster' | 'executable' | 'garbage' | 'per-article' | 'stub-archive' | 'nonsense-piece' | 'tiny-archive'; key: string; sample: boolean }
|
||||
|
||||
/** A file as the judge sees it: its name, parts, groups, poster, and the articles seen with their bytes. */
|
||||
export interface JudgedFile {
|
||||
@@ -148,6 +159,36 @@ export function isNonsensePiece(f: JudgedFile): boolean {
|
||||
)
|
||||
}
|
||||
|
||||
/** RELEASE_LIKE with spaces as separators too: the tiny fakes are posted under names written either way. */
|
||||
const RELEASE_LIKE_SPACED = new RegExp(RELEASE_LIKE.source.replace('[._-]', '[._ -]'), 'i')
|
||||
|
||||
/**
|
||||
* Words of the real tiny archives under release-like names: subtitles, cracks, trainers and keygens (a few kB of
|
||||
* exe is the content), covers and sidecars.
|
||||
*/
|
||||
const TINY_CONTENT =
|
||||
/\b(subs?|subtitles?|subpack|srt|untertitel|ondertitels?|ondertiteling|legendas?|napisy|sottotitoli|cracks?|cracked|keygen|keymaker|trainer|patch|patched|serials?|activator|loader|nocd|no-cd|fix|covers?|artwork|booklet|scans?|proof|sample|info|readme)\b/i
|
||||
|
||||
/**
|
||||
* A tiny archive: one article of TINY_MIN_BYTES to NONSENSE_MIN_BYTES holding one archive under a release-like,
|
||||
* unscrambled name (spaces allowed), outside the groups that post real small archives, and without an e-book,
|
||||
* subtitle, crack or sidecar word: the 2015-17 exe-stub flood in miniature (LOG.md (131)).
|
||||
*/
|
||||
export function isTinyArchive(f: JudgedFile): boolean {
|
||||
return (
|
||||
f.totalParts === 1 &&
|
||||
f.segments === 1 &&
|
||||
f.bytes >= TINY_MIN_BYTES &&
|
||||
f.bytes < NONSENSE_MIN_BYTES &&
|
||||
!f.groups.some((g) => STUB_EXEMPT_GROUPS.test(g)) &&
|
||||
RELEASE_LIKE_SPACED.test(fileStem(f.name)) &&
|
||||
!EBOOK.test(f.name) &&
|
||||
!TINY_CONTENT.test(f.name) &&
|
||||
!isScrambledFile(f.name) &&
|
||||
!isRandomFile(f.name)
|
||||
)
|
||||
}
|
||||
|
||||
export function isRandomFile(name: string) {
|
||||
return looksRandom(fileStem(name))
|
||||
}
|
||||
@@ -221,6 +262,7 @@ export class RuleSet {
|
||||
if (legs !== '') return { rule: 'stub-archive', key: legs, sample }
|
||||
}
|
||||
if (this.rules.dropNonsensePieces && isNonsensePiece(archive)) return { rule: 'nonsense-piece', key: uploadTool(archive.firstMessageId), sample }
|
||||
if (this.rules.dropTinyArchives && isTinyArchive(archive)) return { rule: 'tiny-archive', key: uploadTool(archive.firstMessageId), sample }
|
||||
}
|
||||
if (files.some((f) => !isRandomFile(f.name))) return { rule: null }
|
||||
const first = files[0]
|
||||
|
||||
@@ -163,6 +163,55 @@ test.group('ReleaseJudge', (group) => {
|
||||
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
|
||||
})
|
||||
|
||||
test('judges tiny single-article files under release-like names', async ({ assert }) => {
|
||||
let n = 0
|
||||
const settled = new Date(Date.now() - 3 * 86400_000)
|
||||
const release = async (name: string, extra: Record<string, unknown> = {}) => {
|
||||
n++
|
||||
const key = `jt${String(n).padStart(6, '0')}`.padEnd(40, '0')
|
||||
return Number(
|
||||
await insertRelease({
|
||||
release_key: key,
|
||||
identity_key: key,
|
||||
name,
|
||||
source: 'stem',
|
||||
groups: ['alt.binaries.boneless'],
|
||||
files: 1,
|
||||
segments: 1,
|
||||
bytes: 15_000,
|
||||
posted_at: settled,
|
||||
readable: true,
|
||||
settled_at: settled,
|
||||
...extra,
|
||||
})
|
||||
)
|
||||
}
|
||||
const dotted = await release('Velvet.S02E20.GERMAN.720p.HDTV.x264-ACED')
|
||||
const spaced = await release('Pacific Rim (2013) BDRip DVDr-sailo1')
|
||||
const subs = await release('Some.Film.2019.German.Subs-XYZ')
|
||||
const ebook = await release('Some.Book.2019.epub-GRP')
|
||||
const music = await release('Some.Album.2019.FLAC-GRP', { groups: ['alt.binaries.sounds.flac'] })
|
||||
const big = await release('Big.Film.2019.1080p-GRP', { bytes: 500_000 })
|
||||
const twoFiles = await release('Two.Films.2019.1080p-GRP', { files: 2, segments: 2 })
|
||||
const plain = await release('Holiday photo')
|
||||
const sidecar = await release('Side.Car.2019.1080p-GRP', { sidecar_only: true })
|
||||
const unreadable = await release('Xk9v.2019.Qpz-ab', { readable: false })
|
||||
|
||||
const round = await new ReleaseJudge().full()
|
||||
const verdicts = new Map(
|
||||
(await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [
|
||||
Number(v.release_id),
|
||||
v.rule,
|
||||
])
|
||||
)
|
||||
assert.equal(verdicts.get(dotted), 'tiny')
|
||||
assert.equal(verdicts.get(spaced), 'tiny')
|
||||
for (const id of [subs, ebook, music, big, twoFiles, plain, sidecar, unreadable]) {
|
||||
assert.notEqual(verdicts.get(id), 'tiny')
|
||||
}
|
||||
assert.equal(round.rules.tiny.added, 2)
|
||||
})
|
||||
|
||||
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
|
||||
assert,
|
||||
}) => {
|
||||
|
||||
@@ -52,6 +52,15 @@ test.group('release / isUnreadableName', () => {
|
||||
'EPL__NZBSPLIT__f1fdb433772a1a7771d6b73c875009ff__NZBSPLIT__EPL.2026.Liverpool.vs.Brighton.1080pEN50fps',
|
||||
'iPhone',
|
||||
'Some.Show.S01E01.1080p.mkv',
|
||||
'FastFilms 2.04',
|
||||
'BestCrypt NP v4.0',
|
||||
'EasyHTML 2.35 German',
|
||||
'SchiffsModell - September 2026',
|
||||
'CrossStitcher - Issue 440, October 2026',
|
||||
'Laura15Juli2026',
|
||||
'IHaveAWife - Isa Bella',
|
||||
'DickHDDaily - Santana Red',
|
||||
'1904-SmithsWork-Hasluck',
|
||||
]) {
|
||||
assert.isFalse(isUnreadableName(n), n)
|
||||
}
|
||||
|
||||
@@ -98,8 +98,9 @@ test.group('rules / verdicts', () => {
|
||||
|
||||
test('nonsense pieces: one article of 100 kB to 1 MB under a release-like name', ({ assert }) => {
|
||||
assert.include(rules.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: 'nonsense-piece', key: 'nyuu' })
|
||||
// A 3 kB cue/log sidecar rar with a year in its name: under the floor. A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
|
||||
assert.deepEqual(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: null })
|
||||
// A 3 kB cue/log sidecar rar with a year in its name: under the floor (the tiny-archive rule's outside the music groups). A book (spaces, a format). Two articles, a plain name, a scrambled one: not this rule.
|
||||
assert.deepEqual(rules.judge('k', [{ ...archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000), groups: ['alt.binaries.sounds.lossless'] }]), { rule: null })
|
||||
assert.include(rules.judge('k', [archive('The Smiths-the queen is dead(Remaster)(2011).rar', 'Yenc@power-post.org (Sir Cedrik)', 1, 1, 3_000)]), { rule: 'tiny-archive' })
|
||||
assert.deepEqual(rules.judge('k', [archive('Kevin Ashman - [Roman 02] - The Rise of Caratacus (2020) (epub).rar', 'fake@address.com (Psychopath)', 2, 1, 303_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 382_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Vieweg.Teubner.Verlag.-.Numerische.Mathematik.2011.German.RETAIL.eBook-BitBook.rar', 'saggir@domain.com', 1, 1, 113_000)]), { rule: null })
|
||||
@@ -109,6 +110,31 @@ test.group('rules / verdicts', () => {
|
||||
const off = new RuleSet({ ...DEFAULT_RULES, dropNonsensePieces: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
|
||||
assert.deepEqual(off.judge('k', [archive('Some.Show.S01E01.1080p.WEB-DL.rar', 'poster@example.org', 1, 1, 790_000)]), { rule: null })
|
||||
})
|
||||
|
||||
test('tiny archives: one article of 2-100 kB under a release name, spaces allowed', ({ assert }) => {
|
||||
// A RAR holding "<title>.exe": the 2015-17 flood in miniature (LOG.md (131)).
|
||||
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DD20.Dubbed.DL.1080p.BD.x264-TVS.rar', 'Yanc@power-post.org <Yanc@power-post.org>', 1, 1, 15_084)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||
assert.include(rules.judge('k', [archive('Cicero Magazin fuer politische Kultur Mai 05 2016.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 17_478)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||
// A scene name without a year, a tag or a group suffix: not release-like, so kept.
|
||||
assert.deepEqual(rules.judge('k', [archive('[BigTitsAtWork Brazzers] Eva Angelina (Camera Cums In Handy).rar', 'Yanc@power-post.org', 2, 1, 23_449)]), { rule: null })
|
||||
assert.include(rules.judge('k', [archive('Graphisoft.Archicad.v14.0.GERMAN.iSO.X86-PillePalle.rar', 'WarezGeek@WarezGeek.org (WarezGeek)', 1, 1, 7_869)]), { rule: 'tiny-archive', key: 'nyuu' })
|
||||
// Subtitles, a crack, a trainer, a cover, an e-book: the real tiny archives.
|
||||
assert.deepEqual(rules.judge('k', [archive('The.Big.Bang.Theory.S04E13.720p.HDTV.subs.rar', 'henk@example.org', 1, 1, 38_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Tipard.DVD.Ripper.v10.0.12.Crack-XXX.rar', 'poster@example.org', 1, 1, 60_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Far.Cry.4.v1.7.0.Plus.12.Trainer-FLiNG.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Some Band - Album (2011) covers.rar', 'poster@example.org', 1, 1, 45_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('One.Wrong.Turn.Deanna.Lynn.Sletten.epub.rar', 'fake@address.com (Psychopath)', 1, 1, 38_000)]), { rule: null })
|
||||
// A karaoke zip in its group; a plain name; a scrambled one; two articles; under the floor; above 100 kB is the nonsense-piece rule's; not settled.
|
||||
assert.deepEqual(rules.judge('k', [{ ...archive('KV-1055806 - Walker Hayes - U Gurl (2017).zip', 'home@home.com', 1, 1, 60_000), groups: ['alt.binaries.sounds.karaoke'] }]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('holiday snaps.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('nYDeXeF49dieZ1J7flK-JFzZ3j.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 2, 20_000)]), { rule: null })
|
||||
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 1_500)]), { rule: null })
|
||||
assert.include(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 150_000)]), { rule: 'nonsense-piece' })
|
||||
assert.deepEqual(rules.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)], false), { rule: null })
|
||||
const off = new RuleSet({ ...DEFAULT_RULES, dropTinyArchives: false, samplePercent: 0, priorities: {}, droppedSignatures: [] })
|
||||
assert.deepEqual(off.judge('k', [archive('Humans.S01E04.German.DL.1080p.BD.x264-TVS.rar', 'poster@example.org', 1, 1, 20_000)]), { rule: null })
|
||||
})
|
||||
})
|
||||
|
||||
test.group('rules / upload tools', () => {
|
||||
|
||||
Reference in new issue
Block a user