Single-archive rules: stub archives and nonsense pieces dropped at the edge
The judge gets each file's articles seen and bytes, and whether the window closed on the release. A settled release of one unsplit archive posted to 2+ groups is dropped as a stub archive when its articles are <= 256 KiB with the release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two under 100 MB); releases touching music, e-book, comics, tabs or fonts groups are exempt. A one-article archive of 100 kB-1 MB under a release-like, unscrambled name without spaces or an e-book format is dropped as a nonsense piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces), mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
2f1bbf2538
commit
e470e231d0
8 files changed
+397
-15
No files matched your search
+34
-2
@@ -151,7 +151,9 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||||
indexed groups in the article's Xref, the one with the lowest priority
|
||||
number; see the group list) → group articles into files → files
|
||||
into releases (`lib/release` rules) → drop garbage (random name +
|
||||
dropped signature or tool) → set aside per-article random subjects and
|
||||
dropped signature or tool), bare executables, dropped posters, and the
|
||||
single-archive fakes (stub archives and nonsense pieces, below) → set
|
||||
aside per-article random subjects and
|
||||
obfuscated subjects (counted only; phase 2). Returns per batch: release
|
||||
chunks, edge fragments, probe candidates, counters. Holds no state beyond
|
||||
its lease.
|
||||
@@ -776,7 +778,37 @@ its release and sets `partial` (the store: `complete = false`). A lone
|
||||
article of a multi-part random file is still counted `per-article`, not
|
||||
written; before the block's end a scrambled name counts as random too
|
||||
(`looks_scrambled`: a random subject and poster per article, 160k of
|
||||
alt.binaries.storage's releases; LOG.md (122)). Gone: fragments and the cut-off, `uploads_of` (online now),
|
||||
alt.binaries.storage's releases; LOG.md (122)).
|
||||
|
||||
**Single-archive rules (built 2026-10-31; LOG.md (123)-(127)).** The judge
|
||||
(`names::judge`, `RuleSet.judge`) gets each file's articles seen and their
|
||||
bytes, and whether the release is *settled* (the window closed on it: a
|
||||
par2 can still join an open one; the start edge doesn't count, since a par2
|
||||
follows its rar and 7% of the catalog's releases are start edges). A settled
|
||||
release of one unsplit zip, rar or 7z and nothing else is dropped as a
|
||||
**stub archive** (`stub-archive`, key: the legs tripped) when posted to
|
||||
2+ groups and (A) its articles are ≤ 256 KiB with the release ≥ 1 MB
|
||||
(Power-Post's 250 kB default; a one-article release always has small
|
||||
articles, which exempts e-books and sidecars), or (X) posted to 4+ groups,
|
||||
or (W) by a webmail poster, X and W under 100 MB (the fakes stop at
|
||||
~90 MB; hash-named uploads and real albums above it don't). A release
|
||||
touching a music, e-book, comics, tabs or fonts group is exempt: those
|
||||
communities post real single small archives the same way (karaoke zips to
|
||||
a.b.sounds.karaoke + a.b.test with Power-Post's articles; 16 of 32,308
|
||||
known fakes lost, LOG.md (127)). These are the
|
||||
exe-stub and recycled-installer floods under film and software names:
|
||||
55% of the catalog on 2026-10-30, 99.3% of the known fakes, 0 known
|
||||
content in the 217k sample, verified on 1,500 decoded articles. A settled
|
||||
single archive of one article and 100 kB–1 MB under a release-like,
|
||||
unscrambled name without spaces and without an e-book format in it is
|
||||
dropped as a **nonsense piece** (`nonsense-piece`, key: the upload tool):
|
||||
the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of
|
||||
the catalog; under 100 kB are cue/log sidecars and adverts, and the names
|
||||
with spaces are books and albums. Both rules are switched by the
|
||||
rule set (`dropStubArchives`, `dropNonsensePieces`), sampled like every
|
||||
drop, and mirrored in `lib/rules/rule_set.ts` (the TypeScript indexer
|
||||
judges as each batch completes files, so it passes settled without the
|
||||
window's knowledge). Gone: fragments and the cut-off, `uploads_of` (online now),
|
||||
`check.ts` and the fixture comparison (the parity with the TypeScript
|
||||
indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
|
||||
(incomplete releases are kept). The wire carries `finished`, `edge` and
|
||||
|
||||
Reference in new issue
Block a user