The judge gets each file's articles seen and bytes, and whether the window
closed on the release. A settled release of one unsplit archive posted to 2+
groups is dropped as a stub archive when its articles are <= 256 KiB with the
release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two
under 100 MB); releases touching music, e-book, comics, tabs or fonts groups
are exempt. A one-article archive of 100 kB-1 MB under a release-like,
unscrambled name without spaces or an e-book format is dropped as a nonsense
piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces),
mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.
looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.
The prefetcher's corpus fill skips groups that aren't active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.
`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A plain stop and a chunk that doesn't inflate already posted the batch so far,
so the stop snapshot stood at a stored resume point; a failed stream (the
stalled one's "Peer disconnected") skipped the post, left the batch open and
lost the window with the snapshot refused. Now only a batch the coordinator
refused stays open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A live reply cut short (the server's compressed data ending at ".\r\n" at a
read boundary; the spool keeps what Corpus.write refuses) made the worker give
the group back without a snapshot, losing the window and the batch since its
last post (six times in the first 40 minutes on homelab). Now the batch before
the chunk goes out, the chunk is reported (fetched again, plain the second
time) and the range is asked for again from that chunk (RangeOutcome::Again):
the stream waits for the new copy and the window stays. Test: the coordinator
hands the same range back at its resume point after a corrupt report.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
a.b.storage's stored 2016 ranges are not contiguous (its 2016 window lies in two
article regions); the batch after a gap kept the previous range's end as its
start and the harness found no leased range for it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
acs5152s1.11 is a volume, Broforce.vol00+01.par2.temp a par2: neither an
album's member. replay:corpus reported every release in the scratch catalog
as the group's.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A release the window closed on with one file named by its stem or post, the
name with a number and no archive, volume, sidecar or random name, waits in
the album of its poster and template (the name's digit runs as #, the
extension apart) posted within the window of the album's first file; the
album closes a window after that span and is written as one release
(source album, no file total) named by the members' shared numbers, a lone
member as the release it was. The batch store writes an album's member
names as release_texts kind 6 (member). A counter total of 100 or more under
which one poster started 20 or more releases is a batch counter: those
releases get no file total.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
edge-bench --group and replay:corpus take --to <article> next to --from, so a
replay covers one window of a group's history while the corpus holds others
(the 2016 ranges next to 2021's and 2025's). deploy/replay-window.sh runs a
window on homelab: every group whose window ranges are all stored is replayed
once into usenet_replay through the voyager-3 app image, 4 groups at a time,
looping while the corpus fill completes the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
replay:corpus runs edge-bench --group over a group's complete corpus ranges
in article order (one window, carried from range to range, finished after
the last range; edge/src/lease.rs holds the lease JSON the worker and the
bench share) and stores every batch through BatchStore as the worker
would, then reports releases finished, provisional, complete and the bytes
per header of the NVMe and HDD tables. Refuses a database not named
*_replay or *_scratch. The app image carries edge-bench; compose gets a
replay profile for homelab.
Design amended (REBUILD.md 1, 2, 5): the window is carried by one worker a
group, no block leases and no block merger; provisional writes stay for
the lease ends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
obfuscators give each volume its own extension (or none); for a random name
(no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
dot counts with a digit before it too (5,419 single files of YME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
"….vol4+4.par2": the subject and the total were the key, so every upload of
a total was one release (2,338 files for 20; also partner-of-ssl-news,
brothers-of-usenet). A random stem (no dots or spaces) on a volume file
joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
name up to its first number, when another follows, joins the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Ballers.2014.S01E01.HDTV.x264-KILLERS[ettv] 15_07_2015.rar", "Computer Bild
- Ausgabe 152015 - 15_07_2015.rar": the cut after "] " or " - " left
"15_07_2015.rar" for every item of a Razor1911 dump (1,335 files for 12);
a date alone after the cut means the whole text is the name.
- "MDG005b - (01/90) "IMG_2307.JPG"": a counter in parentheses makes no post
key, so every "IMG" set of the poster was one (609 for 90); an image under
a post token is keyed by it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7ad4b70 also keyed rar uploads by artist and total: "VA - Best Of Shamania
Pro 2K16-(SHB 006)-WEB-2016-ZzZz" became 140 files for 20, Ace_Of_Base 72
for 12. Those group by their full name; only a name without capitals or
spaces (Meow's "blaise_mc-in_hellstill_3-ep-web-2022-known.par2") joins
the tracks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[07/14] - "blaise_mc-in_hellstill_3-ep-web-2022-known.par2"": the tracks and the
00- files are keyed by artist and total, the par2 by its stem, so ~1,900 Meow
and Cat posts were two releases. Under a counter a stem that reads as a scene
music release (artist-…-web|cd|ep|single|vinyl|…-…YYYY(-group)) takes the
artist key. And "goh1Phoz5ahXu.par.vol006+08.par2", "X.part.par2" beside
the volumes: .par/.part before a par2 suffix goes too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0c5ede9 cut "Angry.Birds.Rio.v1.2.2.cracked.READ.NFO-THETA.rar" at .NFO: the
release lost its group, and two groups' releases of one game would merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[01/13] - "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with
"[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the tracks are keyed by
artist and total (SCENE_TRACK), but the cover, nfo, sfv and m3u were releases
of their own (2,630 single files of Meow in a.b.ftn). Under a bare counter a
00- stem takes the same key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
[newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
Panic.mkv"") are one release per show, season and counter total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
goes (Kingdn's episodes, 2,850 single files).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
total of index×10, so every track had a total of its own and was a
release of its own. A total zero-padded wider than its index is no
counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
corpus grows between builds).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
'"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
"-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
("(NELET11.par2) [01/58] - "NELET11.par2"").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
[0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
twice_(extended_mix).mp3""): the artist's album bounded by the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
last word ("<abm> … AB: part2 0573-1002 043.jpg").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
is keyed by its set, not its name without numbers: a poster's 263 diana
sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
spaced suffix ("she has sensual legs 172 _ 172") as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
