Each claim looks its candidates up in probe_cache by message_id; with the
index on the HDD that took 1.27 s a claim, 48 s of each minute. Moved by
hand on homelab; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
~20 GB of the NVMe, read by key only; the catalog was at its cap and history
paused. Moved on homelab by hand (3 min, writers stopped; the cap raised to
100); the migration does the same on a new deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The October partition's three indexes took 2.9 GB of the NVMe, its pkey
never scanned; chunks are read from the HDD anyway. Moved on homelab by hand
(12 s); the migration does the same on a new deploy, parent indexes too so new
partitions follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
obfuscators give each volume its own extension (or none); for a random name
(no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
dot counts with a digit before it too (5,419 single files of YME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
"….vol4+4.par2": the subject and the total were the key, so every upload of
a total was one release (2,338 files for 20; also partner-of-ssl-news,
brothers-of-usenet). A random stem (no dots or spaces) on a volume file
joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
name up to its first number, when another follows, joins the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Oldest inspection first went through the 3-month backfill before today's
spots, whose releases the tail has: 3 h after reset 24, 2 of 16.7k matched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 50 exactly the coordinator still got 502s now and then: a hedged request
closes its connection and opens another while the old one still counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coordinator 38 + its plain connection + prober 10 + spotter + spot inspector:
the spot inspector got 502 Too many connections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a spot that becomes a release of its own needs its NZB (the header
pass's releases have their chunks); spots took 2.7 GB on the NVMe at 9k.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spot older than what the catalog has indexed (the spotter's first backfill
reaches back ~3 months) matched nothing and would have become a release of
its own, a duplicate once the history pass builds the upload. Unmatched
spots are matched again in turns (rematched_at) for 30 days; the 24 h before
a standalone release count from the inspection; a later match deletes the
spot's own release and links the header pass's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every commit waited on the WAL in ./pg on the HDD (WALWrite; the HDD 100%
busy, 56–70 ms a write): prober flushes and claims took most of each minute.
A PG_WAL_DIR volume (default /var/lib/usenet-rebuild-wal) at
/var/lib/postgresql/wal, with data/pg_wal a symlink to it (set up by hand
once, with Postgres stopped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober writes it every flush and the merger looks inner stems up in it,
so it no longer belongs with the write-once payloads on the HDD. Moved on
homelab by hand (prober and merger stopped); the migration is a no-op there
and moves it on a new deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Ballers.2014.S01E01.HDTV.x264-KILLERS[ettv] 15_07_2015.rar", "Computer Bild
- Ausgabe 152015 - 15_07_2015.rar": the cut after "] " or " - " left
"15_07_2015.rar" for every item of a Razor1911 dump (1,335 files for 12);
a date alone after the cut means the whole text is the name.
- "MDG005b - (01/90) "IMG_2307.JPG"": a counter in parentheses makes no post
key, so every "IMG" set of the poster was one (609 for 90); an image under
a post token is keyed by it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7ad4b70 also keyed rar uploads by artist and total: "VA - Best Of Shamania
Pro 2K16-(SHB 006)-WEB-2016-ZzZz" became 140 files for 20, Ace_Of_Base 72
for 12. Those group by their full name; only a name without capitals or
spaces (Meow's "blaise_mc-in_hellstill_3-ep-web-2022-known.par2") joins
the tracks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[07/14] - "blaise_mc-in_hellstill_3-ep-web-2022-known.par2"": the tracks and the
00- files are keyed by artist and total, the par2 by its stem, so ~1,900 Meow
and Cat posts were two releases. Under a counter a stem that reads as a scene
music release (artist-…-web|cd|ep|single|vinyl|…-…YYYY(-group)) takes the
artist key. And "goh1Phoz5ahXu.par.vol006+08.par2", "X.part.par2" beside
the volumes: .par/.part before a par2 suffix goes too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the bytes on disk were checked: at 296.6 GB with 2,300 ranges complete it
marked 100 more (~310 GB). The marked ranges not complete count at the stored
ranges' mean size, and no more are marked than fit under 300 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0c5ede9 cut "Angry.Birds.Rio.v1.2.2.cracked.READ.NFO-THETA.rar" at .NFO: the
release lost its group, and two groups' releases of one game would merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hep, hephaestus and gyges post each file under a subject name of its own
(31,000 one-file releases), but the yEnc name in each file's first article
keeps the upload's random stem ("bX147zx8I0oEgj7nzFCiorCT0ps2jEcb.7z.001"),
one stem per counter total. The merger's second pass joins releases whose
lone inspections share a random stem: one group and poster, the same total,
within 48 h, no more files than the total plus 10. An expression index over
the stem; release_merges records the kind (via) and takes a stem.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every kantodoNNN.bin test upload carries the same par2 set: 64 releases held
it nearby and became one release of 631 files for a set of 10. A set held by
more than 4 releases nearby isn't merged, nor members with more than 3x the
set's files plus 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A par2 index and its repost both hold a set: moving both rows to the target
broke par2_sets' primary key, and the round failed on it every minute from
20:28 UTC (no merges at all). Duplicates among the others go first; a set that
fails is logged and the round goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[01/13] - "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with
"[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the tracks are keyed by
artist and total (SCENE_TRACK), but the cover, nfo, sfv and m3u were releases
of their own (2,630 single files of Meow in a.b.ftn). Under a bare counter a
00- stem takes the same key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The settler marks a settled release's candidates ready (and unreadable when
it had no readable name). Claims took the candidates by joining releases
from random points of the queue; in its newer half nothing had settled, so
a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and
the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over
the claimable rows only, the slices drawn within them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
[newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
Panic.mkv"") are one release per show, season and counter total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fetching, waiting for candidates, idle, inspecting; claims and flushes. The
link had room (222 of 1000 Mbit/s with the coordinator at 239k headers/s,
one test connection ~84 Mbit/s meanwhile) while the prober did ~5/s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
goes (Kingdn's episodes, 2,850 single files).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of a claim filtered ~700k rows by kind through the id index (0.37 s
a slice, 20 slices and up to 5 tiers a claim, the lanes waiting meanwhile).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
total of index×10, so every track had a total of its own and was a
release of its own. A total zero-padded wider than its index is no
counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
corpus grows between builds).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
'"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
"-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
("(NELET11.par2) [01/58] - "NELET11.par2"").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
[0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
twice_(extended_mix).mp3""): the artist's album bounded by the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
last word ("<abm> … AB: part2 0573-1002 043.jpg").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
is keyed by its set, not its name without numbers: a poster's 263 diana
sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
spaced suffix ("she has sensual legs 172 _ 172") as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each image was a release of its own in: "Set 008 10of63 new0464.jpg",
"202/430 princess-079-058.jpg", "( 098 /500) 002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The file total is read from the subject without the file name, as with a
quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
once an image was skipped), and a trailing camera timestamp
("kurozo-2006-7-26-15-21") numbers no set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
(all "stp x"); picture threads of many sets (keira_s04, keira_s05,
perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
\"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
the file name, the real one stayed in the subject: every file a release.
It gives way to a file name with an extension beside it
(nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath 15of19 larabath_2.5.jpg",
"Luisa: (068 of 160)"): the subject is collapsed, the name wasn't,
so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).
Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
obf_runs: how obfuscated articles come in runs (strict, and per poster
and tool with a time gap). obf_kinds: what they are (unquoted subjects
with a counter, random tokens equal to the message-id, other random
tokens, ...), with examples. LOG.md 2026-10-11 (5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.
Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
