Commit Graph
213 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 1c8da1dd12 A long number is a per-post id only under a counter
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m32s
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:41:40 +02:00
Daan MeijerandClaude Opus 5.5 8d5385fdec Split zip volumes (.zNN) and .xNN as volumes, but not .x32/.x64/.x86
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m23s
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 01:47:21 +02:00
Daan MeijerandClaude Opus 5.5 87e15d810d Old rar volumes: .s–.v only, not .x64/.x86
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m36s
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 00:53:38 +02:00
Daan MeijerandClaude Opus 5.5 d4e53ca0d8 More per-file suffixes: old rar volumes, .rev, a part's par2, .bc; a size after an unquoted name
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 1m22s
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
  own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:59:09 +02:00
Daan MeijerandClaude Opus 5.5 2823d42bfa Merge obfuscated uploads by the random stem of their inner names
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 2m32s
hep, hephaestus and gyges post each file under a subject name of its own
(31,000 one-file releases), but the yEnc name in each file's first article
keeps the upload's random stem ("bX147zx8I0oEgj7nzFCiorCT0ps2jEcb.7z.001"),
one stem per counter total. The merger's second pass joins releases whose
lone inspections share a random stem: one group and poster, the same total,
within 48 h, no more files than the total plus 10. An expression index over
the stem; release_merges records the kind (via) and takes a stem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:38:31 +02:00
Daan MeijerandClaude Opus 5.5 942c4901fa Par2 merge: a set held by many releases, or a merge far larger than its set, isn't one upload
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m34s
Every kantodoNNN.bin test upload carries the same par2 set: 64 releases held
it nearby and became one release of 631 files for a set of 10. A set held by
more than 4 releases nearby isn't merged, nor members with more than 3x the
set's files plus 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 22:29:48 +02:00
Daan MeijerandClaude Opus 5.5 c72bf68fd2 Par2 merge: a set held by two merged releases; one failed set doesn't stop a round
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m59s
A par2 index and its repost both hold a set: moving both rows to the target
broke par2_sets' primary key, and the round failed on it every minute from
20:28 UTC (no merges at all). Duplicates among the others go first; a set that
fails is logged and the round goes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 22:28:50 +02:00
Daan MeijerandClaude Opus 5.5 7b99cfdb0e A scene music post's 00- files go with its tracks
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m22s
"[01/13] - "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with
"[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the tracks are keyed by
artist and total (SCENE_TRACK), but the cover, nfo, sfv and m3u were releases
of their own (2,630 single files of Meow in a.b.ftn). Under a bare counter a
00- stem takes the same key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:01:12 +02:00
Daan MeijerandClaude Opus 5.5 f6e54b6c58 CORPUS_AHEAD 16: 8 ranges left 20 of 40 connections idle
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m13s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:17:12 +02:00
Daan MeijerandClaude Opus 5.5 7b436f3f77 Idle connections fetch capture ranges into the corpus
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m6s
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:11:57 +02:00
Daan MeijerandClaude Opus 5.5 0fafe71b21 The poster-tag tolerance only without a counter
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m43s
Test and deploy / deploy (push) Successful in 1m24s
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:52:26 +02:00
Daan MeijerandClaude Opus 5.5 62250081ef A poster tag of 4+ words may share one with the stem
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m39s
Test and deploy / deploy (push) Successful in 1m10s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:24:42 +02:00
Daan MeijerandClaude Opus 5.5 cfd29c8de3 A long hex stem without a counter is the upload
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m22s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:53:53 +02:00
Daan MeijerandClaude Opus 5.5 6d301fc668 Hosts and sponsors without a counter name no upload
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m10s
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:58:25 +02:00
Daan MeijerandClaude Opus 5.5 cdc1cdc0cc Prober: claim only candidates of settled releases, without the join
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Failing after 4m2s
The settler marks a settled release's candidates ready (and unreadable when
it had no readable name). Claims took the candidates by joining releases
from random points of the queue; in its newer half nothing had settled, so
a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and
the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over
the claimable rows only, the slices drawn within them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:12:59 +02:00
Daan MeijerandClaude Opus 5.5 1866e38c25 Identity: per-file hex ids, dated stems, season packs under a bare counter
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m26s
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
  [newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
  ("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
  still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
  Panic.mkv"") are one release per show, season and counter total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:10:40 +02:00
Daan MeijerandClaude Opus 5.5 1cb6a4554f Stems: keyed without case and a scene index's 00-
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m21s
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:04:17 +02:00
Daan MeijerandClaude Opus 5.5 123ed81782 Prober: where the lanes' time goes, in the minute report
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m23s
Fetching, waiting for candidates, idle, inspecting; claims and flushes. The
link had room (222 of 1000 Mbit/s with the coordinator at 239k headers/s,
one test connection ~84 Mbit/s meanwhile) while the prober did ~5/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:58:42 +02:00
Daan MeijerandClaude Opus 5.5 358f5a8571 Names: split rar volumes; a file's size in brackets after its counter
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 1m4s
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
  one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
  goes (Kingdn's episodes, 2,850 single files).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:52:35 +02:00
Daan MeijerandClaude Opus 5.5 3301216aa8 Probe candidates: an index on (kind, id) of the waiting ones for the prober's claims
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 3m27s
A slice of a claim filtered ~700k rows by kind through the id index (0.37 s
a slice, 20 slices and up to 5 tiers a claim, the lanes waiting meanwhile).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:33:27 +02:00
Daan MeijerandClaude Opus 5.5 0727afe968 Prober: pipelined bodies, merge-first order, a probe cache across resets
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m33s
Test and deploy / deploy (push) Successful in 2m26s
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
  once, replies read in order (one at a time it ran 3-10 articles/s on 6
  connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
  releases without a readable name (what the par2 merge needs), then those
  of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
  16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
  probe queue, not the cache; recreated candidates of probed articles are
  answered from it without the news server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:23:01 +02:00
Daan MeijerandClaude Opus 5.5 6d1f2cc2a1 Par2 merge: a set file matching several releases nearby is ambiguous
Test and deploy / test (push) Successful in 1m1s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m4s
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:15:32 +02:00
Daan MeijerandClaude Opus 5.5 8e07478659 Counters: a total padded wider than its index is none; report diff on shared ranges
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m25s
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
  total of index×10, so every track had a total of its own and was a
  release of its own. A total zero-padded wider than its index is no
  counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
  corpus grows between builds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:12:28 +02:00
Daan MeijerandClaude Opus 5.5 1a5035d988 Names: stray quotes, single quotes, hosts, bracketed names; dates, repeats
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m8s
Test and deploy / deploy (push) Successful in 2m6s
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
  '"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
  extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
  not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
  brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
  "-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
  show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
  ("(NELET11.par2) [01/58] - "NELET11.par2"").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:58:04 +02:00
Daan MeijerandClaude Opus 5.5 885e4e6541 Settler: history releases settle after half an hour
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m17s
A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:42:53 +02:00
Daan MeijerandClaude Opus 5.5 3d0dc6ca1f The par2 merge: releases whose files one par2 set describes become one
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m27s
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.

Also: corpus_sweep.py counts the marked ranges without psql's command tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:37:35 +02:00
Daan MeijerandClaude Opus 5.5 9ea2669b2a Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:28:22 +02:00
Daan MeijerandClaude Opus 5.5 d43603a43f Identity: hashed stems under a tag, numbered volumes, scene tracks
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m28s
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
  Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
  day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
  [0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
  twice_(extended_mix).mp3""): the artist's album bounded by the total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:46:10 +02:00
Daan MeijerandClaude Opus 5.5 3fc38edda8 Identity: other quoted text, text separators, ids in image names, more
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
  COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
  files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
  read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
  the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
  ("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
  markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
  last word ("<abm> … AB: part2 0573-1002 043.jpg").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:21 +02:00
Daan MeijerandClaude Opus 5.5 e1d21d1e7c Identity: a set's number before yEnc is no size; names after an arrow
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m6s
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
  the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
  yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
  of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
  Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 06:46:19 +02:00
Daan MeijerandClaude Opus 5.5 18a146e130 Image sets: keyed by set without words around them; px, ImageVenue, spaced repeats
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 2m17s
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
  is keyed by its set, not its name without numbers: a poster's 263 diana
  sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
  spaced suffix ("she has sensual legs 172 _ 172") as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 05:45:53 +02:00
Daan MeijerandClaude Opus 5.5 108d798d97 Image sets: a counter in the name ends the set; copy markers, resolutions, _00 go
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m59s
Each image was a release of its own in: "Set 008  10of63  new0464.jpg",
"202/430  princess-079-058.jpg", "( 098 /500)  002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:45:37 +02:00
Daan MeijerandClaude Opus 5.5 e72a9b1ef5 Identity: unquoted names are no counters; renumbered and timestamped image sets
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m4s
- The file total is read from the subject without the file name, as with a
  quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
  of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
  once an image was skipped), and a trailing camera timestamp
  ("kurozo-2006-7-26-15-21") numbers no set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 03:47:51 +02:00
Daan MeijerandClaude Opus 5.5 f094fb4593 Image sets: a suffix that ends the number before is a repeat too
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m24s
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:45:29 +02:00
Daan MeijerandClaude Opus 5.5 392ff09910 Image sets: suffixes that don't count the images go first
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:45:57 +02:00
Daan MeijerandClaude Opus 5.5 c4f48a6d4c Image sets by their numbered name, digits kept
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m1s
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
  (all "stp x"); picture threads of many sets (keira_s04, keira_s05,
  perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
  made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:45:52 +02:00
Daan MeijerandClaude Opus 5.5 0577b16e5e Dates are no file counters
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m42s
Test and deploy / deploy (push) Successful in 1m11s
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:05:53 +02:00
Daan MeijerandClaude Opus 5.5 8dfccc1c8e Numbered items: also after a disc code (karaoke)
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m20s
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:03 +02:00
Daan MeijerandClaude Opus 5.5 2814c77225 A quoted word that isn't a file name; split parts with two digits
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m41s
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
  of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
  \"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
  the file name, the real one stayed in the subject: every file a release.
  It gives way to a file name with an extension beside it
  (nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
  HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:46:43 +02:00
Daan MeijerandClaude Opus 5.5 625ea23171 More single-file splits from the re-index
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m49s
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath  15of19 larabath_2.5.jpg",
  "Luisa:     (068 of 160)"): the subject is collapsed, the name wasn't,
  so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
  295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
  Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
  numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:15:31 +02:00
Daan MeijerandClaude Opus 5.5 8d400822bd Unquoted album tracks: the name starts at the track number
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Successful in 1m21s
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:41:54 +02:00
Daan MeijerandClaude Opus 5.5 e0a0d0d520 An upload spread over groups is one release
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m55s
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:28:57 +02:00
Daan MeijerandClaude Opus 5.5 df35f4678c A random poster per file is no part of a release key
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m45s
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:55:15 +02:00
Daan MeijerandClaude Opus 5.5 b1213fbea0 Image sets: sizes in brackets and one-word set names
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m19s
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:25:18 +02:00
Daan MeijerandClaude Opus 5.5 47b445ffb0 Numbered items of a set don't take the poster-tag rule
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m44s
Test and deploy / deploy (push) Successful in 1m28s
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:10:58 +02:00
Daan MeijerandClaude Opus 5.5 90f30f445d Subjects without a quoted file name: the name from the rest of the subject
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m25s
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).

Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:52:55 +02:00
Daan MeijerandClaude Opus 5.5 18b9ffdca3 tools: obf_runs and obf_kinds, measuring the obfuscated subjects
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m24s
obf_runs: how obfuscated articles come in runs (strict, and per poster
and tool with a time gap). obf_kinds: what they are (unquoted subjects
with a counter, random tokens equal to the message-id, other random
tokens, ...), with examples. LOG.md 2026-10-11 (5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:03 +02:00
Daan MeijerandClaude Opus 5.5 5109481d67 The Rust worker is the product: edge/, built and checked in CI
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 2m56s
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.

Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:01:18 +02:00
Daan MeijerandClaude Opus 5.5 48f4e982f9 Rebuild: the edge workers' API in its own service
Test and deploy / test (push) Successful in 49s
Test and deploy / deploy (push) Successful in 1m16s
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:49:56 +02:00
Daan MeijerandClaude Opus 5.5 34fea342f5 REBUILD.md is the concept for what runs; the phase 2 backlog
Test and deploy / test (push) Successful in 44s
Test and deploy / deploy (push) Failing after 1m4s
From "draft for approval" to the concept of voyager 2 (voyager 1's
CONCEPT.md and D1-D42 are superseded). Phase 2: what's built (the
prober), and voyager 1's features not yet here: spots, NZB import,
passwords, SFV, RAR walks, 7z listings, body naming of obfuscated
subjects, after-probe garbage drops; where its exported data is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:48:49 +02:00