Commit Graph
205 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 f6e54b6c58 CORPUS_AHEAD 16: 8 ranges left 20 of 40 connections idle
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m13s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:17:12 +02:00
Daan MeijerandClaude Opus 5.5 7b436f3f77 Idle connections fetch capture ranges into the corpus
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m6s
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:11:57 +02:00
Daan MeijerandClaude Opus 5.5 0fafe71b21 The poster-tag tolerance only without a counter
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m43s
Test and deploy / deploy (push) Successful in 1m24s
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:52:26 +02:00
Daan MeijerandClaude Opus 5.5 62250081ef A poster tag of 4+ words may share one with the stem
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m39s
Test and deploy / deploy (push) Successful in 1m10s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:24:42 +02:00
Daan MeijerandClaude Opus 5.5 cfd29c8de3 A long hex stem without a counter is the upload
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m22s
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:53:53 +02:00
Daan MeijerandClaude Opus 5.5 6d301fc668 Hosts and sponsors without a counter name no upload
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m10s
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:58:25 +02:00
Daan MeijerandClaude Opus 5.5 cdc1cdc0cc Prober: claim only candidates of settled releases, without the join
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Failing after 4m2s
The settler marks a settled release's candidates ready (and unreadable when
it had no readable name). Claims took the candidates by joining releases
from random points of the queue; in its newer half nothing had settled, so
a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and
the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over
the claimable rows only, the slices drawn within them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:12:59 +02:00
Daan MeijerandClaude Opus 5.5 1866e38c25 Identity: per-file hex ids, dated stems, season packs under a bare counter
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m26s
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
  [newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
  ("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
  still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
  Panic.mkv"") are one release per show, season and counter total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:10:40 +02:00
Daan MeijerandClaude Opus 5.5 1cb6a4554f Stems: keyed without case and a scene index's 00-
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m21s
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:04:17 +02:00
Daan MeijerandClaude Opus 5.5 123ed81782 Prober: where the lanes' time goes, in the minute report
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m23s
Fetching, waiting for candidates, idle, inspecting; claims and flushes. The
link had room (222 of 1000 Mbit/s with the coordinator at 239k headers/s,
one test connection ~84 Mbit/s meanwhile) while the prober did ~5/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:58:42 +02:00
Daan MeijerandClaude Opus 5.5 358f5a8571 Names: split rar volumes; a file's size in brackets after its counter
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 1m4s
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
  one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
  goes (Kingdn's episodes, 2,850 single files).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:52:35 +02:00
Daan MeijerandClaude Opus 5.5 3301216aa8 Probe candidates: an index on (kind, id) of the waiting ones for the prober's claims
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 3m27s
A slice of a claim filtered ~700k rows by kind through the id index (0.37 s
a slice, 20 slices and up to 5 tiers a claim, the lanes waiting meanwhile).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:33:27 +02:00
Daan MeijerandClaude Opus 5.5 0727afe968 Prober: pipelined bodies, merge-first order, a probe cache across resets
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m33s
Test and deploy / deploy (push) Successful in 2m26s
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
  once, replies read in order (one at a time it ran 3-10 articles/s on 6
  connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
  releases without a readable name (what the par2 merge needs), then those
  of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
  16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
  probe queue, not the cache; recreated candidates of probed articles are
  answered from it without the news server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:23:01 +02:00
Daan MeijerandClaude Opus 5.5 6d1f2cc2a1 Par2 merge: a set file matching several releases nearby is ambiguous
Test and deploy / test (push) Successful in 1m1s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m4s
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:15:32 +02:00
Daan MeijerandClaude Opus 5.5 8e07478659 Counters: a total padded wider than its index is none; report diff on shared ranges
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m25s
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
  total of index×10, so every track had a total of its own and was a
  release of its own. A total zero-padded wider than its index is no
  counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
  corpus grows between builds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:12:28 +02:00
Daan MeijerandClaude Opus 5.5 1a5035d988 Names: stray quotes, single quotes, hosts, bracketed names; dates, repeats
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m8s
Test and deploy / deploy (push) Successful in 2m6s
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
  '"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
  extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
  not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
  brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
  "-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
  show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
  ("(NELET11.par2) [01/58] - "NELET11.par2"").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:58:04 +02:00
Daan MeijerandClaude Opus 5.5 885e4e6541 Settler: history releases settle after half an hour
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m17s
A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:42:53 +02:00
Daan MeijerandClaude Opus 5.5 3d0dc6ca1f The par2 merge: releases whose files one par2 set describes become one
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m27s
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.

Also: corpus_sweep.py counts the marked ranges without psql's command tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:37:35 +02:00
Daan MeijerandClaude Opus 5.5 9ea2669b2a Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:28:22 +02:00
Daan MeijerandClaude Opus 5.5 d43603a43f Identity: hashed stems under a tag, numbered volumes, scene tracks
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m28s
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
  Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
  day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
  [0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
  twice_(extended_mix).mp3""): the artist's album bounded by the total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:46:10 +02:00
Daan MeijerandClaude Opus 5.5 3fc38edda8 Identity: other quoted text, text separators, ids in image names, more
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
  COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
  files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
  read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
  the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
  ("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
  markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
  last word ("<abm> … AB: part2 0573-1002 043.jpg").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:21 +02:00
Daan MeijerandClaude Opus 5.5 e1d21d1e7c Identity: a set's number before yEnc is no size; names after an arrow
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m6s
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
  the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
  yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
  of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
  Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 06:46:19 +02:00
Daan MeijerandClaude Opus 5.5 18a146e130 Image sets: keyed by set without words around them; px, ImageVenue, spaced repeats
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 2m17s
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
  is keyed by its set, not its name without numbers: a poster's 263 diana
  sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
  spaced suffix ("she has sensual legs 172 _ 172") as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 05:45:53 +02:00
Daan MeijerandClaude Opus 5.5 108d798d97 Image sets: a counter in the name ends the set; copy markers, resolutions, _00 go
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m59s
Each image was a release of its own in: "Set 008  10of63  new0464.jpg",
"202/430  princess-079-058.jpg", "( 098 /500)  002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:45:37 +02:00
Daan MeijerandClaude Opus 5.5 e72a9b1ef5 Identity: unquoted names are no counters; renumbered and timestamped image sets
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m4s
- The file total is read from the subject without the file name, as with a
  quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
  of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
  once an image was skipped), and a trailing camera timestamp
  ("kurozo-2006-7-26-15-21") numbers no set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 03:47:51 +02:00
Daan MeijerandClaude Opus 5.5 f094fb4593 Image sets: a suffix that ends the number before is a repeat too
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m24s
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:45:29 +02:00
Daan MeijerandClaude Opus 5.5 392ff09910 Image sets: suffixes that don't count the images go first
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:45:57 +02:00
Daan MeijerandClaude Opus 5.5 c4f48a6d4c Image sets by their numbered name, digits kept
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m1s
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
  (all "stp x"); picture threads of many sets (keira_s04, keira_s05,
  perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
  made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:45:52 +02:00
Daan MeijerandClaude Opus 5.5 0577b16e5e Dates are no file counters
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m42s
Test and deploy / deploy (push) Successful in 1m11s
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:05:53 +02:00
Daan MeijerandClaude Opus 5.5 8dfccc1c8e Numbered items: also after a disc code (karaoke)
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m20s
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:03 +02:00
Daan MeijerandClaude Opus 5.5 2814c77225 A quoted word that isn't a file name; split parts with two digits
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m41s
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
  of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
  \"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
  the file name, the real one stayed in the subject: every file a release.
  It gives way to a file name with an extension beside it
  (nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
  HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:46:43 +02:00
Daan MeijerandClaude Opus 5.5 625ea23171 More single-file splits from the re-index
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m49s
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath  15of19 larabath_2.5.jpg",
  "Luisa:     (068 of 160)"): the subject is collapsed, the name wasn't,
  so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
  295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
  Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
  numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:15:31 +02:00
Daan MeijerandClaude Opus 5.5 8d400822bd Unquoted album tracks: the name starts at the track number
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Successful in 1m21s
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:41:54 +02:00
Daan MeijerandClaude Opus 5.5 e0a0d0d520 An upload spread over groups is one release
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m55s
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:28:57 +02:00
Daan MeijerandClaude Opus 5.5 df35f4678c A random poster per file is no part of a release key
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m45s
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:55:15 +02:00
Daan MeijerandClaude Opus 5.5 b1213fbea0 Image sets: sizes in brackets and one-word set names
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m19s
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:25:18 +02:00
Daan MeijerandClaude Opus 5.5 47b445ffb0 Numbered items of a set don't take the poster-tag rule
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m44s
Test and deploy / deploy (push) Successful in 1m28s
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:10:58 +02:00
Daan MeijerandClaude Opus 5.5 90f30f445d Subjects without a quoted file name: the name from the rest of the subject
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m25s
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).

Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:52:55 +02:00
Daan MeijerandClaude Opus 5.5 18b9ffdca3 tools: obf_runs and obf_kinds, measuring the obfuscated subjects
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m24s
obf_runs: how obfuscated articles come in runs (strict, and per poster
and tool with a time gap). obf_kinds: what they are (unquoted subjects
with a counter, random tokens equal to the message-id, other random
tokens, ...), with examples. LOG.md 2026-10-11 (5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:03 +02:00
Daan MeijerandClaude Opus 5.5 5109481d67 The Rust worker is the product: edge/, built and checked in CI
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 2m56s
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.

Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:01:18 +02:00
Daan MeijerandClaude Opus 5.5 48f4e982f9 Rebuild: the edge workers' API in its own service
Test and deploy / test (push) Successful in 49s
Test and deploy / deploy (push) Successful in 1m16s
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:49:56 +02:00
Daan MeijerandClaude Opus 5.5 34fea342f5 REBUILD.md is the concept for what runs; the phase 2 backlog
Test and deploy / test (push) Successful in 44s
Test and deploy / deploy (push) Failing after 1m4s
From "draft for approval" to the concept of voyager 2 (voyager 1's
CONCEPT.md and D1-D42 are superseded). Phase 2: what's built (the
prober), and voyager 1's features not yet here: spots, NZB import,
passwords, SFV, RAR walks, 7z listings, body naming of obfuscated
subjects, after-probe garbage drops; where its exported data is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:48:49 +02:00
Daan MeijerandClaude Opus 5.5 93eaec8d27 Rebuild: sort indexes that match newest-first
Test and deploy / test (push) Successful in 47s
Test and deploy / deploy (push) Successful in 5m20s
The default page (no search words) took 23-27 s: SearchService sorts
posted_at DESC NULLS LAST, id DESC, the indexes were on posted_at DESC
(nulls first), so Postgres sorted all 68M releases. New indexes match
the sort (also per category), built concurrently; the old ones dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:39:42 +02:00
Daan MeijerandClaude Opus 5.5 92480174b9 Search and the web interface use the prober's findings
Test and deploy / test (push) Successful in 48s
Test and deploy / deploy (push) Successful in 2m27s
- Search matches a word in the name or the inner name (trigram index on
  inner_name, partial, built concurrently); "Readable names" includes
  releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
  name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
  Contents (per probed article its yEnc name, kind, type, encryption and
  the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
  with an inner name is included, titled by it (without its extension).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:33:23 +02:00
Daan MeijerandClaude Opus 5.5 5ab5399b43 Prober: the claim's random starting points drawn in code
Test and deploy / test (push) Successful in 54s
Test and deploy / deploy (push) Successful in 1m21s
Postgres evaluated random() once for all 20 slices: every claim still
came from one place in the queue (214 of 214 taken from a.b.zogg).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:00:23 +02:00
Daan MeijerandClaude Opus 5.5 18f5d27413 Prober: claim from random points of the queue, not in id order
Test and deploy / test (push) Successful in 45s
Test and deploy / deploy (push) Successful in 1m23s
In id order the prober went group by group, and some groups' articles
take 0.6-3 s each (a.b.zogg 2016; repeated fetches the same, pipelining
4 deep no faster), where a sample across all years takes 20-60 ms for
most: 7 articles/s on 6 connections, against 67/s in its first minutes.
A claim now takes 20 slices from random points (353 ms per 200).
tools/body_timing.ts: BODY time for the oldest and newest candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:55:23 +02:00
Daan MeijerandClaude Opus 5.5 4ed7f89951 Prober: no packaging or release-name-plus-extension as inner name
Test and deploy / test (push) Successful in 43s
Test and deploy / deploy (push) Successful in 1m18s
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:44:08 +02:00
Daan MeijerandClaude Opus 5.5 738402aefa Prober: an inspection per candidate; no inner name that is the release's name
Test and deploy / test (push) Successful in 53s
Test and deploy / deploy (push) Successful in 1m53s
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:37:32 +02:00
Daan MeijerandClaude Opus 5.5 8bd6d6aa5f Rebuild: phase 2's prober reads the probe queue
Test and deploy / test (push) Successful in 50s
Test and deploy / deploy (push) Successful in 1m46s
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.

Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:32:49 +02:00
Daan MeijerandClaude Opus 5.5 d72fe2f13f Tests: a corrupt chunk fails its range on the third report
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Successful in 1m31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:22:10 +02:00