Commit Graph
194 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 3301216aa8 Probe candidates: an index on (kind, id) of the waiting ones for the prober's claims
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 3m27s
A slice of a claim filtered ~700k rows by kind through the id index (0.37 s
a slice, 20 slices and up to 5 tiers a claim, the lanes waiting meanwhile).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:33:27 +02:00
Daan MeijerandClaude Opus 5.5 0727afe968 Prober: pipelined bodies, merge-first order, a probe cache across resets
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m33s
Test and deploy / deploy (push) Successful in 2m26s
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
  once, replies read in order (one at a time it ran 3-10 articles/s on 6
  connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
  releases without a readable name (what the par2 merge needs), then those
  of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
  16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
  probe queue, not the cache; recreated candidates of probed articles are
  answered from it without the news server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:23:01 +02:00
Daan MeijerandClaude Opus 5.5 6d1f2cc2a1 Par2 merge: a set file matching several releases nearby is ambiguous
Test and deploy / test (push) Successful in 1m1s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m4s
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:15:32 +02:00
Daan MeijerandClaude Opus 5.5 8e07478659 Counters: a total padded wider than its index is none; report diff on shared ranges
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m25s
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
  total of index×10, so every track had a total of its own and was a
  release of its own. A total zero-padded wider than its index is no
  counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
  corpus grows between builds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:12:28 +02:00
Daan MeijerandClaude Opus 5.5 1a5035d988 Names: stray quotes, single quotes, hosts, bracketed names; dates, repeats
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m8s
Test and deploy / deploy (push) Successful in 2m6s
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
  '"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
  extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
  not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
  brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
  "-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
  show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
  ("(NELET11.par2) [01/58] - "NELET11.par2"").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:58:04 +02:00
Daan MeijerandClaude Opus 5.5 885e4e6541 Settler: history releases settle after half an hour
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m17s
A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:42:53 +02:00
Daan MeijerandClaude Opus 5.5 3d0dc6ca1f The par2 merge: releases whose files one par2 set describes become one
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m27s
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.

Also: corpus_sweep.py counts the marked ranges without psql's command tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:37:35 +02:00
Daan MeijerandClaude Opus 5.5 9ea2669b2a Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:28:22 +02:00
Daan MeijerandClaude Opus 5.5 d43603a43f Identity: hashed stems under a tag, numbered volumes, scene tracks
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m28s
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
  Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
  day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
  [0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
  twice_(extended_mix).mp3""): the artist's album bounded by the total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:46:10 +02:00
Daan MeijerandClaude Opus 5.5 3fc38edda8 Identity: other quoted text, text separators, ids in image names, more
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
  COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
  files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
  read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
  the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
  ("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
  markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
  last word ("<abm> … AB: part2 0573-1002 043.jpg").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:21 +02:00
Daan MeijerandClaude Opus 5.5 e1d21d1e7c Identity: a set's number before yEnc is no size; names after an arrow
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m6s
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
  the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
  yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
  of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
  Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 06:46:19 +02:00
Daan MeijerandClaude Opus 5.5 18a146e130 Image sets: keyed by set without words around them; px, ImageVenue, spaced repeats
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 2m17s
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
  is keyed by its set, not its name without numbers: a poster's 263 diana
  sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
  spaced suffix ("she has sensual legs 172 _ 172") as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 05:45:53 +02:00
Daan MeijerandClaude Opus 5.5 108d798d97 Image sets: a counter in the name ends the set; copy markers, resolutions, _00 go
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m59s
Each image was a release of its own in: "Set 008  10of63  new0464.jpg",
"202/430  princess-079-058.jpg", "( 098 /500)  002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:45:37 +02:00
Daan MeijerandClaude Opus 5.5 e72a9b1ef5 Identity: unquoted names are no counters; renumbered and timestamped image sets
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m4s
- The file total is read from the subject without the file name, as with a
  quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
  of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
  once an image was skipped), and a trailing camera timestamp
  ("kurozo-2006-7-26-15-21") numbers no set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 03:47:51 +02:00
Daan MeijerandClaude Opus 5.5 f094fb4593 Image sets: a suffix that ends the number before is a repeat too
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m24s
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:45:29 +02:00
Daan MeijerandClaude Opus 5.5 392ff09910 Image sets: suffixes that don't count the images go first
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:45:57 +02:00
Daan MeijerandClaude Opus 5.5 c4f48a6d4c Image sets by their numbered name, digits kept
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m1s
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
  (all "stp x"); picture threads of many sets (keira_s04, keira_s05,
  perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
  made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:45:52 +02:00
Daan MeijerandClaude Opus 5.5 0577b16e5e Dates are no file counters
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m42s
Test and deploy / deploy (push) Successful in 1m11s
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:05:53 +02:00
Daan MeijerandClaude Opus 5.5 8dfccc1c8e Numbered items: also after a disc code (karaoke)
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m20s
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:03 +02:00
Daan MeijerandClaude Opus 5.5 2814c77225 A quoted word that isn't a file name; split parts with two digits
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m41s
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
  of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
  \"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
  the file name, the real one stayed in the subject: every file a release.
  It gives way to a file name with an extension beside it
  (nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
  HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:46:43 +02:00
Daan MeijerandClaude Opus 5.5 625ea23171 More single-file splits from the re-index
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m49s
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath  15of19 larabath_2.5.jpg",
  "Luisa:     (068 of 160)"): the subject is collapsed, the name wasn't,
  so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
  295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
  Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
  numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:15:31 +02:00
Daan MeijerandClaude Opus 5.5 8d400822bd Unquoted album tracks: the name starts at the track number
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Successful in 1m21s
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:41:54 +02:00
Daan MeijerandClaude Opus 5.5 e0a0d0d520 An upload spread over groups is one release
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m55s
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:28:57 +02:00
Daan MeijerandClaude Opus 5.5 df35f4678c A random poster per file is no part of a release key
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m45s
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:55:15 +02:00
Daan MeijerandClaude Opus 5.5 b1213fbea0 Image sets: sizes in brackets and one-word set names
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m19s
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:25:18 +02:00
Daan MeijerandClaude Opus 5.5 47b445ffb0 Numbered items of a set don't take the poster-tag rule
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m44s
Test and deploy / deploy (push) Successful in 1m28s
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:10:58 +02:00
Daan MeijerandClaude Opus 5.5 90f30f445d Subjects without a quoted file name: the name from the rest of the subject
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m25s
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).

Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:52:55 +02:00
Daan MeijerandClaude Opus 5.5 18b9ffdca3 tools: obf_runs and obf_kinds, measuring the obfuscated subjects
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m24s
obf_runs: how obfuscated articles come in runs (strict, and per poster
and tool with a time gap). obf_kinds: what they are (unquoted subjects
with a counter, random tokens equal to the message-id, other random
tokens, ...), with examples. LOG.md 2026-10-11 (5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:03 +02:00
Daan MeijerandClaude Opus 5.5 5109481d67 The Rust worker is the product: edge/, built and checked in CI
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 2m56s
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.

Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:01:18 +02:00
Daan MeijerandClaude Opus 5.5 48f4e982f9 Rebuild: the edge workers' API in its own service
Test and deploy / test (push) Successful in 49s
Test and deploy / deploy (push) Successful in 1m16s
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:49:56 +02:00
Daan MeijerandClaude Opus 5.5 34fea342f5 REBUILD.md is the concept for what runs; the phase 2 backlog
Test and deploy / test (push) Successful in 44s
Test and deploy / deploy (push) Failing after 1m4s
From "draft for approval" to the concept of voyager 2 (voyager 1's
CONCEPT.md and D1-D42 are superseded). Phase 2: what's built (the
prober), and voyager 1's features not yet here: spots, NZB import,
passwords, SFV, RAR walks, 7z listings, body naming of obfuscated
subjects, after-probe garbage drops; where its exported data is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:48:49 +02:00
Daan MeijerandClaude Opus 5.5 93eaec8d27 Rebuild: sort indexes that match newest-first
Test and deploy / test (push) Successful in 47s
Test and deploy / deploy (push) Successful in 5m20s
The default page (no search words) took 23-27 s: SearchService sorts
posted_at DESC NULLS LAST, id DESC, the indexes were on posted_at DESC
(nulls first), so Postgres sorted all 68M releases. New indexes match
the sort (also per category), built concurrently; the old ones dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:39:42 +02:00
Daan MeijerandClaude Opus 5.5 92480174b9 Search and the web interface use the prober's findings
Test and deploy / test (push) Successful in 48s
Test and deploy / deploy (push) Successful in 2m27s
- Search matches a word in the name or the inner name (trigram index on
  inner_name, partial, built concurrently); "Readable names" includes
  releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
  name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
  Contents (per probed article its yEnc name, kind, type, encryption and
  the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
  with an inner name is included, titled by it (without its extension).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:33:23 +02:00
Daan MeijerandClaude Opus 5.5 5ab5399b43 Prober: the claim's random starting points drawn in code
Test and deploy / test (push) Successful in 54s
Test and deploy / deploy (push) Successful in 1m21s
Postgres evaluated random() once for all 20 slices: every claim still
came from one place in the queue (214 of 214 taken from a.b.zogg).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:00:23 +02:00
Daan MeijerandClaude Opus 5.5 18f5d27413 Prober: claim from random points of the queue, not in id order
Test and deploy / test (push) Successful in 45s
Test and deploy / deploy (push) Successful in 1m23s
In id order the prober went group by group, and some groups' articles
take 0.6-3 s each (a.b.zogg 2016; repeated fetches the same, pipelining
4 deep no faster), where a sample across all years takes 20-60 ms for
most: 7 articles/s on 6 connections, against 67/s in its first minutes.
A claim now takes 20 slices from random points (353 ms per 200).
tools/body_timing.ts: BODY time for the oldest and newest candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:55:23 +02:00
Daan MeijerandClaude Opus 5.5 4ed7f89951 Prober: no packaging or release-name-plus-extension as inner name
Test and deploy / test (push) Successful in 43s
Test and deploy / deploy (push) Successful in 1m18s
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:44:08 +02:00
Daan MeijerandClaude Opus 5.5 738402aefa Prober: an inspection per candidate; no inner name that is the release's name
Test and deploy / test (push) Successful in 53s
Test and deploy / deploy (push) Successful in 1m53s
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:37:32 +02:00
Daan MeijerandClaude Opus 5.5 8bd6d6aa5f Rebuild: phase 2's prober reads the probe queue
Test and deploy / test (push) Successful in 50s
Test and deploy / deploy (push) Successful in 1m46s
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.

Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:32:49 +02:00
Daan MeijerandClaude Opus 5.5 d72fe2f13f Tests: a corrupt chunk fails its range on the third report
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Successful in 1m31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:22:10 +02:00
Daan MeijerandClaude Opus 5.5 fa52d9256b Rebuild: a chunk reported corrupt gets a second plain try before the range fails
Test and deploy / test (push) Failing after 24s
Test and deploy / deploy (push) Skipped
Range 161820 failed with "doesn't inflate, also fetched plain" though
the chunk was likely the compressed copy again: in flight, or spooled
before the prefetcher's targets were refreshed, when the plain flag was
set. The range now fails only on the third report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:21:44 +02:00
Daan MeijerandClaude Opus 5.5 fd06b16bd9 deploy.sh: remove the images a deploy replaced
Test and deploy / test (push) Successful in 30s
Test and deploy / deploy (push) Successful in 2m1s
Each deploy left a 321 MB app image dangling on the NVMe (13 after a day
of deploys).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:21:39 +02:00
Daan MeijerandClaude Opus 5.5 40df52bafd Rebuild: 8 more ranges being fetched than the connections need
Test and deploy / test (push) Successful in 27s
Test and deploy / deploy (push) Successful in 1m3s
Ranges with a chunk or two left take fewer than perRange connections:
with loops/perRange + 2 ranges, 3.2 of 40 loops were idle on average
over 15 minutes (up to 9.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:36:24 +02:00
Daan MeijerandClaude Opus 5.5 27a26c9614 Rebuild: a release key taken by another group's writer gets the next number
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 2m29s
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:22:49 +02:00
Daan MeijerandClaude Opus 5.5 b87203e41a edge-node: a range whose batch is too large goes on in smaller batches
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Successful in 2m25s
Range 154218 (a.b.hdtv.tv-episodes) looped 13 times on HTTP 413: a
500k-article batch over the server's 256 MB body limit, though under
100k files. On a 413 the worker gives the range back and posts that
range's batches at half the articles and files from then on (again on
another 413, down to 1/64); stored batches stay. The message says how
large the batch was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:07:54 +02:00
Daan MeijerandClaude Opus 5.5 7b07e00f90 Rebuild: the stream waits up to an hour for a chunk
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Successful in 1m1s
Some groups' chunks take 20+ minutes while still coming in (range
154153: 17 MB in 22 minutes); the stream's 15-minute wait failed their
workers (two in 15 minutes). Stalled requests are given up separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:21:24 +02:00
Daan MeijerandClaude Opus 5.5 887df12ead Rebuild: give up stalled requests, not slow ones
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Successful in 56s
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:51:55 +02:00
Daan MeijerandClaude Opus 5.5 dbccec932c Rebuild: a chunk's deadline doubles each time it's hit
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m8s
At 90 s a chunk that always takes longer would be given up forever
(range 155437's worker gave up waiting on its stream). Each deadline a
chunk hits doubles its next one (90, 180, 360 ... up to 16x), reset
once it comes in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:36:36 +02:00
Daan MeijerandClaude Opus 5.5 d81a202172 Rebuild: chunk deadline 90 s
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m1s
At 180 s, 60 chunks were given up in 15 minutes: each held a connection
3 minutes, ~30% of the 40 connections' time, while finished chunks
took 24 s on average (Eweka slow this afternoon).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:06:30 +02:00
Daan MeijerandClaude Opus 5.5 fc1c64d5c0 Rebuild: drop NUL from batches; chunk deadline 180 s
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Successful in 1m0s
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.

The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 13:12:27 +02:00
Daan MeijerandClaude Opus 5.5 f4c0417066 Rebuild: enough ranges being fetched for every connection
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m1s
With 2 connections per range the prefetcher needs 20+ ranges with
chunks left; fully spooled ranges counted as ready (SPOOL_AHEAD_PER_LEASE)
but take no connection, so on small ranges 30 of 40 connections were
busy (13-15 ranges). Fresh ranges now also fill up to loops/perRange + 2
ranges being fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:44:09 +02:00