Commit Graph
187 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 9ea2669b2a Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:28:22 +02:00
Daan MeijerandClaude Opus 5.5 d43603a43f Identity: hashed stems under a tag, numbered volumes, scene tracks
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m28s
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
  Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
  day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
  [0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
  twice_(extended_mix).mp3""): the artist's album bounded by the total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:46:10 +02:00
Daan MeijerandClaude Opus 5.5 3fc38edda8 Identity: other quoted text, text separators, ids in image names, more
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
  COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
  files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
  read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
  the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
  ("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
  markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
  last word ("<abm> … AB: part2 0573-1002 043.jpg").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:21 +02:00
Daan MeijerandClaude Opus 5.5 e1d21d1e7c Identity: a set's number before yEnc is no size; names after an arrow
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m6s
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
  the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
  yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
  of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
  Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 06:46:19 +02:00
Daan MeijerandClaude Opus 5.5 18a146e130 Image sets: keyed by set without words around them; px, ImageVenue, spaced repeats
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 2m17s
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
  is keyed by its set, not its name without numbers: a poster's 263 diana
  sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
  spaced suffix ("she has sensual legs 172 _ 172") as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 05:45:53 +02:00
Daan MeijerandClaude Opus 5.5 108d798d97 Image sets: a counter in the name ends the set; copy markers, resolutions, _00 go
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m59s
Each image was a release of its own in: "Set 008  10of63  new0464.jpg",
"202/430  princess-079-058.jpg", "( 098 /500)  002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:45:37 +02:00
Daan MeijerandClaude Opus 5.5 e72a9b1ef5 Identity: unquoted names are no counters; renumbered and timestamped image sets
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m4s
- The file total is read from the subject without the file name, as with a
  quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
  of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
  once an image was skipped), and a trailing camera timestamp
  ("kurozo-2006-7-26-15-21") numbers no set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 03:47:51 +02:00
Daan MeijerandClaude Opus 5.5 f094fb4593 Image sets: a suffix that ends the number before is a repeat too
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 1m24s
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:45:29 +02:00
Daan MeijerandClaude Opus 5.5 392ff09910 Image sets: suffixes that don't count the images go first
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m1s
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:45:57 +02:00
Daan MeijerandClaude Opus 5.5 c4f48a6d4c Image sets by their numbered name, digits kept
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m1s
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
  (all "stp x"); picture threads of many sets (keira_s04, keira_s05,
  perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
  made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:45:52 +02:00
Daan MeijerandClaude Opus 5.5 0577b16e5e Dates are no file counters
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m42s
Test and deploy / deploy (push) Successful in 1m11s
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:05:53 +02:00
Daan MeijerandClaude Opus 5.5 8dfccc1c8e Numbered items: also after a disc code (karaoke)
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 2m20s
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:03 +02:00
Daan MeijerandClaude Opus 5.5 2814c77225 A quoted word that isn't a file name; split parts with two digits
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m41s
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
  of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
  \"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
  the file name, the real one stayed in the subject: every file a release.
  It gives way to a file name with an extension beside it
  (nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
  HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:46:43 +02:00
Daan MeijerandClaude Opus 5.5 625ea23171 More single-file splits from the re-index
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 1m49s
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath  15of19 larabath_2.5.jpg",
  "Luisa:     (068 of 160)"): the subject is collapsed, the name wasn't,
  so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
  295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
  Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
  numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:15:31 +02:00
Daan MeijerandClaude Opus 5.5 8d400822bd Unquoted album tracks: the name starts at the track number
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Successful in 1m21s
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:41:54 +02:00
Daan MeijerandClaude Opus 5.5 e0a0d0d520 An upload spread over groups is one release
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m55s
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:28:57 +02:00
Daan MeijerandClaude Opus 5.5 df35f4678c A random poster per file is no part of a release key
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m45s
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:55:15 +02:00
Daan MeijerandClaude Opus 5.5 b1213fbea0 Image sets: sizes in brackets and one-word set names
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Successful in 1m19s
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:25:18 +02:00
Daan MeijerandClaude Opus 5.5 47b445ffb0 Numbered items of a set don't take the poster-tag rule
Test and deploy / test (push) Successful in 45s
Test and deploy / edge (push) Successful in 1m44s
Test and deploy / deploy (push) Successful in 1m28s
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:10:58 +02:00
Daan MeijerandClaude Opus 5.5 90f30f445d Subjects without a quoted file name: the name from the rest of the subject
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m25s
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).

Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:52:55 +02:00
Daan MeijerandClaude Opus 5.5 18b9ffdca3 tools: obf_runs and obf_kinds, measuring the obfuscated subjects
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m24s
obf_runs: how obfuscated articles come in runs (strict, and per poster
and tool with a time gap). obf_kinds: what they are (unquoted subjects
with a counter, random tokens equal to the message-id, other random
tokens, ...), with examples. LOG.md 2026-10-11 (5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:03 +02:00
Daan MeijerandClaude Opus 5.5 5109481d67 The Rust worker is the product: edge/, built and checked in CI
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 2m56s
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.

Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:01:18 +02:00
Daan MeijerandClaude Opus 5.5 48f4e982f9 Rebuild: the edge workers' API in its own service
Test and deploy / test (push) Successful in 49s
Test and deploy / deploy (push) Successful in 1m16s
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:49:56 +02:00
Daan MeijerandClaude Opus 5.5 34fea342f5 REBUILD.md is the concept for what runs; the phase 2 backlog
Test and deploy / test (push) Successful in 44s
Test and deploy / deploy (push) Failing after 1m4s
From "draft for approval" to the concept of voyager 2 (voyager 1's
CONCEPT.md and D1-D42 are superseded). Phase 2: what's built (the
prober), and voyager 1's features not yet here: spots, NZB import,
passwords, SFV, RAR walks, 7z listings, body naming of obfuscated
subjects, after-probe garbage drops; where its exported data is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:48:49 +02:00
Daan MeijerandClaude Opus 5.5 93eaec8d27 Rebuild: sort indexes that match newest-first
Test and deploy / test (push) Successful in 47s
Test and deploy / deploy (push) Successful in 5m20s
The default page (no search words) took 23-27 s: SearchService sorts
posted_at DESC NULLS LAST, id DESC, the indexes were on posted_at DESC
(nulls first), so Postgres sorted all 68M releases. New indexes match
the sort (also per category), built concurrently; the old ones dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:39:42 +02:00
Daan MeijerandClaude Opus 5.5 92480174b9 Search and the web interface use the prober's findings
Test and deploy / test (push) Successful in 48s
Test and deploy / deploy (push) Successful in 2m27s
- Search matches a word in the name or the inner name (trigram index on
  inner_name, partial, built concurrently); "Readable names" includes
  releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
  name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
  Contents (per probed article its yEnc name, kind, type, encryption and
  the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
  with an inner name is included, titled by it (without its extension).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:33:23 +02:00
Daan MeijerandClaude Opus 5.5 5ab5399b43 Prober: the claim's random starting points drawn in code
Test and deploy / test (push) Successful in 54s
Test and deploy / deploy (push) Successful in 1m21s
Postgres evaluated random() once for all 20 slices: every claim still
came from one place in the queue (214 of 214 taken from a.b.zogg).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:00:23 +02:00
Daan MeijerandClaude Opus 5.5 18f5d27413 Prober: claim from random points of the queue, not in id order
Test and deploy / test (push) Successful in 45s
Test and deploy / deploy (push) Successful in 1m23s
In id order the prober went group by group, and some groups' articles
take 0.6-3 s each (a.b.zogg 2016; repeated fetches the same, pipelining
4 deep no faster), where a sample across all years takes 20-60 ms for
most: 7 articles/s on 6 connections, against 67/s in its first minutes.
A claim now takes 20 slices from random points (353 ms per 200).
tools/body_timing.ts: BODY time for the oldest and newest candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:55:23 +02:00
Daan MeijerandClaude Opus 5.5 4ed7f89951 Prober: no packaging or release-name-plus-extension as inner name
Test and deploy / test (push) Successful in 43s
Test and deploy / deploy (push) Successful in 1m18s
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:44:08 +02:00
Daan MeijerandClaude Opus 5.5 738402aefa Prober: an inspection per candidate; no inner name that is the release's name
Test and deploy / test (push) Successful in 53s
Test and deploy / deploy (push) Successful in 1m53s
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:37:32 +02:00
Daan MeijerandClaude Opus 5.5 8bd6d6aa5f Rebuild: phase 2's prober reads the probe queue
Test and deploy / test (push) Successful in 50s
Test and deploy / deploy (push) Successful in 1m46s
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.

Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:32:49 +02:00
Daan MeijerandClaude Opus 5.5 d72fe2f13f Tests: a corrupt chunk fails its range on the third report
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Successful in 1m31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:22:10 +02:00
Daan MeijerandClaude Opus 5.5 fa52d9256b Rebuild: a chunk reported corrupt gets a second plain try before the range fails
Test and deploy / test (push) Failing after 24s
Test and deploy / deploy (push) Skipped
Range 161820 failed with "doesn't inflate, also fetched plain" though
the chunk was likely the compressed copy again: in flight, or spooled
before the prefetcher's targets were refreshed, when the plain flag was
set. The range now fails only on the third report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:21:44 +02:00
Daan MeijerandClaude Opus 5.5 fd06b16bd9 deploy.sh: remove the images a deploy replaced
Test and deploy / test (push) Successful in 30s
Test and deploy / deploy (push) Successful in 2m1s
Each deploy left a 321 MB app image dangling on the NVMe (13 after a day
of deploys).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:21:39 +02:00
Daan MeijerandClaude Opus 5.5 40df52bafd Rebuild: 8 more ranges being fetched than the connections need
Test and deploy / test (push) Successful in 27s
Test and deploy / deploy (push) Successful in 1m3s
Ranges with a chunk or two left take fewer than perRange connections:
with loops/perRange + 2 ranges, 3.2 of 40 loops were idle on average
over 15 minutes (up to 9.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:36:24 +02:00
Daan MeijerandClaude Opus 5.5 27a26c9614 Rebuild: a release key taken by another group's writer gets the next number
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 2m29s
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:22:49 +02:00
Daan MeijerandClaude Opus 5.5 b87203e41a edge-node: a range whose batch is too large goes on in smaller batches
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Successful in 2m25s
Range 154218 (a.b.hdtv.tv-episodes) looped 13 times on HTTP 413: a
500k-article batch over the server's 256 MB body limit, though under
100k files. On a 413 the worker gives the range back and posts that
range's batches at half the articles and files from then on (again on
another 413, down to 1/64); stored batches stay. The message says how
large the batch was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:07:54 +02:00
Daan MeijerandClaude Opus 5.5 7b07e00f90 Rebuild: the stream waits up to an hour for a chunk
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Successful in 1m1s
Some groups' chunks take 20+ minutes while still coming in (range
154153: 17 MB in 22 minutes); the stream's 15-minute wait failed their
workers (two in 15 minutes). Stalled requests are given up separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:21:24 +02:00
Daan MeijerandClaude Opus 5.5 887df12ead Rebuild: give up stalled requests, not slow ones
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Successful in 56s
a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:51:55 +02:00
Daan MeijerandClaude Opus 5.5 dbccec932c Rebuild: a chunk's deadline doubles each time it's hit
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m8s
At 90 s a chunk that always takes longer would be given up forever
(range 155437's worker gave up waiting on its stream). Each deadline a
chunk hits doubles its next one (90, 180, 360 ... up to 16x), reset
once it comes in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:36:36 +02:00
Daan MeijerandClaude Opus 5.5 d81a202172 Rebuild: chunk deadline 90 s
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m1s
At 180 s, 60 chunks were given up in 15 minutes: each held a connection
3 minutes, ~30% of the 40 connections' time, while finished chunks
took 24 s on average (Eweka slow this afternoon).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:06:30 +02:00
Daan MeijerandClaude Opus 5.5 fc1c64d5c0 Rebuild: drop NUL from batches; chunk deadline 180 s
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Successful in 1m0s
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.

The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 13:12:27 +02:00
Daan MeijerandClaude Opus 5.5 f4c0417066 Rebuild: enough ranges being fetched for every connection
Test and deploy / test (push) Successful in 25s
Test and deploy / deploy (push) Successful in 1m1s
With 2 connections per range the prefetcher needs 20+ ranges with
chunks left; fully spooled ranges counted as ready (SPOOL_AHEAD_PER_LEASE)
but take no connection, so on small ranges 30 of 40 connections were
busy (13-15 ranges). Fresh ranges now also fill up to loops/perRange + 2
ranges being fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:44:09 +02:00
Daan MeijerandClaude Opus 5.5 40aa78cda3 Rebuild: a deadline per chunk; batches with a huge fragment
Test and deploy / test (push) Successful in 29s
Test and deploy / deploy (push) Successful in 1m30s
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.

Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:30:04 +02:00
Daan MeijerandClaude Opus 5.5 e095232d89 REBUILD.md: the deployment
Test and deploy / test (push) Successful in 26s
Test and deploy / deploy (push) Successful in 38s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 00:10:44 +02:00
Daan MeijerandClaude Opus 5.5 84d3af5511 Rebuild: a proper deployment for voyager 2
Test and deploy / test (push) Successful in 48s
Test and deploy / deploy (push) Successful in 2m12s
Dockerfile with two targets: app (compiled AdonisJS, production
dependencies, Node 24.16 as in CI) and edge (the Rust worker, cargo
caches as build mounts). deploy/compose.yaml, project usenet-rebuild:
postgres (data on the HDD, catalog tablespace on the NVMe), migrate,
web, coordinator, edge; edge-ts and ace behind profiles.
deploy/deploy.sh (~/bin/deploy-usenet-rebuild on homelab): fetches the
commit (must be on rebuild), builds, migrates, recreates web,
coordinator and edge (the worker stopped first), never postgres.

CI deploys a push to rebuild that passes: the deploy key's forced
command passes "deploy-rebuild <sha>" on. A push to master no longer
deploys from this branch's workflow. deploy/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 00:01:38 +02:00
Daan MeijerandClaude Opus 5.5 678b3dd438 Rebuild: spread prefetch connections over ranges and groups
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Skipped
At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.

Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 23:33:43 +02:00
Daan MeijerandClaude Opus 5.5 4bf526ab6e Rebuild: prefetch report samples what the connections do
Test and deploy / test (push) Successful in 44s
Test and deploy / deploy (push) Skipped
Once a second: pool connections with a command in flight, loops
fetching/writing/idle; per request GROUP time, XOVER time to the status
line and the transfer, transfers with a silence over 5 s. The old
"connections busy" (fetch time of the chunks finished in the minute)
said 59-75% while all 40 connections were in flight all the time:
dropped.

Homelab: 39-40 of 40 in flight, no idle loops; the status line comes at
once, the whole 8-35 s is the transfer, with data flowing (few gaps
over 5 s): ~3 MB per chunk at 0.1-0.4 MB/s per connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:40:50 +02:00
Daan MeijerandClaude Opus 5.5 8475969145 Rebuild: hedged chunk fetches and connection max age (both off by default)
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Skipped
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.

Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:13:44 +02:00
Daan MeijerandClaude Opus 5.5 fc8ec9ff48 Rebuild: tools/compress_ab.ts, compressed vs plain XOVER side by side
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Skipped
Each chunk fetched with and without XFEATURE COMPRESS GZIP on two
connections at the same moment. 30 chunks on homelab (24 bottom-500,
6 deferred): medians 2.9 s compressed, 3.2 s plain; 59 vs 350 bytes
per article. Slow fetches (27-52 s) hit either mode, rarely both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:36:43 +02:00