"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The settler marks a settled release's candidates ready (and unreadable when
it had no readable name). Claims took the candidates by joining releases
from random points of the queue; in its newer half nothing had settled, so
a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and
the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over
the claimable rows only, the slices drawn within them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
[newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
Panic.mkv"") are one release per show, season and counter total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
goes (Kingdn's episodes, 2,850 single files).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
total of index×10, so every track had a total of its own and was a
release of its own. A total zero-padded wider than its index is no
counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
corpus grows between builds).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
'"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
"-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
("(NELET11.par2) [01/58] - "NELET11.par2"").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release whose last post is 6 hours old gets nothing from the tail, only
fragments from neighbouring ranges: settled once unchanged for 30 minutes.
Waiting 6 hours after it was read held every history release back from
phase 2 after each reset; with the hourly sweeps' resets the prober (and so
the par2 merge) never had work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
[0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
twice_(extended_mix).mp3""): the artist's album bounded by the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
last word ("<abm> … AB: part2 0573-1002 043.jpg").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
is keyed by its set, not its name without numbers: a poster's 263 diana
sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
spaced suffix ("she has sensual legs 172 _ 172") as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each image was a release of its own in: "Set 008 10of63 new0464.jpg",
"202/430 princess-079-058.jpg", "( 098 /500) 002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The file total is read from the subject without the file name, as with a
quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
once an image was skipped), and a trailing camera timestamp
("kurozo-2006-7-26-15-21") numbers no set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
(all "stp x"); picture threads of many sets (keira_s04, keira_s05,
perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
\"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
the file name, the real one stayed in the subject: every file a release.
It gives way to a file name with an extension beside it
(nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath 15of19 larabath_2.5.jpg",
"Luisa: (068 of 160)"): the subject is collapsed, the name wasn't,
so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).
Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.
Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search matches a word in the name or the inner name (trigram index on
inner_name, partial, built concurrently); "Readable names" includes
releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
Contents (per probed article its yEnc name, kind, type, encryption and
the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
with an inner name is included, titled by it (without its extension).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.
The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)
A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- releases.complete is computed with every batch (files >= the
counter's total) instead of only when a release settles after 6 quiet
hours: the 'complete' filter matched nothing in a fresh catalog.
- A release's file total comes from any counter form (counterTotal:
'(01/10)', 'File 27 of 132', …), not only [i/M]; the post rule's key
keeps the [i/M] total, so keys are unchanged. Both implementations.
- groups.tail: a group's live tail can be paused (its history and owner
priority stay), e.g. to let other groups' history go first.
- Stats: history by articles, and the current pass (history ranges
before those deferred 2000 priorities behind) as its own figure; the
header showed 0.1% of ranges counting 145k deferred ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
So a cover 'x.jpg' joins 'x.part01.rar' in a stem-named post; an image
set like 'IMG_008' keys as the stem 'IMG'. Both implementations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picture groups made a release per image: 501k single-file releases in
the first 24 ranges of the bottom 500 groups (a.b.models 385k), ~120M
for all of them. Counters written out ('File 027 of 132', '[12of42]',
'104 of 472', '[2239-6389]', '02/17', '0097-1574'; only when the first
number is at most the second, so '[1948-12]' stays) and per-file sizes
('76.88 MB', '335555 Bytes') kept every subject unique.
- subjectRemainder removes those counters and sizes, and empty brackets.
- A subject-named release is bounded by its counter's total
(counterTotal: the first form found): uploads a poster names alike
stay apart, an upload's files (par2, nfo) together.
- Images and audio tracks don't take the poster-tag rule: the subject
names the set or album. An image set without a counter: the image's
name up to its last number, without numbers.
- Documents and comics (pdf, epub, mobi, azw3, djvu, cbr, cbz, cb7, chm,
lit, doc, docx) are a release each: collections post hundreds under
one subject ('cw-horror&mystery').
releases:regroup --group re-keys a group's indexed releases together:
cheerleaders 55,741 -> 3,165 releases, models 385,298 -> 40,313, German
audiobooks 4,106 -> 1,724, playstation 3,309 -> 2,345; comics and
e-books a release per item; boneless unchanged. Ported to edge-bench-rs;
check.ts: identical on all captures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release 145637 held four albums: the poster puts '[KiLL.dA.FuCkIn.
EaStErRaBbIt]----' before every upload, and the text outside the quotes
(when it names a release) was the key with the poster. The same made
one release of a poster's albums under '(PowBeertje80 Posted)' (a post
id, with the same [i/M]), of masterposts' films, and of one volume of
every set in a DVD9 post.
Now, when the file's stem describes something itself (12+ letters and
digits, two words of 3+ letters) and the text outside the quotes shares
no word with it (4+ letters as part of the stem, 3-letter words and
numbers whole; boilerplate like www/net/posted not counted; no 5+ digit
per-post id), that text is taken for the poster's tag and the stem
says which release the file is. Albums of track files ('FTA … Ketchup'),
'DIE_ZEIT_35' with '35_China_dl', 'FTD769997' posts, scene names with
abbreviated file names ('grp-sm') and generic sidecars ('file_id.diz')
keep their grouping.
releases:regroup re-keys a sample of the catalog with the current rule:
on homelab's test data 145637 → 4 albums, 111566 → 17 albums of 27
files, the rest of the subject releases that change mostly lose an ad
image ('Flyer-download-2-Day.gif'); stem releases unchanged. Ported to
edge-bench-rs: check.ts still finds all captures identical (ath 2013:
16 → 76 releases on both sides).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api (Newznab: caps, search, tvsearch, movie, music, book, get) for
Prowlarr, Sonarr and Radarr: voyager 1's controller and XML on the new
catalog; readable releases not known to miss files; NEWZNAB_API_KEY.
- /api/v1/releases (q, cat, readable, complete, source, poster, size,
sort, paging), /api/v1/releases/:id with its files (decoded from the
chunks; NzbService.files, shared with the NZB), /api/v1/stats
(estimates, plus history and tail progress), /health. Open, as voyager
1's catalog; the web interface uses it.
- release_category(name): voyager 1's categorizer without the archive
contents, a generated column with a (category, posted_at) index.
- @adonisjs/static for the web interface (next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
