From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
[0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
twice_(extended_mix).mp3""): the artist's album bounded by the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
last word ("<abm> … AB: part2 0573-1002 043.jpg").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
is keyed by its set, not its name without numbers: a poster's 263 diana
sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
spaced suffix ("she has sensual legs 172 _ 172") as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each image was a release of its own in: "Set 008 10of63 new0464.jpg",
"202/430 princess-079-058.jpg", "( 098 /500) 002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The file total is read from the subject without the file name, as with a
quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
once an image was skipped), and a trailing camera timestamp
("kurozo-2006-7-26-15-21") numbers no set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"051 - White leather kate054001_1_1.jpg": the index is inside 054001
(set 054, image 001) and _1_1 repeats it, but only a suffix equal to the
whole number before counted as a repeat, so the set kept 054001 and every
image was a release (Duke's threads: 2,791 releases of 3,251 files). A
suffix that the number before ends with is a repeat as well. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The set bound cut at an image name's last number, but in many threads
that's a suffix, so every image was a set ("It does my heart good to see
Red Dog dancing again": 4,683 releases of 4,730 files): a repeat of the
index (katelg010_10), a year (hgrscans1097_2009), a size
(pj02kristina_24of40_1600), an "of" total (24of40). imageSet drops
those, then cuts at the last number (cutie-071-032 stays cutie-071-).
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without a file counter a subject-named image release was bounded by the
image's name with all numbers removed:
- STIRILING SILVER's sets stp250x…stp465x became one 44k-image release
(all "stp x"); picture threads of many sets (keira_s04, keira_s05,
perla_s01, apneatic-7, -8 …) one release each.
- Free-text names without numbers ("JimAndJoey Kindly Husband-to-Be.JPG")
made each image a set: one thread was 1,177 releases.
The bound is now the name up to its last number with its digits
("stp250x", "keira_s05_l keira_s05_", "apneatic-7-"), and nothing for a
name without numbers. models fixture: 172 -> 754 releases, by set. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Avira AntiVir Rescue System 08-29-2011.rar", "Android Apps and Games
Pack 27-02-2011", "…-cable-18-08-2012-1king.rar": counterTotal read the
date as a counter (8 of 29, 2 of 2011), so the release showed a wrong
file total and counted as incomplete. Dates (2011-08-29, 08-29-2011,
29.08.2011, 02-2011) are taken out before the written-out counters are
read; [i/M] and (i/M) come first as before. Both implementations; tests;
fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Spotlight - 83xx's [16/19] - \"SC8367-12 - Wynette, Tammy - Singing My
Song.zip\"": the disc-and-track code isn't a bare track number, so the
poster-tag rule keyed every track by its own name. A stem starting with
an upper-case code and digits before the track number (SC8367-12,
ZMP050-04) is a numbered item too. Both implementations; tests; fixtures
identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the re-index's single-file releases:
- A quoted text without an extension ("My \"Official\" remaining set's
of Inna … 025.jpg [25/74]", "… lr140x16.jpg ( file 046 of 922)
\"yEnc\"", "\"Dr. Lust\" - … - [12/21] - Ed_DRL_VDR_LM.part09.rar") was
the file name, the real one stayed in the subject: every file a release.
It gives way to a file name with an extension beside it
(nameBesideQuotes); random quoted names with none beside keep theirs.
- Split parts numbered with two digits ("Admiral.EP05.2008.720p.Rus.
HDCLUB.mkv.21", a.b.movies.divx.russian) share their stem, as .001.
Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases of one file whose counter announces many (910k of 3M after the
sixth reset's first minutes), by cause:
- An unquoted name with double spaces ("lara_bath 15of19 larabath_2.5.jpg",
"Luisa: (068 of 160)"): the subject is collapsed, the name wasn't,
so it stayed in the remainder; it's taken out collapsed too.
- Audiobook and album tracks keyed by their own name ("RP - File 268 of
295 - \"… Seeing A Large Cat - D11.16-22.mp3\"", "CANDYD :: \"Candy
Dulfer - Candy Store - 11 - everytime.mp3\""): by the name without its
numbers, as image sets.
- A size without a unit before the closing yEnc ("…mp3\" - 8123 yEnc").
- Random posters with separators ("SOFZRw1-OTEd@…", "…_OOW@…").
Both implementations; tests; fixtures identical (models: 267 -> 172).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"A Strange Light from the East - Eastern Influenced Western Music
(1965-72) - CD2 - 07 - Rascals - Sattva.mp3 (1/43)": unquotedName cut at
the last " - " ("Sattva.mp3"), leaving track number and artist in the
subject, so every track was a release (85665085, 85665050). With a
" - NN - " track or part number the name starts there ("07 - Rascals -
Sattva.mp3"), the rest names the album and disc. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releases 78361820 and 78411563 were two files of one FTD upload
("Yusef_Lateef-Alight_Upon_the_Lake_…-WEB-2026-FTD", [1/6] and [3/6], 4 s
apart) under a new random poster each ("N9vplIaavBCz@aCTVoAsk.JSE",
"FVWWHku0fGAg@QfaTz0g0.RVH"): the stem key includes the poster. When the
poster's own name (before the @) looks random, it's left out of the key;
real posters keep theirs (two uploaders of one scene release at the same
time stay apart), and uploads hours apart stay apart by the re-upload
rule. Both implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old picture groups split every image into a release (the reset's first
hour: 583k releases, 447k of one file; a.b.pictures.erotica.high-heels
262k): "Aletta Ocean - \"100628-035.jpg\" yEnc [347K]" kept its size in
the remainder, and "Corie - \"2877-041.jpg\" yEnc [290K]" named its set
with one word, which namesRelease (two words) refused, so the image's
own name keyed it. Sizes like [347K] (and a cut-off "336573 byt") are
removed; for images a one-word subject (3+ letters) names the set. On
100k high-heels articles: 4,509 releases -> 192. Both implementations;
tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"ai-engineer-complete [401/465] - \"003 Defining Tools with JSON
Schema.mp4\"": the subject's text shares no word with the lesson's
descriptive name, so it was taken for a poster's constant tag and each of
the course's 465 files keyed a release of its own (releases 75941915,
75941916). A stem starting with a 1-3 digit index and a separator is an
item of a set the subject names, as audio tracks and images already
were; uploads hours apart stay apart by the re-upload rule. Both
implementations; tests; fixtures identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most "obfuscated" subjects of older and smaller groups name their file
without quotes ("DVDFab 8.2.0.5.rar (006/332)", fta's "q3B9…part023.rar
(104/137)", "New_Super_Mario_Bros._[Keygen].exe (1/2)") or with one
quote ("[keira_s04_0037_l.jpg\" yEnc] (2/2)"): 100% of ath 2013's and
fta's, 94% of models' (LOG.md 2026-10-11 (5)). With a segment counter and
no quoted name, unquotedName takes the name before a lone quote (after a
"[", else after the last space), else the text when it ends in a file
name (after "] " or " - "), else its first word that is one; a file
counter stuck in a bracket is dropped. The release identity takes such a
name out of the subject as it would quotes (else every image of a set
was a release: 9,130 instead of 267 on the models fixture).
Both implementations; a fourth fixture (fta, 25k lines) and tests; check
identical on all four. Truly random subjects (ngPost hex = message-id,
random tokens) stay obfuscated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User: Rust is the product (option a). tools/edge-bench-rs moves to edge/
(crate edge; binaries edge-node, the worker, and edge-bench, the replay).
CI's new job edge runs cargo test and edge/check-fixtures.sh: both
RangeIndexers on three committed captures (edge/fixtures, 25k overview
lines each of a.b.models, a.b.ath and a.b.boneless in 5 chunks, 3.5 MB),
compared field by field with check.ts; deploy waits for it. The
TypeScript RangeIndexer stays as check.ts's reference. README: what it
is now; the benchmark findings as history.
Random names: looksRandom missed long tokens of capitals and digits
("51294P1CUKR2R35X", release 74539068, readable and a Newznab title).
12+ capitals and digits switching 6+ times between them are random
(COD4MW2REMASTERED switches 4 times, BEATLES1964REMASTER twice); both
implementations, identical on the fixtures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.
The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk not in after NNTP_HEDGE_AFTER_SECONDS (10) is asked again on one
of NNTP_HEDGE_CONNECTIONS spare connections; the first answer wins and
the other connection is closed (lib/nntp/hedge.ts, tested). Connections
can be retired after NNTP_CONNECTION_MAX_MINUTES (±25%). The prefetch
report counts hedges and slow first requests by connection age.
Homelab, 15 min each: hedging on (40 + 6) 280k articles/s, off 273k.
The second request won about half of its races, but ~25-35% of first
requests were slow (>10 s) at every connection age, so neither helps:
Eweka is slow under our load, not on particular connections. Both
off by default; the age statistics stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
So a cover 'x.jpg' joins 'x.part01.rar' in a stem-named post; an image
set like 'IMG_008' keys as the stem 'IMG'. Both implementations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picture groups made a release per image: 501k single-file releases in
the first 24 ranges of the bottom 500 groups (a.b.models 385k), ~120M
for all of them. Counters written out ('File 027 of 132', '[12of42]',
'104 of 472', '[2239-6389]', '02/17', '0097-1574'; only when the first
number is at most the second, so '[1948-12]' stays) and per-file sizes
('76.88 MB', '335555 Bytes') kept every subject unique.
- subjectRemainder removes those counters and sizes, and empty brackets.
- A subject-named release is bounded by its counter's total
(counterTotal: the first form found): uploads a poster names alike
stay apart, an upload's files (par2, nfo) together.
- Images and audio tracks don't take the poster-tag rule: the subject
names the set or album. An image set without a counter: the image's
name up to its last number, without numbers.
- Documents and comics (pdf, epub, mobi, azw3, djvu, cbr, cbz, cb7, chm,
lit, doc, docx) are a release each: collections post hundreds under
one subject ('cw-horror&mystery').
releases:regroup --group re-keys a group's indexed releases together:
cheerleaders 55,741 -> 3,165 releases, models 385,298 -> 40,313, German
audiobooks 4,106 -> 1,724, playstation 3,309 -> 2,345; comics and
e-books a release per item; boneless unchanged. Ported to edge-bench-rs;
check.ts: identical on all captures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release 145637 held four albums: the poster puts '[KiLL.dA.FuCkIn.
EaStErRaBbIt]----' before every upload, and the text outside the quotes
(when it names a release) was the key with the poster. The same made
one release of a poster's albums under '(PowBeertje80 Posted)' (a post
id, with the same [i/M]), of masterposts' films, and of one volume of
every set in a DVD9 post.
Now, when the file's stem describes something itself (12+ letters and
digits, two words of 3+ letters) and the text outside the quotes shares
no word with it (4+ letters as part of the stem, 3-letter words and
numbers whole; boilerplate like www/net/posted not counted; no 5+ digit
per-post id), that text is taken for the poster's tag and the stem
says which release the file is. Albums of track files ('FTA … Ketchup'),
'DIE_ZEIT_35' with '35_China_dl', 'FTD769997' posts, scene names with
abbreviated file names ('grp-sm') and generic sidecars ('file_id.diz')
keep their grouping.
releases:regroup re-keys a sample of the catalog with the current rule:
on homelab's test data 145637 → 4 albums, 111566 → 17 albums of 27
files, the rest of the subject releases that change mostly lose an ad
image ('Flyer-download-2-Day.gif'); stem releases unchanged. Ported to
edge-bench-rs: check.ts still finds all captures identical (ath 2013:
16 → 76 releases on both sides).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nzb (buildNzb, parseNzb) from the current app; NzbService builds a
release's NZB from its chunks; GET /api/v1/releases?q= (every word of the
name) and GET /api/v1/releases/:id/nzb behind the api guard. A test stores
a batch, finds the release and reads its NZB back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nyuu's 12 capital+small letter pairs in 15 bytes (10 bits a pair), its
timestamp as a delta within the file, the suffix once per file;
ngPost-style 32 hex digits in 16 bytes; other ids as zstd text as before.
Recent boneless: 25.2 → 20.3 B per segment (and 25.0 → 19.5 on the next
range); a 2013 ath range stays at 17.9 (JBinUp ids are text). Every chunk
of three real ranges decodes to itself (edge:bench --verify). Encoding is
no slower than v1 (less text for zstd). Version 1 chunks stay readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview chunks in, per batch out: releases as encoded chunks, probe
candidates, counts per rule (crossposts per owner group, obfuscated,
garbage, per-article, incomplete), a sample of what was dropped, and
fragments near the range's edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nntp (XOVER client, Xref, GROUP), lib/headers (subjects, file keys,
segments from overview lines; the yEnc path stays behind for phase 2),
lib/release (identity, random names, now in lib), lib/files (aggregator,
volume names), lib/rules (posting tools, owner by priority, release
verdicts), lib/chunks (release chunk codec). Their unit tests came along;
new ones for codec, rules and volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka answers 'no such article' after ~450 ms; a quarter of inspection
requests were for missing articles and each failed job was retried up
to three times. Files get content_type 'missing' instead; homelab also
records it from failure reports of nodes not rebuilt yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 sets, NZBs and SFVs were sniffed from their first segment and then
read whole, downloading that segment a second time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured with 20 relay connections, plain and gzip both gave ~1.5 chunks/s
for 2026 a.b.ath articles; plain used many times the CPU and traffic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For 2026 a.b.ath articles Eweka's gzip replies have a long tail (p90 12.4 s
per 20k vs 3.9 s plain; 6.1k vs 10.5k headers/s). SEED_NNTP_COMPRESS turns
compression back on; plain replies are deflated for nodes asynchronously.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:plan plans every part of a group's history no range covers (it
started below the lowest known range and left a.b.ath's 13 most recent
months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One poster adds the same 22 GB hidden '.1917-….mkv' decoy to every 7z
upload; the largest-entry rule named 840 releases after it. Entries with
a hidden path part are skipped; release:rename --hidden renames the
existing ones from their stored listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
85% of the 7z archives inspected have encrypted headers (-mhe). Their
encoded header lists the 7zAES coder, so the packed header stream is no
longer downloaded and bsdtar no longer runs for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection lists only the entries whose headers are in the first segment
of a first volume. A walk hops from header to header, fetching one segment
per entry and volume, in its own queue (one job per collection worker) on
NNTP connections inspection isn't waiting for. content:walk-archives queues
releases inspected earlier (CONCEPT D40).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
