One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.
A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
obfuscators give each volume its own extension (or none); for a random name
(no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
dot counts with a digit before it too (5,419 single files of YME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
"….vol4+4.par2": the subject and the total were the key, so every upload of
a total was one release (2,338 files for 20; also partner-of-ssl-news,
brothers-of-usenet). A random stem (no dots or spaces) on a volume file
joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
name up to its first number, when another follows, joins the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Ballers.2014.S01E01.HDTV.x264-KILLERS[ettv] 15_07_2015.rar", "Computer Bild
- Ausgabe 152015 - 15_07_2015.rar": the cut after "] " or " - " left
"15_07_2015.rar" for every item of a Razor1911 dump (1,335 files for 12);
a date alone after the cut means the whole text is the name.
- "MDG005b - (01/90) "IMG_2307.JPG"": a counter in parentheses makes no post
key, so every "IMG" set of the poster was one (609 for 90); an image under
a post token is keyed by it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7ad4b70 also keyed rar uploads by artist and total: "VA - Best Of Shamania
Pro 2K16-(SHB 006)-WEB-2016-ZzZz" became 140 files for 20, Ace_Of_Base 72
for 12. Those group by their full name; only a name without capitals or
spaces (Meow's "blaise_mc-in_hellstill_3-ep-web-2022-known.par2") joins
the tracks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[07/14] - "blaise_mc-in_hellstill_3-ep-web-2022-known.par2"": the tracks and the
00- files are keyed by artist and total, the par2 by its stem, so ~1,900 Meow
and Cat posts were two releases. Under a counter a stem that reads as a scene
music release (artist-…-web|cd|ep|single|vinyl|…-…YYYY(-group)) takes the
artist key. And "goh1Phoz5ahXu.par.vol006+08.par2", "X.part.par2" beside
the volumes: .par/.part before a par2 suffix goes too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0c5ede9 cut "Angry.Birds.Rio.v1.2.2.cracked.READ.NFO-THETA.rar" at .NFO: the
release lost its group, and two groups' releases of one game would merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[01/13] - "00-ammar-turk_uit_de_kroeg-single-web-nl-2022-cover.jpg"" with
"[05/13] - "01-ammar-turk_uit_de_kroeg.mp3"": the tracks are keyed by
artist and total (SCENE_TRACK), but the cover, nfo, sfv and m3u were releases
of their own (2,630 single files of Meow in a.b.ftn). Under a bare counter a
00- stem takes the same key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"[ MacGyver.S01E03.DVDRip.XviD-MEDiEVAL ]-[04/38] - "sample.macgy.s01e03-med.avi""
shares one word with the sample's stem: 6225008 took the name for a tag and
split the episode 17 + 21. A counted subject names its upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "21.Jump.Street.2012.BDRip.AC3.German.XviD-POE.part01.rar"":
"German" in the stem made the tag a name, and the poster's readable uploads
were releases of 1,422 and 1,535 files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"German Board - [Usenet4ever.info] und [SecretUsenet.com] - "<md5>-u4e.part6.rar"":
no counter, so the subject route lumped 11 uploads into one release of 14,450
files. HASHED also takes one 24+ hex run, so the stem keys each upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"(www.Thunder-News.org) >XXX< <Sponsored by Secretusenet> - "ps-itb10b.r00""
has no counter to bound it, so a poster's uploads were one subject release
(1,135 files). Without a counter the remainder drops www hosts and
"Sponsored by X"; what's left names the release or the stem does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the corpus sweep (695 ranges):
- A bracketed hex id per file ("[PRiVATE] Rosewood.S02E08… [1f1793f68d5321]
[newzNZB] [01/22]") goes from the subject: nEwZ[NZB]'s 32,101 single files.
- A stem that starts with a date is specific enough for the poster-tag rule
("09-30-2008_CTC.r00" under "#sterntuary - Coast To Coast Radio Show":
still one release of 25,188 files, the stem being too short to "describe").
- Episodes under a bare counter ("[01/36] "Pokemon - S02E01 - Pallet Party
Panic.mkv"") are one release per show, season and counter total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"00-laura_van_dam_-_capture_radio_113-cable-10-10-2026-talion.m3u" and
.sfv beside "Laura_van_Dam_-_Capture_Radio_113-CABLE-10-10-2026-TALiON.par2"
were two releases of one upload (160731958, 160731584).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "rld-gtre.r13.002": a split volume after a rar volume is stripped like
one after .rar (usenet-town's posts were 1,620 single files).
- "[26/32] - [4929] "…part24.rar"": the bracketed size after the counter
goes (Kingdn's episodes, 2,850 single files).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Very Best Of The Who [01/010]", "[02/020]", "[07/070]": a posting tool's
total of index×10, so every track had a total of its own and was a
release of its own. A total zero-padded wider than its index is no
counter ("[001/010]" still is).
- corpus_sweep.py report compares builds over the ranges both replayed (the
corpus grows between builds).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the first corpus sweep (183 ranges):
- A stray quote took the last quote pair ('"x.mp3" "Album_BY_WaRLoRD',
'"x.nfo" " yEnc', '"x.nzb" yEnc"'): the last quoted part with an
extension is the name. Each file was a release of its own.
- A single-quoted name at the end ("… [001/106] 'You, Me And Dupree.nfo'"),
not the host before it (www.x2l.nl).
- Text ending in a host ("… www.libranews.nl") is no file name; a name in
brackets comes first ("[01/77][LIBRATEAM_M2219009.par2]",
"-^-[Mnozil Brass - Seven.nzb]-^-").
- A leading date is no item number ("09-30-2008_BAK_JRS.r00": a daily
show's 32,621 files were one release under the poster's tag).
- The file name repeated elsewhere in the subject goes too
("(NELET11.par2) [01/58] - "NELET11.par2"").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the top-100 groups:
- A hashed stem under text without a counter is the upload ("Seit 1753
Tagen - usenet4ever.info - "055631a1_b6c820bc_…-u4e.part01.rar""): the
day's uploads were one release of 3,747 files.
- A numbered volume as the extension, within the [i/M] total ("(????)
[0659/2195] - "DN.VPL PCK.1508"", "TKY.ASN PCK.691."): keyed without it.
- Scene tracks under a bare counter ("[06/14] - "02-repiet_-_dont_think_
twice_(extended_mix).mp3""): the artist's album bounded by the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Quoted text other than the file name stays in the remainder: ">>> CREATIVE
COMMONS NZB <<< "0190 alarm 5.0 " - File 1 of 9: "…rar"" made 12,605
files of all uploads one release.
- namesRelease accepts words joined by , & : ; \ / + ("Posting & deleting,
read 0 file "E:\VladModels\y038_Irina\185" …"): such image sets fell to
the stem rule ("P8250001.JPG"), one release over all folders.
- imageSet: a 5+ digit number left in the set is a per-image id
("sss-70556-3474182-23-10825418-pc", "Alena p 26663_…"): no set. Copy
markers may repeat ("diana-050-052[1][1]").
- unquotedName: an image after numbered text without a separator is the
last word ("<abm> … AB: part2 0573-1002 043.jpg").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-less size before the closing yEnc ('"x.mp3" - 8123 yEnc') took
the set number in 'Newstar Destiny I set 1 [1 of 60] "destiny-001-001.jpg"
yEnc' once the quotes were gone: 97 sets of 60 were one release. A number
of fewer than 4 digits after a word stays.
- unquotedName also cuts after "-> " ("… -> [0001 of 0076] ->
Exclusive-tgirls_000000-lg.jpg"): the arrow was part of the file name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image whose subject has no words ("##[ "diana-040-001.jpg" (01/68)")
is keyed by its set, not its name without numbers: a poster's 263 diana
sets were one release of 17,808 images.
- imageSet strips "-2048px" and ImageVenue's "_123_244lo", and reads a
spaced suffix ("she has sensual legs 172 _ 172") as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each image was a release of its own in: "Set 008 10of63 new0464.jpg",
"202/430 princess-079-058.jpg", "( 098 /500) 002 148.jpg" (the counter
is part of the name: the set is the text before it),
"sexypattycake_0903040445[1].jpg" (a copy's marker), "092_081_2000x1500.jpg"
(a resolution) and "KP0027_197_00.jpg" (a constant suffix).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The file total is read from the subject without the file name, as with a
quoted one: "shyla-shy-09-0195.jpg hardcore …" read as 9 of 195 made each
of 866 images a release of its own.
- imageSet: "_N" right after a number is a renumbering ("kate108lg029_28"
once an image was skipped), and a trailing camera timestamp
("kurozo-2006-7-26-15-21") numbers no set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
