The first run scanned the 58M-row pending table on the HDD once per
6-hour window and starved the prober and the batch store. Now one scan
classes the finished rows and seeds the record, and the waiting rows go
by windows of created_at through their index. The prober's start puts a
previous run's claims back with a status scan of the same table, 20+
minutes after every deploy: a partial index on the taken rows (migration
6900) makes it instant.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The production dry run of releases:readable flagged 5.37M of 19.8M rows,
and its first sample held real names: "CrossStitcher - Issue 440",
"SchiffsModell", "FastFilms 2.04", "BestCrypt NP v4.0", "SmithsWork",
"Laura15Juli2026". Two CamelCase words with few vowels have four runs and
no lone letter: a scramble now needs five runs, a lone letter or an inner
capital run besides its few vowels; the consonant-run clause is gone
(it counted y); a token whose letter runs are all words or acronyms with
numbers between them is neutral. On a 49k-row production sample the
flagged share went from 27.4% to 26.4% with the letters-only flags all
scrambles; the evaluation sets are unchanged (LOG.md (129)).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).
- lib/release/classes.ts, names::pending_class, SQL pending_class():
a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
target's article; candidates carry bytes now), pending_classes with
each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
the unexplored, at most a quarter of a claim from one), the cheapest
target first within one, then the oldest of the rest; records classes
like signatures and drops them at the same thresholds; discards their
waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
dropped class out of the pending store (counted as garbage), the
explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
catalog release's names a pending release the probe didn't, and sets
the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
the finished ones; rules:classes publishes the dropped classes;
releases:readable also marks the rows' candidates unreadable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Searches with the readable filter on returned scrambles: with a random
4-letter query nearly every hit was one ("Zx5AjNaK-mh7CFg6Hf8uu",
"Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ", "bd5b...-addon"), and
15% of a 6,000-row sample of the catalog's readable names were. looksRandom
knows single-token shapes only; separators, spaces, a random "extension"
of any length and a packaging word hid the rest.
isUnreadableName (lib/release/readable.ts) and names::is_unreadable_name
read a name token by token: words (one case or Capitalized), scrambles
(letter/digit and case switches, lone letters between capitals, inner
capital runs, few vowels, hex) and neutral tokens (digits, codes like
S02E10 and 1080p, CamelCase, packaging words); a name is unreadable when
its scrambles outweigh its words. Scene tags' "i" for "I" and the
NZBSPLIT hash are read as such. On the evaluation sets: 554 of 559
random-search names caught, 0 of 630 real names flagged (LOG.md (129)).
The edge's readable flag and the prober's inner names use it; the drop
rules keep looksRandom, which is narrower. An unnamed release goes to the
pending store (decision f), so more of the obfuscated groups' releases
wait for a probe now. releases:readable clears the flag on the existing
catalog's rows by name (a dry run by default).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first five hours in production (LOG.md (128)) dropped 522 passworded
videos of one a.b.erotica poster: ROT13-named volumes (".cnegN.rar" is
".partN.rar") of 1-270 MB in 258 kB articles to two groups, which the
article-size leg took for stubs. The cap that X and W already had now
applies to A as well (the catalog sample has 6 A-leg hits over 100 MB,
4 of them exe-only), and a ROT13 volume name is no single archive.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The judge gets each file's articles seen and bytes, and whether the window
closed on the release. A settled release of one unsplit archive posted to 2+
groups is dropped as a stub archive when its articles are <= 256 KiB with the
release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two
under 100 MB); releases touching music, e-book, comics, tabs or fonts groups
are exempt. A one-article archive of 100 kB-1 MB under a release-like,
unscrambled name without spaces or an e-book format is dropped as a nonsense
piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces),
mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.
looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.
The prefetcher's corpus fill skips groups that aren't active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"^zed" finds "Zed Dead", "The_Zed_Files", not "Facialized": a regex on
release_texts, (^|[^[:alnum:]]) before the word, which the trigram index
serves (homelab: 551 texts for ^zed against 6,606 for zed, 35 ms warm).
Words without ^ match anywhere as before; the search box says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pending namer's TypeScript re-index gave a file short of parts on the
server to the range path's fragment rule, so a partial release the edge
wrote (Bob's hex-named uploads in chello, ratcave, font, storage) never came
out again: 10 of 148k named releases moved. RangeIndexer's partialFiles
keeps such a file in its release, marked partial; tools/namer_check.ts
replays sample rows from the corpus (10 of 10 found with it, 1 without).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The next claim started at half a batch of four pipeline rounds a lane,
3-6 s of work, while a claim takes 1.5-4 s and more; at 20-24 lanes the
lanes waited for candidates up to 19% of their time. queueLevels: the
claim starts below eight rounds of every lane's pipeline (~13 s) and
fills the queue to twice that. Taken candidates go back to waiting at
start-up as before, so a bigger queue costs nothing on a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every group is tiled from article 1 now: one range with status 'ancient'
covers what had expired on the server before the group was planned.
Nothing selects it (leases, prefetcher, corpus fill, tail merger all take
'planned'), the status panel reports it apart, the search's history stats
leave it out. Migration 6700 adds it for the groups planned before;
Planner.historyRows (now a pure function, unit-tested) for new groups.
The status bands are ordered by an explicit key: the ancient row at
priority 0 tied "history 0-999" with "window 0".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The sampling ranges in other years are capture too, at history priorities: they made a band each. The database
size walked the HDD partitions for minutes and the probe counts took 10 s on 11 M rows: dropped, and the partial
indexes' reltuples instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.
`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A release the window hadn't closed on at a lease's end is written provisionally
for its continuation to finish; the first run's range leases left 109k such
rows at their blocks' end edges, 606 of them complete, shown as still
collecting (release 10697). The coordinator now settles a provisional release
that is complete and quiet for six hours as the finish would: settled_at set,
its waiting probe candidates made claimable. Incomplete ones keep waiting.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A round added a row a group whenever 100k articles or 15 minutes had passed,
tuned for a tail leased within minutes; with nothing leasing the tail the rows
piled up (127k planned tail ranges of a median 14k articles, 97% adjacent). Now
a round grows the group's last range when it is a planned, unspooled, untaken
row at the tail's priority, up to RANGE_ARTICLES, and only then adds rows; a
range taken meanwhile gets the slice as a row. ranges:merge-tail (--dry-run)
packs adjacent planned, uncaptured, unspooled, untaken tail ranges of a group
and priority into ranges of RANGE_ARTICLES under the lease lock.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A plain stop and a chunk that doesn't inflate already posted the batch so far,
so the stop snapshot stood at a stored resume point; a failed stream (the
stalled one's "Peer disconnected") skipped the post, left the batch open and
lost the window with the snapshot refused. Now only a batch the coordinator
refused stays open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The corpus fill claims chunks of a leased range ahead of the leased run (the
prefetcher's claim keys are per range and chunk), stores them in the corpus
only, and the run leaves them there. The stream looked at the corpus once per
chunk and then polled the spool only: when the store came seconds after that
look, the stream waited its whole hour and the worker lost the group and its
open batch (three groups on 2026-10-27, LOG.md (118)). Now the wait loop checks
both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lease bounds alone don't select a window: the history planner's oldest
buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026
history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the
2016 corpus window sat at voyager 2's leftover priorities (done when the window
was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next
and the prefetcher's run-ahead take capture ranges only; the run-ahead now also
keeps to the lease bounds. Test: a group with history at 0 only isn't leased,
the history range after the window isn't handed out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A live reply cut short (the server's compressed data ending at ".\r\n" at a
read boundary; the spool keeps what Corpus.write refuses) made the worker give
the group back without a snapshot, losing the window and the batch since its
last post (six times in the first 40 minutes on homelab). Now the batch before
the chunk goes out, the chunk is reported (fetched again, plain the second
time) and the range is asked for again from that chunk (RangeOutcome::Again):
the stream waits for the new copy and the window stays. Test: the coordinator
hands the same range back at its resume point after a corrupt report.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
xoverRaw gives one zlib/gzip stream, as the corpus stores it; the namer fed
the bytes as text and found nothing. The prober had no corpus mount, so every
move went to the server.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CI's fresh database had no release_chunks partition yet: the batch store
creates it on its own path, the namer's test wrote a chunk without it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
deploy.sh and the nightly resume leave the edge stopped while .cache/no-edge
exists (range by range it would split releases at every edge with no merger
left). deploy/replay-window.sh takes DB and OUT and passes
REPLAY_INTO_CATALOG=1, which lets replay:corpus write a database not named
*_replay or *_scratch (never the test one).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
a.b.storage's stored 2016 ranges are not contiguous (its 2016 window lies in two
article regions); the batch after a gap kept the previous range's end as its
start and the harness found no leased range for it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
acs5152s1.11 is a volume, Broforce.vol00+01.par2.temp a par2: neither an
album's member. replay:corpus reported every release in the scratch catalog
as the group's.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A release the window closed on with one file named by its stem or post, the
name with a number and no archive, volume, sidecar or random name, waits in
the album of its poster and template (the name's digit runs as #, the
extension apart) posted within the window of the album's first file; the
album closes a window after that span and is written as one release
(source album, no file total) named by the members' shared numbers, a lone
member as the release it was. The batch store writes an album's member
names as release_texts kind 6 (member). A counter total of 100 or more under
which one poster started 20 or more releases is a batch counter: those
releases get no file total.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
deploy/replay-window.sh passes the compose app environment (HOST, PORT, APP_URL,
SPOOL_DIR, CHUNK_TABLESPACE) the ace command validates. The resume script starts
only the coordinator (the corpus fill) while .cache/v2-reset exists: the voyager 2
catalog was reset on 2026-10-26 and its workers would fill it again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The app image failed to build ("cannot copy from stage edge-build, it needs to be defined before current stage app").
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
edge-bench --group and replay:corpus take --to <article> next to --from, so a
replay covers one window of a group's history while the corpus holds others
(the 2016 ranges next to 2021's and 2025's). deploy/replay-window.sh runs a
window on homelab: every group whose window ranges are all stored is replayed
once into usenet_replay through the voyager-3 app image, 4 groups at a time,
looping while the corpus fill completes the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
replay:corpus runs edge-bench --group over a group's complete corpus ranges
in article order (one window, carried from range to range, finished after
the last range; edge/src/lease.rs holds the lease JSON the worker and the
bench share) and stores every batch through BatchStore as the worker
would, then reports releases finished, provisional, complete and the bytes
per header of the NVMe and HDD tables. Refuses a database not named
*_replay or *_scratch. The app image carries edge-bench; compose gets a
replay profile for homelab.
Design amended (REBUILD.md 1, 2, 5): the window is carried by one worker a
group, no block leases and no block merger; provisional writes stay for
the lease ends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 512 candidates were the lowest ids at the lowest priority; a group's
ranges have consecutive ids, so once a window's small groups were stored
they were all one group's, two targets on two connections (10 ranges an
hour). Ordered by the per-group rank now: the window spans 85 groups.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.
Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sixteen groups never got through on one connection (a reply timed out
after 2 minutes); now a timed-out point is skipped and logged, the
group goes on, and stops after 3 failures. estimate() ignored --missing's
filter: it dated 0 groups on the second run.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A 24 h posting-time window per block instead of write-early-merge-later,
albums for counterless series, poster/group-set ids and binary keys in
releases, cold tables on the HDD, the corpus test before the reset.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The NVMe floor is 4 GB since Oct 23; at 12 the prober had claimed nothing
since Oct 23 15:30 (7-10 GB free).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.
A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sets stored in one batch share created_at, so paging on created_at < last
skipped ~10k of 61k sets; the timestamp now also stays text so microseconds
survive the round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prefetcher fetched capture ranges into the corpus 16 at a time in id
order, and ids run group by group: the 16 were one or two groups', the
per-group connection limit (4) let 4-8 loops work and left the rest
idle, ~30k articles/s for the first batch of 19,955 capture ranges. Now
at most perGroup / perRange (2) of a group are taken, so the 16 come
from 8+ groups and every loop gets work (and the pool asks for more).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The name pass put every holder of a set into its merge: a par2 posted
again years later ("Auto Bild Klassik Februar 02-2014", held by a 2014
and a 2026 release) would have joined the repost to the old upload. Now
the newest holder anchors it and only holders in its group within 12
hours take part. Every round, three sets failed on the probe candidates'
unique key: a member held a candidate of an article the target had.
merge() now drops such a candidate (the target's, or a lower member's,
stays), as it already does for par2 sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.
category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merges delete releases, so the count of a key's releases fell behind its
numbers (83 releases up to key:129): the 20 numbers tried were all taken
and the range's batch failed for good. Spot releases (spot:key) are not
numbered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The incremental judge counted each changed release's files twice, each
index entry a heap fetch: 24-32 s a chunk. file_hashes by (hash16k,
size, release_id) replaces (hash16k); the other releases of a file are
judged only when the chunk brings it to 20.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The judge's fake rule reads all of it every round: 17-18 min on the HDD,
~3 on the NVMe. Migration 4600 is a no-op now; 4800 moves it back where
4600 ran. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round took 22-29 min an hour: every match went through the insert's
conflict check (60-150 s a rule), scraped and locked each scanned all of
releases. The spot inspector's releases now add their poster to posters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober's output, appended and read a release at a time: 40% of the
NVMe's growth while probing. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jsonb can't hold \u0000, and ON CONFLICT can't take a row twice in one
statement: either failed the whole flush, its outcomes lost and its
candidates taken until a restart. NULs are dropped from strings; the
probe_cache insert takes each message-id once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each claim looks its candidates up in probe_cache by message_id; with the
index on the HDD that took 1.27 s a claim, 48 s of each minute. Moved by
hand on homelab; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
