A round added a row a group whenever 100k articles or 15 minutes had passed,
tuned for a tail leased within minutes; with nothing leasing the tail the rows
piled up (127k planned tail ranges of a median 14k articles, 97% adjacent). Now
a round grows the group's last range when it is a planned, unspooled, untaken
row at the tail's priority, up to RANGE_ARTICLES, and only then adds rows; a
range taken meanwhile gets the slice as a row. ranges:merge-tail (--dry-run)
packs adjacent planned, uncaptured, unspooled, untaken tail ranges of a group
and priority into ranges of RANGE_ARTICLES under the lease lock.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A plain stop and a chunk that doesn't inflate already posted the batch so far,
so the stop snapshot stood at a stored resume point; a failed stream (the
stalled one's "Peer disconnected") skipped the post, left the batch open and
lost the window with the snapshot refused. Now only a batch the coordinator
refused stays open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The corpus fill claims chunks of a leased range ahead of the leased run (the
prefetcher's claim keys are per range and chunk), stores them in the corpus
only, and the run leaves them there. The stream looked at the corpus once per
chunk and then polled the spool only: when the store came seconds after that
look, the stream waited its whole hour and the worker lost the group and its
open batch (three groups on 2026-10-27, LOG.md (118)). Now the wait loop checks
both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lease bounds alone don't select a window: the history planner's oldest
buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026
history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the
2016 corpus window sat at voyager 2's leftover priorities (done when the window
was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next
and the prefetcher's run-ahead take capture ranges only; the run-ahead now also
keeps to the lease bounds. Test: a group with history at 0 only isn't leased,
the history range after the window isn't handed out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A live reply cut short (the server's compressed data ending at ".\r\n" at a
read boundary; the spool keeps what Corpus.write refuses) made the worker give
the group back without a snapshot, losing the window and the batch since its
last post (six times in the first 40 minutes on homelab). Now the batch before
the chunk goes out, the chunk is reported (fetched again, plain the second
time) and the range is asked for again from that chunk (RangeOutcome::Again):
the stream waits for the new copy and the window stays. Test: the coordinator
hands the same range back at its resume point after a corrupt report.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
xoverRaw gives one zlib/gzip stream, as the corpus stores it; the namer fed
the bytes as text and found nothing. The prober had no corpus mount, so every
move went to the server.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CI's fresh database had no release_chunks partition yet: the batch store
creates it on its own path, the namer's test wrote a chunk without it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
deploy.sh and the nightly resume leave the edge stopped while .cache/no-edge
exists (range by range it would split releases at every edge with no merger
left). deploy/replay-window.sh takes DB and OUT and passes
REPLAY_INTO_CATALOG=1, which lets replay:corpus write a database not named
*_replay or *_scratch (never the test one).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
a.b.storage's stored 2016 ranges are not contiguous (its 2016 window lies in two
article regions); the batch after a gap kept the previous range's end as its
start and the harness found no leased range for it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
acs5152s1.11 is a volume, Broforce.vol00+01.par2.temp a par2: neither an
album's member. replay:corpus reported every release in the scratch catalog
as the group's.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A release the window closed on with one file named by its stem or post, the
name with a number and no archive, volume, sidecar or random name, waits in
the album of its poster and template (the name's digit runs as #, the
extension apart) posted within the window of the album's first file; the
album closes a window after that span and is written as one release
(source album, no file total) named by the members' shared numbers, a lone
member as the release it was. The batch store writes an album's member
names as release_texts kind 6 (member). A counter total of 100 or more under
which one poster started 20 or more releases is a batch counter: those
releases get no file total.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
deploy/replay-window.sh passes the compose app environment (HOST, PORT, APP_URL,
SPOOL_DIR, CHUNK_TABLESPACE) the ace command validates. The resume script starts
only the coordinator (the corpus fill) while .cache/v2-reset exists: the voyager 2
catalog was reset on 2026-10-26 and its workers would fill it again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The app image failed to build ("cannot copy from stage edge-build, it needs to be defined before current stage app").
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
edge-bench --group and replay:corpus take --to <article> next to --from, so a
replay covers one window of a group's history while the corpus holds others
(the 2016 ranges next to 2021's and 2025's). deploy/replay-window.sh runs a
window on homelab: every group whose window ranges are all stored is replayed
once into usenet_replay through the voyager-3 app image, 4 groups at a time,
looping while the corpus fill completes the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
replay:corpus runs edge-bench --group over a group's complete corpus ranges
in article order (one window, carried from range to range, finished after
the last range; edge/src/lease.rs holds the lease JSON the worker and the
bench share) and stores every batch through BatchStore as the worker
would, then reports releases finished, provisional, complete and the bytes
per header of the NVMe and HDD tables. Refuses a database not named
*_replay or *_scratch. The app image carries edge-bench; compose gets a
replay profile for homelab.
Design amended (REBUILD.md 1, 2, 5): the window is carried by one worker a
group, no block leases and no block merger; provisional writes stay for
the lease ends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 512 candidates were the lowest ids at the lowest priority; a group's
ranges have consecutive ids, so once a window's small groups were stored
they were all one group's, two targets on two connections (10 ranges an
hour). Ordered by the per-group rank now: the window spans 85 groups.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.
Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sixteen groups never got through on one connection (a reply timed out
after 2 minutes); now a timed-out point is skipped and logged, the
group goes on, and stops after 3 failures. estimate() ignored --missing's
filter: it dated 0 groups on the second run.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A 24 h posting-time window per block instead of write-early-merge-later,
albums for counterless series, poster/group-set ids and binary keys in
releases, cold tables on the HDD, the corpus test before the reset.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The NVMe floor is 4 GB since Oct 23; at 12 the prober had claimed nothing
since Oct 23 15:30 (7-10 GB free).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.
A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sets stored in one batch share created_at, so paging on created_at < last
skipped ~10k of 61k sets; the timestamp now also stays text so microseconds
survive the round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prefetcher fetched capture ranges into the corpus 16 at a time in id
order, and ids run group by group: the 16 were one or two groups', the
per-group connection limit (4) let 4-8 loops work and left the rest
idle, ~30k articles/s for the first batch of 19,955 capture ranges. Now
at most perGroup / perRange (2) of a group are taken, so the 16 come
from 8+ groups and every loop gets work (and the pool asks for more).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The name pass put every holder of a set into its merge: a par2 posted
again years later ("Auto Bild Klassik Februar 02-2014", held by a 2014
and a 2026 release) would have joined the repost to the old upload. Now
the newest holder anchors it and only holders in its group within 12
hours take part. Every round, three sets failed on the probe candidates'
unique key: a member held a candidate of an article the target had.
merge() now drops such a candidate (the target's, or a lower member's,
stays), as it already does for par2 sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.
category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
