Migration 7400 moves pending_releases (with its two HDD indexes),
release_inspections and probe_cache to the default tablespace (nvme on
homelab): the edge's batch upserts into pending_releases took minutes each
on the HDD and the prober's flushes waited on them (LOG.md (137)). Homelab
moved them by hand with the services stopped: 66 GB in 10.5 minutes.
With everything the prober touches on the NVMe, the nightly pause leaves the
prober running (usenet-rebuild-pause.sh), and a deploy while paused
recreates it with node-api and web (deploy.sh).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
writePending upserted pending_releases in chunks of 1,000 rows, each chunk
followed by its class upsert, so one transaction locked releases, classes,
releases, classes, and a later chunk's keys and classes could sort before an
earlier chunk's: the opposite of the ladder's record(), which holds all its
rows and then takes all its classes. Four deadlocks between the two class
upserts in six hours (LOG.md (136)). Now all rows are sorted once and the
classes upserted once, after the last chunk.
Migration 7300 moves probe_candidates and its indexes to the default
tablespace (nvme on homelab): the catalog claim's UPDATE from random points
took 2-3 minutes on the HDD beside the sweep and the autovacuum, with all
lanes waiting. Homelab moved it by hand (83 s, ~8 GB, services stopped); the
migration is a no-op there, and without a default tablespace.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A poster of 2.7 kB broke the index build (a btree key is at most a third of a page) and left an invalid index
that IF NOT EXISTS would have taken for the real one: the migration drops it first and indexes md5(poster), and
the sweep asks by it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The discard sweep of a dropped signature read the whole pending_releases table per signature (a parallel seq
scan over the poster test, four minutes on the HDD that starved the lanes' claims): a partial index on
(group_id, tool, poster) WHERE status = 0 makes it a range, and the sweep queries a poster's rows by it and a
tool's and group's by the ladder index.
The batch store's pending upsert, the record, and the sweep deadlocked on pending_releases and pending_classes
rows locked in different orders, and a deadlocked flush dropped its outcomes: lockOrder() sorts every
multi-row statement's set the same way, and a flush that deadlocks anyway keeps its outcomes for the next.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After aa40222 the prober still idled: a dropped signature's range of the
ladder index is minutes of HDD (309 dropped, 242 with a poster tested on
the rows) and the sweep read the same ranges again every five minutes
inside the claim. Now each dropped class and signature is swept once
(swept_at, migration 7100; null again when the record drops it anew),
20 of each per sweep in rounds of 10k rows, in a loop of its own in the
prober beside the lanes and the namer (LOG.md (133)).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tiny-archive rule (dropTinyArchives, both indexers): a settled single
archive of one article and 2-100 kB under a release-like name, spaces
allowed, outside the exempt groups and without an e-book, subtitle,
crack, cover or sidecar word: the 2015-17 exe-stub flood in miniature,
1.66M catalog rows (LOG.md (131), (133)).
PendingLadder.discard scanned every waiting row against the dropped
signatures and classes, 9+ minutes a sweep inside the claim, stalling
every lane: now driven from the dropped classes and signatures into the
indexes. The namer feeds the indexer the span's overview text chunk by
chunk (one string of it topped V8's limit) and gives up spans over 3M
articles. Migration 7000: a partial index on the taken probe candidates
for the prober's start.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first run scanned the 58M-row pending table on the HDD once per
6-hour window and starved the prober and the batch store. Now one scan
classes the finished rows and seeds the record, and the waiting rows go
by windows of created_at through their index. The prober's start puts a
previous run's claims back with a status scan of the same table, 20+
minutes after every deploy: a partial index on the taken rows (migration
6900) makes it instant.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).
- lib/release/classes.ts, names::pending_class, SQL pending_class():
a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
target's article; candidates carry bytes now), pending_classes with
each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
the unexplored, at most a quarter of a claim from one), the cheapest
target first within one, then the oldest of the rest; records classes
like signatures and drops them at the same thresholds; discards their
waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
dropped class out of the pending store (counted as garbage), the
explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
catalog release's names a pending release the probe didn't, and sets
the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
the finished ones; rules:classes publishes the dropped classes;
releases:readable also marks the rows' candidates unreadable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every group is tiled from article 1 now: one range with status 'ancient'
covers what had expired on the server before the group was planned.
Nothing selects it (leases, prefetcher, corpus fill, tail merger all take
'planned'), the status panel reports it apart, the search's history stats
leave it out. Migration 6700 adds it for the groups planned before;
Planner.historyRows (now a pure function, unit-tested) for new groups.
The status bands are ordered by an explicit key: the ancient row at
priority 0 tied "history 0-999" with "window 0".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.
`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.
category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The incremental judge counted each changed release's files twice, each
index entry a heap fetch: 24-32 s a chunk. file_hashes by (hash16k,
size, release_id) replaces (hash16k); the other releases of a file are
judged only when the chunk brings it to 20.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The judge's fake rule reads all of it every round: 17-18 min on the HDD,
~3 on the NVMe. Migration 4600 is a no-op now; 4800 moves it back where
4600 ran. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round took 22-29 min an hour: every match went through the insert's
conflict check (60-150 s a rule), scraped and locked each scanned all of
releases. The spot inspector's releases now add their poster to posters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober's output, appended and read a release at a time: 40% of the
NVMe's growth while probing. Homelab moved it by hand; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each claim looks its candidates up in probe_cache by message_id; with the
index on the HDD that took 1.27 s a claim, 48 s of each minute. Moved by
hand on homelab; a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
~20 GB of the NVMe, read by key only; the catalog was at its cap and history
paused. Moved on homelab by hand (3 min, writers stopped; the cap raised to
100); the migration does the same on a new deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The October partition's three indexes took 2.9 GB of the NVMe, its pkey
never scanned; chunks are read from the HDD anyway. Moved on homelab by hand
(12 s); the migration does the same on a new deploy, parent indexes too so new
partitions follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a spot that becomes a release of its own needs its NZB (the header
pass's releases have their chunks); spots took 2.7 GB on the NVMe at 9k.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spot older than what the catalog has indexed (the spotter's first backfill
reaches back ~3 months) matched nothing and would have become a release of
its own, a duplicate once the history pass builds the upload. Unmatched
spots are matched again in turns (rematched_at) for 30 days; the 24 h before
a standalone release count from the inspection; a later match deletes the
spot's own release and links the header pass's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober writes it every flush and the merger looks inner stems up in it,
so it no longer belongs with the write-once payloads on the HDD. Moved on
homelab by hand (prober and merger stopped); the migration is a no-op there
and moves it on a new deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hep, hephaestus and gyges post each file under a subject name of its own
(31,000 one-file releases), but the yEnc name in each file's first article
keeps the upload's random stem ("bX147zx8I0oEgj7nzFCiorCT0ps2jEcb.7z.001"),
one stem per counter total. The merger's second pass joins releases whose
lone inspections share a random stem: one group and poster, the same total,
within 48 h, no more files than the total plus 10. An expression index over
the stem; release_merges records the kind (via) and takes a stem.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The settler marks a settled release's candidates ready (and unreadable when
it had no readable name). Claims took the candidates by joining releases
from random points of the queue; in its newer half nothing had settled, so
a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and
the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over
the claimable rows only, the slices drawn within them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of a claim filtered ~700k rows by kind through the id index (0.37 s
a slice, 20 slices and up to 5 tiers a claim, the lanes waiting meanwhile).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- XoverClient.bodies: PROBE_PIPELINE (8) BODY commands per connection at
once, replies read in order (one at a time it ran 3-10 articles/s on 6
connections against a queue of 1M).
- Claims in tiers: lone files first, then par2 sets and archives of
releases without a readable name (what the par2 merge needs), then those
of readable releases.
- probe_cache: final answers by message-id (status, inspection with the
16 KiB hash and par2 file list), PROBE_VERSION 2. A reset empties the
probe queue, not the cache; recreated candidates of probed articles are
answered from it without the news server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.
edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default page (no search words) took 23-27 s: SearchService sorts
posted_at DESC NULLS LAST, id DESC, the indexes were on posted_at DESC
(nulls first), so Postgres sorted all 68M releases. New indexes match
the sort (also per category), built concurrently; the old ones dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search matches a word in the name or the inner name (trigram index on
inner_name, partial, built concurrently); "Readable names" includes
releases with an inner name.
- unlocked=1 (chip "No password") leaves out passworded releases.
- Results carry passworded and innerName: a Password badge, the inner
name under the release name.
- The release view: Inside and Inspected facts, a passworded notice, and
Contents (per probed article its yEnc name, kind, type, encryption and
the names inside) from release_inspections.
- Newznab: the password attribute from the prober; an obfuscated release
with an inner name is included, titled by it (without its extension).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
