Commit Graph
355 Commits
Author SHA1 Message Date
Daan MeijerandClaude Fable 5.1 d7171a5ca4 Pending ladder: the sweep and the record lock their rows in key order
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m4s
discardKeys updated with `release_key = ANY(subquery)`, which locks rows in
the plan's scan order, and deadlocked against a batch upsert that takes its
rows by key (LOG.md (137)). It now updates from a subquery that selects the
rows ORDER BY release_key FOR UPDATE, so the locks are taken sorted (the
plan: LockRows above Sort). record() runs the same SELECT ... FOR UPDATE over
its taken rows before its UPDATE ... FROM jsonb_to_recordset, whose join
order the planner chooses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 00:32:09 +01:00
Daan MeijerandClaude Fable 5.1 eadae82b0c The prober's tables on the NVMe; the prober runs through the night
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 1m30s
Migration 7400 moves pending_releases (with its two HDD indexes),
release_inspections and probe_cache to the default tablespace (nvme on
homelab): the edge's batch upserts into pending_releases took minutes each
on the HDD and the prober's flushes waited on them (LOG.md (137)). Homelab
moved them by hand with the services stopped: 66 GB in 10.5 minutes.

With everything the prober touches on the NVMe, the nightly pause leaves the
prober running (usenet-rebuild-pause.sh), and a deploy while paused
recreates it with node-api and web (deploy.sh).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 23:57:59 +01:00
Daan MeijerandClaude Fable 5.1 e02a7c89b4 REBUILD.md: probe_candidates back on the NVMe (migration 7300)
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:52 +01:00
Daan MeijerandClaude Fable 5.1 6ef7ddca92 Batch store: one lock order for the pending upsert; probe_candidates on the NVMe
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m33s
writePending upserted pending_releases in chunks of 1,000 rows, each chunk
followed by its class upsert, so one transaction locked releases, classes,
releases, classes, and a later chunk's keys and classes could sort before an
earlier chunk's: the opposite of the ladder's record(), which holds all its
rows and then takes all its classes. Four deadlocks between the two class
upserts in six hours (LOG.md (136)). Now all rows are sorted once and the
classes upserted once, after the last chunk.

Migration 7300 moves probe_candidates and its indexes to the default
tablespace (nvme on homelab): the catalog claim's UPDATE from random points
took 2-3 minutes on the HDD beside the sweep and the autovacuum, with all
lanes waiting. Homelab moved it by hand (83 s, ~8 GB, services stopped); the
migration is a no-op there, and without a default tablespace.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 19:52:09 +01:00
Daan MeijerandClaude Fable 5.1 a71de56a7c Judge: the tiny rule, the catalog's side of the edge's tiny-archive rule
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 4m23s
One file of one article, 2–100 kB, under a readable release-like name outside the music, e-book and comic
groups, without an e-book, subtitle, crack or sidecar word: the 2015–17 flood's 1.5M rows that the edge
indexed before its rule. The same tests as isTinyArchive in Postgres's regex dialect; no join on the
candidates (98% of a sample have an archive candidate, the rest lone par2 files and scrambled names).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 13:15:03 +01:00
Daan MeijerandClaude Fable 5.1 411c36eec0 Prober: the signature index by the poster's md5
Test and deploy / test (push) Successful in 3m5s
Test and deploy / deploy (push) Successful in 1h2m47s
A poster of 2.7 kB broke the index build (a btree key is at most a third of a page) and left an invalid index
that IF NOT EXISTS would have taken for the real one: the migration drops it first and indexes md5(poster), and
the sweep asks by it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 12:08:53 +01:00
Daan MeijerandClaude Fable 5.1 8e008da364 Prober: the sweep by a signature index, pending rows locked in one order, deadlocked flushes kept
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Failing after 21m32s
The discard sweep of a dropped signature read the whole pending_releases table per signature (a parallel seq
scan over the poster test, four minutes on the HDD that starved the lanes' claims): a partial index on
(group_id, tool, poster) WHERE status = 0 makes it a range, and the sweep queries a poster's rows by it and a
tool's and group's by the ladder index.

The batch store's pending upsert, the record, and the sweep deadlocked on pending_releases and pending_classes
rows locked in different orders, and a deadlocked flush dropped its outcomes: lockOrder() sorts every
multi-row statement's set the same way, and a flush that deadlocks anyway keeps its outcomes for the next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 10:36:19 +01:00
Daan MeijerandClaude Fable 5.1 5d3b578322 The discard sweep beside the lanes, each dropped signature swept once
Test and deploy / test (push) Successful in 2m57s
Test and deploy / deploy (push) Successful in 38m14s
After aa40222 the prober still idled: a dropped signature's range of the
ladder index is minutes of HDD (309 dropped, 242 with a poster tested on
the rows) and the sweep read the same ranges again every five minutes
inside the claim. Now each dropped class and signature is swept once
(swept_at, migration 7100; null again when the record drops it anew),
20 of each per sweep in rounds of 10k rows, in a loop of its own in the
prober beside the lanes and the namer (LOG.md (133)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 21:12:51 +01:00
Daan MeijerandClaude Fable 5.1 aa4022295a Tiny-archive rule; the ladder's discard by index; the namer in chunks
Test and deploy / test (push) Successful in 3m1s
Test and deploy / deploy (push) Successful in 32m21s
The tiny-archive rule (dropTinyArchives, both indexers): a settled single
archive of one article and 2-100 kB under a release-like name, spaces
allowed, outside the exempt groups and without an e-book, subtitle,
crack, cover or sidecar word: the 2015-17 exe-stub flood in miniature,
1.66M catalog rows (LOG.md (131), (133)).

PendingLadder.discard scanned every waiting row against the dropped
signatures and classes, 9+ minutes a sweep inside the claim, stalling
every lane: now driven from the dropped classes and signatures into the
indexes. The namer feeds the indexer the span's overview text chunk by
chunk (one string of it topped V8's limit) and gives up spans over 3M
articles. Migration 7000: a partial index on the taken probe candidates
for the prober's start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 20:23:16 +01:00
Daan MeijerandClaude Fable 5.1 4d46db291e pending:classes without table scans; an index for the prober's startup reset
Test and deploy / test (push) Successful in 2m55s
Test and deploy / deploy (push) Successful in 25m6s
The first run scanned the 58M-row pending table on the HDD once per
6-hour window and starved the prober and the batch store. Now one scan
classes the finished rows and seeds the record, and the waiting rows go
by windows of created_at through their index. The prober's start puts a
previous run's claims back with a status scan of the same table, 20+
minutes after every deploy: a partial index on the taken rows (migration
6900) makes it instant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:33:48 +01:00
Daan MeijerandClaude Fable 5.1 3612bf3e43 Readable names: CamelCase product names and words with numbers stay readable
Test and deploy / test (push) Successful in 2m49s
Test and deploy / deploy (push) Successful in 3m24s
The production dry run of releases:readable flagged 5.37M of 19.8M rows,
and its first sample held real names: "CrossStitcher - Issue 440",
"SchiffsModell", "FastFilms 2.04", "BestCrypt NP v4.0", "SmithsWork",
"Laura15Juli2026". Two CamelCase words with few vowels have four runs and
no lone letter: a scramble now needs five runs, a lone letter or an inner
capital run besides its few vowels; the consonant-run clause is gone
(it counted y); a token whose letter runs are all words or acronyms with
numbers between them is neutral. On a 49k-row production sample the
flagged share went from 27.4% to 26.4% with the letters-only flags all
scrambles; the evaluation sets are unchanged (LOG.md (129)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 13:22:54 +01:00
Daan MeijerandClaude Fable 5.1 44c0c73b05 Pending ladder by class, twins by content, dropped classes at the edge
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Successful in 40m8s
The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).

- lib/release/classes.ts, names::pending_class, SQL pending_class():
  a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
  target's article; candidates carry bytes now), pending_classes with
  each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
  the unexplored, at most a quarter of a claim from one), the cheapest
  target first within one, then the oldest of the rest; records classes
  like signatures and drops them at the same thresholds; discards their
  waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
  dropped class out of the pending store (counted as garbage), the
  explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
  catalog release's names a pending release the probe didn't, and sets
  the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
  the finished ones; rules:classes publishes the dropped classes;
  releases:readable also marks the rows' candidates unreadable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 12:22:32 +01:00
Daan MeijerandClaude Fable 5.1 81a5dce45a Readable names: a token-by-token classifier for the readable flag
Test and deploy / test (push) Successful in 3m36s
Test and deploy / deploy (push) Skipped
Searches with the readable filter on returned scrambles: with a random
4-letter query nearly every hit was one ("Zx5AjNaK-mh7CFg6Hf8uu",
"Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ", "bd5b...-addon"), and
15% of a 6,000-row sample of the catalog's readable names were. looksRandom
knows single-token shapes only; separators, spaces, a random "extension"
of any length and a packaging word hid the rest.

isUnreadableName (lib/release/readable.ts) and names::is_unreadable_name
read a name token by token: words (one case or Capitalized), scrambles
(letter/digit and case switches, lone letters between capitals, inner
capital runs, few vowels, hex) and neutral tokens (digits, codes like
S02E10 and 1080p, CamelCase, packaging words); a name is unreadable when
its scrambles outweigh its words. Scene tags' "i" for "I" and the
NZBSPLIT hash are read as such. On the evaluation sets: 554 of 559
random-search names caught, 0 of 630 real names flagged (LOG.md (129)).

The edge's readable flag and the prober's inner names use it; the drop
rules keep looksRandom, which is narrower. An unnamed release goes to the
pending store (decision f), so more of the obfuscated groups' releases
wait for a probe now. releases:readable clears the flag on the existing
catalog's rows by name (a dry run by default).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 10:01:31 +01:00
Daan MeijerandClaude Fable 5.1 2643266559 Stub-archive rule: 100 MB cap on every leg, ROT13 volume names are volumes
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Successful in 4m1s
The first five hours in production (LOG.md (128)) dropped 522 passworded
videos of one a.b.erotica poster: ROT13-named volumes (".cnegN.rar" is
".partN.rar") of 1-270 MB in 258 kB articles to two groups, which the
article-size leg took for stubs. The cap that X and W already had now
applies to A as well (the catalog sample has 6 A-leg hits over 100 MB,
4 of them exe-only), and a ROT13 volume name is no single archive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 05:49:37 +01:00
Daan MeijerandClaude Fable 5.1 e470e231d0 Single-archive rules: stub archives and nonsense pieces dropped at the edge
Test and deploy / test (push) Successful in 3m0s
Test and deploy / deploy (push) Successful in 4m21s
The judge gets each file's articles seen and bytes, and whether the window
closed on the release. A settled release of one unsplit archive posted to 2+
groups is dropped as a stub archive when its articles are <= 256 KiB with the
release >= 1 MB, or it is in 4+ groups, or by a webmail poster (those two
under 100 MB); releases touching music, e-book, comics, tabs or fonts groups
are exempt. A one-article archive of 100 kB-1 MB under a release-like,
unscrambled name without spaces or an e-book format is dropped as a nonsense
piece. Both switched by the rule set (dropStubArchives, dropNonsensePieces),
mirrored in lib/rules, sampled like every drop (LOG.md (123)-(127)).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 11:35:46 +01:00
Daan MeijerandClaude Opus 5.5 2f1bbf2538 REBUILD.md: lone articles under scrambled names
Test and deploy / test (push) Successful in 3m8s
Test and deploy / deploy (push) Successful in 2m10s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:20:04 +01:00
Daan MeijerandClaude Opus 5.5 cfce8884d4 Lone articles under scrambled names are per-article; inactive groups get no corpus fill
Test and deploy / test (push) Successful in 3m6s
Test and deploy / deploy (push) Skipped
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.

looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.

The prefetcher's corpus fill skips groups that aren't active.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:19:11 +01:00
Daan MeijerandClaude Opus 5.5 5132ab9083 Search: a word with a leading ^ matches at a word's start only
Test and deploy / test (push) Successful in 3m5s
Test and deploy / deploy (push) Successful in 2m7s
"^zed" finds "Zed Dead", "The_Zed_Files", not "Facialized": a regex on
release_texts, (^|[^[:alnum:]]) before the word, which the trigram index
serves (homelab: 551 texts for ^zed against 6,606 for zed, 35 ms warm).
Words without ^ match anywhere as before; the search box says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 20:19:52 +01:00
Daan MeijerandClaude Opus 5.5 3f0b5b8be1 Subject names lose the brackets the counters left unmatched
Test and deploy / test (push) Successful in 3m3s
Test and deploy / deploy (push) Successful in 2m13s
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:04:06 +01:00
Daan MeijerandClaude Opus 5.5 c6f9409e7f REBUILD.md: the namer's failures were the partial releases, 52k of 210k, not nearly all
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:07:24 +01:00
Daan MeijerandClaude Opus 5.5 8cf965af80 The namer re-reads partial releases as the Rust window writes them
Test and deploy / test (push) Successful in 3m10s
Test and deploy / deploy (push) Successful in 5m38s
The pending namer's TypeScript re-index gave a file short of parts on the
server to the range path's fragment rule, so a partial release the edge
wrote (Bob's hex-named uploads in chello, ratcave, font, storage) never came
out again: 10 of 148k named releases moved. RangeIndexer's partialFiles
keeps such a file in its release, marked partial; tools/namer_check.ts
replays sample rows from the corpus (10 of 10 found with it, 1 without).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 10:41:50 +01:00
Daan MeijerandClaude Opus 5.5 0ccb850ea4 Prober: claim ahead of the lanes, not when they run dry
Test and deploy / deploy (push) Successful in 1m34s
Test and deploy / test (push) Successful in 3m6s
The next claim started at half a batch of four pipeline rounds a lane,
3-6 s of work, while a claim takes 1.5-4 s and more; at 20-24 lanes the
lanes waited for candidates up to 19% of their time. queueLevels: the
claim starts below eight rounds of every lane's pipeline (~13 s) and
fills the queue to twice that. Taken candidates go back to waiting at
start-up as before, so a bigger queue costs nothing on a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 14:52:48 +01:00
Daan MeijerandClaude Fable 5.1 68b480c97e An "ancient" range a group: articles 1 … low watermark − 1, never filled
Test and deploy / deploy (push) Successful in 3m49s
Test and deploy / test (push) Successful in 3m19s
Every group is tiled from article 1 now: one range with status 'ancient'
covers what had expired on the server before the group was planned.
Nothing selects it (leases, prefetcher, corpus fill, tail merger all take
'planned'), the status panel reports it apart, the search's history stats
leave it out. Migration 6700 adds it for the groups planned before;
Planner.historyRows (now a pure function, unit-tested) for new groups.
The status bands are ordered by an explicit key: the ancient row at
priority 0 tied "history 0-999" with "window 0".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-28 09:12:05 +01:00
Daan MeijerandClaude Fable 5.1 c18c320927 coordinator:status: the capture windows are priorities 0–2, the probe backlog from the planner's estimates
Test and deploy / test (push) Successful in 3m12s
Test and deploy / deploy (push) Successful in 2m5s
The sampling ranges in other years are capture too, at history priorities: they made a band each. The database
size walked the HDD partitions for minutes and the probe counts took 10 s on 11 M rows: dropped, and the partial
indexes' reltuples instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 14:00:08 +01:00
Daan MeijerandClaude Fable 5.1 680751e26a Lease lanes for the tail, a mixed lease order, coordinator:status, the edge end to end
Test and deploy / test (push) Successful in 3m14s
Test and deploy / deploy (push) Successful in 2m30s
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail
planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades
through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads`
(EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The
lease order within a lane alternates the group with the most planned articles and the longest unheld one. The
takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range
is held by a holder gone quiet can be taken over.

`coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with
their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the
corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot
file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 13:47:31 +01:00
Daan MeijerandClaude Fable 5.1 db96642a72 A complete provisional release quiet for six hours is settled
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m52s
A release the window hadn't closed on at a lease's end is written provisionally
for its continuation to finish; the first run's range leases left 109k such
rows at their blocks' end edges, 606 of them complete, shown as still
collecting (release 10697). The coordinator now settles a provisional release
that is complete and quiet for six hours as the finish would: settled_at set,
its waiting probe candidates made claimable. Incomplete ones keep waiting.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:04:31 +01:00
Daan MeijerandClaude Fable 5.1 1810abd7d8 The tail planner grows a group's open range; ranges:merge-tail packs the slivers
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m22s
Test and deploy / deploy (push) Successful in 2m53s
A round added a row a group whenever 100k articles or 15 minutes had passed,
tuned for a tail leased within minutes; with nothing leasing the tail the rows
piled up (127k planned tail ranges of a median 14k articles, 97% adjacent). Now
a round grows the group's last range when it is a planned, unspooled, untaken
row at the tail's priority, up to RANGE_ARTICLES, and only then adds rows; a
range taken meanwhile gets the slice as a row. ranges:merge-tail (--dry-run)
packs adjacent planned, uncaptured, unspooled, untaken tail ranges of a group
and priority into ranges of RANGE_ARTICLES under the lease lock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:46:47 +01:00
Daan MeijerandClaude Fable 5.1 e3957e5af5 The edge posts the open batch before a failed stream gives the group back
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m13s
Test and deploy / deploy (push) Successful in 1m18s
A plain stop and a chunk that doesn't inflate already posted the batch so far,
so the stop snapshot stood at a stored resume point; a failed stream (the
stalled one's "Peer disconnected") skipped the post, left the batch open and
lost the window with the snapshot refused. Now only a batch the coordinator
refused stays open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:06:52 +01:00
Daan MeijerandClaude Fable 5.1 31fc204c76 The range stream polls the corpus as well as the spool while it waits
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m13s
Test and deploy / deploy (push) Successful in 4m11s
The corpus fill claims chunks of a leased range ahead of the leased run (the
prefetcher's claim keys are per range and chunk), stores them in the corpus
only, and the run leaves them there. The stream looked at the corpus once per
chunk and then polled the spool only: when the store came seconds after that
look, the stream waited its whole hour and the worker lost the group and its
open batch (three groups on 2026-10-27, LOG.md (118)). Now the wait loop checks
both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 09:19:16 +01:00
Daan MeijerandClaude Fable 5.1 4af5e06400 LEASE_CAPTURE_ONLY: group leases of the corpus windows' ranges only
Test and deploy / test (push) Successful in 1m6s
Test and deploy / edge (push) Successful in 2m44s
Test and deploy / deploy (push) Skipped
The lease bounds alone don't select a window: the history planner's oldest
buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026
history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the
2016 corpus window sat at voyager 2's leftover priorities (done when the window
was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next
and the prefetcher's run-ahead take capture ranges only; the run-ahead now also
keeps to the lease bounds. Test: a group with history at 0 only isn't leased,
the history range after the window isn't handed out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 23:26:31 +01:00
Daan MeijerandClaude Fable 5.1 29e0cc8f6e The window worker keeps its window over a chunk that doesn't inflate
Test and deploy / test (push) Successful in 1m14s
Test and deploy / edge (push) Successful in 2m37s
Test and deploy / deploy (push) Successful in 3m7s
A live reply cut short (the server's compressed data ending at ".\r\n" at a
read boundary; the spool keeps what Corpus.write refuses) made the worker give
the group back without a snapshot, losing the window and the batch since its
last post (six times in the first 40 minutes on homelab). Now the batch before
the chunk goes out, the chunk is reported (fetched again, plain the second
time) and the range is asked for again from that chunk (RangeOutcome::Again):
the stream waits for the new copy and the window stays. Test: the coordinator
hands the same range back at its resume point after a corrupt report.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 22:53:52 +01:00
Daan MeijerandClaude Fable 5.1 1f6e3316ec The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m16s
A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 18:22:18 +01:00
Daan MeijerandClaude Fable 5.1 3c4fe324a1 The namer inflates the XOVER reply and reads the corpus from the prober's container
Test and deploy / test (push) Successful in 57s
Test and deploy / edge (push) Successful in 2m19s
Test and deploy / deploy (push) Successful in 1m58s
xoverRaw gives one zlib/gzip stream, as the corpus stores it; the namer fed
the bytes as text and found nothing. The prober had no corpus mount, so every
move went to the server.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 17:21:34 +01:00
Daan MeijerandClaude Fable 5.1 f1381a9b43 The namer makes sure of the chunk month partition before its write
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 1m4s
CI's fresh database had no release_chunks partition yet: the batch store
creates it on its own path, the namer's test wrote a chunk without it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 17:08:19 +01:00
Daan MeijerandClaude Fable 5.1 8f5f83b4ea Deploying voyager 3 without its edge worker: the no-edge marker, history replayed into the catalog
Test and deploy / test (push) Failing after 51s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Skipped
deploy.sh and the nightly resume leave the edge stopped while .cache/no-edge
exists (range by range it would split releases at every edge with no merger
left). deploy/replay-window.sh takes DB and OUT and passes
REPLAY_INTO_CATALOG=1, which lets replay:corpus write a database not named
*_replay or *_scratch (never the test one).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 16:30:56 +01:00
Daan MeijerandClaude Fable 5.1 91984d3625 edge-bench --group: a range after a gap starts its batch at the range
Test and deploy / test (push) Failing after 56s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Skipped
a.b.storage's stored 2016 ranges are not contiguous (its 2016 window lies in two
article regions); the batch after a gap kept the previous range's end as its
start and the harness found no leased range for it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 13:45:08 +01:00
Daan MeijerandClaude Fable 5.1 e135076fcd The pending ladder in the prober: unnamed releases probed cheapest first, signatures learned, the named moved into the catalog
Test and deploy / test (push) Failing after 48s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Skipped
pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 13:41:30 +01:00
Daan MeijerandClaude Fable 5.1 9cf85e09d8 Nothing enters the catalog without a usable name: the pending store (decision f amended)
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Skipped
An unnamed release (no readable file name) is written to pending_releases
(migration 6300, the HDD) with its keys, counts, article span, upload tool
and probe targets, instead of releases, release_texts, posters and a chunk.
The edge sends it with an empty payload and its tool; the batch store splits
a batch by readable. The pending ladder (probes cheapest first, signature
verdicts, the move into the catalog) comes next. The replay report shows the
pending releases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 13:18:14 +01:00
Daan MeijerandClaude Fable 5.1 6860e6ccaf Albums: no numbered volumes or stray suffixes; the replay report counts the group's releases
Test and deploy / test (push) Successful in 1m10s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Skipped
acs5152s1.11 is a volume, Broforce.vol00+01.par2.temp a par2: neither an
album's member. replay:corpus reported every release in the scratch catalog
as the group's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 12:28:40 +01:00
Daan MeijerandClaude Fable 5.1 4d0610ca06 Albums in the Rust edge, and the batch-counter rule (design 3)
Test and deploy / test (push) Successful in 1m4s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Skipped
A release the window closed on with one file named by its stem or post, the
name with a number and no archive, volume, sidecar or random name, waits in
the album of its poster and template (the name's digit runs as #, the
extension apart) posted within the window of the album's first file; the
album closes a window after that span and is written as one release
(source album, no file total) named by the members' shared numbers, a lone
member as the release it was. The batch store writes an album's member
names as release_texts kind 6 (member). A counter total of 100 or more under
which one poster started 20 or more releases is a batch counter: those
releases get no file total.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 12:22:30 +01:00
Daan MeijerandClaude Fable 5.1 219fe4fd4c REBUILD.md: voyager 2's final numbers for group sets and albums (LOG 112)
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 08:12:03 +01:00
Daan MeijerandClaude Fable 5.1 f437929181 Homelab: the replay container's app environment; the nightly resume honours the v2 reset
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Skipped
deploy/replay-window.sh passes the compose app environment (HOST, PORT, APP_URL,
SPOOL_DIR, CHUNK_TABLESPACE) the ace command validates. The resume script starts
only the coordinator (the corpus fill) while .cache/v2-reset exists: the voyager 2
catalog was reset on 2026-10-26 and its workers would fill it again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 08:09:45 +01:00
Daan MeijerandClaude Fable 5.1 f300a2fd81 Dockerfile: the edge-build stage before the app stage, which copies edge-bench from it
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m28s
Test and deploy / deploy (push) Skipped
The app image failed to build ("cannot copy from stage edge-build, it needs to be defined before current stage app").

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 08:00:14 +01:00
Daan MeijerandClaude Fable 5.1 ae735361a0 Replay harness: a --to bound and the homelab window driver
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Skipped
edge-bench --group and replay:corpus take --to <article> next to --from, so a
replay covers one window of a group's history while the corpus holds others
(the 2016 ranges next to 2021's and 2025's). deploy/replay-window.sh runs a
window on homelab: every group whose window ranges are all stored is replayed
once into usenet_replay through the voyager-3 app image, 4 groups at a time,
looping while the corpus fill completes the rest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 07:56:39 +01:00
Daan MeijerandClaude Fable 5.1 72e973d006 The replay harness: a group's corpus through one carried window into a scratch catalog
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m43s
Test and deploy / deploy (push) Skipped
replay:corpus runs edge-bench --group over a group's complete corpus ranges
in article order (one window, carried from range to range, finished after
the last range; edge/src/lease.rs holds the lease JSON the worker and the
bench share) and stores every batch through BatchStore as the worker
would, then reports releases finished, provisional, complete and the bytes
per header of the NVMe and HDD tables. Refuses a database not named
*_replay or *_scratch. The app image carries edge-bench; compose gets a
replay profile for homelab.

Design amended (REBUILD.md 1, 2, 5): the window is carried by one worker a
group, no block leases and no block merger; provisional writes stay for
the lease ends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-26 00:27:09 +01:00
Daan Meijer cf51db047f Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Skipped
2026-10-25 22:43:14 +01:00
Daan MeijerandClaude Fable 5.1 e779f67c77 Prefetcher: corpus candidates a range of every group at a time
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m24s
The 512 candidates were the lowest ids at the lowest priority; a group's
ranges have consecutive ids, so once a window's small groups were stored
they were all one group's, two targets on two connections (10 ranges an
hour). Ordered by the per-group rank now: the window spans 85 groups.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 22:43:01 +01:00
Daan MeijerandClaude Fable 5.1 f8cb17903b voyager 3: the window in the Rust edge
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:28:35 +01:00
Daan MeijerandClaude Fable 5.1 2580f20a43 voyager 3: the batch store writes releases once, by id references and bytea keys
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Skipped
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.

Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:04:03 +01:00
Daan MeijerandClaude Fable 5.1 bc4e17e86f voyager 3: release_texts, the searchable strings of a release (migration 6200)
Test and deploy / test (push) Failing after 52s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Skipped
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 20:33:28 +01:00