From 0a4d863f42b09ff9e7523fb87bf975d73a10f383 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Sun, 25 Oct 2026 00:54:14 +0200 Subject: [PATCH] REBUILD.md: voyager 3 design (window indexing, albums, slimmer catalog) A 24 h posting-time window per block instead of write-early-merge-later, albums for counterless series, poster/group-set ids and binary keys in releases, cold tables on the HDD, the corpus test before the reset. Co-Authored-By: Claude Fable 5.1 --- REBUILD.md | 103 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 102 insertions(+), 1 deletion(-) diff --git a/REBUILD.md b/REBUILD.md index 8c44f10..dd6f2c5 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -5,7 +5,8 @@ rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its decisions D1–D42 are superseded; its domain model still holds, and its features not yet here are the phase 2 backlog below. -Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs. +Branch `rebuild`; its successor is designed under *Voyager 3* below (branch +`voyager-3`). Phase 1 finds releases from headers and makes their NZBs. Everything learned from articles (names from yEnc headers, par2, archive listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14). @@ -352,6 +353,106 @@ processes, one NNTP connection each: `spot-text`, and the spot), not on the catalog row; the catalog only says `passworded`. +## Voyager 3: window indexing (design, 2026-10-25) + +Branch `voyager-3` (a push to it only runs the tests; `rebuild` keeps +running on homelab until the reset). Goal: the whole history (~1,220 B +articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe +and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today +(144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is +≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources: +LOG.md 2026-10-24 (102). + +What is wrong today: a release is written as soon as one range sees it, +updated by every later range that finds more parts, merged afterwards +(fragments, par2, names) and settled by wall-clock quiet time, which means +little in a backfill. That churn is most of the WAL (0.6 TB a day) and the +20 k write IOPS bursts, and leaves bloat. Half the rows are single-article +releases, a third of them one poster's image dump (27.7 M rows, LOG.md +(102)). + +1. **Blocks.** A group's history is still fetched as ranges of 2 M + articles in chunks of 100 k (the spool, the corpus, the prefetcher stay), + but it is *leased* as blocks: consecutive ranges covering at least 30 + days of posting time (`ranges.posted_low/high`), so the window below is + small against a block (≤ 3% of a block's releases touch an edge). A + small group is one block; boneless has a block a week or so. Blocks of + one group run in parallel on different workers, in any order. + +2. **The window.** The worker reads a block in article order and keeps its + open files and releases in memory (counters, names, sizes; the segment + lists spooled to a file per block on the worker's disk). The *frontier* + is the median Date of the last 4,096 articles with a plausible date + (inside the block's span ± 7 days); Date headers alone aren't trusted + (clock skew, bogus years). A release is **finished** when the frontier + is 24 h past its last part (`WINDOW_HOURS`), or at the block's end: + written once, with its completeness decided then (every announced file + there; par2 decides later for those without a counter), never updated by + the header pass again. Incomplete releases are kept (user, 2026-10-24). + A fill posted after the window is a release of its own (the judge's + `repeated` rule already knows reposts). Releases still open at a block's + end are written flagged `edge`; the block merger (replaces the fragment + merger) joins the edge releases of adjacent blocks of a group by + identity key once both are done, and clears the flag. + The settler goes: `settled_at` is the finish. The **tail** keeps its + latency: a tail lease emits a release provisionally as today and the + window finalizes it once (two writes, not one per range). + +3. **Albums.** Files of one poster in one owner group whose subjects differ + only by a number sequence, or share a directory-like prefix, posted + within the window, become one release (`source = 'album'`): its name the + shared template (or the prefix), its files kept by name so each stays + findable; at least two files, else the single stays a release. This is + for subjects that name no release (today's `post`/`stem` singles); a + poster's ten different cracks in a day stay ten releases. + +4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` → + `posters(id)` (the table gets an id; 33 B → 4), `group_set_id` → + `group_sets` (the ~2,600 distinct group combinations; 85 B jsonb → 4), + `release_key` and `identity_key` as 20-byte `bytea` (41 B each → 20), + `source` a smallint. ~150 of ~430 B a row; the poster and key indexes + shrink with them (~12 GB of 50 today); `releases_category_newest_idx` + (5.3 GB, used once) goes, category browsing filters the newest index. + Today's 85 GB would be ~55. Cold tables move to the HDD tablespace: + `release_inspections` and `probe_candidates` (28 GB, written once, not + read by search); `probe_cache` is there already. + +5. **Memory and disk at the edge.** The busiest group posts ~50 M articles + a day; at a file per ~50 articles that is ~1 M open files of ~100 B in + memory per block, and their segment lists (~25 B each) in the block's + spool file, read back when a release is finished. Per worker thread a + few hundred MB at worst. + +6. **The corpus stays as it is.** Measured on 300 unfiltered chunks + (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of + those 73% are random 32-hex tokens, 27% `[PRiVATE]` hashes, ~0.3% posts + with a readable name and no counter. The filter (crosspost, counterless) + loses nothing voyager 3 would index; albums work from filtered chunks + (the image dump's subjects carry "(1/1)"). Indexing without a counter at + all (obfuscated uploads, ~31% of all lines) would need bodies, and the + raw corpus (2,328 ranges) for a sweep. + +7. **Where it lives.** The window, blocks and albums are in the Rust edge + (`edge/`), the batch store, the planner and leases; the TypeScript + RangeIndexer keeps the scanner (the corpus filter, tests) and drops its + parity with the Rust worker for the window (the `check.ts` comparison + covered the range path, which goes). New migrations rebuild the schema + from a reset, no in-place conversion of the 81 M rows. + +8. **Test before the reset.** Replay the corpus (the raw ranges and the + captures, several B articles) with the new edge into a scratch database + on bulk: NVMe bytes per article (the one number that decides whether + 4 TB is enough), the share of albums and incomplete releases, and a + sample of releases against today's. Speed is measured after the reset, + on the NVMe. + +9. **The reset.** Tail first, then history by priority, read from the + corpus where stored. Survives: `probe_cache` (HDD, by message-id), + `spots`, `groups` and their priorities, `nntp_shares`, the corpus. + Recomputed: releases, chunks, inspections, candidates, par2 data, + verdicts, file hashes. Resets are paused (2026-10-12); this one waits + for the user's go-ahead, and for the new NVMe. + ## Decisions a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by