REBUILD.md: voyager 3 design (window indexing, albums, slimmer catalog)

A 24 h posting-time window per block instead of write-early-merge-later,
albums for counterless series, poster/group-set ids and binary keys in
releases, cold tables on the HDD, the corpus test before the reset.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

REBUILD.md

@@ -5,7 +5,8 @@ rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its
 decisions D1–D42 are superseded; its domain model still holds, and its
 features not yet here are the phase 2 backlog below.
 
-Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
+Branch `rebuild`; its successor is designed under *Voyager 3* below (branch
+`voyager-3`). Phase 1 finds releases from headers and makes their NZBs.
 Everything learned from articles (names from yEnc headers, par2, archive
 listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
 never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
@@ -352,6 +353,106 @@ processes, one NNTP connection each:
   `spot-text`, and the spot), not on the catalog row; the catalog only says
   `passworded`.
 
+## Voyager 3: window indexing (design, 2026-10-25)
+
+Branch `voyager-3` (a push to it only runs the tests; `rebuild` keeps
+running on homelab until the reset). Goal: the whole history (~1,220 B
+articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe
+and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today
+(144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is
+≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources:
+LOG.md 2026-10-24 (102).
+
+What is wrong today: a release is written as soon as one range sees it,
+updated by every later range that finds more parts, merged afterwards
+(fragments, par2, names) and settled by wall-clock quiet time, which means
+little in a backfill. That churn is most of the WAL (0.6 TB a day) and the
+20 k write IOPS bursts, and leaves bloat. Half the rows are single-article
+releases, a third of them one poster's image dump (27.7 M rows, LOG.md
+(102)).
+
+1. **Blocks.** A group's history is still fetched as ranges of 2 M
+   articles in chunks of 100 k (the spool, the corpus, the prefetcher stay),
+   but it is *leased* as blocks: consecutive ranges covering at least 30
+   days of posting time (`ranges.posted_low/high`), so the window below is
+   small against a block (≤ 3% of a block's releases touch an edge). A
+   small group is one block; boneless has a block a week or so. Blocks of
+   one group run in parallel on different workers, in any order.
+
+2. **The window.** The worker reads a block in article order and keeps its
+   open files and releases in memory (counters, names, sizes; the segment
+   lists spooled to a file per block on the worker's disk). The *frontier*
+   is the median Date of the last 4,096 articles with a plausible date
+   (inside the block's span ± 7 days); Date headers alone aren't trusted
+   (clock skew, bogus years). A release is **finished** when the frontier
+   is 24 h past its last part (`WINDOW_HOURS`), or at the block's end:
+   written once, with its completeness decided then (every announced file
+   there; par2 decides later for those without a counter), never updated by
+   the header pass again. Incomplete releases are kept (user, 2026-10-24).
+   A fill posted after the window is a release of its own (the judge's
+   `repeated` rule already knows reposts). Releases still open at a block's
+   end are written flagged `edge`; the block merger (replaces the fragment
+   merger) joins the edge releases of adjacent blocks of a group by
+   identity key once both are done, and clears the flag.
+   The settler goes: `settled_at` is the finish. The **tail** keeps its
+   latency: a tail lease emits a release provisionally as today and the
+   window finalizes it once (two writes, not one per range).
+
+3. **Albums.** Files of one poster in one owner group whose subjects differ
+   only by a number sequence, or share a directory-like prefix, posted
+   within the window, become one release (`source = 'album'`): its name the
+   shared template (or the prefix), its files kept by name so each stays
+   findable; at least two files, else the single stays a release. This is
+   for subjects that name no release (today's `post`/`stem` singles); a
+   poster's ten different cracks in a day stay ten releases.
+
+4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` →
+   `posters(id)` (the table gets an id; 33 B → 4), `group_set_id` →
+   `group_sets` (the ~2,600 distinct group combinations; 85 B jsonb → 4),
+   `release_key` and `identity_key` as 20-byte `bytea` (41 B each → 20),
+   `source` a smallint. ~150 of ~430 B a row; the poster and key indexes
+   shrink with them (~12 GB of 50 today); `releases_category_newest_idx`
+   (5.3 GB, used once) goes, category browsing filters the newest index.
+   Today's 85 GB would be ~55. Cold tables move to the HDD tablespace:
+   `release_inspections` and `probe_candidates` (28 GB, written once, not
+   read by search); `probe_cache` is there already.
+
+5. **Memory and disk at the edge.** The busiest group posts ~50 M articles
+   a day; at a file per ~50 articles that is ~1 M open files of ~100 B in
+   memory per block, and their segment lists (~25 B each) in the block's
+   spool file, read back when a release is finished. Per worker thread a
+   few hundred MB at worst.
+
+6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
+   (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
+   those 73% are random 32-hex tokens, 27% `[PRiVATE]` hashes, ~0.3% posts
+   with a readable name and no counter. The filter (crosspost, counterless)
+   loses nothing voyager 3 would index; albums work from filtered chunks
+   (the image dump's subjects carry "(1/1)"). Indexing without a counter at
+   all (obfuscated uploads, ~31% of all lines) would need bodies, and the
+   raw corpus (2,328 ranges) for a sweep.
+
+7. **Where it lives.** The window, blocks and albums are in the Rust edge
+   (`edge/`), the batch store, the planner and leases; the TypeScript
+   RangeIndexer keeps the scanner (the corpus filter, tests) and drops its
+   parity with the Rust worker for the window (the `check.ts` comparison
+   covered the range path, which goes). New migrations rebuild the schema
+   from a reset, no in-place conversion of the 81 M rows.
+
+8. **Test before the reset.** Replay the corpus (the raw ranges and the
+   captures, several B articles) with the new edge into a scratch database
+   on bulk: NVMe bytes per article (the one number that decides whether
+   4 TB is enough), the share of albums and incomplete releases, and a
+   sample of releases against today's. Speed is measured after the reset,
+   on the NVMe.
+
+9. **The reset.** Tail first, then history by priority, read from the
+   corpus where stored. Survives: `probe_cache` (HDD, by message-id),
+   `spots`, `groups` and their priorities, `nntp_shares`, the corpus.
+   Recomputed: releases, chunks, inspections, candidates, par2 data,
+   verdicts, file hashes. Resets are paused (2026-10-12); this one waits
+   for the user's go-ahead, and for the new NVMe.
+
 ## Decisions
 
 a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by