Rebuild: spec draft (header-first indexer)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

REBUILD.md

@@ -0,0 +1,74 @@
+# Rebuild: header-first indexer (spec, draft for approval)
+
+Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
+Everything learned from articles (names from yEnc headers, par2, archive
+listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
+never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
+
+## Target
+
+7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
+`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
+main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
+dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
+≥ 25 B overview lines/day (~50 connections), so ~3 weeks.
+
+## Parts
+
+1. **Coordinator** (server). Plans each group's history as ranges of 2 M
+   articles, leases them to workers, tracks progress and retries. The
+   group list (owner rule) and the garbage signature table go with every
+   lease. Fetches XOVER for relay workers (gzip stream passed on as is).
+   Order: oldest era first (2008–2018 holds 63% of the readable files on
+   17% of the lines), then forward.
+
+2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if
+   CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order →
+   parse subjects → skip crossposts outside their main group (first listed
+   group in Xref among the indexed ones) → group articles into files → files
+   into releases (`lib/release` rules) → drop garbage (random name +
+   dropped signature or tool) → set aside per-article random subjects and
+   obfuscated subjects (counted only; phase 2). Returns per batch: release
+   chunks, edge fragments, counters. Holds no state beyond its lease.
+
+3. **Chunk store** (bulk, append-only). One record per release per batch:
+   its files' names, sizes and segment lists, compactly encoded (template
+   codec for message-ids, zstd). Written once, never updated; a re-run of a
+   range replaces that range's chunks (no per-file dedup). ~3 TB.
+
+4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
+   name, poster, groups, posted_at, file count, size, segments, expected
+   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
+   aggregates once per batch (one upsert per release per batch), plus a
+   small name index for search. ~115 M rows, ~50–100 GB.
+
+5. **Fragments**. Files crossing a range edge: kept by range and article
+   number; merged when the neighbouring range is done, then written as a
+   chunk. Never staged per segment.
+
+6. **Settling**. A release is settled when the ranges covering its posting
+   span are done and no chunk arrived for a day: completeness is computed
+   then (expected vs found files and segments). Phase 2 picks its work from
+   settled releases.
+
+7. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
+   search (names) and get. Web UI later.
+
+## Not in phase 1
+
+Inspection and archive walks, par2/SFV naming, spots, NZB imports, the
+after-probe garbage deletes, live indexing (the current indexer keeps
+running and does those meanwhile).
+
+## Decisions needed
+
+a. Chunk payload: a write-once `bytea` table in Postgres (partitioned by
+   month, HDD), or append-only files on disk with an offset in the catalog.
+   Proposal: Postgres first (one system, backups, transactions); files only
+   if writes still limit.
+b. Same Postgres database with separate tables (`rb_*`), or a separate
+   database. Proposal: separate database on the same server.
+c. The group list: fixed top 1000 from LIST ACTIVE now, or re-ranked later.
+d. Reposts (same release posted again): separate releases, or merged by
+   name and size in the catalog. Proposal: separate, merged later in phase 2.
+e. Headers through the relay (as now), or workers with their own accounts.