Rebuild spec: d and e decided; coordinator prefetches ranges into a spool

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

REBUILD.md

@@ -0,0 +1,125 @@
+# Rebuild: header-first indexer (spec, draft for approval)
+
+Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
+Everything learned from articles (names from yEnc headers, par2, archive
+listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
+never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
+
+## Target
+
+7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
+`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
+main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
+dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
+≥ 25 B overview lines/day (~50 connections), so ~3 weeks.
+
+## Parts
+
+1. **Coordinator** (server). Plans each group's history as ranges of 2 M
+   articles, leases them to workers, tracks progress and retries. The
+   group list (owner rule) and the garbage signature table go with every
+   lease. Fetches XOVER for the workers (relay: workers have no NNTP
+   account; the gzip stream is passed on as is).
+
+   **Prefetch**: the coordinator fetches the next ranges ahead of the
+   workers, in chunks of 50 k lines and in order, into a spool on disk
+   (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
+   sequentially). It keeps about two ranges ahead per active lease, capped
+   by size (default 10 GB), and fetches with all its header connections all
+   the time, independent of how fast workers are. A lease goes to a range
+   that is already (partly) spooled; a worker reads its chunks from the
+   spool, waiting only for chunks still being fetched, and falls back to a
+   direct relay fetch for a range that isn't spooled. Spooled chunks are
+   deleted when their range's results are stored, and fetched again if a
+   range is leased again later.
+   Order: oldest era first (2008–2018 holds 63% of the readable files on
+   17% of the lines), then forward.
+
+2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if
+   CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order →
+   parse subjects → skip crossposts outside their owner group (of the
+   indexed groups in the article's Xref, the one with the lowest priority
+   number; see the group list) → group articles into files → files
+   into releases (`lib/release` rules) → drop garbage (random name +
+   dropped signature or tool) → set aside per-article random subjects and
+   obfuscated subjects (counted only; phase 2). Returns per batch: release
+   chunks, edge fragments, probe candidates, counters. Holds no state beyond
+   its lease.
+
+   **Probe candidates**: per release, the articles phase 2 should read, as
+   (release key, kind, message-id, file in its chunk): the first article of
+   the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
+   index (smallest `.par2`), and of a lone random-named file; for a 7z set
+   also its last volume (the listing is in the tail). At most a few per
+   release (~115 M releases → ~150–250 M articles), not one per file. Per-
+   article and obfuscated subjects get none (they need every article's body).
+
+3. **Chunk store** (bulk, append-only). One record per release per batch:
+   its files' names, sizes and segment lists, compactly encoded (template
+   codec for message-ids, zstd). Written once, never updated; a re-run of a
+   range replaces that range's chunks (no per-file dedup). ~3 TB.
+
+4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
+   name, poster, groups, posted_at, file count, size, segments, expected
+   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
+   aggregates once per batch (one upsert per release per batch), plus a
+   small name index for search. ~115 M rows, ~50–100 GB.
+
+5. **Rules and recovery**. What a worker skips isn't stored (message-ids
+   are the bulk: garbage alone would be ~2.5 TB); a skipped range is read
+   again instead (headers are cheap: ~53 B per overview line compressed).
+   - *Per range, per rule and signature*: how many articles and files the
+     rule dropped (garbage per (tool, group), per-article random subjects,
+     obfuscated subjects, and crossposts per owner group). A few rows per
+     range.
+   - *Rule versions*: every range records the version of the rule set it ran
+     with. Retracting or changing a rule makes the ranges where it matched
+     stale; the coordinator leases them again for that rule only, and the
+     worker returns just what the rule had dropped.
+   - *Sample*: ~1% of what each rule drops (by a hash of the release key) is
+     kept with full file details and probe candidates, so a rule can be
+     checked, and its losses measured in phase 2, before it is retracted.
+
+6. **Probe queue**. The candidates, in the catalog's database: one small row
+   each (release, kind, message-id, chunk reference), priority by the
+   release's state (settled first) and kind (random names before readable
+   ones). Phase 2 takes them from here; phase 1 only fills it.
+
+7. **Fragments**. Files crossing a range edge: kept by range and article
+   number; merged when the neighbouring range is done, then written as a
+   chunk. Never staged per segment.
+
+8. **Settling**. A release is settled when the ranges covering its posting
+   span are done and no chunk arrived for a day: completeness is computed
+   then (expected vs found files and segments). Phase 2 picks its work from
+   settled releases.
+
+9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
+   search (names) and get. Web UI later.
+
+## Group list
+
+Editable at any time: an added group gets its ranges planned, a removed one
+no new leases. Each group gets a priority number when it is added (the first
+list by size, later additions after it); a crosspost belongs to the indexed
+group in its Xref with the lowest number. So adding a group never takes
+articles from groups already done, and removing one re-reads only the ranges
+of other groups whose counts show crossposts skipped for it (part 5). Start:
+the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
+2 M articles; boneless alone 80 k).
+
+## Not in phase 1
+
+Inspection and archive walks, par2/SFV naming, spots, NZB imports, the
+after-probe garbage deletes, live indexing (the current indexer keeps
+running and does those meanwhile).
+
+## Decisions
+
+a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
+   month, on the HDD; files on disk only if writes still limit. **Accepted.**
+b. A separate database on the same server. **Accepted.**
+c. Group list: editable, owner by priority number (above). **Accepted.**
+d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
+e. Headers through the coordinator's relay, with prefetch (part 1).
+   **Accepted.**