From 16e504a8fb6c874071e7d1800e0a0ffa32150f48 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Thu, 8 Oct 2026 19:37:26 +0200 Subject: [PATCH] Rebuild: spec draft (header-first indexer) Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 REBUILD.md diff --git a/REBUILD.md b/REBUILD.md new file mode 100644 index 0000000..15d2eb3 --- /dev/null +++ b/REBUILD.md @@ -0,0 +1,74 @@ +# Rebuild: header-first indexer (spec, draft for approval) + +Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs. +Everything learned from articles (names from yEnc headers, par2, archive +listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it +never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14). + +## Target + +7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest +`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their +main group; 140 B kept segments in ~1.25 B files and ~115 M releases after +dropping garbage; 28 TB of compressed overview traffic. Throughput goal: +≥ 25 B overview lines/day (~50 connections), so ~3 weeks. + +## Parts + +1. **Coordinator** (server). Plans each group's history as ranges of 2 M + articles, leases them to workers, tracks progress and retries. The + group list (owner rule) and the garbage signature table go with every + lease. Fetches XOVER for relay workers (gzip stream passed on as is). + Order: oldest era first (2008–2018 holds 63% of the readable files on + 17% of the lines), then forward. + +2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if + CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order → + parse subjects → skip crossposts outside their main group (first listed + group in Xref among the indexed ones) → group articles into files → files + into releases (`lib/release` rules) → drop garbage (random name + + dropped signature or tool) → set aside per-article random subjects and + obfuscated subjects (counted only; phase 2). Returns per batch: release + chunks, edge fragments, counters. Holds no state beyond its lease. + +3. **Chunk store** (bulk, append-only). One record per release per batch: + its files' names, sizes and segment lists, compactly encoded (template + codec for message-ids, zstd). Written once, never updated; a re-run of a + range replaces that range's chunks (no per-file dedup). ~3 TB. + +4. **Release catalog** (hot, Postgres on NVMe). One row per release: key, + name, poster, groups, posted_at, file count, size, segments, expected + file count (`[i/N]`), has-par2, readable-name flag, state. Updated with + aggregates once per batch (one upsert per release per batch), plus a + small name index for search. ~115 M rows, ~50–100 GB. + +5. **Fragments**. Files crossing a range edge: kept by range and article + number; merged when the neighbouring range is done, then written as a + chunk. Never staged per segment. + +6. **Settling**. A release is settled when the ranges covering its posting + span are done and no chunk arrived for a day: completeness is computed + then (expected vs found files and segments). Phase 2 picks its work from + settled releases. + +7. **NZB and API**. NZB built from a release's chunks; a minimal Newznab + search (names) and get. Web UI later. + +## Not in phase 1 + +Inspection and archive walks, par2/SFV naming, spots, NZB imports, the +after-probe garbage deletes, live indexing (the current indexer keeps +running and does those meanwhile). + +## Decisions needed + +a. Chunk payload: a write-once `bytea` table in Postgres (partitioned by + month, HDD), or append-only files on disk with an offset in the catalog. + Proposal: Postgres first (one system, backups, transactions); files only + if writes still limit. +b. Same Postgres database with separate tables (`rb_*`), or a separate + database. Proposal: separate database on the same server. +c. The group list: fixed top 1000 from LIST ACTIVE now, or re-ranked later. +d. Reposts (same release posted again): separate releases, or merged by + name and size in the catalog. Proposal: separate, merged later in phase 2. +e. Headers through the relay (as now), or workers with their own accounts.