usenet-indexer/REBUILD.md

Rebuild: header-first indexer (spec, draft for approval)

Branch rebuild. Phase 1 finds releases from headers and makes their NZBs. Everything learned from articles (names from yEnc headers, par2, archive listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest alt.binaries.* groups. Estimated: 528 B overview lines; 435 B in their main group; 140 B kept segments in ~1.25 B files and ~115 M releases after dropping garbage; 28 TB of compressed overview traffic. Throughput goal: ≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

Parts

1. Coordinator (server). Plans each group's history as ranges of 2 M articles, leases them to workers, tracks progress and retries. The group list (owner rule) and the garbage signature table go with every lease. Fetches XOVER for relay workers (gzip stream passed on as is). Order: oldest era first (2008–2018 holds 63% of the readable files on 17% of the lines), then forward.

2. Edge worker (node, TypeScript reusing lib/; Rust later only if CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order → parse subjects → skip crossposts outside their main group (first listed group in Xref among the indexed ones) → group articles into files → files into releases (lib/release rules) → drop garbage (random name + dropped signature or tool) → set aside per-article random subjects and obfuscated subjects (counted only; phase 2). Returns per batch: release chunks, edge fragments, probe candidates, counters. Holds no state beyond its lease.

   Probe candidates: per release, the articles phase 2 should read, as (release key, kind, message-id, file in its chunk): the first article of the first archive volume (.part01.rar, .rar, .7z.001), of the par2 index (smallest .par2), and of a lone random-named file; for a 7z set also its last volume (the listing is in the tail). At most a few per release (~115 M releases → ~150–250 M articles), not one per file. Per- article and obfuscated subjects get none (they need every article's body).

3. Chunk store (bulk, append-only). One record per release per batch: its files' names, sizes and segment lists, compactly encoded (template codec for message-ids, zstd). Written once, never updated; a re-run of a range replaces that range's chunks (no per-file dedup). ~3 TB.

4. Release catalog (hot, Postgres on NVMe). One row per release: key, name, poster, groups, posted_at, file count, size, segments, expected file count ([i/N]), has-par2, readable-name flag, state. Updated with aggregates once per batch (one upsert per release per batch), plus a small name index for search. ~115 M rows, ~50–100 GB.

5. Probe queue. The candidates, in the catalog's database: one small row each (release, kind, message-id, chunk reference), priority by the release's state (settled first) and kind (random names before readable ones). Phase 2 takes them from here; phase 1 only fills it.

6. Fragments. Files crossing a range edge: kept by range and article number; merged when the neighbouring range is done, then written as a chunk. Never staged per segment.

7. Settling. A release is settled when the ranges covering its posting span are done and no chunk arrived for a day: completeness is computed then (expected vs found files and segments). Phase 2 picks its work from settled releases.

8. NZB and API. NZB built from a release's chunks; a minimal Newznab search (names) and get. Web UI later.

Not in phase 1

Inspection and archive walks, par2/SFV naming, spots, NZB imports, the after-probe garbage deletes, live indexing (the current indexer keeps running and does those meanwhile).

Decisions needed

a. Chunk payload: a write-once bytea table in Postgres (partitioned by month, HDD), or append-only files on disk with an offset in the catalog. Proposal: Postgres first (one system, backups, transactions); files only if writes still limit. b. Same Postgres database with separate tables (rb_*), or a separate database. Proposal: separate database on the same server. c. The group list: fixed top 1000 from LIST ACTIVE now, or re-ranked later. d. Reposts (same release posted again): separate releases, or merged by name and size in the catalog. Proposal: separate, merged later in phase 2. e. Headers through the relay (as now), or workers with their own accounts.