usenet-indexer/REBUILD.md

# Rebuild: header-first indexer (spec, draft for approval)

Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
Everything learned from articles (names from yEnc headers, par2, archive
listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

## Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

## Parts

1. **Coordinator** (server). Plans each group's history as ranges of 2 M
   articles, leases them to workers, tracks progress and retries. The
   group list (owner rule) and the garbage signature table go with every
   lease. Fetches XOVER for relay workers (gzip stream passed on as is).
   Order: oldest era first (2008–2018 holds 63% of the readable files on
   17% of the lines), then forward.

2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if
   CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order →
   parse subjects → skip crossposts outside their owner group (of the
   indexed groups in the article's Xref, the one with the lowest priority
   number; see the group list) → group articles into files → files
   into releases (`lib/release` rules) → drop garbage (random name +
   dropped signature or tool) → set aside per-article random subjects and
   obfuscated subjects (counted only; phase 2). Returns per batch: release
   chunks, edge fragments, probe candidates, counters. Holds no state beyond
   its lease.

   **Probe candidates**: per release, the articles phase 2 should read, as
   (release key, kind, message-id, file in its chunk): the first article of
   the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
   index (smallest `.par2`), and of a lone random-named file; for a 7z set
   also its last volume (the listing is in the tail). At most a few per
   release (~115 M releases → ~150–250 M articles), not one per file. Per-
   article and obfuscated subjects get none (they need every article's body).

3. **Chunk store** (bulk, append-only). One record per release per batch:
   its files' names, sizes and segment lists, compactly encoded (template
   codec for message-ids, zstd). Written once, never updated; a re-run of a
   range replaces that range's chunks (no per-file dedup). ~3 TB.

4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
   name, poster, groups, posted_at, file count, size, segments, expected
   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
   aggregates once per batch (one upsert per release per batch), plus a
   small name index for search. ~115 M rows, ~50–100 GB.

5. **Rules and recovery**. What a worker skips isn't stored (message-ids
   are the bulk: garbage alone would be ~2.5 TB); a skipped range is read
   again instead (headers are cheap: ~53 B per overview line compressed).
   - *Per range, per rule and signature*: how many articles and files the
     rule dropped (garbage per (tool, group), per-article random subjects,
     obfuscated subjects, and crossposts per owner group). A few rows per
     range.
   - *Rule versions*: every range records the version of the rule set it ran
     with. Retracting or changing a rule makes the ranges where it matched
     stale; the coordinator leases them again for that rule only, and the
     worker returns just what the rule had dropped.
   - *Sample*: ~1% of what each rule drops (by a hash of the release key) is
     kept with full file details and probe candidates, so a rule can be
     checked, and its losses measured in phase 2, before it is retracted.

6. **Probe queue**. The candidates, in the catalog's database: one small row
   each (release, kind, message-id, chunk reference), priority by the
   release's state (settled first) and kind (random names before readable
   ones). Phase 2 takes them from here; phase 1 only fills it.

7. **Fragments**. Files crossing a range edge: kept by range and article
   number; merged when the neighbouring range is done, then written as a
   chunk. Never staged per segment.

8. **Settling**. A release is settled when the ranges covering its posting
   span are done and no chunk arrived for a day: completeness is computed
   then (expected vs found files and segments). Phase 2 picks its work from
   settled releases.

9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
   search (names) and get. Web UI later.

## Group list

Editable at any time: an added group gets its ranges planned, a removed one
no new leases. Each group gets a priority number when it is added (the first
list by size, later additions after it); a crosspost belongs to the indexed
group in its Xref with the lowest number. So adding a group never takes
articles from groups already done, and removing one re-reads only the ranges
of other groups whose counts show crossposts skipped for it (part 5). Start:
the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
2 M articles; boneless alone 80 k).

## Not in phase 1

Inspection and archive walks, par2/SFV naming, spots, NZB imports, the
after-probe garbage deletes, live indexing (the current indexer keeps
running and does those meanwhile).

## Decisions

a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
   month, on the HDD; files on disk only if writes still limit. **Accepted.**
b. A separate database on the same server. **Accepted.**
c. Group list: editable, owner by priority number (above). **Accepted.**
d. Reposts (same release posted again): separate releases, or merged by
   name and size in the catalog. Proposal: separate, merged later in phase 2.
e. Headers through the relay (as now), or workers with their own accounts.