usenet-indexer/REBUILD.md

Rebuild: header-first indexer (spec, draft for approval)

Branch rebuild. Phase 1 finds releases from headers and makes their NZBs. Everything learned from articles (names from yEnc headers, par2, archive listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

Status (2026-10-09)

Built on this branch (a fresh AdonisJS 7 app; phase 1 code imported from the current app by pipeline stage): coordinator (planner, NNTP pool, prefetch spool, leases, rule sets, fragment merger, settling), the node API (lease, framed stream, batches, finish/release/heartbeat), the edge worker (edge:node, --processes), the catalog and chunk store, NZBs and a name search. Measured on the workstation (4 cores, 100 Mbit, Eweka):

• Edge worker, one core: ~210–215 k headers/s on recent a.b.boneless, ~330–340 k on a 2013 a.b.ath range; ~0.7–0.8 GB peak per process. (Start: 73–103 k/s, 1.0–1.2 GB.) What's left is spread out (the per-file key ~12%, GC ~10%, zlib ~10%, the scan ~6%).
• NNTP: Eweka takes ~1.5–2 s per XOVER command; ~22–30 k lines/s per connection in chunks of 100 k when the line allows; connections don't slow each other.
• Bandwidth: compressed XOVER is ~69 B per article on recent boneless (100 k-article chunks of ~7 MB). The workstation's 100 Mbit line carries ~140–150 k articles/s, which is where its end-to-end runs now stop (10 connections 90–99% busy, ~89 Mbit/s).
• End to end at that rate (one 4-core machine): workers about half busy each; server side ~17% (coordinator) + ~8% (web) + ~3% (Postgres) of a core. (Before this round's fixes Postgres was at 32%: the merger's neighbour query read every range per file.)
• Storage: chunks 17.9–20.3 B per segment (format v2); a release ~1.1 KB.
• For the whole target (528 B overview lines): ~36 TB of compressed XOVER to download. At 50 connections (~1–1.25 M lines/s, ~0.6–0.7 Gbit/s) ~5–6 days, ~5–6 worker cores, two coordinator shards (one process handles ~60 MB/s), under a core for web and Postgres. The line is the limit below ~0.7 Gbit/s: at 1 Gbit, ~3.5 days at best.

Not built yet: re-running ranges for a retracted rule (the counts and rule versions are recorded), deployment (Dockerfile, compose), phase 2.

Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest alt.binaries.* groups. Estimated: 528 B overview lines; 435 B in their main group; 140 B kept segments in ~1.25 B files and ~115 M releases after dropping garbage; 28 TB of compressed overview traffic. Throughput goal: ≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

Parts

1. Coordinator (server). Plans each group's history as ranges of 2 M articles, leases them to workers, tracks progress and retries. The group list (owner rule) and the garbage signature table go with every lease. Fetches XOVER for the workers (relay: workers have no NNTP account; the gzip stream is passed on as is).

   Prefetch: the coordinator fetches the next ranges ahead of the workers, in chunks of 100 k lines and in order, into a spool on disk (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read sequentially). It keeps about two ranges ahead per active lease, capped by size (default 10 GB), and fetches with all its header connections all the time, independent of how fast workers are. A lease goes to a range that is already (partly) spooled, else to the next planned one (leased ranges are fetched first); a worker reads its chunks from the spool, waiting only for chunks still being fetched. A worker gets its whole range in one streaming request (GET /ranges/:id/stream): the spooled chunks in order, waiting for those still being fetched. It still parses chunk by chunk (2 M lines are ~560 MB of text) and posts results per batch, moving the range's resume point. Chunks matter on the NNTP side: a compressed XOVER reply can't be resumed, old ranges answer slowly, and chunks of one range can be fetched in parallel. Spooled chunks are deleted when their range's results are stored, and fetched again if a range is leased again later. Order: oldest era first (2008–2018 holds 63% of the readable files on 17% of the lines), then forward. One coordinator process handles ~60 MB/s of compressed XOVER (its event loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over processes with coordinator:run --shard=i/n (ranges with id % n = i − 1, each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and settles).

2. Edge worker (node, TypeScript reusing lib/). Measured (edge:bench, 2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013, 103 k/s on a recent a.b.boneless range (2 M headers each), ~1.2 GB peak per process; the 25 B lines/day goal needs ~3–4 cores over all nodes, so no Rust. Per lease: compressed chunks of 10–50 k lines, in order → parse subjects → skip crossposts outside their owner group (of the indexed groups in the article's Xref, the one with the lowest priority number; see the group list) → group articles into files → files into releases (lib/release rules) → drop garbage (random name + dropped signature or tool) → set aside per-article random subjects and obfuscated subjects (counted only; phase 2). Returns per batch: release chunks, edge fragments, probe candidates, counters. Holds no state beyond its lease.

   Probe candidates: per release, the articles phase 2 should read, as (release key, kind, message-id, file in its chunk): the first article of the first archive volume (.part01.rar, .rar, .7z.001), of the par2 index (smallest .par2), and of a lone random-named file; for a 7z set also its last volume (the listing is in the tail). At most a few per release (~115 M releases → ~150–250 M articles), not one per file. Per- article and obfuscated subjects get none (they need every article's body).

3. Chunk store (bulk, append-only). One record per release per batch: its files' names, sizes and segment lists, compactly encoded (template codec for message-ids, zstd). Written once, never updated. A batch is stored in one transaction and only when it starts at its range's resume point, so a batch sent twice or late is refused and the catalog's sums stay right (no per-file dedup); a recovery run for a retracted rule only adds what that rule had dropped. ~3 TB.

4. Release catalog (hot, Postgres on NVMe). One row per release: key, name, poster, groups, posted_at, file count, size, segments, expected file count ([i/N]), has-par2, readable-name flag, state. Updated with aggregates once per batch (one upsert per release per batch), plus a small name index for search. ~115 M rows, ~50–100 GB.

5. Rules and recovery. What a worker skips isn't stored (message-ids are the bulk: garbage alone would be ~2.5 TB); a skipped range is read again instead (headers are cheap: ~53 B per overview line compressed).

   • Per range, per rule and signature: how many articles and files the rule dropped (garbage per (tool, group), per-article random subjects, obfuscated subjects, and crossposts per owner group). A few rows per range.
   • Rule versions: every range records the version of the rule set it ran with. Retracting or changing a rule makes the ranges where it matched stale; the coordinator leases them again for that rule only, and the worker returns just what the rule had dropped.
   • Sample: ~1% of what each rule drops (by a hash of the release key) is kept with full file details and probe candidates, so a rule can be checked, and its losses measured in phase 2, before it is retracted.

6. Probe queue. The candidates, in the catalog's database: one small row each (release, kind, message-id, chunk reference), priority by the release's state (settled first) and kind (random names before readable ones). Phase 2 takes them from here; phase 1 only fills it.

7. Fragments. Files crossing a range edge: kept by range and article number; merged when the neighbouring range is done, then written as a chunk. Never staged per segment.

8. Settling. A release is settled when the ranges covering its posting span are done and no chunk arrived for a day: completeness is computed then (expected vs found files and segments). Phase 2 picks its work from settled releases.

9. NZB and API. NZB built from a release's chunks; a minimal Newznab search (names) and get. Web UI later.

Group list

Editable at any time: an added group gets its ranges planned, a removed one no new leases. Each group gets a priority number when it is added (the first list by size, later additions after it); a crosspost belongs to the indexed group in its Xref with the lowest number. So adding a group never takes articles from groups already done, and removing one re-reads only the ranges of other groups whose counts show crossposts skipped for it (part 5). Start: the 1000 largest alt.binaries.* groups by LIST ACTIVE (~266 k ranges of 2 M articles; boneless alone 80 k).

Not in phase 1

Inspection and archive walks, par2/SFV naming, spots, NZB imports, the after-probe garbage deletes, live indexing (the current indexer keeps running and does those meanwhile).

Decisions

a. Chunk payload: a write-once bytea table in Postgres, partitioned by month, on the HDD; files on disk only if writes still limit. Accepted. b. A separate database on the same server. Accepted. c. Group list: editable, owner by priority number (above). Accepted. d. Reposts: separate releases in phase 1, merged in phase 2. Accepted. e. Headers through the coordinator's relay, with prefetch (part 1). Accepted.