usenet-indexer/REBUILD.md

# Voyager 2: header-first indexer (concept)

The concept for what runs (since 2026-10-11; it started as the spec of a
rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its
decisions D1–D42 are superseded; its domain model still holds, and its
features not yet here are the phase 2 backlog below.

Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
Everything learned from articles (names from yEnc headers, par2, archive
listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

## Status (2026-10-09)

Built on this branch (a fresh AdonisJS 7 app; phase 1 code imported from
the current app by pipeline stage): coordinator (planner, NNTP pool,
prefetch spool, leases, rule sets, fragment merger, settling), the node API
(lease, framed stream, batches, finish/release/heartbeat), the edge worker
(`edge:node`, `--processes`), the catalog and chunk store, NZBs and a name
search. Measured on the workstation (4 cores, 100 Mbit, Eweka):

- Edge worker, one core: ~210–215 k headers/s on recent a.b.boneless,
  ~330–340 k on a 2013 a.b.ath range; ~0.7–0.8 GB peak per process.
  (Start: 73–103 k/s, 1.0–1.2 GB.) What's left is spread out (the
  per-file key ~12%, GC ~10%, zlib ~10%, the scan ~6%).
- NNTP: Eweka takes ~1.5–2 s per XOVER command; ~22–30 k lines/s per
  connection in chunks of 100 k when the line allows; connections don't
  slow each other.
- Bandwidth: compressed XOVER is ~69 B per article on recent boneless
  (100 k-article chunks of ~7 MB). The workstation's 100 Mbit line carries
  ~140–150 k articles/s, which is where its end-to-end runs now stop (10
  connections 90–99% busy, ~89 Mbit/s).
- End to end at that rate (one 4-core machine): workers about half busy
  each; server side ~17% (coordinator) + ~8% (web) + ~3% (Postgres) of a
  core. (Before this round's fixes Postgres was at 32%: the merger's
  neighbour query read every range per file.)
- On homelab (12 cores, 1 Gbit; 2026-10-09, separate containers and
  database, `/mnt/bulk/usenet-rebuild`): 20 connections fetched ~630 k
  articles/s (~31 k per connection, ~3 s per 100 k chunk, ~250 Mbit/s at
  ~50 B per article on older boneless) and 8 worker processes indexed all of
  it, waiting for the spool (~366 M headers in ~12 minutes). Server side at
  that rate: coordinator 0.4–0.5 cores (1.3–1.7 before it stopped
  inflating replies to check them), web 0.3–0.45, Postgres (on the HDD)
  under 0.15. Worker per core there: ~195 k headers/s (boneless 2026),
  ~275 k (ath 2013).
- Storage: chunks 17.9–20.3 B per segment (format v2); a release ~1.1 KB.
- For the whole target (528 B overview lines): ~36 TB of compressed XOVER
  to download. At 50 connections (~1–1.25 M lines/s, ~0.6–0.7 Gbit/s)
  ~5–6 days, ~5–6 worker cores, two coordinator shards (one process
  handles ~60 MB/s), under a core for web and Postgres. The line is the
  limit below ~0.7 Gbit/s: at 1 Gbit, ~3.5 days at best.

Where workers run: total speed is set by the account's NNTP connections
(~30 k articles/s each), not by worker CPU, so remote workers don't make
it faster; they take CPU off homelab (at 1.25 M/s ~7 worker cores) at the
cost of relaying the compressed XOVER (≤ ~0.7 Gbit/s up, homelab has 1
Gbit/s). Plan: coordinator, web, Postgres and a few workers on homelab,
most workers on batcave, a few on poweredge-kantoor with
`edge:node --max-mbit` (e.g. 50 of its 160 Mbit/s ≈ 100–125 k articles/s;
the cap is shared by the node's processes).

Also built (2026-10-09): the live tail (new articles planned every 2
minutes from one LIST ACTIVE per name prefix, at priority −1: leased and
fetched before history); the catalog API, Newznab (`/api`) and the web
interface, from voyager 1 (the current build), with categories from the
name; voyager 1's 29 learned garbage signatures (`rules:signatures`,
`database/data/voyager1_signatures.tsv`).

Deployment (2026-10-10): `Dockerfile` (targets app and edge),
`deploy/compose.yaml` (project usenet-rebuild on homelab), `deploy/deploy.sh`;
a push to `rebuild` that passes CI is deployed. See `deploy/README.md`.

Not built yet: re-running ranges for a retracted rule (the counts and rule
versions are recorded), decision f, phase
2 (inspection of the probe queue: yEnc names, encryption, archives, par2;
passwords; spots).

## Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

## Parts

1. **Coordinator** (server). Plans each group's history as ranges of 2 M
   articles, leases them to workers, tracks progress and retries. The
   group list (owner rule) and the garbage signature table go with every
   lease. Fetches XOVER for the workers (relay: workers have no NNTP
   account; the gzip stream is passed on as is).

   **Prefetch**: the coordinator fetches the next ranges ahead of the
   workers, in chunks of 100 k lines and in order, into a spool on disk
   (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
   sequentially). It keeps about two ranges ahead per active lease, capped
   by size (default 10 GB), and fetches with all its header connections all
   the time, independent of how fast workers are. A lease goes to a range
   that is already (partly) spooled, else to the next planned one (leased
   ranges are fetched first); a worker reads its chunks from the spool,
   waiting only for chunks still being fetched. A worker gets its
   whole range in one streaming request (`GET /ranges/:id/stream`): the
   spooled chunks in order, waiting for those still being fetched. It still
   parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
   per batch, moving the range's resume point. Chunks matter on the NNTP
   side: a compressed XOVER reply can't be resumed, old ranges answer
   slowly, and chunks of one range can be fetched in parallel. Spooled chunks are
   deleted when their range's results are stored, and fetched again if a
   range is leased again later.
   Order: oldest era first (2008–2018 holds 63% of the readable files on
   17% of the lines), then forward.
   One coordinator process handles ~60 MB/s of compressed XOVER (its event
   loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over
   processes with `coordinator:run --shard=i/n` (ranges with id % n = i − 1,
   each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and
   settles).

2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
   2026-10-09, after two rounds of work): ~200–215 k headers/s per core on
   recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak
   per process. **Rust** (`edge/`, the same RangeIndexer path
   over the same captures; `check.ts` finds every output identical, the
   decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1
   CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory
   (0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the
   regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line
   scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores
   instead of ~6–7; the speed is still set by the NNTP connections. The
   cost: a second implementation of lib/headers, lib/release, lib/rules and
   the chunk format, kept identical (check.ts) for as long as both exist.
   Findings, how to run both and how to compare their outputs:
   `edge/README.md`. Since 2026-10-09 the Rust worker
   (`edge-node`, same crate) is the one running on homelab: ~8× less CPU
   per header and 20× less memory than `edge:node` in production.
   Per lease: compressed chunks of 10–50 k lines, in order →
   parse subjects → skip crossposts outside their owner group (of the
   indexed groups in the article's Xref, the one with the lowest priority
   number; see the group list) → group articles into files → files
   into releases (`lib/release` rules) → drop garbage (random name +
   dropped signature or tool) → set aside per-article random subjects and
   obfuscated subjects (counted only; phase 2). Returns per batch: release
   chunks, edge fragments, probe candidates, counters. Holds no state beyond
   its lease.

   **Probe candidates**: per release, the articles phase 2 should read, as
   (release key, kind, message-id, file in its chunk): the first article of
   the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
   index (smallest `.par2`), and of a lone random-named file; for a 7z set
   also its last volume (the listing is in the tail). At most a few per
   release (~115 M releases → ~150–250 M articles), not one per file. Per-
   article and obfuscated subjects get none (they need every article's body).

3. **Chunk store** (bulk, append-only). One record per release per batch:
   its files' names, sizes and segment lists, compactly encoded (template
   codec for message-ids, zstd). Written once, never updated. A batch is
   stored in one transaction and only when it starts at its range's resume
   point, so a batch sent twice or late is refused and the catalog's sums
   stay right (no per-file dedup); a recovery run for a retracted rule only
   adds what that rule had dropped. ~3 TB.

4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
   name, poster, groups, posted_at, file count, size, segments, expected
   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
   aggregates once per batch (one upsert per release per batch), plus a
   small name index for search. ~115 M rows, ~50–100 GB.

5. **Rules and recovery**. What a worker skips isn't stored (message-ids
   are the bulk: garbage alone would be ~2.5 TB); a skipped range is read
   again instead (headers are cheap: ~53 B per overview line compressed).
   - *Per range, per rule and signature*: how many articles and files the
     rule dropped (garbage per (tool, group), per-article random subjects,
     obfuscated subjects, and crossposts per owner group). A few rows per
     range.
   - *Rule versions*: every range records the version of the rule set it ran
     with. Retracting or changing a rule makes the ranges where it matched
     stale; the coordinator leases them again for that rule only, and the
     worker returns just what the rule had dropped.
   - *Sample*: ~1% of what each rule drops (by a hash of the release key) is
     kept with full file details and probe candidates, so a rule can be
     checked, and its losses measured in phase 2, before it is retracted.

6. **Probe queue**. The candidates, in the catalog's database: one small row
   each (release, kind, message-id, chunk reference), priority by the
   release's state (settled first) and kind (random names before readable
   ones). Phase 2 takes them from here; phase 1 only fills it.

7. **Fragments**. Files crossing a range edge: kept by range and article
   number; merged when the neighbouring range is done, then written as a
   chunk. Never staged per segment.

8. **Settling**. A release is settled when the ranges covering its posting
   span are done and no chunk arrived for a day: completeness is computed
   then (expected vs found files and segments). Phase 2 picks its work from
   settled releases.

9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
   search (names) and get. Web UI later.

## Group list

Editable at any time: an added group gets its ranges planned, a removed one
no new leases. Each group gets a priority number when it is added (the first
list by size, later additions after it); a crosspost belongs to the indexed
group in its Xref with the lowest number. So adding a group never takes
articles from groups already done, and removing one re-reads only the ranges
of other groups whose counts show crossposts skipped for it (part 5). Start:
the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
2 M articles; boneless alone 80 k).

## Corpus

The grouping sweeps' dataset (2026-10-12; LOG.md 2026-10-12 (12)): the XOVER
chunks of ranges marked `capture`, kept on the HDD in
`corpus/<group>/<start>-<end>/` (edge-bench's layout, `range.json` once
complete), up to 300 GB. The prefetcher stores them as it fetches and reads
them from there afterwards, so a reset re-indexes them without the news
server. `deploy/corpus_sweep.py` marks more ranges each sweep (`capture`,
spread over groups and eras, the top-100 groups first), replays every stored
range with the deployed build's edge-bench (`replay`, ~seconds a range) and
reports suspects and the change since the previous build (`report`). A rule
change is judged on the whole corpus before a reset.

## Phase 2

Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It
reads the probe queue's first articles (`lone`, `par2`, `archive`) and
stores per release whether it's passworded and its inner name, which search,
the web interface and Newznab use.

Built: the par2 merge (2026-10-12; `releases:merge`, the `merger` service,
LOG.md 2026-10-12 (13)). The prober stores per probed file start the MD5 of
its first 16 KiB (`file_hashes`) and per par2 body its set's file list
(`par2_files`, `par2_sets`), which holds the same hash per file whatever the
file was renamed to. Every minute the merger merges the releases of one
group, posted within 48 hours of a release holding the set's par2, whose
files the set describes, into the one with the most files (chunks,
candidates, inspections, counts; the par2's shared name when none was
readable); `release_merges` keeps what went where. Obfuscated uploads whose
files were each a release of their own (`lone` candidates) become one.
It reads what's new an hour (`created_at`) at a time, its place kept after
each, and merges only the sets held or matched by two releases or more; a
file whose first 16 KiB and size more than 20 files share is a common one and
matches nothing (LOG.md 2026-10-20 (85)).
A set's file list is dropped once it can't merge anything more: complete (all
its files in its holder), judged (every holder junk), or final (every holder
inside a run of done ranges, those ranges' candidates probed, its hashes seen
by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).

**Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
homelab: `voyager1-tables.dump` with 2.76 M spots, upload signatures,
signature probes, dropped releases and groups; `voyager1-release-passwords.csv`
with 68,032 releases with a password):

| Feature | Voyager 1 | In voyager 2 |
|---|---|---|
| Spotnet spots | D29 | the spotter and the spot inspector (below); new spots only, the export's not imported |
| Posted NZBs read into releases (also NZBs inside archives) | D26, D28 | not yet |
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
| Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
| Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |

Live indexing (the tail) runs since 2026-10-09.

**Junk** (voyager 1's second filter, D42). After the header pass's rules
(garbage signatures, executables, per-article and obfuscated subjects), the
judge (`releases:judge`, service `judge`) judges releases by what the
prober, the merges and the spots found: every 2 minutes the releases whose
facts changed (noted by triggers in `judge_dirty`), daily every release (for
what time changes: a day after settling, ranges read beside it). In this
order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
*locked* (passworded, no password from a spot, its NZB or the title),
*unusable* (random names, no inner name, no par2 or inner-name merge, no
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
set's version) hide releases from search (`judged` shows them); a verdict
whose rule no longer matches is taken back. Nothing is dropped yet.

**Sidecars** (the merger's third pass). An upload's .nfo, .sfv or .nzb
that came apart from it (another name, another counter: "[00/80] … .nzb"
beside "[01/94] … .part01.rar") was a release of its own. The settler
queues small settled releases (≤ 3 files, < 1 MB) in `fold_candidates`;
one whose files are all sidecars joins the poster's bigger release in the
same group posted within 10 minutes: the one sharing a word of 4+ letters
with its subject, else the one under the same counter total, else none
(`release_merges.via` 'sidecar').

**Spots** (after voyager 1's D29; `lib/spotnet` ported as is). Two
processes, one NNTP connection each:

- **The spotter** (`spots:read`, service `spotter`) reads free.pt's
  overviews after the last article it saw (`spot_state`; the first time the
  last `SPOT_BACKFILL`, 20,000, ~3 months), every `SPOT_EVERY_SECONDS`
  (120). For each spot it reads the full header: the signature is checked
  there as Spotweb does (Spotnet's keys 2–4; self-signed key 7 by hashcash),
  and the X-XML gives the description, the website and the NZB's articles.
  A password the title (`{{pw}}`) or the description names
  (`passwordFromSpot`) is kept with the spot, with its source. A
  delete/dispose message marks its target `removed` (a personal one only
  from the same spotter within 5 days) and takes its links and passwords
  back from the releases.
- **The spot inspector** (`spots:inspect`, service `spot-inspector`) looks
  inside each listed spot, newest first: it reads the compressed NZB (≤
  `SPOT_MAX_NZB_SEGMENTS` articles in alt.binaries.ftd), lists its files
  in `spot_files` (named as the header pass names them, `parseSubject`),
  and matches each file to the release the header pass built for it: the
  same release identity (`releaseIdentity` over the file's subject and
  poster) in one of its groups, the release posted nearest the file within
  48 h. Matched releases are linked in `release_spots`. Placeholder NZBs
  (`dummy`), gone articles (`nzb-gone`) and unreadable ones are noted on the
  spot. Unmatched spots are matched again every hour; after
  `SPOT_STANDALONE_AFTER_HOURS` (24) a spot becomes a release of its own
  (source `spot`), whose NZB is the spot's (kept gzipped in `spots.nzb`).
- **Passwords are kept apart** in `release_passwords` (release, password,
  source: `nzb` for the NZB's `<meta type="password">`, else `spot-title` or
  `spot-text`, and the spot), not on the catalog row; the catalog only says
  `passworded`.

## Decisions

a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
   month, on the HDD; files on disk only if writes still limit. **Accepted.**
b. A separate database on the same server. **Accepted.**
c. Group list: editable, owner by priority number (above). **Accepted.**
d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
e. Headers through the coordinator's relay, with prefetch (part 1).
   **Accepted.**
f. Random-named releases (no file with a readable name) keep no chunk in
   the header pass: their file keys and probe candidates only, until
   phase 2 names them (bodies) and fetches their segments again. The full
   history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
   kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
g. The account's NNTP connections shared by priority (`nntp_shares`): the
   coordinator (headers) before the prober, each between its minimum and
   maximum, 5 kept for the spotter, the spot inspector, the plain
   connection and reconnects. A process grows only into connections the
   others closed, so the account never goes over its limit. Replaces the
   fixed split (38 + 8), under which the prober stayed at 8 while history
   was paused and the coordinator's connections idle (LOG.md 2026-10-18
   (75)). **Accepted** 2026-10-18.
h. Idle NNTP connections closed: one with nothing to do for
   `NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
   (the coordinator's pool on its next sync, the plain one too; a dry
   prober lane; the spotter between rounds; the spot inspector with no
   spots). Eweka dropped the quiet ones and the next command failed
   ("Spot inspection failed: NNTP connection closed"), and idle ones took
   the account's connections from downloads. **Accepted** 2026-10-22.