89 lines
4.6 KiB
Markdown
89 lines
4.6 KiB
Markdown
# Rebuild: header-first indexer (spec, draft for approval)
|
||
|
||
Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
|
||
Everything learned from articles (names from yEnc headers, par2, archive
|
||
listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
|
||
never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
|
||
|
||
## Target
|
||
|
||
7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
|
||
`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
|
||
main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
|
||
dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||
≥ 25 B overview lines/day (~50 connections), so ~3 weeks.
|
||
|
||
## Parts
|
||
|
||
1. **Coordinator** (server). Plans each group's history as ranges of 2 M
|
||
articles, leases them to workers, tracks progress and retries. The
|
||
group list (owner rule) and the garbage signature table go with every
|
||
lease. Fetches XOVER for relay workers (gzip stream passed on as is).
|
||
Order: oldest era first (2008–2018 holds 63% of the readable files on
|
||
17% of the lines), then forward.
|
||
|
||
2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if
|
||
CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order →
|
||
parse subjects → skip crossposts outside their main group (first listed
|
||
group in Xref among the indexed ones) → group articles into files → files
|
||
into releases (`lib/release` rules) → drop garbage (random name +
|
||
dropped signature or tool) → set aside per-article random subjects and
|
||
obfuscated subjects (counted only; phase 2). Returns per batch: release
|
||
chunks, edge fragments, probe candidates, counters. Holds no state beyond
|
||
its lease.
|
||
|
||
**Probe candidates**: per release, the articles phase 2 should read, as
|
||
(release key, kind, message-id, file in its chunk): the first article of
|
||
the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
|
||
index (smallest `.par2`), and of a lone random-named file; for a 7z set
|
||
also its last volume (the listing is in the tail). At most a few per
|
||
release (~115 M releases → ~150–250 M articles), not one per file. Per-
|
||
article and obfuscated subjects get none (they need every article's body).
|
||
|
||
3. **Chunk store** (bulk, append-only). One record per release per batch:
|
||
its files' names, sizes and segment lists, compactly encoded (template
|
||
codec for message-ids, zstd). Written once, never updated; a re-run of a
|
||
range replaces that range's chunks (no per-file dedup). ~3 TB.
|
||
|
||
4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
|
||
name, poster, groups, posted_at, file count, size, segments, expected
|
||
file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
|
||
aggregates once per batch (one upsert per release per batch), plus a
|
||
small name index for search. ~115 M rows, ~50–100 GB.
|
||
|
||
5. **Probe queue**. The candidates, in the catalog's database: one small row
|
||
each (release, kind, message-id, chunk reference), priority by the
|
||
release's state (settled first) and kind (random names before readable
|
||
ones). Phase 2 takes them from here; phase 1 only fills it.
|
||
|
||
6. **Fragments**. Files crossing a range edge: kept by range and article
|
||
number; merged when the neighbouring range is done, then written as a
|
||
chunk. Never staged per segment.
|
||
|
||
7. **Settling**. A release is settled when the ranges covering its posting
|
||
span are done and no chunk arrived for a day: completeness is computed
|
||
then (expected vs found files and segments). Phase 2 picks its work from
|
||
settled releases.
|
||
|
||
8. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
|
||
search (names) and get. Web UI later.
|
||
|
||
## Not in phase 1
|
||
|
||
Inspection and archive walks, par2/SFV naming, spots, NZB imports, the
|
||
after-probe garbage deletes, live indexing (the current indexer keeps
|
||
running and does those meanwhile).
|
||
|
||
## Decisions needed
|
||
|
||
a. Chunk payload: a write-once `bytea` table in Postgres (partitioned by
|
||
month, HDD), or append-only files on disk with an offset in the catalog.
|
||
Proposal: Postgres first (one system, backups, transactions); files only
|
||
if writes still limit.
|
||
b. Same Postgres database with separate tables (`rb_*`), or a separate
|
||
database. Proposal: separate database on the same server.
|
||
c. The group list: fixed top 1000 from LIST ACTIVE now, or re-ranked later.
|
||
d. Reposts (same release posted again): separate releases, or merged by
|
||
name and size in the catalog. Proposal: separate, merged later in phase 2.
|
||
e. Headers through the relay (as now), or workers with their own accounts.
|