diff --git a/REBUILD.md b/REBUILD.md index 15d2eb3..46a933a 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -29,7 +29,16 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: into releases (`lib/release` rules) → drop garbage (random name + dropped signature or tool) → set aside per-article random subjects and obfuscated subjects (counted only; phase 2). Returns per batch: release - chunks, edge fragments, counters. Holds no state beyond its lease. + chunks, edge fragments, probe candidates, counters. Holds no state beyond + its lease. + + **Probe candidates**: per release, the articles phase 2 should read, as + (release key, kind, message-id, file in its chunk): the first article of + the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2 + index (smallest `.par2`), and of a lone random-named file; for a 7z set + also its last volume (the listing is in the tail). At most a few per + release (~115 M releases → ~150–250 M articles), not one per file. Per- + article and obfuscated subjects get none (they need every article's body). 3. **Chunk store** (bulk, append-only). One record per release per batch: its files' names, sizes and segment lists, compactly encoded (template @@ -42,16 +51,21 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: aggregates once per batch (one upsert per release per batch), plus a small name index for search. ~115 M rows, ~50–100 GB. -5. **Fragments**. Files crossing a range edge: kept by range and article +5. **Probe queue**. The candidates, in the catalog's database: one small row + each (release, kind, message-id, chunk reference), priority by the + release's state (settled first) and kind (random names before readable + ones). Phase 2 takes them from here; phase 1 only fills it. + +6. **Fragments**. Files crossing a range edge: kept by range and article number; merged when the neighbouring range is done, then written as a chunk. Never staged per segment. -6. **Settling**. A release is settled when the ranges covering its posting +7. **Settling**. A release is settled when the ranges covering its posting span are done and no chunk arrived for a day: completeness is computed then (expected vs found files and segments). Phase 2 picks its work from settled releases. -7. **NZB and API**. NZB built from a release's chunks; a minimal Newznab +8. **NZB and API**. NZB built from a release's chunks; a minimal Newznab search (names) and get. Web UI later. ## Not in phase 1