diff --git a/REBUILD.md b/REBUILD.md index 46a933a..5bc0f7b 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -51,21 +51,35 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: aggregates once per batch (one upsert per release per batch), plus a small name index for search. ~115 M rows, ~50–100 GB. -5. **Probe queue**. The candidates, in the catalog's database: one small row +5. **Rules and recovery**. What a worker skips isn't stored (message-ids + are the bulk: garbage alone would be ~2.5 TB); a skipped range is read + again instead (headers are cheap: ~53 B per overview line compressed). + - *Per range, per rule and signature*: how many articles and files the + rule dropped (garbage per (tool, group), per-article random subjects, + obfuscated subjects, crossposts). A few rows per range. + - *Rule versions*: every range records the version of the rule set it ran + with. Retracting or changing a rule makes the ranges where it matched + stale; the coordinator leases them again for that rule only, and the + worker returns just what the rule had dropped. + - *Sample*: ~1% of what each rule drops (by a hash of the release key) is + kept with full file details and probe candidates, so a rule can be + checked, and its losses measured in phase 2, before it is retracted. + +6. **Probe queue**. The candidates, in the catalog's database: one small row each (release, kind, message-id, chunk reference), priority by the release's state (settled first) and kind (random names before readable ones). Phase 2 takes them from here; phase 1 only fills it. -6. **Fragments**. Files crossing a range edge: kept by range and article +7. **Fragments**. Files crossing a range edge: kept by range and article number; merged when the neighbouring range is done, then written as a chunk. Never staged per segment. -7. **Settling**. A release is settled when the ranges covering its posting +8. **Settling**. A release is settled when the ranges covering its posting span are done and no chunk arrived for a day: completeness is computed then (expected vs found files and segments). Phase 2 picks its work from settled releases. -8. **NZB and API**. NZB built from a release's chunks; a minimal Newznab +9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab search (names) and get. Web UI later. ## Not in phase 1