From 76ecef81afe7bd836eaa4d3d6f8a602d97227d99 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Thu, 8 Oct 2026 23:54:20 +0200 Subject: [PATCH] Rebuild spec: a, b, c decided; group list with priority-based crosspost owners Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/REBUILD.md b/REBUILD.md index 5bc0f7b..061dbf4 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -24,8 +24,9 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: 2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order → - parse subjects → skip crossposts outside their main group (first listed - group in Xref among the indexed ones) → group articles into files → files + parse subjects → skip crossposts outside their owner group (of the + indexed groups in the article's Xref, the one with the lowest priority + number; see the group list) → group articles into files → files into releases (`lib/release` rules) → drop garbage (random name + dropped signature or tool) → set aside per-article random subjects and obfuscated subjects (counted only; phase 2). Returns per batch: release @@ -56,7 +57,8 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: again instead (headers are cheap: ~53 B per overview line compressed). - *Per range, per rule and signature*: how many articles and files the rule dropped (garbage per (tool, group), per-article random subjects, - obfuscated subjects, crossposts). A few rows per range. + obfuscated subjects, and crossposts per owner group). A few rows per + range. - *Rule versions*: every range records the version of the rule set it ran with. Retracting or changing a rule makes the ranges where it matched stale; the coordinator leases them again for that rule only, and the @@ -82,21 +84,29 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: 9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab search (names) and get. Web UI later. +## Group list + +Editable at any time: an added group gets its ranges planned, a removed one +no new leases. Each group gets a priority number when it is added (the first +list by size, later additions after it); a crosspost belongs to the indexed +group in its Xref with the lowest number. So adding a group never takes +articles from groups already done, and removing one re-reads only the ranges +of other groups whose counts show crossposts skipped for it (part 5). Start: +the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of +2 M articles; boneless alone 80 k). + ## Not in phase 1 Inspection and archive walks, par2/SFV naming, spots, NZB imports, the after-probe garbage deletes, live indexing (the current indexer keeps running and does those meanwhile). -## Decisions needed +## Decisions -a. Chunk payload: a write-once `bytea` table in Postgres (partitioned by - month, HDD), or append-only files on disk with an offset in the catalog. - Proposal: Postgres first (one system, backups, transactions); files only - if writes still limit. -b. Same Postgres database with separate tables (`rb_*`), or a separate - database. Proposal: separate database on the same server. -c. The group list: fixed top 1000 from LIST ACTIVE now, or re-ranked later. +a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by + month, on the HDD; files on disk only if writes still limit. **Accepted.** +b. A separate database on the same server. **Accepted.** +c. Group list: editable, owner by priority number (above). **Accepted.** d. Reposts (same release posted again): separate releases, or merged by name and size in the catalog. Proposal: separate, merged later in phase 2. e. Headers through the relay (as now), or workers with their own accounts.