From 06f5d81fbb8c1f17e27b22e39e6da355637c0579 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Thu, 8 Oct 2026 23:57:17 +0200 Subject: [PATCH] Rebuild spec: d and e decided; coordinator prefetches ranges into a spool Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/REBUILD.md b/REBUILD.md index 061dbf4..b312dc3 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -18,7 +18,20 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: 1. **Coordinator** (server). Plans each group's history as ranges of 2 M articles, leases them to workers, tracks progress and retries. The group list (owner rule) and the garbage signature table go with every - lease. Fetches XOVER for relay workers (gzip stream passed on as is). + lease. Fetches XOVER for the workers (relay: workers have no NNTP + account; the gzip stream is passed on as is). + + **Prefetch**: the coordinator fetches the next ranges ahead of the + workers, in chunks of 50 k lines and in order, into a spool on disk + (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read + sequentially). It keeps about two ranges ahead per active lease, capped + by size (default 10 GB), and fetches with all its header connections all + the time, independent of how fast workers are. A lease goes to a range + that is already (partly) spooled; a worker reads its chunks from the + spool, waiting only for chunks still being fetched, and falls back to a + direct relay fetch for a range that isn't spooled. Spooled chunks are + deleted when their range's results are stored, and fetched again if a + range is leased again later. Order: oldest era first (2008–2018 holds 63% of the readable files on 17% of the lines), then forward. @@ -107,6 +120,6 @@ a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by month, on the HDD; files on disk only if writes still limit. **Accepted.** b. A separate database on the same server. **Accepted.** c. Group list: editable, owner by priority number (above). **Accepted.** -d. Reposts (same release posted again): separate releases, or merged by - name and size in the catalog. Proposal: separate, merged later in phase 2. -e. Headers through the relay (as now), or workers with their own accounts. +d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.** +e. Headers through the coordinator's relay, with prefetch (part 1). + **Accepted.**