From 699395c51f30e7dac9854136695239a4e57996ac Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Fri, 9 Oct 2026 00:01:03 +0200 Subject: [PATCH] Rebuild spec: a worker streams its whole range in one request Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/REBUILD.md b/REBUILD.md index b312dc3..8211339 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -29,7 +29,13 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: the time, independent of how fast workers are. A lease goes to a range that is already (partly) spooled; a worker reads its chunks from the spool, waiting only for chunks still being fetched, and falls back to a - direct relay fetch for a range that isn't spooled. Spooled chunks are + direct relay fetch for a range that isn't spooled. A worker gets its + whole range in one streaming request (`GET /ranges/:id/stream`): the + spooled chunks in order, waiting for those still being fetched. It still + parses chunk by chunk (2 M lines are ~560 MB of text) and posts results + per batch, moving the range's resume point. Chunks matter on the NNTP + side: a compressed XOVER reply can't be resumed, old ranges answer + slowly, and chunks of one range can be fetched in parallel. Spooled chunks are deleted when their range's results are stored, and fetched again if a range is leased again later. Order: oldest era first (2008–2018 holds 63% of the readable files on