Rebuild spec: a worker streams its whole range in one request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 00:01:03 +02:00
1 parent 06f5d81fbb
commit 699395c51f
1 file changed
+7 -1
+7 -1
View File
@@ -29,7 +29,13 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
the time, independent of how fast workers are. A lease goes to a range
that is already (partly) spooled; a worker reads its chunks from the
spool, waiting only for chunks still being fetched, and falls back to a
direct relay fetch for a range that isn't spooled. Spooled chunks are
direct relay fetch for a range that isn't spooled. A worker gets its
whole range in one streaming request (`GET /ranges/:id/stream`): the
spooled chunks in order, waiting for those still being fetched. It still
parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
per batch, moving the range's resume point. Chunks matter on the NNTP
side: a compressed XOVER reply can't be resumed, old ranges answer
slowly, and chunks of one range can be fetched in parallel. Spooled chunks are
deleted when their range's results are stored, and fetched again if a
range is leased again later.
Order: oldest era first (2008–2018 holds 63% of the readable files on