Rebuild spec: a worker streams its whole range in one request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
06f5d81fbb
commit
699395c51f
1 file changed
+7
-1
+7
-1
@@ -29,7 +29,13 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||||
the time, independent of how fast workers are. A lease goes to a range
|
||||
that is already (partly) spooled; a worker reads its chunks from the
|
||||
spool, waiting only for chunks still being fetched, and falls back to a
|
||||
direct relay fetch for a range that isn't spooled. Spooled chunks are
|
||||
direct relay fetch for a range that isn't spooled. A worker gets its
|
||||
whole range in one streaming request (`GET /ranges/:id/stream`): the
|
||||
spooled chunks in order, waiting for those still being fetched. It still
|
||||
parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
|
||||
per batch, moving the range's resume point. Chunks matter on the NNTP
|
||||
side: a compressed XOVER reply can't be resumed, old ranges answer
|
||||
slowly, and chunks of one range can be fetched in parallel. Spooled chunks are
|
||||
deleted when their range's results are stored, and fetched again if a
|
||||
range is leased again later.
|
||||
Order: oldest era first (2008–2018 holds 63% of the readable files on
|
||||
|
||||
Reference in new issue
Block a user