Rebuild spec: a worker streams its whole range in one request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
06f5d81fbb
commit
699395c51f
1 file changed
+7
-1
+7
-1
@@ -29,7 +29,13 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
|||||||
the time, independent of how fast workers are. A lease goes to a range
|
the time, independent of how fast workers are. A lease goes to a range
|
||||||
that is already (partly) spooled; a worker reads its chunks from the
|
that is already (partly) spooled; a worker reads its chunks from the
|
||||||
spool, waiting only for chunks still being fetched, and falls back to a
|
spool, waiting only for chunks still being fetched, and falls back to a
|
||||||
direct relay fetch for a range that isn't spooled. Spooled chunks are
|
direct relay fetch for a range that isn't spooled. A worker gets its
|
||||||
|
whole range in one streaming request (`GET /ranges/:id/stream`): the
|
||||||
|
spooled chunks in order, waiting for those still being fetched. It still
|
||||||
|
parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
|
||||||
|
per batch, moving the range's resume point. Chunks matter on the NNTP
|
||||||
|
side: a compressed XOVER reply can't be resumed, old ranges answer
|
||||||
|
slowly, and chunks of one range can be fetched in parallel. Spooled chunks are
|
||||||
deleted when their range's results are stored, and fetched again if a
|
deleted when their range's results are stored, and fetched again if a
|
||||||
range is leased again later.
|
range is leased again later.
|
||||||
Order: oldest era first (2008–2018 holds 63% of the readable files on
|
Order: oldest era first (2008–2018 holds 63% of the readable files on
|
||||||
|
|||||||
Reference in new issue
Block a user