Rebuild: the coordinator spools replies without inflating them

On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.

".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 07:40:06 +02:00
1 parent 53bfea8d15
commit a979b5d11c
6 files changed
+60 -10

No files matched your search

+5
View File
@@ -53,6 +53,11 @@ class Spool {
return readFile(this.path(rangeId, span))
}
/** One chunk, e.g. one that doesn't inflate (a false end of reply; see XoverClient.compressedRaw). */
async removeChunk(rangeId: number, span: ChunkSpan) {
await rm(this.path(rangeId, span), { force: true })
}
async remove(rangeId: number) {
await rm(join(this.dir, String(rangeId)), { recursive: true, force: true })
}