Rebuild: the coordinator spools replies without inflating them

On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.

".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 07:40:06 +02:00
1 parent 53bfea8d15
commit a979b5d11c
6 files changed
+60 -10

No files matched your search

+16 -5
View File
@@ -233,20 +233,31 @@ export class XoverClient {
}
/**
* A zlib/gzip stream followed by a "." line: complete once it ends in
* ".\r\n" and inflates. The pieces are joined only then: joining them on
* every arrival was quadratic, ~15% of a prefetcher's time (LOG.md 2026-10-09).
* A zlib/gzip stream followed by a "." line. The pieces are joined only
* once the data ends in ".\r\n": joining them on every arrival was
* quadratic, ~15% of a prefetcher's time (LOG.md 2026-10-09).
*
* Raw (the coordinator's spool) it is complete then: inflating every reply
* to be sure cost the coordinator ~1.5 cores at ~630 k articles/s (mostly
* GC of the inflated text). ".\r\n" can also end a network read inside
* the compressed data (~1 in 16 M reads): that chunk doesn't inflate on
* the worker, which reports it and the chunk is fetched again (REBUILD.md,
* prefetch), and the rest of the reply breaks this connection's next
* command. With the text wanted, it must inflate.
*/
private async compressedRaw(wantText: boolean): Promise<{ raw: Buffer; text: string }> {
while (true) {
if (this.length > DOT_LINE.length && this.endsWith(DOT_LINE)) {
const all = this.buffer()
const raw = all.subarray(0, all.length - DOT_LINE.length)
if (!wantText) {
this.take(all.length)
return { raw, text: '' }
}
try {
// On a thread-pool thread: a chunk's ~25 MB of text took the coordinator's event loop ~60 ms.
const inflated = await unzipAsync(raw)
this.take(all.length)
return { raw: Buffer.from(raw), text: wantText ? inflated.toString('utf8') : '' }
return { raw, text: inflated.toString('utf8') }
} catch {
// ".\r\n" inside the compressed data: wait for more.
}