Rebuild: the coordinator spools replies without inflating them
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator used 1.3-1.7 cores, mostly GC of the text it inflated only to know a compressed reply was complete, plus a copy of every reply. A raw reply is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at the same rate. ".\r\n" can also end a network read inside the compressed data (~1 in 16M reads, a few dozen times over the whole target). Such a chunk doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt), the spooled chunk is removed and the range goes back to the queue; the prefetcher fetches it again. The rest of that reply breaks the connection's next command, so the connection is replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
53bfea8d15
commit
a979b5d11c
6 files changed
+60
-10
No files matched your search
@@ -233,20 +233,31 @@ export class XoverClient {
|
||||
}
|
||||
|
||||
/**
|
||||
* A zlib/gzip stream followed by a "." line: complete once it ends in
|
||||
* ".\r\n" and inflates. The pieces are joined only then: joining them on
|
||||
* every arrival was quadratic, ~15% of a prefetcher's time (LOG.md 2026-10-09).
|
||||
* A zlib/gzip stream followed by a "." line. The pieces are joined only
|
||||
* once the data ends in ".\r\n": joining them on every arrival was
|
||||
* quadratic, ~15% of a prefetcher's time (LOG.md 2026-10-09).
|
||||
*
|
||||
* Raw (the coordinator's spool) it is complete then: inflating every reply
|
||||
* to be sure cost the coordinator ~1.5 cores at ~630 k articles/s (mostly
|
||||
* GC of the inflated text). ".\r\n" can also end a network read inside
|
||||
* the compressed data (~1 in 16 M reads): that chunk doesn't inflate on
|
||||
* the worker, which reports it and the chunk is fetched again (REBUILD.md,
|
||||
* prefetch), and the rest of the reply breaks this connection's next
|
||||
* command. With the text wanted, it must inflate.
|
||||
*/
|
||||
private async compressedRaw(wantText: boolean): Promise<{ raw: Buffer; text: string }> {
|
||||
while (true) {
|
||||
if (this.length > DOT_LINE.length && this.endsWith(DOT_LINE)) {
|
||||
const all = this.buffer()
|
||||
const raw = all.subarray(0, all.length - DOT_LINE.length)
|
||||
if (!wantText) {
|
||||
this.take(all.length)
|
||||
return { raw, text: '' }
|
||||
}
|
||||
try {
|
||||
// On a thread-pool thread: a chunk's ~25 MB of text took the coordinator's event loop ~60 ms.
|
||||
const inflated = await unzipAsync(raw)
|
||||
this.take(all.length)
|
||||
return { raw: Buffer.from(raw), text: wantText ? inflated.toString('utf8') : '' }
|
||||
return { raw, text: inflated.toString('utf8') }
|
||||
} catch {
|
||||
// ".\r\n" inside the compressed data: wait for more.
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user