Rebuild: chunks the server's compression cuts short are fetched plain
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Skipped

Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)

A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 13:51:25 +02:00
1 parent 3737ea1542
commit cfe85abc3e
7 files changed
+121 -7

No files matched your search

+8 -1
View File
@@ -67,7 +67,14 @@ class Leases {
const range = await this.leased(rangeId, nodeId)
const span = Spool.chunksOf({ start: Number(range.start_id), end: Number(range.end_id) }).find((s) => s.from === from)
if (span) await Spool.removeChunk(rangeId, span)
await db.from('ranges').where('id', rangeId).update({ status: 'planned', node_id: null })
const plain: number[] = (range.plain_chunks ?? []).map(Number)
if (plain.includes(from)) {
// Fetched plain (deflated here) and still not inflating: not the server's compression; set aside.
await db.from('ranges').where('id', rangeId).update({ status: 'failed', node_id: null, error: `chunk ${from} doesn't inflate, also fetched plain` })
return
}
// The server's compressed reply is cut short (the same bytes every time): fetched plain next time.
await db.rawQuery(`UPDATE ranges SET status = 'planned', node_id = NULL, plain_chunks = array_append(plain_chunks, ?::bigint) WHERE id = ?`, [from, rangeId])
}
/** A stopping worker gives its range back; the next one resumes it. */