Rebuild: chunks the server's compression cuts short are fetched plain
Eweka sends some ranges' compressed XOVER truncated, the same bytes every time, while the plain reply is whole (a.b.models 3317215-3367214: 1,675,600 bytes that don't inflate, twice; 50,000 lines plain; tools/false_end.ts). Not a false end of reply. The worker's corrupt report removed the chunk and the range was fetched again, with the same bytes: one range was reported 540 times. (Before this morning's change the inflate check waited for more data until the 10-minute timeout, like voyager 1's 'chunk never answered' ranges.) A reported chunk's start goes into ranges.plain_chunks; the prefetcher fetches those over one connection without XFEATURE COMPRESS GZIP (NntpPool.usePlain, opened on demand) and deflates them here. A plain chunk reported again sets the range aside (status failed, error). The capture (edge:bench --capture) fetches such a chunk plain too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3737ea1542
commit
cfe85abc3e
7 files changed
+121
-7
No files matched your search
@@ -98,10 +98,19 @@ test.group('Node API', (group) => {
|
||||
await Promise.all(Spool.chunksOf({ start: 1, end: 300 }).map((span) => Spool.has(rangeId, span))),
|
||||
[true, false, true]
|
||||
)
|
||||
const range = await db.from('ranges').where('id', rangeId).first()
|
||||
let range = await db.from('ranges').where('id', rangeId).first()
|
||||
assert.equal(range.status, 'planned')
|
||||
assert.isNull(range.node_id)
|
||||
// Fetched plain next time (the server's compressed reply was cut short).
|
||||
assert.deepEqual(range.plain_chunks.map(Number), [101])
|
||||
// No longer this node's: a second report is refused.
|
||||
;(await client.post(`/api/v1/node/ranges/${rangeId}/corrupt`).bearerToken(token).json({ from: 101 })).assertStatus(409)
|
||||
// Leased again and the plain chunk still doesn't inflate: the range is set aside, not leased in a loop.
|
||||
await Spool.write(rangeId, { from: 101, to: 200 }, chunk(111))
|
||||
;(await client.post('/api/v1/node/lease').bearerToken(token)).assertStatus(200)
|
||||
;(await client.post(`/api/v1/node/ranges/${rangeId}/corrupt`).bearerToken(token).json({ from: 101 })).assertStatus(200)
|
||||
range = await db.from('ranges').where('id', rangeId).first()
|
||||
assert.equal(range.status, 'failed')
|
||||
assert.include(range.error, 'also fetched plain')
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user