Rebuild: chunks the server's compression cuts short are fetched plain
Eweka sends some ranges' compressed XOVER truncated, the same bytes every time, while the plain reply is whole (a.b.models 3317215-3367214: 1,675,600 bytes that don't inflate, twice; 50,000 lines plain; tools/false_end.ts). Not a false end of reply. The worker's corrupt report removed the chunk and the range was fetched again, with the same bytes: one range was reported 540 times. (Before this morning's change the inflate check waited for more data until the 10-minute timeout, like voyager 1's 'chunk never answered' ranges.) A reported chunk's start goes into ranges.plain_chunks; the prefetcher fetches those over one connection without XFEATURE COMPRESS GZIP (NntpPool.usePlain, opened on demand) and deflates them here. A plain chunk reported again sets the range aside (status failed, error). The capture (edge:bench --capture) fetches such a chunk plain too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3737ea1542
commit
cfe85abc3e
7 files changed
+121
-7
No files matched your search
+11
-1
@@ -78,7 +78,17 @@ export default class EdgeBench extends BaseCommand {
|
||||
for (let from = this.start; from <= end; from += this.chunk, index++) {
|
||||
const to = Math.min(from + this.chunk - 1, end)
|
||||
const started = Date.now()
|
||||
const raw = await client.xoverRaw(from, to)
|
||||
let raw = await client.xoverRaw(from, to)
|
||||
// Eweka cuts some ranges' compressed reply short (the same bytes every time): fetched plain then.
|
||||
try {
|
||||
unzipSync(raw)
|
||||
} catch {
|
||||
const plain = await XoverClient.connect({ host: env.get('NNTP_HOST')!, port: env.get('NNTP_PORT')!, user: env.get('NNTP_USER'), password: env.get('NNTP_PASSWORD'), secure: env.get('NNTP_SECURE', true), compress: false, timeoutMs: 300_000 })
|
||||
await plain.group(this.group)
|
||||
raw = await plain.xoverRaw(from, to)
|
||||
plain.close()
|
||||
this.logger.warning(`chunk ${index}: the compressed reply didn't inflate; fetched plain`)
|
||||
}
|
||||
await writeFile(join(this.dir, `${String(index).padStart(4, '0')}-${from}-${to}.gz`), raw)
|
||||
this.logger.info(`chunk ${index}: ${from}–${to}, ${(raw.length / 1e6).toFixed(1)} MB in ${((Date.now() - started) / 1000).toFixed(1)} s`)
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user