Rebuild: chunks the server's compression cuts short are fetched plain
Eweka sends some ranges' compressed XOVER truncated, the same bytes every time, while the plain reply is whole (a.b.models 3317215-3367214: 1,675,600 bytes that don't inflate, twice; 50,000 lines plain; tools/false_end.ts). Not a false end of reply. The worker's corrupt report removed the chunk and the range was fetched again, with the same bytes: one range was reported 540 times. (Before this morning's change the inflate check waited for more data until the 10-minute timeout, like voyager 1's 'chunk never answered' ranges.) A reported chunk's start goes into ranges.plain_chunks; the prefetcher fetches those over one connection without XFEATURE COMPRESS GZIP (NntpPool.usePlain, opened on demand) and deflates them here. A plain chunk reported again sets the range aside (status failed, error). The capture (edge:bench --capture) fetches such a chunk plain too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3737ea1542
commit
cfe85abc3e
7 files changed
+121
-7
No files matched your search
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* Whether a compressed XOVER reply that doesn't inflate is the server's: the
|
||||
* range fetched raw twice (compared with a saved copy), and once uncompressed.
|
||||
*
|
||||
* node --env-file=.env --import=@poppinss/ts-exec tools/false_end.ts <group> <from> <to> [saved.gz]
|
||||
*/
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { unzipSync } from 'node:zlib'
|
||||
import { XoverClient } from '#lib/nntp/index'
|
||||
|
||||
const [group, from, to, saved] = process.argv.slice(2)
|
||||
const env = process.env
|
||||
const connect = (compress: boolean) =>
|
||||
XoverClient.connect({ host: env.NNTP_HOST!, port: Number(env.NNTP_PORT ?? 563), user: env.NNTP_USER, password: env.NNTP_PASSWORD, secure: (env.NNTP_SECURE ?? 'true') !== 'false', compress, timeoutMs: 120_000 })
|
||||
const gz = await connect(true)
|
||||
await gz.group(group)
|
||||
for (let i = 0; i < 2; i++) {
|
||||
const raw = await gz.xoverRaw(Number(from), Number(to))
|
||||
let inflates = 'inflates'
|
||||
try {
|
||||
unzipSync(raw)
|
||||
} catch (e: any) {
|
||||
inflates = `doesn't inflate (${e.message})`
|
||||
}
|
||||
const same = saved ? (Buffer.compare(raw, readFileSync(saved)) === 0 ? ', same bytes as saved' : ', differs from saved') : ''
|
||||
console.log(`compressed #${i + 1}: ${raw.length} bytes, ${inflates}${same}`)
|
||||
}
|
||||
gz.close()
|
||||
const plain = await connect(false)
|
||||
await plain.group(group)
|
||||
const lines = await plain.xover(Number(from), Number(to))
|
||||
console.log(`plain: ${lines.length} overview lines`)
|
||||
plain.close()
|
||||
Reference in new issue
Block a user