Rebuild: chunks the server's compression cuts short are fetched plain
Eweka sends some ranges' compressed XOVER truncated, the same bytes every time, while the plain reply is whole (a.b.models 3317215-3367214: 1,675,600 bytes that don't inflate, twice; 50,000 lines plain; tools/false_end.ts). Not a false end of reply. The worker's corrupt report removed the chunk and the range was fetched again, with the same bytes: one range was reported 540 times. (Before this morning's change the inflate check waited for more data until the 10-minute timeout, like voyager 1's 'chunk never answered' ranges.) A reported chunk's start goes into ranges.plain_chunks; the prefetcher fetches those over one connection without XFEATURE COMPRESS GZIP (NntpPool.usePlain, opened on demand) and deflates them here. A plain chunk reported again sets the range aside (status failed, error). The capture (edge:bench --capture) fetches such a chunk plain too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3737ea1542
commit
cfe85abc3e
7 files changed
+121
-7
No files matched your search
@@ -75,7 +75,41 @@ class NntpPool {
|
||||
}
|
||||
|
||||
/** Closes every connection, also those in use: their commands fail. */
|
||||
private plain: Promise<XoverClient> | null = null
|
||||
private plainQueue: Promise<unknown> = Promise.resolve()
|
||||
|
||||
/**
|
||||
* One connection without XFEATURE COMPRESS GZIP, opened when first needed,
|
||||
* one command at a time: for chunks whose compressed reply the server
|
||||
* cuts short (Leases.corrupt). Its replies are deflated here.
|
||||
*/
|
||||
async usePlain<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
|
||||
const run = this.plainQueue.then(async () => {
|
||||
this.plain ??= XoverClient.connect({
|
||||
host: env.get('NNTP_HOST')!,
|
||||
port: env.get('NNTP_PORT', 563),
|
||||
user: env.get('NNTP_USER'),
|
||||
password: env.get('NNTP_PASSWORD'),
|
||||
secure: env.get('NNTP_SECURE', true),
|
||||
compress: false,
|
||||
timeoutMs: coordinatorConfig.nntpTimeoutMs,
|
||||
})
|
||||
const client = await this.plain
|
||||
try {
|
||||
return await work(client)
|
||||
} catch (error) {
|
||||
this.plain = null
|
||||
client.close()
|
||||
throw error
|
||||
}
|
||||
})
|
||||
this.plainQueue = run.catch(() => {})
|
||||
return run
|
||||
}
|
||||
|
||||
async close() {
|
||||
void this.plain?.then((c) => c.close()).catch(() => {})
|
||||
this.plain = null
|
||||
for (const client of [...this.idle.splice(0), ...this.busy]) client.close()
|
||||
this.busy.clear()
|
||||
this.open = 0
|
||||
|
||||
Reference in new issue
Block a user