Rebuild: give up stalled requests, not slow ones
Test and deploy / test (push) Successful in 24s
Test and deploy / deploy (push) Successful in 56s

a.b.0day.stuffz sends steadily but slowly (~400 kB a minute, 5-7
minutes per 100k chunk): the fixed deadline (90 s, doubling up to 360 s)
gave its chunks up again and again, and two of its ranges' worker
streams timed out. A stall is a request that got less than
NNTP_STALL_KB_PER_MINUTE (100) in the last minute (the stuck replies
got ~25 kB a minute); then its connection is closed and the chunk
fetched again. Replaces NNTP_CHUNK_DEADLINE_SECONDS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-10 16:51:55 +02:00
1 parent dbccec932c
commit 887df12ead
3 files changed
+29 -24

No files matched your search

+8 -8
View File
@@ -38,15 +38,15 @@ export default {
*/
nntpTimeoutMs: env.get('NNTP_TIMEOUT_MS', 2 * 60 * 1000),
/**
* A chunk still coming in after this long is given up (its connection
* closed) and fetched again: a reply that trickles never goes quiet long
* enough for nntpTimeoutMs, and kept a worker waiting on one chunk for an
* hour (LOG.md 2026-10-10 (2)). Chunks take 3–35 s; at 300 s, 258 were
* given up in 3 hours (~1.4 a minute, each holding a connection 5 minutes);
* at 180 s 60 in 15 minutes, ~30% of 40 connections' time, while finished
* chunks took 24 s on average.
* A request that got less than this in the last minute (kB) is given up,
* its connection closed, and the chunk fetched again: a reply that trickles
* (6,204 → 6,229 kB in a minute) never goes quiet long enough for
* nntpTimeoutMs, and kept a worker waiting on one chunk for an hour. Not a
* deadline: some groups send steadily but slowly (a.b.0day.stuffz ~400 kB a
* minute, 5–7 minutes a chunk), and a fixed 90–360 s gave those up again
* and again (LOG.md 2026-10-10 (3)).
*/
nntpChunkDeadlineMs: env.get('NNTP_CHUNK_DEADLINE_SECONDS', 90) * 1000,
nntpStallBytesPerMinute: env.get('NNTP_STALL_KB_PER_MINUTE', 100) * 1000,
/**
* A chunk not fetched after this long is asked again on another
* connection; the first answer wins and the other connection is closed.