Daan MeijerandClaude Opus 5.5
40aa78cda3
A worker waited on one chunk of a leased range for an hour (range 146477: five 15-minute stream timeouts, then the range took 63 s; 88 "stream: Peer disconnected" overnight). The silence timeout doesn't catch a reply that trickles: a chunk still coming in after NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed, and fetched again. The report names the oldest request in flight (range, chunk, age, bytes in) and counts deadlines. Batches failed with "Maximum call stack size exceeded" (8 overnight): Math.min(...segments) on a fragment with more segments than the stack takes arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
85 lines
4.1 KiB
TypeScript
85 lines
4.1 KiB
TypeScript
import env from '#start/env'
|
||
import app from '@adonisjs/core/services/app'
|
||
|
||
/** The coordinator (REBUILD.md, part 1). */
|
||
export default {
|
||
/** Articles per range. */
|
||
rangeArticles: env.get('RANGE_ARTICLES', 2_000_000),
|
||
/**
|
||
* Articles per XOVER chunk (spool files, stream frames). Eweka takes
|
||
* ~1.5–2 s per command whatever its size: one connection read 9–15k
|
||
* lines/s in chunks of 25k, ~20k in 50k, ~30k in 100k and 200k (LOG.md
|
||
* 2026-10-09).
|
||
*/
|
||
chunkArticles: env.get('CHUNK_ARTICLES', 100_000),
|
||
/**
|
||
* The live tail (REBUILD.md, coordinator): new articles become ranges
|
||
* every `everySeconds` (one LIST ACTIVE per name prefix), leased and
|
||
* fetched before any history. A group gets a tail range once it has
|
||
* `minArticles` new (a chunk) or its oldest unplanned article waited
|
||
* `maxDelayMinutes`, so a quiet group doesn't make a range every round.
|
||
*/
|
||
tail: {
|
||
everySeconds: env.get('TAIL_EVERY_SECONDS', 120),
|
||
minArticles: env.get('TAIL_MIN_ARTICLES', 100_000),
|
||
maxDelayMinutes: env.get('TAIL_MAX_DELAY_MINUTES', 15),
|
||
/** Before the history's 0 (oldest) … 1000 (newest). */
|
||
priority: -1,
|
||
},
|
||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||
/** NNTP connections the coordinator fetches headers with. */
|
||
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
|
||
/**
|
||
* A reply with no data for this long fails, and its connection is replaced.
|
||
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
|
||
* a fresh one answers in seconds; LOG.md 2026-10-09 (14)): at 10 minutes
|
||
* the prefetcher nearly stopped. Data arrives within seconds otherwise.
|
||
*/
|
||
nntpTimeoutMs: env.get('NNTP_TIMEOUT_MS', 2 * 60 * 1000),
|
||
/**
|
||
* A chunk still coming in after this long is given up (its connection
|
||
* closed) and fetched again: a reply that trickles never goes quiet long
|
||
* enough for nntpTimeoutMs, and kept a worker waiting on one chunk for an
|
||
* hour (LOG.md 2026-10-10 (2)). Chunks take 3–35 s.
|
||
*/
|
||
nntpChunkDeadlineMs: env.get('NNTP_CHUNK_DEADLINE_SECONDS', 5 * 60) * 1000,
|
||
/**
|
||
* A chunk not fetched after this long is asked again on another
|
||
* connection; the first answer wins and the other connection is closed.
|
||
* Eweka stalls single requests: the same chunk took 27–52 s on one
|
||
* connection and 1–9 s on another at the same moment; median ~3 s
|
||
* (LOG.md 2026-10-09 (15)). Measured in production (40 + 6 connections vs
|
||
* 40, 15 min each): 280k vs 273k articles/s, within the noise; the second
|
||
* request won half the time. Off by default.
|
||
*/
|
||
nntpHedgeAfterMs: env.get('NNTP_HEDGE_AFTER_SECONDS', 10) * 1000,
|
||
/** Of nntpConnections, kept for those second requests (the prefetcher's loops are the rest). 0: no hedging. */
|
||
nntpHedgeConnections: env.get('NNTP_HEDGE_CONNECTIONS', 0),
|
||
/**
|
||
* A connection is closed after this long (when it's free), and a fresh one
|
||
* opened. 0: never. Requests were as often slow (>10 s) on connections
|
||
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
|
||
*/
|
||
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
|
||
spool: {
|
||
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
|
||
/** Ranges kept fetched ahead, per active lease. */
|
||
aheadPerLease: env.get('SPOOL_AHEAD_PER_LEASE', 4),
|
||
/** At least this many ranges ahead, so the first leases start at once. */
|
||
minAhead: env.get('SPOOL_MIN_AHEAD', 2),
|
||
maxBytes: env.get('SPOOL_MAX_GB', 10) * 1e9,
|
||
/**
|
||
* At most this many connections on one range, and on one group, at a
|
||
* time (0: no limit): spreads the requests over ranges and groups instead
|
||
* of all on the next chunks of the first ranges. Homelab, 40 connections,
|
||
* 12 min each, alternated: unlimited (on ~10 ranges) 281k and 167k
|
||
* articles/s, 2 per range and 4 per group (on ~22) 319k and 296k, with
|
||
* shorter chunk times (11.5 vs 13–24 s; LOG.md 2026-10-09 (18)). Needs
|
||
* targets for it: aheadPerLease 4.
|
||
*/
|
||
perRange: env.get('PREFETCH_PER_RANGE', 2),
|
||
perGroup: env.get('PREFETCH_PER_GROUP', 4),
|
||
},
|
||
}
|