NNTP shares: the account's connections shared by priority
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 53s

The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-18 11:20:09 +02:00
1 parent 7eba53043a
commit 39480c486f
10 files changed
+373 -75

No files matched your search

+17 -2
View File
@@ -31,8 +31,21 @@ export default {
},
/** A leased range without a heartbeat for this long goes back to the queue. */
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
/** NNTP connections the coordinator fetches headers with. */
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
/**
* The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
* all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
* and 2 for a hedged request's or a reconnect's new connection while the old one closes).
*/
nntpShares: {
total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
everyMs: 15_000,
/** Headers before probes: history takes what it can use, the prober the rest. */
coordinatorPriority: 2,
proberPriority: 1,
},
/**
* A reply with no data for this long fails, and its connection is replaced.
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
@@ -70,11 +83,13 @@ export default {
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
probe: {
/**
* NNTP connections the prober reads articles with, besides the
* At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
*/
connections: env.get('PROBE_CONNECTIONS', 6),
/** Of them, always (its share's minimum; nntpShares). */
minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
/** Candidates claimed at a time. */
batch: 400,
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */