NNTP shares: the account's connections shared by priority
The coordinator (38) and the prober (8) had a fixed split; with history paused the coordinator's stood idle while the prober's were its limit. nntp_shares: each process syncs every 15 s (priority, minimum, what it could use, what it has open) and gets what it may open: minimums first, then by priority what each wants, growing only into connections the others closed or gave up. The pool's size and the prober's lanes follow the grant; the coordinator doubles its want while work waits for a connection, else asks for its peak in use and a margin; the prober wants its maximum unless the queue is dry. Claims scale with the lanes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7eba53043a
commit
39480c486f
10 files changed
+373
-75
No files matched your search
+17
-2
@@ -31,8 +31,21 @@ export default {
|
||||
},
|
||||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||||
/** NNTP connections the coordinator fetches headers with. */
|
||||
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
|
||||
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
|
||||
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
|
||||
/**
|
||||
* The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
|
||||
* all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
|
||||
* and 2 for a hedged request's or a reconnect's new connection while the old one closes).
|
||||
*/
|
||||
nntpShares: {
|
||||
total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
|
||||
everyMs: 15_000,
|
||||
/** Headers before probes: history takes what it can use, the prober the rest. */
|
||||
coordinatorPriority: 2,
|
||||
proberPriority: 1,
|
||||
},
|
||||
/**
|
||||
* A reply with no data for this long fails, and its connection is replaced.
|
||||
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
|
||||
@@ -70,11 +83,13 @@ export default {
|
||||
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
|
||||
probe: {
|
||||
/**
|
||||
* NNTP connections the prober reads articles with, besides the
|
||||
* At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
|
||||
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
|
||||
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
|
||||
*/
|
||||
connections: env.get('PROBE_CONNECTIONS', 6),
|
||||
/** Of them, always (its share's minimum; nntpShares). */
|
||||
minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
|
||||
/** Candidates claimed at a time. */
|
||||
batch: 400,
|
||||
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
|
||||
|
||||
Reference in new issue
Block a user