Daan MeijerandClaude Opus 5.5
39480c486f
The coordinator (38) and the prober (8) had a fixed split; with history paused the coordinator's stood idle while the prober's were its limit. nntp_shares: each process syncs every 15 s (priority, minimum, what it could use, what it has open) and gets what it may open: minimums first, then by priority what each wants, growing only into connections the others closed or gave up. The pool's size and the prober's lanes follow the grant; the coordinator doubles its want while work waits for a connection, else asks for its peak in use and a margin; the prober wants its maximum unless the queue is dry. Claims scale with the lanes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
91 lines
3.7 KiB
TypeScript
91 lines
3.7 KiB
TypeScript
import db from '@adonisjs/lucid/services/db'
|
|
import coordinatorConfig from '#config/coordinator'
|
|
|
|
/** A share whose process hasn't synced for this long is gone: its connections closed with it. */
|
|
const STALE_SECONDS = 60
|
|
/** Syncs one at a time (pg_advisory_xact_lock). */
|
|
const LOCK = 2_026_101_875
|
|
|
|
export interface Share {
|
|
name: string
|
|
priority: number
|
|
/** Kept whatever the others want (as far as the total goes, by priority). */
|
|
min: number
|
|
/** What it could use now. */
|
|
want: number
|
|
/** Connections it has open (or may have: a lane without one yet counts). */
|
|
open: number
|
|
}
|
|
|
|
/** What each share should have: its minimum first, then by priority what it wants, up to the total. */
|
|
export function targets(total: number, shares: Share[]): Map<string, number> {
|
|
const ordered = [...shares].sort((a, b) => b.priority - a.priority || a.name.localeCompare(b.name))
|
|
const target = new Map<string, number>()
|
|
let left = total
|
|
for (const s of ordered) {
|
|
const t = Math.min(s.min, left)
|
|
target.set(s.name, t)
|
|
left -= t
|
|
}
|
|
for (const s of ordered) {
|
|
const more = Math.min(Math.max(s.want - target.get(s.name)!, 0), left)
|
|
target.set(s.name, target.get(s.name)! + more)
|
|
left -= more
|
|
}
|
|
return target
|
|
}
|
|
|
|
/**
|
|
* The connections `self` may have: its target, but no more than the others leave over. Another share holds what it
|
|
* has open or was granted, whichever is more (it may open up to its grant until its next sync tells it otherwise),
|
|
* so the account never goes over the total: a share grows only into what the others closed or gave up.
|
|
*/
|
|
export function grant(total: number, self: Share, others: (Share & { granted: number })[]): number {
|
|
const target = targets(total, [self, ...others]).get(self.name)!
|
|
const held = others.reduce((sum, o) => sum + Math.max(o.open, o.granted), 0)
|
|
return Math.max(0, Math.min(target, total - held))
|
|
}
|
|
|
|
/**
|
|
* A process's share of the NNTP account's connections (LOG.md 2026-10-18 (75)): the coordinator and the prober had a
|
|
* fixed split, and with history paused the coordinator's 38 stood idle while the prober's 8 were its limit. Each
|
|
* syncs every nntpShares.everyMs (what it wants and has open) and gets what it may open; one wanting less closes
|
|
* connections as they come free, and the other grows as those show up closed.
|
|
*/
|
|
export class NntpShare {
|
|
granted = 0
|
|
|
|
constructor(
|
|
readonly name: string,
|
|
private priority: number,
|
|
readonly min: number,
|
|
readonly max: number
|
|
) {}
|
|
|
|
async sync(want: number, open: number): Promise<number> {
|
|
const self: Share = { name: this.name, priority: this.priority, min: this.min, want: Math.min(Math.max(want, this.min), this.max), open }
|
|
const total = coordinatorConfig.nntpShares.total
|
|
this.granted = await db.transaction(async (trx) => {
|
|
await trx.rawQuery(`SELECT pg_advisory_xact_lock(?)`, [LOCK])
|
|
const { rows } = await trx.rawQuery(
|
|
`SELECT name, priority, min, want, open, granted FROM nntp_shares WHERE name <> ? AND heartbeat_at > now() - make_interval(secs => ?)`,
|
|
[this.name, STALE_SECONDS]
|
|
)
|
|
const granted = grant(total, self, rows)
|
|
await trx.rawQuery(
|
|
`INSERT INTO nntp_shares (name, priority, min, want, open, granted, heartbeat_at) VALUES (?, ?, ?, ?, ?, ?, now())
|
|
ON CONFLICT (name) DO UPDATE SET priority = excluded.priority, min = excluded.min, want = excluded.want,
|
|
open = excluded.open, granted = excluded.granted, heartbeat_at = now()`,
|
|
[this.name, this.priority, this.min, self.want, open, granted]
|
|
)
|
|
return granted
|
|
})
|
|
return this.granted
|
|
}
|
|
|
|
/** Stopping: its connections are closed, the others needn't wait for the row to go stale. */
|
|
async leave() {
|
|
await db.from('nntp_shares').where('name', this.name).delete()
|
|
}
|
|
}
|