Daan MeijerandClaude Opus 5.5
7df5a638ca
Two processes, one NNTP connection each (the coordinator 40 -> 38): - spots:read (service spotter): free.pt's overviews after the last article seen, each spot's full header verified (lib/spotnet from voyager 1, as Spotweb), its description, website and NZB articles; the password its title or text names; delete/dispose messages applied. - spots:inspect (service spot-inspector): each spot's compressed NZB read, its files listed (spot_files), each matched to the header pass's release by the same release identity in one of its groups within 48 h (release_spots); unmatched spots matched again hourly, after 24 h a release of their own (source spot) with the spot's NZB. - release_passwords: per release and password, its source (nzb, spot-title, spot-text) and spot; the catalog keeps only passworded. XoverClient.head(); PROBE_CONNECTIONS default 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
123 lines
6.1 KiB
TypeScript
123 lines
6.1 KiB
TypeScript
import env from '#start/env'
|
||
import app from '@adonisjs/core/services/app'
|
||
|
||
/** The coordinator (REBUILD.md, part 1). */
|
||
export default {
|
||
/** Articles per range. */
|
||
rangeArticles: env.get('RANGE_ARTICLES', 2_000_000),
|
||
/**
|
||
* Articles per XOVER chunk (spool files, stream frames). Eweka takes
|
||
* ~1.5–2 s per command whatever its size: one connection read 9–15k
|
||
* lines/s in chunks of 25k, ~20k in 50k, ~30k in 100k and 200k (LOG.md
|
||
* 2026-10-09).
|
||
*/
|
||
chunkArticles: env.get('CHUNK_ARTICLES', 100_000),
|
||
/**
|
||
* The live tail (REBUILD.md, coordinator): new articles become ranges
|
||
* every `everySeconds` (one LIST ACTIVE per name prefix), leased and
|
||
* fetched before any history. A group gets a tail range once it has
|
||
* `minArticles` new (a chunk) or its oldest unplanned article waited
|
||
* `maxDelayMinutes`, so a quiet group doesn't make a range every round.
|
||
*/
|
||
tail: {
|
||
everySeconds: env.get('TAIL_EVERY_SECONDS', 120),
|
||
minArticles: env.get('TAIL_MIN_ARTICLES', 100_000),
|
||
maxDelayMinutes: env.get('TAIL_MAX_DELAY_MINUTES', 15),
|
||
/** Before the history's 0 (oldest) … 1000 (newest). */
|
||
priority: -1,
|
||
},
|
||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||
/** NNTP connections the coordinator fetches headers with. */
|
||
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
|
||
/**
|
||
* A reply with no data for this long fails, and its connection is replaced.
|
||
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
|
||
* a fresh one answers in seconds; LOG.md 2026-10-09 (14)): at 10 minutes
|
||
* the prefetcher nearly stopped. Data arrives within seconds otherwise.
|
||
*/
|
||
nntpTimeoutMs: env.get('NNTP_TIMEOUT_MS', 2 * 60 * 1000),
|
||
/**
|
||
* A request that got less than this in the last minute (kB) is given up,
|
||
* its connection closed, and the chunk fetched again: a reply that trickles
|
||
* (6,204 → 6,229 kB in a minute) never goes quiet long enough for
|
||
* nntpTimeoutMs, and kept a worker waiting on one chunk for an hour. Not a
|
||
* deadline: some groups send steadily but slowly (a.b.0day.stuffz ~400 kB a
|
||
* minute, 5–7 minutes a chunk), and a fixed 90–360 s gave those up again
|
||
* and again (LOG.md 2026-10-10 (3)).
|
||
*/
|
||
nntpStallBytesPerMinute: env.get('NNTP_STALL_KB_PER_MINUTE', 100) * 1000,
|
||
/**
|
||
* A chunk not fetched after this long is asked again on another
|
||
* connection; the first answer wins and the other connection is closed.
|
||
* Eweka stalls single requests: the same chunk took 27–52 s on one
|
||
* connection and 1–9 s on another at the same moment; median ~3 s
|
||
* (LOG.md 2026-10-09 (15)). Measured in production (40 + 6 connections vs
|
||
* 40, 15 min each): 280k vs 273k articles/s, within the noise; the second
|
||
* request won half the time. Off by default.
|
||
*/
|
||
nntpHedgeAfterMs: env.get('NNTP_HEDGE_AFTER_SECONDS', 10) * 1000,
|
||
/** Of nntpConnections, kept for those second requests (the prefetcher's loops are the rest). 0: no hedging. */
|
||
nntpHedgeConnections: env.get('NNTP_HEDGE_CONNECTIONS', 0),
|
||
/**
|
||
* A connection is closed after this long (when it's free), and a fresh one
|
||
* opened. 0: never. Requests were as often slow (>10 s) on connections
|
||
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
|
||
*/
|
||
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
|
||
probe: {
|
||
/**
|
||
* NNTP connections the prober reads articles with, besides the
|
||
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
|
||
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
|
||
*/
|
||
connections: env.get('PROBE_CONNECTIONS', 6),
|
||
/** Candidates claimed at a time. */
|
||
batch: 400,
|
||
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
|
||
pipeline: env.get('PROBE_PIPELINE', 8),
|
||
},
|
||
/**
|
||
* Spotnet spots (app/services/spotter.ts, spot_inspector.ts): the group the spots are posted in, how many of
|
||
* its last articles the spotter reads the first time (~215 spots a day: ~3 months), how often it looks again,
|
||
* the most articles an NZB may take, and how long an inspected spot whose files match no release waits for
|
||
* the header pass before it becomes a release of its own.
|
||
*/
|
||
spots: {
|
||
group: env.get('SPOT_GROUP', 'free.pt'),
|
||
backfill: env.get('SPOT_BACKFILL', 20_000),
|
||
everySeconds: env.get('SPOT_EVERY_SECONDS', 120),
|
||
maxNzbSegments: env.get('SPOT_MAX_NZB_SEGMENTS', 64),
|
||
standaloneAfterHours: env.get('SPOT_STANDALONE_AFTER_HOURS', 24),
|
||
},
|
||
/** Stored XOVER chunks of ranges marked `capture`, for the grouping sweeps (app/services/corpus.ts). Off when unset. */
|
||
corpus: {
|
||
dir: env.get('CORPUS_DIR'),
|
||
/**
|
||
* Capture ranges fetched into the corpus by connections the spool leaves idle, this many at a time (8, 2 connections each, left 20 of 40 idle): the
|
||
* spool only fetches what workers lease soon, and the tail comes first, so after a reset the capture
|
||
* ranges waited behind it for hours (1,400 marked, none stored, 34 of 40 connections idle; LOG.md 2026-10-12 (27)).
|
||
*/
|
||
ahead: env.get('CORPUS_AHEAD', 16),
|
||
},
|
||
spool: {
|
||
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
|
||
/** Ranges kept fetched ahead, per active lease. */
|
||
aheadPerLease: env.get('SPOOL_AHEAD_PER_LEASE', 4),
|
||
/** At least this many ranges ahead, so the first leases start at once. */
|
||
minAhead: env.get('SPOOL_MIN_AHEAD', 2),
|
||
maxBytes: env.get('SPOOL_MAX_GB', 10) * 1e9,
|
||
/**
|
||
* At most this many connections on one range, and on one group, at a
|
||
* time (0: no limit): spreads the requests over ranges and groups instead
|
||
* of all on the next chunks of the first ranges. Homelab, 40 connections,
|
||
* 12 min each, alternated: unlimited (on ~10 ranges) 281k and 167k
|
||
* articles/s, 2 per range and 4 per group (on ~22) 319k and 296k, with
|
||
* shorter chunk times (11.5 vs 13–24 s; LOG.md 2026-10-09 (18)). Needs
|
||
* targets for it: aheadPerLease 4.
|
||
*/
|
||
perRange: env.get('PREFETCH_PER_RANGE', 2),
|
||
perGroup: env.get('PREFETCH_PER_GROUP', 4),
|
||
},
|
||
}
|