Rebuild: spread prefetch connections over ranges and groups
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Skipped

At most PREFETCH_PER_RANGE (2) connections on a range and
PREFETCH_PER_GROUP (4) on a group; SPOOL_AHEAD_PER_LEASE 2 -> 4 so there
are targets enough. The report says how many ranges and groups are in
flight.

Homelab, 40 connections, alternated 12 min each: unlimited (~10 ranges
in flight) 281k and 167k articles/s; spread (~22 ranges) 319k and 296k,
chunks 11.5 s against 13-24 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 23:33:43 +02:00
1 parent 4bf526ab6e
commit 678b3dd438
3 files changed
+46 -6

No files matched your search

+12 -1
View File
@@ -58,9 +58,20 @@ export default {
spool: {
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
/** Ranges kept fetched ahead, per active lease. */
aheadPerLease: env.get('SPOOL_AHEAD_PER_LEASE', 2),
aheadPerLease: env.get('SPOOL_AHEAD_PER_LEASE', 4),
/** At least this many ranges ahead, so the first leases start at once. */
minAhead: env.get('SPOOL_MIN_AHEAD', 2),
maxBytes: env.get('SPOOL_MAX_GB', 10) * 1e9,
/**
* At most this many connections on one range, and on one group, at a
* time (0: no limit): spreads the requests over ranges and groups instead
* of all on the next chunks of the first ranges. Homelab, 40 connections,
* 12 min each, alternated: unlimited (on ~10 ranges) 281k and 167k
* articles/s, 2 per range and 4 per group (on ~22) 319k and 296k, with
* shorter chunk times (11.5 vs 13–24 s; LOG.md 2026-10-09 (18)). Needs
* targets for it: aheadPerLease 4.
*/
perRange: env.get('PREFETCH_PER_RANGE', 2),
perGroup: env.get('PREFETCH_PER_GROUP', 4),
},
}