Idle connections fetch capture ranges into the corpus
Test and deploy / test (push) Successful in 42s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m6s

The spool only fetches what workers lease soon, in priority order, and the
tail never ends: after a reset the 1,400 capture ranges waited behind it
while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture
ranges whose chunks aren't all stored are now targets after everything else,
fetched into the corpus only (no spool); a later lease reads them from disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-12 20:11:57 +02:00
1 parent 0fafe71b21
commit 7b436f3f77
5 files changed
+106 -3

No files matched your search

+6
View File
@@ -80,6 +80,12 @@ export default {
/** Stored XOVER chunks of ranges marked `capture`, for the grouping sweeps (app/services/corpus.ts). Off when unset. */
corpus: {
dir: env.get('CORPUS_DIR'),
/**
* Capture ranges fetched into the corpus by connections the spool leaves idle, this many at a time: the
* spool only fetches what workers lease soon, and the tail comes first, so after a reset the capture
* ranges waited behind it for hours (1,400 marked, none stored, 34 of 40 connections idle; LOG.md 2026-10-12 (27)).
*/
ahead: env.get('CORPUS_AHEAD', 8),
},
spool: {
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),