The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m16s

A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-26 18:22:18 +01:00
1 parent 3c4fe324a1
commit 1f6e3316ec
19 files changed
+1358 -31

No files matched your search

+12
View File
@@ -31,6 +31,13 @@ export default {
},
/** A leased range without a heartbeat for this long goes back to the queue. */
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
/**
* The ranges group leases hand out (REBUILD.md design 1), by priority: the tail is negative, the capture windows
* 0–2 (2016, 2021, 2025), the rest of the history above (its position in the group, or 4000+). Bounds so a
* deployment can run one window first (homelab 2026-10-26: 0..0, the 2016 window); unbounded by default.
*/
leasePriorityMin: env.get('LEASE_MIN_PRIORITY', -1_000_000),
leasePriorityMax: env.get('LEASE_MAX_PRIORITY', 1_000_000),
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
@@ -146,6 +153,11 @@ export default {
*/
filter: env.get('CORPUS_FILTER', true),
},
/**
* The group windows' snapshots (REBUILD.md, "Voyager 3: window indexing", 5; app/services/group_leases.ts): a
* file a group on bulk, next to the spool by default.
*/
snapshotDir: env.get('SNAPSHOT_DIR', `${env.get('SPOOL_DIR', app.tmpPath('spool'))}/snapshots`),
spool: {
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
/** Ranges kept fetched ahead, per active lease. */