The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c4fe324a1
commit
1f6e3316ec
19 files changed
+1358
-31
No files matched your search
@@ -31,6 +31,13 @@ export default {
|
||||
},
|
||||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||||
/**
|
||||
* The ranges group leases hand out (REBUILD.md design 1), by priority: the tail is negative, the capture windows
|
||||
* 0–2 (2016, 2021, 2025), the rest of the history above (its position in the group, or 4000+). Bounds so a
|
||||
* deployment can run one window first (homelab 2026-10-26: 0..0, the 2016 window); unbounded by default.
|
||||
*/
|
||||
leasePriorityMin: env.get('LEASE_MIN_PRIORITY', -1_000_000),
|
||||
leasePriorityMax: env.get('LEASE_MAX_PRIORITY', 1_000_000),
|
||||
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
|
||||
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
|
||||
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
|
||||
@@ -146,6 +153,11 @@ export default {
|
||||
*/
|
||||
filter: env.get('CORPUS_FILTER', true),
|
||||
},
|
||||
/**
|
||||
* The group windows' snapshots (REBUILD.md, "Voyager 3: window indexing", 5; app/services/group_leases.ts): a
|
||||
* file a group on bulk, next to the spool by default.
|
||||
*/
|
||||
snapshotDir: env.get('SNAPSHOT_DIR', `${env.get('SPOOL_DIR', app.tmpPath('spool'))}/snapshots`),
|
||||
spool: {
|
||||
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
|
||||
/** Ranges kept fetched ahead, per active lease. */
|
||||
|
||||
Reference in new issue
Block a user