The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m16s

A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-26 18:22:18 +01:00
1 parent 3c4fe324a1
commit 1f6e3316ec
19 files changed
+1358 -31

No files matched your search

+1 -1
View File
@@ -1,7 +1,7 @@
export { RangeIndexer, postedSpan, chunkOf, candidatesOf, releaseKeyOf, uploadsOf, UPLOAD_GAP_MS, type EdgeBatch, type ReleaseChunk, type DroppedChunk, type ProbeCandidate, type CandidateKind, type RuleCount, type Fragment, type RangeIndexerOptions } from './range_indexer.js'
export { encodeFrame, decodeFrames, isKeepalive, KEEPALIVE, type Frame } from './stream.js'
export { toWire, fromWire, type WireBatch } from './wire.js'
export type { Lease } from './lease.js'
export type { Lease, GroupLease, NextRange } from './lease.js'
export type { EdgeSegment, FileMeta } from './scan.js'
export { throttle } from './throttle.js'
export { filterOverviews, undelta, type ChunkDrops } from './corpus_filter.js'
+15
View File
@@ -5,3 +5,18 @@ export interface Lease {
range: { id: number; group: string; start: number; end: number; next: number }
rules: Rules
}
/**
* A group lease (REBUILD.md, "Voyager 3: window indexing", 1): the group a worker thread holds, the rules, and the
* snapshot of the group's window when there is one (its next article; the body comes from GET groups/:id/snapshot).
*/
export interface GroupLease {
group: { id: number; name: string; priority: number }
rules: Rules
snapshot: { next: number } | null
}
/** The group's next range in article order, leased to the holder (POST groups/:id/next; 204 when caught up). */
export interface NextRange {
range: { id: number; start: number; end: number; next: number }
}