The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c4fe324a1
commit
1f6e3316ec
19 files changed
+1358
-31
No files matched your search
@@ -11,6 +11,40 @@ pub struct LeaseJson {
|
||||
pub rules: RulesJson,
|
||||
}
|
||||
|
||||
/// A group lease (lib/edge/lease.ts GroupLease): the group a worker thread holds, the rules, the window's snapshot if any.
|
||||
#[derive(Deserialize)]
|
||||
pub struct GroupLeaseJson {
|
||||
pub group: GroupJson,
|
||||
pub rules: RulesJson,
|
||||
pub snapshot: Option<SnapshotJson>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct GroupJson {
|
||||
pub id: i64,
|
||||
pub name: String,
|
||||
pub priority: i64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct SnapshotJson {
|
||||
pub next: i64,
|
||||
}
|
||||
|
||||
/// The group's next range (lib/edge/lease.ts NextRange).
|
||||
#[derive(Deserialize)]
|
||||
pub struct NextRangeJson {
|
||||
pub range: RangeSpanJson,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Clone, Copy)]
|
||||
pub struct RangeSpanJson {
|
||||
pub id: i64,
|
||||
pub start: i64,
|
||||
pub end: i64,
|
||||
pub next: i64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct RangeJson {
|
||||
pub id: i64,
|
||||
|
||||
Reference in new issue
Block a user