Daan MeijerandClaude Fable 5.1
1f6e3316ec
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
100 lines
2.7 KiB
Rust
100 lines
2.7 KiB
Rust
//! A lease as the coordinator sends it (lib/edge/lease.ts): the range and the rules, JSON with the TypeScript
|
|
//! names. The edge-node worker takes one per range; edge-bench's replay takes the rules alone (`--rules`).
|
|
|
|
use crate::names::Rules;
|
|
use serde::Deserialize;
|
|
use std::collections::HashMap;
|
|
|
|
#[derive(Deserialize)]
|
|
pub struct LeaseJson {
|
|
pub range: RangeJson,
|
|
pub rules: RulesJson,
|
|
}
|
|
|
|
/// A group lease (lib/edge/lease.ts GroupLease): the group a worker thread holds, the rules, the window's snapshot if any.
|
|
#[derive(Deserialize)]
|
|
pub struct GroupLeaseJson {
|
|
pub group: GroupJson,
|
|
pub rules: RulesJson,
|
|
pub snapshot: Option<SnapshotJson>,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
pub struct GroupJson {
|
|
pub id: i64,
|
|
pub name: String,
|
|
pub priority: i64,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
pub struct SnapshotJson {
|
|
pub next: i64,
|
|
}
|
|
|
|
/// The group's next range (lib/edge/lease.ts NextRange).
|
|
#[derive(Deserialize)]
|
|
pub struct NextRangeJson {
|
|
pub range: RangeSpanJson,
|
|
}
|
|
|
|
#[derive(Deserialize, Clone, Copy)]
|
|
pub struct RangeSpanJson {
|
|
pub id: i64,
|
|
pub start: i64,
|
|
pub end: i64,
|
|
pub next: i64,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
pub struct RangeJson {
|
|
pub id: i64,
|
|
pub group: String,
|
|
pub start: i64,
|
|
pub end: i64,
|
|
pub next: i64,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct RulesJson {
|
|
pub version: i64,
|
|
pub priorities: HashMap<String, u32>,
|
|
#[serde(default)]
|
|
pub dropped_signatures: Vec<String>,
|
|
#[serde(default)]
|
|
pub dropped_tools: Option<Vec<String>>,
|
|
#[serde(default)]
|
|
pub per_article_max_parts: Option<u32>,
|
|
#[serde(default)]
|
|
pub sample_percent: Option<f64>,
|
|
#[serde(default)]
|
|
pub drop_executables: Option<bool>,
|
|
#[serde(default)]
|
|
pub dropped_posters: Option<Vec<String>>,
|
|
}
|
|
|
|
impl RulesJson {
|
|
/// The rules with today's defaults under them (a field the stored set doesn't have keeps the default).
|
|
pub fn rules(self) -> Rules {
|
|
let mut rules = Rules::defaults(self.priorities.into_iter().collect());
|
|
rules.version = self.version;
|
|
rules.dropped_signatures = self.dropped_signatures.into_iter().collect();
|
|
if let Some(tools) = self.dropped_tools {
|
|
rules.dropped_tools = tools.into_iter().collect();
|
|
}
|
|
if let Some(n) = self.per_article_max_parts {
|
|
rules.per_article_max_parts = n;
|
|
}
|
|
if let Some(p) = self.sample_percent {
|
|
rules.sample_percent = p;
|
|
}
|
|
if let Some(e) = self.drop_executables {
|
|
rules.drop_executables = e;
|
|
}
|
|
if let Some(posters) = self.dropped_posters {
|
|
rules.dropped_posters = posters;
|
|
}
|
|
rules
|
|
}
|
|
}
|