Daan MeijerandClaude Fable 5.1
1f6e3316ec
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
36 lines
713 B
TOML
36 lines
713 B
TOML
[package]
|
|
name = "edge"
|
|
version = "0.1.0"
|
|
edition = "2021"
|
|
description = "The edge worker (edge-node) and its benchmark (edge-bench): voyager 3's window indexing in Rust"
|
|
|
|
[[bin]]
|
|
name = "edge-bench"
|
|
path = "src/main.rs"
|
|
|
|
[[bin]]
|
|
name = "edge-node"
|
|
path = "src/bin/edge-node.rs"
|
|
|
|
[dependencies]
|
|
base64 = "0.22"
|
|
bincode = "1.3"
|
|
chrono = "0.4"
|
|
flate2 = { version = "1", default-features = false, features = ["zlib-rs"] }
|
|
libdeflater = "1.26.1"
|
|
md-5 = "0.10"
|
|
memchr = "2.8.3"
|
|
mimalloc = "0.1.52"
|
|
regex = "1"
|
|
rustc-hash = "2"
|
|
serde = { version = "1.0.229", features = ["derive"] }
|
|
serde_json = "1.0.151"
|
|
sha1 = "0.10"
|
|
signal-hook = "0.4.5"
|
|
ureq = "3.4.2"
|
|
zstd = "0.13"
|
|
|
|
[profile.release]
|
|
lto = true
|
|
codegen-units = 1
|