The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m16s

A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-26 18:22:18 +01:00
1 parent 3c4fe324a1
commit 1f6e3316ec
19 files changed
+1358 -31

No files matched your search

+16 -4
View File
@@ -234,12 +234,24 @@ them:
```sh
cargo build --release
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--ranges N] [--max-mbit M]
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]
```
- **One range per thread** (default one per core): one thread reads the
coordinator's stream and inflates a frame ahead, another posts the
batches in order (one may wait), the indexer goes on meanwhile.
- **One group per thread** (default one per core; REBUILD.md, "Voyager 3:
window indexing", 1 and 5): a thread leases a group (`POST
groups/lease`), takes its ranges in article order (`POST
groups/:id/next`), streams each range's chunks (`GET ranges/:id/stream`:
the corpus or the spool, the worker can't tell) through one carried
window, posts a batch at least at every range end and finishes the
range; one thread reads the stream and inflates a frame ahead, another
posts the batches in order, the indexer goes on meanwhile. The window's
snapshot (bincode, gzip) goes to the coordinator every
`--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch,
and when the group is given back: caught up (`next` answers 204: the
tail), a stop, a lost lease or a failure; the next holder restores it
(`GET groups/:id/snapshot`) and continues at its article. `--groups N`
stops after N groups; `--ranges N` is the old per-range lease
(`POST lease`), one range per thread without a carried window.
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
409 means the lease is lost (the range stops, isn't given back); a
batch once 500k articles or 100k files wait (a picture group's batches