The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c4fe324a1
commit
1f6e3316ec
19 files changed
+1358
-31
No files matched your search
+16
-4
@@ -234,12 +234,24 @@ them:
|
||||
|
||||
```sh
|
||||
cargo build --release
|
||||
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--ranges N] [--max-mbit M]
|
||||
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]
|
||||
```
|
||||
|
||||
- **One range per thread** (default one per core): one thread reads the
|
||||
coordinator's stream and inflates a frame ahead, another posts the
|
||||
batches in order (one may wait), the indexer goes on meanwhile.
|
||||
- **One group per thread** (default one per core; REBUILD.md, "Voyager 3:
|
||||
window indexing", 1 and 5): a thread leases a group (`POST
|
||||
groups/lease`), takes its ranges in article order (`POST
|
||||
groups/:id/next`), streams each range's chunks (`GET ranges/:id/stream`:
|
||||
the corpus or the spool, the worker can't tell) through one carried
|
||||
window, posts a batch at least at every range end and finishes the
|
||||
range; one thread reads the stream and inflates a frame ahead, another
|
||||
posts the batches in order, the indexer goes on meanwhile. The window's
|
||||
snapshot (bincode, gzip) goes to the coordinator every
|
||||
`--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch,
|
||||
and when the group is given back: caught up (`next` answers 204: the
|
||||
tail), a stop, a lost lease or a failure; the next holder restores it
|
||||
(`GET groups/:id/snapshot`) and continues at its article. `--groups N`
|
||||
stops after N groups; `--ranges N` is the old per-range lease
|
||||
(`POST lease`), one range per thread without a carried window.
|
||||
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
|
||||
409 means the lease is lost (the range stops, isn't given back); a
|
||||
batch once 500k articles or 100k files wait (a picture group's batches
|
||||
|
||||
Reference in new issue
Block a user