Lease lanes for the tail, a mixed lease order, coordinator:status, the edge end to end
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads` (EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The lease order within a lane alternates the group with the most planned articles and the longest unheld one. The takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range is held by a holder gone quiet can be taken over. `coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
db96642a72
commit
680751e26a
21 files changed
+1362
-180
No files matched your search
+20
-3
@@ -234,9 +234,20 @@ them:
|
||||
|
||||
```sh
|
||||
cargo build --release
|
||||
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]
|
||||
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--tail-threads T] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]
|
||||
```
|
||||
|
||||
- **Two lanes** (2026-10-27): `--tail-threads T` (EDGE_TAIL_THREADS; one of
|
||||
two or more threads by default) puts that many threads on the groups'
|
||||
*tail* lane (`?lane=tail` on the group routes): the tail planner's
|
||||
ranges, through a window and snapshot of their own
|
||||
(`<group>-tail.gz`). The other threads work the history lane: the
|
||||
windows and the history the coordinator's bounds hand out. A tail thread
|
||||
never wades through a group's history first, and a held history group's
|
||||
tail isn't stuck behind its days of ranges; the coordinator orders each
|
||||
lane's leases by alternating the group with the most planned articles
|
||||
and the longest unheld one.
|
||||
|
||||
- **One group per thread** (default one per core; REBUILD.md, "Voyager 3:
|
||||
window indexing", 1 and 5): a thread leases a group (`POST
|
||||
groups/lease`), takes its ranges in article order (`POST
|
||||
@@ -273,5 +284,11 @@ COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-nod
|
||||
same (releases with every column, chunks, candidates, rule counts,
|
||||
fragments with their JSON).
|
||||
- **On homelab:** compose service `edge` (deploy/compose.yaml), the
|
||||
`edge` image target of the Dockerfile, `--threads $EDGE_THREADS` (8);
|
||||
it talks to `node-api`.
|
||||
`edge` image target of the Dockerfile, `--threads $EDGE_THREADS` (8),
|
||||
EDGE_TAIL_THREADS (1); it talks to `node-api`.
|
||||
- **End to end:** `tests/e2e/edge_node.spec.ts` runs this binary
|
||||
(`edge/target/release/edge-node`, or EDGE_NODE_BIN) against the test
|
||||
server, database, spool and corpus: a chunk the corpus fill stores while
|
||||
the stream waits, a failed stream and the restart that restores the
|
||||
snapshot, a restart whose snapshot file is gone, a takeover. `node ace
|
||||
test e2e` after `cargo build --release`; CI builds the binary first.
|
||||
Reference in new issue
Block a user