From e3957e5af580940893e3ada2d9796e4870553ab9 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Tue, 27 Oct 2026 10:06:52 +0100 Subject: [PATCH] The edge posts the open batch before a failed stream gives the group back A plain stop and a chunk that doesn't inflate already posted the batch so far, so the stop snapshot stood at a stored resume point; a failed stream (the stalled one's "Peer disconnected") skipped the post, left the batch open and lost the window with the snapshot refused. Now only a batch the coordinator refused stays open. Co-Authored-By: Claude Fable 5.1 --- REBUILD.md | 5 ++++- edge/README.md | 7 +++++-- edge/src/bin/edge-node.rs | 7 ++++--- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/REBUILD.md b/REBUILD.md index 626d574..4c135e4 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -557,7 +557,10 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md them in the corpus only, and the run leaves them there; a stream that looked at the corpus once, seconds before that store, waited its whole hour on the spool and the worker lost the group and its open batch - (three groups on the first morning). + (three groups on the first morning). A failed stream now posts the + open batch before the group goes back, like a stop does, so the + window's snapshot survives it; only a batch the coordinator refused + (lease lost, too large) still loses the window. 6. **The corpus stays as it is.** Measured on 300 unfiltered chunks (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of diff --git a/edge/README.md b/edge/README.md index 8b135d3..54c612d 100644 --- a/edge/README.md +++ b/edge/README.md @@ -248,8 +248,11 @@ COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-nod snapshot (bincode, gzip) goes to the coordinator every `--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch, and when the group is given back: caught up (`next` answers 204: the - tail), a stop, a lost lease or a failure; the next holder restores it - (`GET groups/:id/snapshot`) and continues at its article. `--groups N` + tail), a stop, a lost lease or a failure; a stop or a failed stream + posts the open batch first, so the snapshot stands at its resume point + (a batch the coordinator refused can't, and that window is lost); the + next holder restores it (`GET groups/:id/snapshot`) and continues at + its article. `--groups N` stops after N groups; `--ranges N` is the old per-range lease (`POST lease`), one range per thread without a carried window. - **As `commands/edge_node.ts`:** retries for network trouble and 5xx; diff --git a/edge/src/bin/edge-node.rs b/edge/src/bin/edge-node.rs index cf0b1e1..35d76f1 100644 --- a/edge/src/bin/edge-node.rs +++ b/edge/src/bin/edge-node.rs @@ -629,9 +629,10 @@ fn run_group_range(api: &Arc, group_id: i64, group_name: &str, range: Range } } } - // Stopped mid-range (or at a chunk that doesn't inflate): the batch so far goes out, so the snapshot stands at - // a stored resume point, and a chunk reported corrupt is read again from its own start. - if !reached_end && failure.is_none() && post_error.lock().unwrap().is_none() && indexer.next() > batch_from { + // Stopped mid-range (a stop, a stream that failed, a chunk that doesn't inflate): the batch so far goes out, + // so the snapshot stands at a stored resume point and the window survives the group going back; a chunk + // reported corrupt is read again from its own start. Only a batch the coordinator refused stays open. + if !reached_end && post_error.lock().unwrap().is_none() && indexer.next() > batch_from { let _ = post_tx.send(indexer.take()); } drop(post_tx);