The edge posts the open batch before a failed stream gives the group back
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m13s
Test and deploy / deploy (push) Successful in 1m18s

A plain stop and a chunk that doesn't inflate already posted the batch so far,
so the stop snapshot stood at a stored resume point; a failed stream (the
stalled one's "Peer disconnected") skipped the post, left the batch open and
lost the window with the snapshot refused. Now only a batch the coordinator
refused stays open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-27 10:06:52 +01:00
1 parent 31fc204c76
commit e3957e5af5
3 files changed
+13 -6

No files matched your search

+4 -1
View File
@@ -557,7 +557,10 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
them in the corpus only, and the run leaves them there; a stream that
looked at the corpus once, seconds before that store, waited its whole
hour on the spool and the worker lost the group and its open batch
(three groups on the first morning).
(three groups on the first morning). A failed stream now posts the
open batch before the group goes back, like a stop does, so the
window's snapshot survives it; only a batch the coordinator refused
(lease lost, too large) still loses the window.
6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
(30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
+5 -2
View File
@@ -248,8 +248,11 @@ COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-nod
snapshot (bincode, gzip) goes to the coordinator every
`--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch,
and when the group is given back: caught up (`next` answers 204: the
tail), a stop, a lost lease or a failure; the next holder restores it
(`GET groups/:id/snapshot`) and continues at its article. `--groups N`
tail), a stop, a lost lease or a failure; a stop or a failed stream
posts the open batch first, so the snapshot stands at its resume point
(a batch the coordinator refused can't, and that window is lost); the
next holder restores it (`GET groups/:id/snapshot`) and continues at
its article. `--groups N`
stops after N groups; `--ranges N` is the old per-range lease
(`POST lease`), one range per thread without a carried window.
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
+4 -3
View File
@@ -629,9 +629,10 @@ fn run_group_range(api: &Arc<Api>, group_id: i64, group_name: &str, range: Range
}
}
}
// Stopped mid-range (or at a chunk that doesn't inflate): the batch so far goes out, so the snapshot stands at
// a stored resume point, and a chunk reported corrupt is read again from its own start.
if !reached_end && failure.is_none() && post_error.lock().unwrap().is_none() && indexer.next() > batch_from {
// Stopped mid-range (a stop, a stream that failed, a chunk that doesn't inflate): the batch so far goes out,
// so the snapshot stands at a stored resume point and the window survives the group going back; a chunk
// reported corrupt is read again from its own start. Only a batch the coordinator refused stays open.
if !reached_end && post_error.lock().unwrap().is_none() && indexer.next() > batch_from {
let _ = post_tx.send(indexer.take());
}
drop(post_tx);