The edge posts the open batch before a failed stream gives the group back
A plain stop and a chunk that doesn't inflate already posted the batch so far, so the stop snapshot stood at a stored resume point; a failed stream (the stalled one's "Peer disconnected") skipped the post, left the batch open and lost the window with the snapshot refused. Now only a batch the coordinator refused stays open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
31fc204c76
commit
e3957e5af5
3 files changed
+13
-6
No files matched your search
+4
-1
@@ -557,7 +557,10 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
|
||||
them in the corpus only, and the run leaves them there; a stream that
|
||||
looked at the corpus once, seconds before that store, waited its whole
|
||||
hour on the spool and the worker lost the group and its open batch
|
||||
(three groups on the first morning).
|
||||
(three groups on the first morning). A failed stream now posts the
|
||||
open batch before the group goes back, like a stop does, so the
|
||||
window's snapshot survives it; only a batch the coordinator refused
|
||||
(lease lost, too large) still loses the window.
|
||||
|
||||
6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
|
||||
(30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
|
||||
|
||||
+5
-2
@@ -248,8 +248,11 @@ COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-nod
|
||||
snapshot (bincode, gzip) goes to the coordinator every
|
||||
`--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch,
|
||||
and when the group is given back: caught up (`next` answers 204: the
|
||||
tail), a stop, a lost lease or a failure; the next holder restores it
|
||||
(`GET groups/:id/snapshot`) and continues at its article. `--groups N`
|
||||
tail), a stop, a lost lease or a failure; a stop or a failed stream
|
||||
posts the open batch first, so the snapshot stands at its resume point
|
||||
(a batch the coordinator refused can't, and that window is lost); the
|
||||
next holder restores it (`GET groups/:id/snapshot`) and continues at
|
||||
its article. `--groups N`
|
||||
stops after N groups; `--ranges N` is the old per-range lease
|
||||
(`POST lease`), one range per thread without a carried window.
|
||||
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
|
||||
|
||||
@@ -629,9 +629,10 @@ fn run_group_range(api: &Arc<Api>, group_id: i64, group_name: &str, range: Range
|
||||
}
|
||||
}
|
||||
}
|
||||
// Stopped mid-range (or at a chunk that doesn't inflate): the batch so far goes out, so the snapshot stands at
|
||||
// a stored resume point, and a chunk reported corrupt is read again from its own start.
|
||||
if !reached_end && failure.is_none() && post_error.lock().unwrap().is_none() && indexer.next() > batch_from {
|
||||
// Stopped mid-range (a stop, a stream that failed, a chunk that doesn't inflate): the batch so far goes out,
|
||||
// so the snapshot stands at a stored resume point and the window survives the group going back; a chunk
|
||||
// reported corrupt is read again from its own start. Only a batch the coordinator refused stays open.
|
||||
if !reached_end && post_error.lock().unwrap().is_none() && indexer.next() > batch_from {
|
||||
let _ = post_tx.send(indexer.take());
|
||||
}
|
||||
drop(post_tx);
|
||||
|
||||
Reference in new issue
Block a user