The replay harness: a group's corpus through one carried window into a scratch catalog
replay:corpus runs edge-bench --group over a group's complete corpus ranges in article order (one window, carried from range to range, finished after the last range; edge/src/lease.rs holds the lease JSON the worker and the bench share) and stores every batch through BatchStore as the worker would, then reports releases finished, provisional, complete and the bytes per header of the NVMe and HDD tables. Refuses a database not named *_replay or *_scratch. The app image carries edge-bench; compose gets a replay profile for homelab. Design amended (REBUILD.md 1, 2, 5): the window is carried by one worker a group, no block leases and no block merger; provisional writes stay for the lease ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
cf51db047f
commit
72e973d006
10 files changed
+735
-151
No files matched your search
+20
-2
@@ -113,11 +113,29 @@ files, segments, payload size, candidates, fragments and samples. The
|
||||
summary can hide compensating differences, so compare the outputs
|
||||
themselves.
|
||||
|
||||
## Comparing the output
|
||||
## Replaying a group through one window
|
||||
|
||||
A corpus group directory (`<corpus>/<group>/<start>-<end>/`, each range with
|
||||
its `range.json`) is replayed in article order with the window carried from
|
||||
range to range, the end finished after the last range (REBUILD.md, "The
|
||||
replay harness"):
|
||||
|
||||
```sh
|
||||
./target/release/edge-bench --group /corpus/alt.binaries.ath --rules rules.json --batch-chunks 5 --wire - > batches.jsonl
|
||||
# --from <article> skips ranges ending before it, --limit <n> stops after n ranges, --no-finish leaves the window open
|
||||
```
|
||||
|
||||
`--rules` takes the lease's rules JSON (`src/lease.rs`: the catalog's rule
|
||||
set and group priorities, as `replay:corpus` writes it); without it the
|
||||
defaults with only the group indexed apply. `--wire -` writes each batch as
|
||||
the worker posts it to stdout, one JSON per line, and the report to stderr;
|
||||
`node ace replay:corpus` reads them into the batch store. Each range end
|
||||
prints the frontier, the releases out and the releases and files still open.
|
||||
|
||||
## The dump
|
||||
|
||||
```sh
|
||||
./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl
|
||||
cd .. && node --import=@poppinss/ts-exec edge/check.ts /path/to/capture /tmp/out.jsonl
|
||||
```
|
||||
|
||||
`--dump` writes every output of every batch as JSON lines:
|
||||
|
||||
Reference in new issue
Block a user