voyager 3: the window in the Rust edge
The indexer finishes files and releases by posting time instead of the range path: the frontier is the median Date of the last 1,024 plausible dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and a release is finished, judged and written once the frontier is 24 h past its last part, flagged edge when posted within a day of the block's start. At the block's end the rest is written provisionally as edges for the block merger. Files short of parts stay in their release (partial: the store writes complete = false). Gone: fragments and the cut-off, uploads_of (online now), check.ts and the fixture comparison; CI replays the fixtures through edge-bench instead. The wire carries finished, edge and partial. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
2580f20a43
commit
f8cb17903b
13 files changed
+457
-413
No files matched your search
+12
-9
@@ -6,18 +6,21 @@ from the node API, reads their prefetched XOVER chunks, finds releases and
|
||||
posts batches. `edge-bench` (`src/main.rs`) replays a captured range through
|
||||
the same indexing (`src/lib.rs`) for timing and for comparing.
|
||||
|
||||
The TypeScript `RangeIndexer` (`lib/edge`, `node ace edge:node`) is the
|
||||
reference: `check.ts` runs both on a capture and compares every output field
|
||||
by field. An identity or rule change goes into both and has to come out
|
||||
identical. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs `cargo test`
|
||||
and `check-fixtures.sh`, which compares both on the committed captures in
|
||||
`fixtures/` (25k overview lines each of a.b.models, a.b.ath and
|
||||
a.b.boneless, in 5 chunks); a push to `rebuild` deploys only when they
|
||||
match.
|
||||
Since voyager 3 (2026-10-25) the Rust indexer is the one implementation of
|
||||
the window (REBUILD.md, "Voyager 3: window indexing"; `src/indexer.rs`):
|
||||
files and releases are finished when the frontier (the median Date of the
|
||||
last 1,024 plausibly dated articles) is 24 h past their last part, or at
|
||||
the block's end (provisionally, as edges), and written once. The
|
||||
TypeScript `RangeIndexer` (`lib/edge`) keeps the range path for the corpus
|
||||
scanner and the tests; the field-by-field comparison (`check.ts`) went
|
||||
with the parity. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs
|
||||
`cargo test` and `bench-fixtures.sh`, which replays the committed captures
|
||||
in `fixtures/` (25k overview lines each of a.b.ath, a.b.boneless, a.b.fta
|
||||
and a.b.models, in 5 chunks).
|
||||
|
||||
```sh
|
||||
cargo test --release --manifest-path edge/Cargo.toml
|
||||
edge/check-fixtures.sh # from the repository root
|
||||
edge/bench-fixtures.sh # from the repository root
|
||||
```
|
||||
|
||||
In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores
|
||||
|
||||
Reference in new issue
Block a user