voyager 3: the window in the Rust edge
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped

The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-25 21:28:35 +01:00
1 parent 2580f20a43
commit f8cb17903b
13 files changed
+457 -413

No files matched your search

+52 -20
View File
@@ -401,21 +401,26 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
2. **The window.** The worker reads a block in article order and keeps its
open files and releases in memory (counters, names, sizes; the segment
lists spooled to a file per block on the worker's disk). The *frontier*
is the median Date of the last 4,096 articles with a plausible date
(inside the block's span ± 7 days); Date headers alone aren't trusted
(clock skew, bogus years). A release is **finished** when the frontier
is 24 h past its last part (`WINDOW_HOURS`), or at the block's end:
written once, with its completeness decided then (every announced file
there; par2 decides later for those without a counter), never updated by
the header pass again. Incomplete releases are kept (user, 2026-10-24).
A fill posted after the window is a release of its own (the judge's
`repeated` rule already knows reposts). Releases still open at a block's
end are written flagged `edge`; the block merger (replaces the fragment
merger) joins the edge releases of adjacent blocks of a group by
identity key once both are done, and clears the flag.
The settler goes: `settled_at` is the finish. The **tail** keeps its
latency: a tail lease emits a release provisionally as today and the
window finalizes it once (two writes, not one per range).
is the median Date of the last 1,024 articles with a plausible date
(within 7 days of the frontier; 1,024 such dates in a row that agree
among themselves move it, the group having paused); Date headers alone
aren't trusted (clock skew, bogus years). 1,024, not more: the median
trails the newest article by half the ring, a day in a group of 500
articles a day. A release is **finished** when the frontier is 24 h past
its last part (`WINDOW_MS`): written once, with its completeness decided
then (every announced file there and no file short of parts; par2
decides later for those without a counter), never updated by the header
pass again. Incomplete releases are kept (user, 2026-10-24). A fill
posted after the window is a release of its own (the judge's `repeated`
rule already knows reposts). At the block's end the releases the window
hasn't closed on are written **provisionally** (`settled_at` null) and
flagged `edge`, as are the finished releases posted within 24 h of the
block's first frontier; the block merger (replaces the fragment merger)
joins the edge releases of adjacent blocks of a group by identity key
once both are done, finishes them and clears the flag. The settler goes:
`settled_at` is the finish. The **tail** needs no path of its own: a
tail lease's open releases at its end are such provisional edges, the
next lease's continuation joins them.
3. **Albums.** Files of one poster in one owner group whose subjects differ
only by a number sequence, or share a directory-like prefix, posted
@@ -571,11 +576,38 @@ service compare the source code. Tests insert fixtures through
`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
and groups by name, source by name). The unit and functional suites pass.
Next: the window and albums in the Rust edge (`finished`, `edge`, the
provisional tail and its finish), block leases and the planner (then the
store keys by the leased block and the lease columns leave `ranges`), the
block merger, the prober writing `release_texts` kind 2, the corpus replay
harness.
### The window in the Rust edge (built 2026-10-25)
`edge/src/indexer.rs`: the scanner and the file aggregator as before; then
the window instead of the range path. `Frontier` keeps a ring of the last
`FRONTIER_ARTICLES` (1,024) plausible dates (within `PLAUSIBLE_MS`, 7
days, of the frontier; 1,024 implausible dates in a row that agree move
it), recomputed once per chunk; established after 64 dated articles
(`settle` at the block's end for smaller blocks); its first value is the
block's start in posting time. Per chunk: completed files, then the files
whose last part is before frontier − `WINDOW_MS` (24 h; undated ones idle
for `idle` chunks), go into the open release of their key whose posting
span is within `UPLOAD_GAP_MS` (3 h) of theirs, else start one (the
upload number per key, distinct within a batch); releases whose last part
is before frontier − `WINDOW_MS` are judged and written `finished`, as an
`edge` when posted within `WINDOW_MS` of the block's start. At the end
(`finish`) everything left is written: finished when the window closed on
it, else `finished: false, edge: true`. A file short of parts stays in
its release and sets `partial` (the store: `complete = false`). A lone
article of a multi-part random file is still counted `per-article`, not
written. Gone: fragments and the cut-off, `uploads_of` (online now),
`check.ts` and the fixture comparison (the parity with the TypeScript
indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
(incomplete releases are kept). The wire carries `finished`, `edge` and
`partial`; the store reads them (`partial` into `complete`). Memory: the
window holds a day of a group's segments in memory (the busiest group's
~50 M); the spool file per block of the design is not built yet.
Next: albums in the edge (design 3), block leases and the planner (then
the store keys by the leased block and the lease columns leave `ranges`;
until then every range end leaves provisional edges), the block merger
(joins and finishes the edges), the prober writing `release_texts` kind 2,
the corpus replay harness.
## Decisions