voyager 3: the window in the Rust edge
The indexer finishes files and releases by posting time instead of the range path: the frontier is the median Date of the last 1,024 plausible dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and a release is finished, judged and written once the frontier is 24 h past its last part, flagged edge when posted within a day of the block's start. At the block's end the rest is written provisionally as edges for the block merger. Files short of parts stay in their release (partial: the store writes complete = false). Gone: fragments and the cut-off, uploads_of (online now), check.ts and the fixture comparison; CI replays the fixtures through edge-bench instead. The wire carries finished, edge and partial. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
2580f20a43
commit
f8cb17903b
13 files changed
+457
-413
No files matched your search
+52
-20
@@ -401,21 +401,26 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
|
||||
2. **The window.** The worker reads a block in article order and keeps its
|
||||
open files and releases in memory (counters, names, sizes; the segment
|
||||
lists spooled to a file per block on the worker's disk). The *frontier*
|
||||
is the median Date of the last 4,096 articles with a plausible date
|
||||
(inside the block's span ± 7 days); Date headers alone aren't trusted
|
||||
(clock skew, bogus years). A release is **finished** when the frontier
|
||||
is 24 h past its last part (`WINDOW_HOURS`), or at the block's end:
|
||||
written once, with its completeness decided then (every announced file
|
||||
there; par2 decides later for those without a counter), never updated by
|
||||
the header pass again. Incomplete releases are kept (user, 2026-10-24).
|
||||
A fill posted after the window is a release of its own (the judge's
|
||||
`repeated` rule already knows reposts). Releases still open at a block's
|
||||
end are written flagged `edge`; the block merger (replaces the fragment
|
||||
merger) joins the edge releases of adjacent blocks of a group by
|
||||
identity key once both are done, and clears the flag.
|
||||
The settler goes: `settled_at` is the finish. The **tail** keeps its
|
||||
latency: a tail lease emits a release provisionally as today and the
|
||||
window finalizes it once (two writes, not one per range).
|
||||
is the median Date of the last 1,024 articles with a plausible date
|
||||
(within 7 days of the frontier; 1,024 such dates in a row that agree
|
||||
among themselves move it, the group having paused); Date headers alone
|
||||
aren't trusted (clock skew, bogus years). 1,024, not more: the median
|
||||
trails the newest article by half the ring, a day in a group of 500
|
||||
articles a day. A release is **finished** when the frontier is 24 h past
|
||||
its last part (`WINDOW_MS`): written once, with its completeness decided
|
||||
then (every announced file there and no file short of parts; par2
|
||||
decides later for those without a counter), never updated by the header
|
||||
pass again. Incomplete releases are kept (user, 2026-10-24). A fill
|
||||
posted after the window is a release of its own (the judge's `repeated`
|
||||
rule already knows reposts). At the block's end the releases the window
|
||||
hasn't closed on are written **provisionally** (`settled_at` null) and
|
||||
flagged `edge`, as are the finished releases posted within 24 h of the
|
||||
block's first frontier; the block merger (replaces the fragment merger)
|
||||
joins the edge releases of adjacent blocks of a group by identity key
|
||||
once both are done, finishes them and clears the flag. The settler goes:
|
||||
`settled_at` is the finish. The **tail** needs no path of its own: a
|
||||
tail lease's open releases at its end are such provisional edges, the
|
||||
next lease's continuation joins them.
|
||||
|
||||
3. **Albums.** Files of one poster in one owner group whose subjects differ
|
||||
only by a number sequence, or share a directory-like prefix, posted
|
||||
@@ -571,11 +576,38 @@ service compare the source code. Tests insert fixtures through
|
||||
`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
|
||||
and groups by name, source by name). The unit and functional suites pass.
|
||||
|
||||
Next: the window and albums in the Rust edge (`finished`, `edge`, the
|
||||
provisional tail and its finish), block leases and the planner (then the
|
||||
store keys by the leased block and the lease columns leave `ranges`), the
|
||||
block merger, the prober writing `release_texts` kind 2, the corpus replay
|
||||
harness.
|
||||
### The window in the Rust edge (built 2026-10-25)
|
||||
|
||||
`edge/src/indexer.rs`: the scanner and the file aggregator as before; then
|
||||
the window instead of the range path. `Frontier` keeps a ring of the last
|
||||
`FRONTIER_ARTICLES` (1,024) plausible dates (within `PLAUSIBLE_MS`, 7
|
||||
days, of the frontier; 1,024 implausible dates in a row that agree move
|
||||
it), recomputed once per chunk; established after 64 dated articles
|
||||
(`settle` at the block's end for smaller blocks); its first value is the
|
||||
block's start in posting time. Per chunk: completed files, then the files
|
||||
whose last part is before frontier − `WINDOW_MS` (24 h; undated ones idle
|
||||
for `idle` chunks), go into the open release of their key whose posting
|
||||
span is within `UPLOAD_GAP_MS` (3 h) of theirs, else start one (the
|
||||
upload number per key, distinct within a batch); releases whose last part
|
||||
is before frontier − `WINDOW_MS` are judged and written `finished`, as an
|
||||
`edge` when posted within `WINDOW_MS` of the block's start. At the end
|
||||
(`finish`) everything left is written: finished when the window closed on
|
||||
it, else `finished: false, edge: true`. A file short of parts stays in
|
||||
its release and sets `partial` (the store: `complete = false`). A lone
|
||||
article of a multi-part random file is still counted `per-article`, not
|
||||
written. Gone: fragments and the cut-off, `uploads_of` (online now),
|
||||
`check.ts` and the fixture comparison (the parity with the TypeScript
|
||||
indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
|
||||
(incomplete releases are kept). The wire carries `finished`, `edge` and
|
||||
`partial`; the store reads them (`partial` into `complete`). Memory: the
|
||||
window holds a day of a group's segments in memory (the busiest group's
|
||||
~50 M); the spool file per block of the design is not built yet.
|
||||
|
||||
Next: albums in the edge (design 3), block leases and the planner (then
|
||||
the store keys by the leased block and the lease columns leave `ranges`;
|
||||
until then every range end leaves provisional edges), the block merger
|
||||
(joins and finishes the edges), the prober writing `release_texts` kind 2,
|
||||
the corpus replay harness.
|
||||
|
||||
## Decisions
|
||||
|
||||
|
||||
Reference in new issue
Block a user