voyager 3: the window in the Rust edge
The indexer finishes files and releases by posting time instead of the range path: the frontier is the median Date of the last 1,024 plausible dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and a release is finished, judged and written once the frontier is 24 h past its last part, flagged edge when posted within a day of the block's start. At the block's end the rest is written provisionally as edges for the block merger. Files short of parts stay in their release (partial: the store writes complete = false). Gone: fragments and the cut-off, uploads_of (online now), check.ts and the fixture comparison; CI replays the fixtures through edge-bench instead. The wire carries finished, edge and partial. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
2580f20a43
commit
f8cb17903b
13 files changed
+457
-413
No files matched your search
@@ -31,8 +31,10 @@ export interface ReleaseChunk {
|
||||
* later chunk of the same release key (the totals then are the release's; the payload only the files not sent yet).
|
||||
*/
|
||||
finished?: boolean
|
||||
/** Still open at its block's end: the block merger joins it with the neighbour block's part. */
|
||||
/** Still open at its block's end, or posted within the window of its start: the block merger joins it with the neighbour block's part. */
|
||||
edge?: boolean
|
||||
/** A file of the release misses parts (the window closed on it): not complete, whatever the counter says. */
|
||||
partial?: boolean
|
||||
payload: Buffer
|
||||
}
|
||||
|
||||
@@ -95,7 +97,8 @@ export interface EdgeBatch {
|
||||
samples: DroppedChunk[]
|
||||
candidates: ProbeCandidate[]
|
||||
counts: RuleCount[]
|
||||
fragments: Fragment[]
|
||||
/** The range path's cut-off files (the TypeScript indexer); the Rust edge's window has none, the store ignores them. */
|
||||
fragments?: Fragment[]
|
||||
}
|
||||
|
||||
export interface RangeIndexerOptions {
|
||||
@@ -210,7 +213,7 @@ export class RangeIndexer {
|
||||
|
||||
/** Files completed and fragments cut since the last take: how big the next batch is. */
|
||||
get pending(): number {
|
||||
let files = this.batch.fragments.length
|
||||
let files = this.batch.fragments!.length
|
||||
for (const list of this.completed.values()) files += list.length
|
||||
return files
|
||||
}
|
||||
@@ -294,7 +297,7 @@ export class RangeIndexer {
|
||||
fragment.segments.push([s.number, s.messageId, s.bytes, s.articleNumber])
|
||||
if (Number.isFinite(s.postedAt) && (fragment.postedAt === null || s.postedAt < fragment.postedAt)) fragment.postedAt = s.postedAt
|
||||
}
|
||||
this.batch.fragments.push(fragment)
|
||||
this.batch.fragments!.push(fragment)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user