Rebuild: end-to-end fixes — spool accounting, keep-alive frames, resume point, overlapped worker
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped

End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).

- Prefetcher: spooled_at was set against a total counted from the range's
  resume point, which shrinks as the worker advances: the last chunks were
  never fetched and the worker waited. Now counted against the whole
  range, only new chunks counted, and leased ranges are always checked
  against the files. Planned ranges ahead are bounded (it had spooled ever
  more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
  (Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
  (it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
  current one, and doesn't wait for a post before indexing on (posts stay
  in order; a failed post stops the range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 01:31:19 +02:00
1 parent db882f0b48
commit fa7d4f3394
8 files changed
+134 -37

No files matched your search

+1 -1
View File
@@ -1,4 +1,4 @@
export { RangeIndexer, type EdgeBatch, type ReleaseChunk, type DroppedChunk, type ProbeCandidate, type CandidateKind, type RuleCount, type Fragment, type RangeIndexerOptions } from './range_indexer.js'
export { encodeFrame, decodeFrames, type Frame } from './stream.js'
export { encodeFrame, decodeFrames, isKeepalive, KEEPALIVE, type Frame } from './stream.js'
export { toWire, fromWire, type WireBatch } from './wire.js'
export type { Lease } from './lease.js'
+3 -1
View File
@@ -83,6 +83,8 @@ export interface RangeIndexerOptions {
/** Inclusive article numbers. */
start: number
end: number
/** Where this worker starts: the range's resume point (a range taken over midway); its first batch starts here. */
resumeAt?: number
rules: Rules
/** Chunks without a new part before an open file is given up (as a fragment near an edge). */
idleChunks?: number
@@ -112,7 +114,7 @@ export class RangeIndexer {
this.rules = new RuleSet(options.rules)
this.idle = options.idleChunks ?? 5
this.edge = options.edgeArticles ?? 50_000
this.batch = this.empty(options.start)
this.batch = this.empty(options.resumeAt ?? options.start)
}
/** One chunk of overview lines (parsed); `next` is the first article after it. */
+7
View File
@@ -11,6 +11,13 @@ export interface Frame {
const HEADER = 20
/** An empty frame, sent while the next chunk is still being fetched, so the response doesn't go quiet. */
export const KEEPALIVE: Frame = { from: 0, to: 0, data: Buffer.alloc(0) }
export function isKeepalive(frame: Frame) {
return frame.data.length === 0 && frame.from === 0
}
export function encodeFrame(frame: Frame): Buffer {
const head = Buffer.alloc(HEADER)
head.writeBigUInt64BE(BigInt(frame.from), 0)