Lease lanes for the tail, a mixed lease order, coordinator:status, the edge end to end
A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads` (EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The lease order within a lane alternates the group with the most planned articles and the longest unheld one. The takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range is held by a holder gone quiet can be taken over. `coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
db96642a72
commit
680751e26a
21 files changed
+1362
-180
No files matched your search
+28
-13
@@ -425,8 +425,9 @@ export class Prefetcher {
|
||||
)
|
||||
// Leased ranges, checked against the spool's files whatever spooled_at says: a worker waits for them.
|
||||
const leased = await rows(db.query()).where('r.status', 'leased').orderBy('r.leased_at')
|
||||
// The groups workers hold (group_leases; REBUILD.md design 1): the prefetcher runs ahead of their cursors.
|
||||
const held: number[] = (
|
||||
// The groups workers hold (group_leases; REBUILD.md design 1), by lane: the prefetcher runs ahead of their
|
||||
// cursors (a group's history and tail lanes are two cursors).
|
||||
const held: { group: number; lane: string }[] = (
|
||||
await db
|
||||
.from('group_leases')
|
||||
.whereNotNull('node_id')
|
||||
@@ -435,8 +436,8 @@ export class Prefetcher {
|
||||
'>=',
|
||||
new Date(Date.now() - coordinatorConfig.leaseTtlSeconds * 1000)
|
||||
)
|
||||
.select('group_id')
|
||||
).map((l) => Number(l.group_id))
|
||||
.select('group_id', 'lane')
|
||||
).map((l) => ({ group: Number(l.group_id), lane: l.lane }))
|
||||
// Planned ranges ahead: those started already, then new ones up to SPOOL_AHEAD_PER_LEASE per lease.
|
||||
const ahead = Math.max(
|
||||
coordinatorConfig.spool.minAhead,
|
||||
@@ -460,22 +461,36 @@ export class Prefetcher {
|
||||
const want = Math.max(ahead - Number(ready), short)
|
||||
// The catalog at its cap on the NVMe: no history fetched ahead (CatalogSpace).
|
||||
const full = await CatalogSpace.full()
|
||||
// Ahead of each held group's cursor: its next ranges in article order (within what the group leases hand out:
|
||||
// LEASE_MIN/MAX_PRIORITY, LEASE_CAPTURE_ONLY), a range of every held group before any group's second; without
|
||||
// held groups (no window worker running) nothing is fetched ahead for workers.
|
||||
// Ahead of each held cursor: the lane's next ranges in article order (the history lane's within what the group
|
||||
// leases hand out: LEASE_MIN/MAX_PRIORITY, LEASE_CAPTURE_ONLY, and none at the catalog's cap; the tail lane's
|
||||
// at the tail's priority), a range of every held cursor before any cursor's second; without held groups (no
|
||||
// window worker running) nothing is fetched ahead for workers.
|
||||
const { leasePriorityMin, leasePriorityMax, leaseCaptureOnly } = coordinatorConfig
|
||||
const tailPriority = coordinatorConfig.tail.priority
|
||||
const heldHistory = full ? [] : held.filter((h) => h.lane === 'history').map((h) => h.group)
|
||||
const heldTail = held.filter((h) => h.lane === 'tail').map((h) => h.group)
|
||||
const fresh =
|
||||
room && want > 0 && held.length > 0
|
||||
room && want > 0 && heldHistory.length + heldTail.length > 0
|
||||
? await rows(db.query())
|
||||
.where('r.status', 'planned')
|
||||
.where('r.spooled_chunks', 0)
|
||||
.whereRaw('r.next_id <= r.end_id')
|
||||
.whereIn('r.group_id', held)
|
||||
.whereBetween('r.priority', [leasePriorityMin, leasePriorityMax])
|
||||
.where((q: any) => (leaseCaptureOnly ? q.where('r.capture', true) : q))
|
||||
.where((q: any) => (full ? q.where('r.priority', '<', 0) : q))
|
||||
.where((q: any) =>
|
||||
q
|
||||
.where((h: any) =>
|
||||
h
|
||||
.whereIn('r.group_id', heldHistory)
|
||||
.where('r.priority', '>', tailPriority)
|
||||
.whereBetween('r.priority', [leasePriorityMin, leasePriorityMax])
|
||||
.where((c: any) => (leaseCaptureOnly ? c.where('r.capture', true) : c))
|
||||
)
|
||||
.orWhere((t: any) =>
|
||||
t.whereIn('r.group_id', heldTail).where('r.priority', '<=', tailPriority)
|
||||
)
|
||||
)
|
||||
.orderByRaw(
|
||||
'row_number() OVER (PARTITION BY r.group_id ORDER BY r.start_id), r.priority, r.start_id'
|
||||
'row_number() OVER (PARTITION BY r.group_id, r.priority <= ? ORDER BY r.start_id), r.priority, r.start_id',
|
||||
[tailPriority]
|
||||
)
|
||||
.limit(want)
|
||||
: []
|
||||
|
||||
Reference in new issue
Block a user