The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5)
A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c4fe324a1
commit
1f6e3316ec
19 files changed
+1358
-31
No files matched your search
@@ -0,0 +1,33 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Group leases (REBUILD.md, "Voyager 3: window indexing", 1 and 5; 2026-10-26): a worker thread holds a group, not
|
||||
* a range, and indexes its ranges in article order through one carried window. One row a group: the holder and
|
||||
* its heartbeat, the range it works, and the snapshot of its window (the article it stands at: the next one to
|
||||
* index) that the coordinator keeps on bulk. Ranges stay what is planned, fetched and finished.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`
|
||||
CREATE TABLE group_leases (
|
||||
group_id integer PRIMARY KEY REFERENCES groups (id) ON DELETE CASCADE,
|
||||
-- The holder; null between holders. A holder quiet for LEASE_TTL_SECONDS is taken over.
|
||||
node_id integer,
|
||||
leased_at timestamptz,
|
||||
heartbeat_at timestamptz,
|
||||
-- The range the holder works (null between ranges).
|
||||
range_id bigint,
|
||||
-- The window's snapshot: the article it stands at (a resume point), and when it was taken; null: none.
|
||||
snapshot_next bigint,
|
||||
snapshot_at timestamptz,
|
||||
ranges_done integer NOT NULL DEFAULT 0,
|
||||
headers bigint NOT NULL DEFAULT 0,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
)`)
|
||||
this.schema.raw(`CREATE INDEX group_leases_held_idx ON group_leases (heartbeat_at) WHERE node_id IS NOT NULL`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP TABLE IF EXISTS group_leases`)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user