4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Fable 5.1	680751e26a	Lease lanes for the tail, a mixed lease order, coordinator:status, the edge end to end ... A group now has two lease lanes (migration 6600): `history` works the ranges the bounds hand out, `tail` the tail planner's, each with its own holder, window, snapshot and takeover redo, so a thread on the tail never wades through a group's history and a held group's tail isn't stuck behind it. The edge puts `--tail-threads` (EDGE_TAIL_THREADS, 1 of 2+ by default) of its threads on the tail lane; the group routes take `?lane=`. The lease order within a lane alternates the group with the most planned articles and the longest unheld one. The takeover redo puts back only the lane's ranges the window did after the snapshot, and a group whose last range is held by a holder gone quiet can be taken over. `coordinator:status` prints the catalog's room, the ranges by band, the last hour's rates, the held groups with their snapshots and the phase-2 backlogs. `tests/e2e` runs the real edge-node binary against the node API: the corpus-fill race, a failed stream and the restart that restores the snapshot, a restart without the snapshot file, a takeover (STREAM_CHUNK_WAIT_MINUTES is a setting now). CI builds the binary before the tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-27 13:47:31 +01:00
Daan MeijerandClaude Fable 5.1	4af5e06400	LEASE_CAPTURE_ONLY: group leases of the corpus windows' ranges only ... The lease bounds alone don't select a window: the history planner's oldest buckets are priority 0-2 as well, so the window worker at 0..0 also took 2026 history at 0 (25k ranges in 40 groups, fetched live) while 1,739 ranges of the 2016 corpus window sat at voyager 2's leftover priorities (done when the window was marked, planned again by the reset). With LEASE_CAPTURE_ONLY the lease, next and the prefetcher's run-ahead take capture ranges only; the run-ahead now also keeps to the lease bounds. Test: a group with history at 0 only isn't leased, the history range after the window isn't handed out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-26 23:26:31 +01:00
Daan MeijerandClaude Fable 5.1	29e0cc8f6e	The window worker keeps its window over a chunk that doesn't inflate ... A live reply cut short (the server's compressed data ending at ".\r\n" at a read boundary; the spool keeps what Corpus.write refuses) made the worker give the group back without a snapshot, losing the window and the batch since its last post (six times in the first 40 minutes on homelab). Now the batch before the chunk goes out, the chunk is reported (fetched again, plain the second time) and the range is asked for again from that chunk (RangeOutcome::Again): the stream waits for the new copy and the window stays. Test: the coordinator hands the same range back at its resume point after a corrupt report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-26 22:53:52 +01:00
Daan MeijerandClaude Fable 5.1	1f6e3316ec	The window worker: group leases, the stream serving the corpus, the snapshot through the API (design 1 and 5) ... A worker thread leases a group and takes its ranges in article order through one carried window; the coordinator keeps one cursor a group (group_leases, migration 6500) and hands out the next range, the rules and nothing about where the chunks are: the range stream serves a stored range's chunks from the corpus file and a live one's from the spool. The window's snapshot (bincode: the frontier, the open files, releases and albums, the upload numbers, the batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and when the group is given back, and the next holder restores it; a takeover after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again. The prefetcher runs ahead of held groups' cursors and skips chunks the corpus has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node holds groups by default; --ranges keeps the old per-range path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-26 18:22:18 +01:00