4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	2b4f673ddb	Rebuild: index lookups for the merger's neighbours and the prefetcher's ranges ... The merger's neighbour query matched "end_id = ? OR start_id = ?", which no index serves: it read every range of the group per file, 5.5 s a round and most of Postgres's time in an end-to-end run. Now the range holding the article before and after, by (group_id, start_id): 0.08 ms. The prefetcher's leased and started ranges (every 2 s) get partial indexes: 15-25 ms -> <0.1 ms with 145k planned ranges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:59:30 +02:00
Daan MeijerandClaude Opus 5.5	821de668d0	Rebuild: the merger stops when a pass over the file keys decides nothing ... Waiting files counted as work, so the merge loop re-read them for its whole 30 s budget every minute: Postgres used ~a core even without workers. Now ~4% at rest, and storing a boneless range costs Postgres ~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads are stored EXTERNAL (zstd already; no pglz attempt). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:49:45 +02:00
Daan MeijerandClaude Opus 5.5	31493a98be	Rebuild: merger rounds with one neighbour query, a cursor and a time budget; planning skips failing groups ... The neighbour check was three queries per file (~2,000 files/minute); now one per round of 20k files, with a cursor so waiting files don't take every round's places, and rounds until nothing is left or 30 s. On the dev backlog (old-style fragments): ~18k files/minute in the 30 s budget; at full throughput the new fragment rule leaves ~1,200 files/minute. A group that fails GROUP (gone from the server) no longer stops planning the others; edge:bench's groups are inactive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:25:45 +02:00
Daan MeijerandClaude Opus 5.5	b862201991	Rebuild: the fragment merger ... Files cut by a range's edge are put together once the ranges around them are done (coordinator:run, every minute): a complete file is judged by the rules and written as a release chunk with probe candidates; one still missing parts while both neighbouring ranges are done is dropped and counted (incomplete/edge). A neighbour not planned yet counts as not done, except before the group's oldest article. BatchStore.writeReleases is the shared write path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:19:53 +02:00