Rebuild: the merger stops when a pass over the file keys decides nothing
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped

Waiting files counted as work, so the merge loop re-read them for its
whole 30 s budget every minute: Postgres used ~a core even without
workers. Now ~4% at rest, and storing a boneless range costs Postgres
~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads
are stored EXTERNAL (zstd already; no pglz attempt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 02:49:45 +02:00
1 parent 73fc9aabdf
commit 821de668d0
3 files changed
+15 -4

No files matched your search

+5 -2
View File
@@ -12,6 +12,8 @@ export interface MergeResult {
completed: number
incomplete: number
waiting: number
/** This round reached the end of the file keys (the next one starts over). */
wrapped: boolean
}
/** File keys per round. */
@@ -29,14 +31,15 @@ class FragmentMerger {
private cursor = ''
async run(): Promise<MergeResult> {
const result: MergeResult = { files: 0, completed: 0, incomplete: 0, waiting: 0 }
const result: MergeResult = { files: 0, completed: 0, incomplete: 0, waiting: 0, wrapped: false }
const { rows: keys } = await db.rawQuery(
`SELECT DISTINCT f.file_key FROM fragments f JOIN ranges r ON r.id = f.range_id
WHERE r.status = 'done' AND f.file_key > ? ORDER BY f.file_key LIMIT ?`,
[this.cursor, BATCH]
)
// At the end of the keys: the next round starts over.
this.cursor = keys.length < BATCH ? '' : keys.at(-1).file_key
result.wrapped = keys.length < BATCH
this.cursor = result.wrapped ? '' : keys.at(-1).file_key
if (keys.length === 0) return result
const rules = new RuleSet(await RuleSets.current())
const rows = await db