Rebuild: the merger stops when a pass over the file keys decides nothing
Waiting files counted as work, so the merge loop re-read them for its whole 30 s budget every minute: Postgres used ~a core even without workers. Now ~4% at rest, and storing a boneless range costs Postgres ~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads are stored EXTERNAL (zstd already; no pglz attempt). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
73fc9aabdf
commit
821de668d0
3 files changed
+15
-4
No files matched your search
@@ -12,6 +12,8 @@ export interface MergeResult {
|
||||
completed: number
|
||||
incomplete: number
|
||||
waiting: number
|
||||
/** This round reached the end of the file keys (the next one starts over). */
|
||||
wrapped: boolean
|
||||
}
|
||||
|
||||
/** File keys per round. */
|
||||
@@ -29,14 +31,15 @@ class FragmentMerger {
|
||||
private cursor = ''
|
||||
|
||||
async run(): Promise<MergeResult> {
|
||||
const result: MergeResult = { files: 0, completed: 0, incomplete: 0, waiting: 0 }
|
||||
const result: MergeResult = { files: 0, completed: 0, incomplete: 0, waiting: 0, wrapped: false }
|
||||
const { rows: keys } = await db.rawQuery(
|
||||
`SELECT DISTINCT f.file_key FROM fragments f JOIN ranges r ON r.id = f.range_id
|
||||
WHERE r.status = 'done' AND f.file_key > ? ORDER BY f.file_key LIMIT ?`,
|
||||
[this.cursor, BATCH]
|
||||
)
|
||||
// At the end of the keys: the next round starts over.
|
||||
this.cursor = keys.length < BATCH ? '' : keys.at(-1).file_key
|
||||
result.wrapped = keys.length < BATCH
|
||||
this.cursor = result.wrapped ? '' : keys.at(-1).file_key
|
||||
if (keys.length === 0) return result
|
||||
const rules = new RuleSet(await RuleSets.current())
|
||||
const rows = await db
|
||||
|
||||
Reference in new issue
Block a user