Prober: the sweep by a signature index, pending rows locked in one order, deadlocked flushes kept
Test and deploy / test (push) Successful in 2m59s
Test and deploy / deploy (push) Failing after 21m32s

The discard sweep of a dropped signature read the whole pending_releases table per signature (a parallel seq
scan over the poster test, four minutes on the HDD that starved the lanes' claims): a partial index on
(group_id, tool, poster) WHERE status = 0 makes it a range, and the sweep queries a poster's rows by it and a
tool's and group's by the ladder index.

The batch store's pending upsert, the record, and the sweep deadlocked on pending_releases and pending_classes
rows locked in different orders, and a deadlocked flush dropped its outcomes: lockOrder() sorts every
multi-row statement's set the same way, and a flush that deadlocks anyway keeps its outcomes for the next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-11-03 10:36:19 +01:00
1 parent 5d3b578322
commit 8e008da364
5 files changed
+110 -24

No files matched your search

+7 -3
View File
@@ -9,7 +9,7 @@ import { SIDECAR } from '#services/sidecar_folder'
import { identityKey, releaseKey } from '#lib/release/keys'
import { sourceCode } from '#lib/release/source'
import { uploadTool } from '#lib/rules/index'
import { byEffort } from '#services/pending_ladder'
import { byEffort, lockOrder } from '#services/pending_ladder'
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
/**
@@ -232,7 +232,8 @@ class BatchStore {
target_bytes = coalesce(pending_releases.target_bytes, excluded.target_bytes)
WHERE NOT pending_releases.finished
RETURNING release_key, tool, klass, (xmax = 0) AS inserted`,
[groupId, JSON.stringify(rows)]
// In the record's lock order: the upsert locks a finished row too, before its WHERE passes it by.
[groupId, JSON.stringify(lockOrder(rows, (r) => r.release_key))]
)
written += done.length
// Each class seen gets its row in the ladder's record, so the claim explores it (pending_classes).
@@ -248,7 +249,10 @@ class BatchStore {
`INSERT INTO pending_classes (tool, group_id, klass, waiting)
SELECT v.tool, ?, v.klass, v.waiting FROM jsonb_to_recordset(?::jsonb) AS v(tool text, klass int, waiting int)
ON CONFLICT (tool, group_id, klass) DO UPDATE SET waiting = pending_classes.waiting + excluded.waiting`,
[groupId, JSON.stringify([...classes.values()])]
[
groupId,
JSON.stringify(lockOrder(classes.values(), (c) => `${c.tool}\t${groupId}\t${c.klass}`)),
]
)
}
}