Prober: claim only candidates of settled releases, without the join
The settler marks a settled release's candidates ready (and unreadable when it had no readable name). Claims took the candidates by joining releases from random points of the queue; in its newer half nothing had settled, so a slice scanned to the end (0.3 s each, 20 slices and 5 tiers a claim) and the prober stood still: 2.4M waiting, 0 probed a minute. Now an index over the claimable rows only, the slices drawn within them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1866e38c25
commit
cdc1cdc0cc
5 files changed
+63
-11
No files matched your search
+12
-6
@@ -21,14 +21,20 @@ class Settler {
|
||||
async run(): Promise<number> {
|
||||
let settled = 0
|
||||
while (true) {
|
||||
const { rowCount } = await db.rawQuery(
|
||||
`UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
|
||||
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
|
||||
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)`,
|
||||
// Their probe candidates become claimable (Prober.claim), the unreadable ones first.
|
||||
const { rows } = await db.rawQuery(
|
||||
`WITH settled AS (
|
||||
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
|
||||
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
|
||||
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
|
||||
RETURNING id, readable),
|
||||
marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id)
|
||||
SELECT count(*)::int AS n FROM settled`,
|
||||
[QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
|
||||
)
|
||||
settled += rowCount ?? 0
|
||||
if ((rowCount ?? 0) < BATCH) return settled
|
||||
const n = rows[0].n
|
||||
settled += n
|
||||
if (n < BATCH) return settled
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user