Prober: claim range per value of unreadable, from the index ends
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m59s

Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-18 11:00:11 +02:00
1 parent a5e0399ee6
commit 7eba53043a
2 files changed
+36 -1

No files matched your search

+10 -1
View File
@@ -167,8 +167,17 @@ export class Prober {
if (want <= 0) return
// The starting points chosen here: in the query Postgres drew random() once for all slices.
// Over the claimable ones of this tier only: the rest of the queue isn't settled yet.
// Per value of unreadable, each an end of the index: over `unreadable >= false` Postgres walked every waiting
// entry of the kind, 2.5 minutes a claim at 28M archive candidates (LOG.md 2026-10-18 (74)).
const values = unreadable ? [true] : [false, true]
const ends = values
.map(() => `SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable = ?`)
.join(' UNION ALL ')
const [{ lo, hi }] = (
await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable >= ?`, [kind, unreadable])
await db.rawQuery(
`SELECT min(lo) AS lo, max(hi) AS hi FROM (${ends}) e`,
values.flatMap((v) => [kind, v])
)
).rows
if (lo === null) continue
const starts = Array.from({ length: SLICES }, () => Number(lo) + Math.floor(Math.random() * (Number(hi) - Number(lo) + 1)))