Scheduler and worker loops survive a database restart
index-scheduler's loop ended when Postgres restarted (the active-groups query was outside its try), leaving live indexing stopped for ~3 hours while the process stayed up. Each step's error is now caught and retried next round; worker:spots and worker:deferred wait 30 s and claim again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
61192690a7
commit
e3e3ecc93d
3 files changed
+60
-41
No files matched your search
@@ -117,11 +117,18 @@ export default class IndexScheduler extends BaseCommand {
|
||||
}
|
||||
}
|
||||
|
||||
// Run immediately, then wait a full interval after each run so runs never overlap.
|
||||
// Run immediately, then wait a full interval after each run so runs never overlap. A failed
|
||||
// round (e.g. Postgres restarting) is logged and the next one tries again: an error that escaped
|
||||
// here once ended the loop while the process stayed up, and live indexing stopped for ~3 hours
|
||||
// (LOG.md 2026-10-07 (3)).
|
||||
while (!Shutdown.requested) {
|
||||
await schedule()
|
||||
await backfill()
|
||||
await split()
|
||||
for (const step of [schedule, backfill, split]) {
|
||||
try {
|
||||
await step()
|
||||
} catch (err: any) {
|
||||
this.logger.error(`Scheduler round failed, retrying next round: ${err.message}`)
|
||||
}
|
||||
}
|
||||
await Shutdown.sleep(indexerConfig.intervalMs)
|
||||
}
|
||||
await this.pool.shutdown()
|
||||
|
||||
Reference in new issue
Block a user