Inspection backlog in Postgres; BullMQ gets a working set
Test and deploy / test (push) Successful in 2m14s
Test and deploy / deploy (push) Successful in 2m51s

5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-07 22:12:13 +02:00
1 parent e7e338213a
commit 35d0b2da65
11 files changed
+219 -24

No files matched your search

+3 -3
View File
@@ -1,8 +1,8 @@
import db from '@adonisjs/lucid/services/db'
import FileStore from '#services/FileStore'
import SfvService from '#services/SfvService'
import QueueService from '#services/QueueService'
import ContentService from '#services/ContentService'
import InspectionQueue from '#services/InspectionQueue'
export interface BinaryFilesResult {
/** New files. */
@@ -134,8 +134,8 @@ class BinaryFileService {
// By the posted name, as before: inspection skips later volumes and readable media names anyway.
const jobs = added
.filter((file) => ContentService.isCandidate(file.posted))
.map((file) => ({ name: 'process-collection', data: { fileId: file.id }, opts: { jobId: `file-${file.id}` } }))
if (jobs.length > 0) await QueueService.collectionQueue.addBulk(jobs)
.map((file) => file.id)
if (jobs.length > 0) await InspectionQueue.add(jobs)
result.inspections = jobs.length
return result
}