Inspection backlog in Postgres; BullMQ gets a working set
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single thread. Producers now add file ids to inspection_queue; index-scheduler feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis moves the existing backlog over once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e7e338213a
commit
35d0b2da65
11 files changed
+219
-24
No files matched your search
@@ -107,6 +107,7 @@ export default await Env.create(new URL('../', import.meta.url), {
|
||||
FILE_KEYS_FILL_SECONDS: Env.schema.number.optional(),
|
||||
GC_AWAIT_INSPECTION_SECONDS: Env.schema.number.optional(),
|
||||
GC_MERGE_INTERVAL_MS: Env.schema.number.optional(),
|
||||
INSPECTION_FEED_MAX: Env.schema.number.optional(),
|
||||
ARCHIVE_WALK_MAX_SEGMENTS: Env.schema.number.optional(),
|
||||
DEFERRED_SETTLE_MS: Env.schema.number.optional(),
|
||||
DEFERRED_EXPAND_MAX_PARTS: Env.schema.number.optional(),
|
||||
|
||||
Reference in new issue
Block a user