Stop refilling parts of binaries that are already a file
Test and deploy / test (push) Successful in 52s
Test and deploy / deploy (push) Successful in 1m23s

Crossposts, reposts and backfill ranges keep bringing articles of files
that worker:file already created. Ingest inserted their header parts again,
and GC's copied-parts pass (which searched all binaries for parts on every
batch) removed only 150k-450k per sweep: parts grew to 68M rows, 56M of
them belonging to finished files, and the 23 GB table paused the backfill.

- Ingest skips header parts (no per-part CRC) for binaries with a file_id.
- GC drops copied parts with a cursor over binaries, 2000 per statement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-04 02:53:11 +02:00
1 parent d3e774a489
commit 2178847513
3 files changed
+60 -14

No files matched your search

+24 -11
View File
@@ -12,6 +12,8 @@ import Shutdown from '#services/Shutdown'
const CLEAN_BATCH = 1000
const SCAN_COUNT = 1000
/** Binaries per statement when dropping copied parts (a few hundred parts each at most). */
const COPIED_PARTS_BATCH = 2000
export default class GcScheduler extends BaseCommand {
public static commandName = 'gc:scheduler'
@@ -173,21 +175,32 @@ export default class GcScheduler extends BaseCommand {
private singleCursor = 0
private releaseCursor = 0
/** Parts of header-derived binaries that are a file already (worker:file now drops them itself). */
/**
* Header parts of binaries that are a file already. worker:file drops them
* when it creates the file, and ingest no longer adds them afterwards; this
* catches what slipped through. A cursor over binaries (as for single
* articles) visits each one once per pass, instead of searching all of
* them for parts on every batch (LOG.md 2026-10-04).
*/
private copiedCursor = 0
private async dropCopiedParts(budget: number): Promise<number> {
let total = 0
while (Date.now() < budget) {
const result = await db.rawQuery(
// Header-derived parts (no per-part CRC) of binaries that are a file now.
`DELETE FROM parts WHERE pcrc32 IS NULL AND binary_id IN (
SELECT b.id FROM binaries b
WHERE b.file_id IS NOT NULL
AND EXISTS (SELECT 1 FROM parts p WHERE p.binary_id = b.id AND p.pcrc32 IS NULL)
LIMIT ?)`,
[500]
const { rows } = await db.rawQuery(
`WITH batch AS (
SELECT id FROM binaries WHERE id > ? AND file_id IS NOT NULL ORDER BY id LIMIT ?
), gone AS (
DELETE FROM parts p USING batch WHERE p.binary_id = batch.id AND p.pcrc32 IS NULL
RETURNING 1)
SELECT (SELECT max(id) FROM batch) AS last, (SELECT count(*) FROM gone) AS deleted`,
[this.copiedCursor, COPIED_PARTS_BATCH]
)
total += result.rowCount ?? 0
if (!result.rowCount) break
total += Number(rows[0].deleted)
if (rows[0].last === null) {
this.copiedCursor = 0
break
}
this.copiedCursor = Number(rows[0].last)
}
return total
}