Stop refilling parts of binaries that are already a file
Crossposts, reposts and backfill ranges keep bringing articles of files that worker:file already created. Ingest inserted their header parts again, and GC's copied-parts pass (which searched all binaries for parts on every batch) removed only 150k-450k per sweep: parts grew to 68M rows, 56M of them belonging to finished files, and the 23 GB table paused the backfill. - Ingest skips header parts (no per-part CRC) for binaries with a file_id. - GC drops copied parts with a cursor over binaries, 2000 per statement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d3e774a489
commit
2178847513
3 files changed
+60
-14
No files matched your search
+24
-11
@@ -12,6 +12,8 @@ import Shutdown from '#services/Shutdown'
|
||||
|
||||
const CLEAN_BATCH = 1000
|
||||
const SCAN_COUNT = 1000
|
||||
/** Binaries per statement when dropping copied parts (a few hundred parts each at most). */
|
||||
const COPIED_PARTS_BATCH = 2000
|
||||
|
||||
export default class GcScheduler extends BaseCommand {
|
||||
public static commandName = 'gc:scheduler'
|
||||
@@ -173,21 +175,32 @@ export default class GcScheduler extends BaseCommand {
|
||||
private singleCursor = 0
|
||||
private releaseCursor = 0
|
||||
|
||||
/** Parts of header-derived binaries that are a file already (worker:file now drops them itself). */
|
||||
/**
|
||||
* Header parts of binaries that are a file already. worker:file drops them
|
||||
* when it creates the file, and ingest no longer adds them afterwards; this
|
||||
* catches what slipped through. A cursor over binaries (as for single
|
||||
* articles) visits each one once per pass, instead of searching all of
|
||||
* them for parts on every batch (LOG.md 2026-10-04).
|
||||
*/
|
||||
private copiedCursor = 0
|
||||
private async dropCopiedParts(budget: number): Promise<number> {
|
||||
let total = 0
|
||||
while (Date.now() < budget) {
|
||||
const result = await db.rawQuery(
|
||||
// Header-derived parts (no per-part CRC) of binaries that are a file now.
|
||||
`DELETE FROM parts WHERE pcrc32 IS NULL AND binary_id IN (
|
||||
SELECT b.id FROM binaries b
|
||||
WHERE b.file_id IS NOT NULL
|
||||
AND EXISTS (SELECT 1 FROM parts p WHERE p.binary_id = b.id AND p.pcrc32 IS NULL)
|
||||
LIMIT ?)`,
|
||||
[500]
|
||||
const { rows } = await db.rawQuery(
|
||||
`WITH batch AS (
|
||||
SELECT id FROM binaries WHERE id > ? AND file_id IS NOT NULL ORDER BY id LIMIT ?
|
||||
), gone AS (
|
||||
DELETE FROM parts p USING batch WHERE p.binary_id = batch.id AND p.pcrc32 IS NULL
|
||||
RETURNING 1)
|
||||
SELECT (SELECT max(id) FROM batch) AS last, (SELECT count(*) FROM gone) AS deleted`,
|
||||
[this.copiedCursor, COPIED_PARTS_BATCH]
|
||||
)
|
||||
total += result.rowCount ?? 0
|
||||
if (!result.rowCount) break
|
||||
total += Number(rows[0].deleted)
|
||||
if (rows[0].last === null) {
|
||||
this.copiedCursor = 0
|
||||
break
|
||||
}
|
||||
this.copiedCursor = Number(rows[0].last)
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user