Partition files by age (id)
New rows went into free space scattered through the 9.6 GB heap on the HDD (left behind by inspection's updates): one random HDD read per insert, and seed batches took a median 13 s to store. The existing table becomes the first partition as it is (a CHECK scan, no rewrite; indexes, its foreign key and the triggers are taken over by the partitioned parent); new partitions of 4M ids go in the same tablespace and stay in memory while they fill. - Every partition has a unique file_key index; FileStore.insert looks keys up in all partitions first (a key can't be unique across partitions). - FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead. - Stats count files from the partitions; seed:node waits ~20 minutes for the API before giving a range up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
41859636d3
commit
f1773744c0
8 files changed
+310
-8
No files matched your search
@@ -8,6 +8,7 @@ import RedisService from '#services/RedisService'
|
||||
import IndexMaintenance from '#services/IndexMaintenance'
|
||||
import ReleaseMergeService from '#services/ReleaseMergeService'
|
||||
import StagingService from '#services/StagingService'
|
||||
import FilePartitions from '#services/FilePartitions'
|
||||
import Shutdown from '#services/Shutdown'
|
||||
|
||||
const CLEAN_BATCH = 1000
|
||||
@@ -61,6 +62,9 @@ export default class GcScheduler extends BaseCommand {
|
||||
}
|
||||
}
|
||||
|
||||
// Inserts fail without a partition for their id (D39).
|
||||
if (!this.dryRun) await FilePartitions.ensure()
|
||||
|
||||
const trimmed = await this.trimBodyQueue()
|
||||
if (trimmed) this.logger.info(`Body queue: ${this.verb('drop', 'dropped')} ${trimmed} oldest waiting jobs.`)
|
||||
|
||||
|
||||
@@ -282,7 +282,8 @@ class SeedApi {
|
||||
if (status >= 400 && status < 500 && status !== 404)
|
||||
throw new Error(`Seed API ${path}: ${status} ${text()}`)
|
||||
// Network errors, 404 and 5xx: storing a batch twice is harmless, so try again.
|
||||
if (attempt >= 8) throw new Error(`Seed API ${path}: ${status || 'no response'} ${text()}`)
|
||||
// ~20 minutes in all, longer than a lease lasts: a restart or maintenance of the API is waited out.
|
||||
if (attempt >= 14) throw new Error(`Seed API ${path}: ${status || 'no response'} ${text()}`)
|
||||
const wait = Math.min(120_000, 2000 * 2 ** (attempt - 1))
|
||||
this.warn(`Seed API ${path}: ${status || text()}; retrying in ${wait / 1000} s`)
|
||||
await sleep(wait)
|
||||
|
||||
Reference in new issue
Block a user