Partition files by age (id)
Test and deploy / test (push) Successful in 54s
Test and deploy / deploy (push) Successful in 7m42s

New rows went into free space scattered through the 9.6 GB heap on the HDD
(left behind by inspection's updates): one random HDD read per insert, and
seed batches took a median 13 s to store. The existing table becomes the
first partition as it is (a CHECK scan, no rewrite; indexes, its foreign
key and the triggers are taken over by the partitioned parent); new
partitions of 4M ids go in the same tablespace and stay in memory while
they fill.

- Every partition has a unique file_key index; FileStore.insert looks keys
  up in all partitions first (a key can't be unique across partitions).
- FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead.
- Stats count files from the partitions; seed:node waits ~20 minutes for
  the API before giving a range up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-05 15:57:51 +02:00
1 parent 41859636d3
commit f1773744c0
8 files changed
+310 -8

No files matched your search

+2 -1
View File
@@ -282,7 +282,8 @@ class SeedApi {
if (status >= 400 && status < 500 && status !== 404)
throw new Error(`Seed API ${path}: ${status} ${text()}`)
// Network errors, 404 and 5xx: storing a batch twice is harmless, so try again.
if (attempt >= 8) throw new Error(`Seed API ${path}: ${status || 'no response'} ${text()}`)
// ~20 minutes in all, longer than a lease lasts: a restart or maintenance of the API is waited out.
if (attempt >= 14) throw new Error(`Seed API ${path}: ${status || 'no response'} ${text()}`)
const wait = Math.min(120_000, 2000 * 2 ** (attempt - 1))
this.warn(`Seed API ${path}: ${status || text()}; retrying in ${wait / 1000} s`)
await sleep(wait)