Partition files by age (id)
New rows went into free space scattered through the 9.6 GB heap on the HDD (left behind by inspection's updates): one random HDD read per insert, and seed batches took a median 13 s to store. The existing table becomes the first partition as it is (a CHECK scan, no rewrite; indexes, its foreign key and the triggers are taken over by the partitioned parent); new partitions of 4M ids go in the same tablespace and stay in memory while they fill. - Every partition has a unique file_key index; FileStore.insert looks keys up in all partitions first (a key can't be unique across partitions). - FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead. - Stats count files from the partitions; seed:node waits ~20 minutes for the API before giving a range up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
41859636d3
commit
f1773744c0
8 files changed
+310
-8
No files matched your search
@@ -390,7 +390,8 @@ class SearchService {
|
||||
SELECT r.name, r.size, r.file_count, r.encrypted, coalesce(r.source, 'unknown') AS source
|
||||
FROM releases r TABLESAMPLE SYSTEM (0.5) WHERE ${VISIBLE_SQL})
|
||||
SELECT (SELECT reltuples FROM pg_class WHERE relname = 'releases_category_idx') AS releases,
|
||||
(SELECT reltuples FROM pg_class WHERE relname = 'files') AS files,
|
||||
-- files is partitioned (D39): its partitions hold the counts.
|
||||
(SELECT sum(c.reltuples) FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid WHERE i.inhparent = 'files'::regclass) AS files,
|
||||
(SELECT count(*) FROM sample) AS sampled,
|
||||
(SELECT count(*) FROM sample WHERE is_readable_name(name)) AS sampled_readable,
|
||||
-- Indexed data only: a spot listing's size is what the spot announces.
|
||||
|
||||
Reference in new issue
Block a user