file_keys: fill in the background, use it once complete
Test and deploy / test (push) Successful in 1m37s
Test and deploy / deploy (push) Successful in 2m23s

Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-06 14:54:12 +02:00
1 parent 693e6c3244
commit f0ff8e200f
8 files changed
+163 -18

No files matched your search

+9 -2
View File
@@ -1,4 +1,5 @@
import db from '@adonisjs/lucid/services/db'
import FileKeys from '#services/FileKeys'
import type { QueryClientContract } from '@adonisjs/lucid/types/database'
import { packSegments, unpackSegments, type Segment } from '#lib/nzb/segments'
@@ -44,8 +45,8 @@ class FileStore {
* Inserts files, skipping those whose `file_key` exists. Returns the rows
* that were inserted (id, filename, file_key). `files` is partitioned by id
* (CONCEPT.md D39) and each partition has its own unique file_key index:
* file_keys (a 64-bit hash of every key, filled by the insert trigger) says
* which keys may exist, and only those are looked up in the partitions; two
* once file_keys (a 64-bit hash of every key) is complete, it says which keys
* may exist, and only those are looked up in the partitions; two
* writers adding the same new key at once meet in the newest partition's index.
*/
async insert(
@@ -55,8 +56,14 @@ class FileStore {
if (files.length === 0) return []
const keys = [...new Set(files.map((file) => file.file_key).filter((key): key is string => typeof key === 'string'))]
const known = new Set<string>()
const filtered = await FileKeys.complete()
for (let i = 0; i < keys.length; i += 5000) {
const chunk = keys.slice(i, i + 5000)
if (!filtered) {
const found = await client.knexQuery().from('files').whereIn('file_key', chunk).select('file_key')
for (const row of found) known.add(row.file_key)
continue
}
const { rows } = await client.rawQuery(
`SELECT k.key FROM unnest(?::text[]) AS k(key)
WHERE EXISTS (SELECT 1 FROM file_keys f WHERE f.key_hash = hashtextextended(k.key, 0))`,