file_keys: fill in the background, use it once complete
Copying all existing keys in the migration read the files rows from the HDD for over half an hour and held up the deploy. gc:scheduler now copies them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks every partition until the fill is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
693e6c3244
commit
f0ff8e200f
8 files changed
+163
-18
No files matched your search
@@ -1,4 +1,5 @@
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import FileKeys from '#services/FileKeys'
|
||||
import type { QueryClientContract } from '@adonisjs/lucid/types/database'
|
||||
import { packSegments, unpackSegments, type Segment } from '#lib/nzb/segments'
|
||||
|
||||
@@ -44,8 +45,8 @@ class FileStore {
|
||||
* Inserts files, skipping those whose `file_key` exists. Returns the rows
|
||||
* that were inserted (id, filename, file_key). `files` is partitioned by id
|
||||
* (CONCEPT.md D39) and each partition has its own unique file_key index:
|
||||
* file_keys (a 64-bit hash of every key, filled by the insert trigger) says
|
||||
* which keys may exist, and only those are looked up in the partitions; two
|
||||
* once file_keys (a 64-bit hash of every key) is complete, it says which keys
|
||||
* may exist, and only those are looked up in the partitions; two
|
||||
* writers adding the same new key at once meet in the newest partition's index.
|
||||
*/
|
||||
async insert(
|
||||
@@ -55,8 +56,14 @@ class FileStore {
|
||||
if (files.length === 0) return []
|
||||
const keys = [...new Set(files.map((file) => file.file_key).filter((key): key is string => typeof key === 'string'))]
|
||||
const known = new Set<string>()
|
||||
const filtered = await FileKeys.complete()
|
||||
for (let i = 0; i < keys.length; i += 5000) {
|
||||
const chunk = keys.slice(i, i + 5000)
|
||||
if (!filtered) {
|
||||
const found = await client.knexQuery().from('files').whereIn('file_key', chunk).select('file_key')
|
||||
for (const row of found) known.add(row.file_key)
|
||||
continue
|
||||
}
|
||||
const { rows } = await client.rawQuery(
|
||||
`SELECT k.key FROM unnest(?::text[]) AS k(key)
|
||||
WHERE EXISTS (SELECT 1 FROM file_keys f WHERE f.key_hash = hashtextextended(k.key, 0))`,
|
||||
|
||||
Reference in new issue
Block a user