Spotnet import, header backfill, staging and counter grouping, faster search

Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
  and NZBs; a spot renames its release, since Spotnet names are more
  trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
  have" from "needs a password we don't have".

Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
  and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
  uploads that were split across posts.

Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
  the NVMe or HDD part of the database gets too large.

Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
  their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
  part count and day.

Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
  per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
  requests. Count only up to 1,000 matches for the page, which made filter
  clicks take seconds (Newznab keeps the 10,000 cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

app/services/FileStore.ts

@@ -0,0 +1,55 @@
+import db from '@adonisjs/lucid/services/db'
+import type { QueryClientContract } from '@adonisjs/lucid/types/database'
+import { packSegments, unpackSegments, type Segment } from '#lib/nzb/segments'
+
+/**
+ * The only code that knows how `files` keeps its segment lists (CONCEPT.md
+ * D31): compressed in `files.segments`, with the first segment's message-id
+ * in the indexed `first_message_id` for lookups. Read them with
+ * `segmentsOf(file)`; write new files with `FileStore.insert`.
+ */
+
+/** A new `files` row, with its segments as a list. */
+export type NewFile = Record<string, unknown> & { segments: Segment[] }
+
+const unpacked = new WeakMap<object, Segment[]>()
+
+/** The segments of a `files` row (a row from `db.from('files')`), in order. */
+export function segmentsOf(file: { segments?: Buffer | Segment[] | null }): Segment[] {
+  const { segments } = file
+  if (!segments) return []
+  if (Array.isArray(segments)) return segments
+  let list = unpacked.get(file)
+  if (!list) {
+    list = unpackSegments(segments)
+    unpacked.set(file, list)
+  }
+  return list
+}
+
+/** Stored form of a segment list, for the File model's column. */
+export function storedSegments(segments: Segment[]) {
+  return packSegments(segments)
+}
+
+export function loadedSegments(stored: Buffer | null): Segment[] {
+  return stored ? unpackSegments(stored) : []
+}
+
+class FileStore {
+  /**
+   * Inserts files, skipping those whose `file_key` exists. Returns the rows
+   * that were inserted (id and filename).
+   */
+  async insert(files: NewFile[], client: Pick<QueryClientContract, 'knexQuery'> = db): Promise<{ id: number; filename: string }[]> {
+    if (files.length === 0) return []
+    const rows = files.map(({ segments, ...file }) => {
+      const sorted = [...segments].sort((a, b) => a.number - b.number)
+      return { ...file, segments: packSegments(sorted), first_message_id: sorted[0]?.id ?? null }
+    })
+    const inserted = await client.knexQuery().table('files').insert(rows).onConflict('file_key').ignore().returning(['id', 'filename'])
+    return inserted.map((row: any) => ({ id: Number(row.id), filename: row.filename }))
+  }
+}
+
+export default new FileStore()