Spotnet import, header backfill, staging and counter grouping, faster search

Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
  and NZBs; a spot renames its release, since Spotnet names are more
  trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
  have" from "needs a password we don't have".

Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
  and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
  uploads that were split across posts.

Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
  the NVMe or HDD part of the database gets too large.

Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
  their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
  part count and day.

Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
  per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
  requests. Count only up to 1,000 matches for the page, which made filter
  clicks take seconds (Newznab keeps the 10,000 cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

commands/workers/Spots.ts

@@ -6,6 +6,10 @@ import NntpService from '#services/NntpService'
 import SpotService from '#services/SpotService'
 
 const IDLE_MS = 30_000
+/** Spots claimed at a time; most of a history backfill is listed in bulk. */
+const BATCH = 500
+/** Imported spots whose details are read per idle round. */
+const DETAILS_BATCH = 50
 
 /**
  * Works through queued Spotnet spots (CONCEPT.md D29): verifies each one,
@@ -29,17 +33,22 @@ export default class SpotsWorker extends BaseCommand {
     let done = 0
     try {
       while (!Shutdown.requested && (!this.limit || done < this.limit)) {
-        const batch = await SpotService.claim(Math.min(concurrency * 4, this.limit ? this.limit - done : Infinity))
+        const batch = await SpotService.claim(Math.min(BATCH, this.limit ? this.limit - done : Infinity))
         if (batch.length === 0) {
           if (this.limit) break
-          await Shutdown.sleep(IDLE_MS)
+          // Idle: read the details of imported spots that don't have them yet (newest first).
+          if ((await SpotService.fillDetails(DETAILS_BATCH)) === 0) await Shutdown.sleep(IDLE_MS)
           continue
         }
+        // Old, verified spots are listed in bulk; the rest need their header (and maybe NZB) read.
+        const listable = batch.filter((spot) => SpotService.isListable(spot))
+        const single = batch.filter((spot) => !SpotService.isListable(spot))
+        counts.listed = (counts.listed ?? 0) + (await SpotService.list(listable))
         let next = 0
         await Promise.all(
           Array.from({ length: concurrency }, async () => {
-            while (next < batch.length) {
-              const status = await SpotService.process(batch[next++])
+            while (next < single.length) {
+              const status = await SpotService.process(single[next++])
               counts[status] = (counts[status] ?? 0) + 1
             }
           })