Spotnet import, header backfill, staging and counter grouping, faster search

Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
  and NZBs; a spot renames its release, since Spotnet names are more
  trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
  have" from "needs a password we don't have".

Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
  and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
  uploads that were split across posts.

Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
  the NVMe or HDD part of the database gets too large.

Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
  their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
  part count and day.

Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
  per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
  requests. Count only up to 1,000 matches for the page, which made filter
  clicks take seconds (Newznab keeps the 10,000 cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

config/gc.ts

@@ -5,7 +5,7 @@ import env from '#start/env'
  */
 export default {
   /** How often `gc:scheduler` sweeps. */
-  intervalMs: env.get('GC_INTERVAL_MS', 10 * 60 * 1000),
+  intervalMs: env.get('GC_INTERVAL_MS', 3 * 60 * 1000),
 
   jobs: {
     /** Completed jobs are only kept for inspection; keep them briefly. */
@@ -34,14 +34,34 @@ export default {
      * complete from headers. A real file gets its next articles in minutes.
      */
     singleArticleAfterSeconds: env.get('GC_SINGLE_ARTICLE_SECONDS', 2 * 60 * 60),
+    /**
+     * Completed header-derived binaries without a fingerprint (size, MD5 of
+     * the first 16 KiB): their file exists and no par2 set can name them
+     * later, so they go sooner than staleAfterSeconds.
+     */
+    unfingerprintedAfterSeconds: env.get('GC_UNFINGERPRINTED_SECONDS', 6 * 60 * 60),
   },
 
+  /**
+   * Staged lone segments (CONCEPT.md D35) are dropped after this long. A
+   * real file's next segment comes in the same or the next fetch range,
+   * seconds to minutes later; at backfill speed two hours of waiting
+   * segments were 12M rows, 10 GB.
+   */
+  stagingTtlSeconds: env.get('STAGING_TTL_SECONDS', 30 * 60),
+
   /** Releases without files or binaries (their binaries were collected) are deleted after this long. */
   emptyReleaseAfterSeconds: env.get('GC_EMPTY_RELEASE_SECONDS', 2 * 60 * 60),
 
-  /** Bulk deletes run in batches of this size, for at most this long per sweep. */
+  /**
+   * Bulk deletes run in batches of this size, each kind (copied parts,
+   * single-article and unfingerprinted binaries, empty releases) for at most
+   * this long per sweep, so one backlog can't starve the others.
+   */
   deleteBatch: 20_000,
-  deleteBudgetMs: env.get('GC_DELETE_BUDGET_MS', 3 * 60 * 1000),
+  deleteBudgetMs: env.get('GC_DELETE_BUDGET_MS', 60 * 1000),
+  /** Bloated indexes are rebuilt at most this often (IndexMaintenance). */
+  reindexIntervalMs: env.get('GC_REINDEX_INTERVAL_MS', 7 * 24 * 60 * 60 * 1000),
 
   /**
    * Deferred files (CONCEPT.md D28) are deleted this long after they were

config/indexer.ts

@@ -39,6 +39,43 @@ export default {
    * binaries. Add one with `group:add free.pt --backfill 5000`.
    */
   spotGroups: ['free.pt'],
+  /**
+   * Spots posted within this many days are imported (their NZB read and its
+   * files added). Older spots are listed: title, size and date, with the NZB
+   * read on first download.
+   */
+  spotImportDays: env.get('SPOT_IMPORT_DAYS', 30),
+
+  /**
+   * Groups whose lone header segments wait in `staged_segments` until their
+   * file shows a second one (CONCEPT.md D35). Comma-separated; `*` for all.
+   */
+  stagingGroups: (env.get('STAGING_GROUPS', '') ?? '').split(',').map((g) => g.trim()).filter(Boolean),
+
+  /**
+   * Groups whose staged segments also group by poster, file counter `[i/M]`
+   * and part count (CONCEPT.md D36), for uploads with a random subject per
+   * article. Needs staging. Comma-separated; `*` for all.
+   */
+  counterGroupingGroups: (env.get('COUNTER_GROUPING_GROUPS', '') ?? '').split(',').map((g) => g.trim()).filter(Boolean),
+
+  /**
+   * Header backfill (`group:backfill`, CONCEPT.md D32): low-priority,
+   * header-only fetch ranges walking back through a group's history.
+   */
+  backfill: {
+    /** Backfill ranges waiting or running at a time, over all groups. */
+    maxQueued: env.get('BACKFILL_MAX_QUEUED', 40),
+    /** BullMQ priority (higher runs later; live ranges have none, so they go first). */
+    priority: 10,
+    /**
+     * No new backfill ranges while the database's NVMe part (default
+     * tablespace) or its HDD part (tablespace `hdd`, the files table) is
+     * larger than this. Checked every scheduler tick.
+     */
+    maxDefaultGb: env.get('BACKFILL_MAX_NVME_GB', 45),
+    maxHddGb: env.get('BACKFILL_MAX_HDD_GB', 150),
+  },
 
   /** worker:deferred leaves a file alone until it has had no new articles for this long. */
   deferredSettleMs: env.get('DEFERRED_SETTLE_MS', 15 * 60 * 1000),

config/nntp.ts

@@ -8,4 +8,6 @@ export default {
   secure: env.get('NNTP_SECURE', false),
   /** Connections per process. Mind the provider's limit across all processes. */
   poolSize: env.get('NNTP_POOL_SIZE', 10),
+  /** A command without an answer this long counts as a dead connection (XOVER of 20k headers takes ~0.2 s). */
+  commandTimeoutMs: env.get('NNTP_COMMAND_TIMEOUT_MS', 120_000),
 }