releases:recategorize: a camelCase name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

REBUILD.md

@@ -189,13 +189,22 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
    number; merged when the neighbouring range is done, then written as a
    chunk. Never staged per segment.
 
-8. **Settling**. A release is settled when the ranges covering its posting
+8. **Settling**. A release is settled when nothing was added to it for 6
-   span are done and no chunk arrived for a day: completeness is computed
+   hours (30 minutes once its last post is that old: history), and its
-   then (expected vs found files and segments). Phase 2 picks its work from
+   completeness decided: complete when every file its [i/N] counter
-   settled releases.
+   announced was found. Without a counter, by its par2 sets once their index
+   is probed (`par2_completeness`): every file a set describes is there by
+   name or by its first 16 KiB, or, renamed, as many files and bytes; decided
+   again after a par2 merge and before the pruner drops a set's file list.
+   Segments aren't compared. Phase 2 picks its work from settled releases.
 
-9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
+9. **NZB and API**. NZB built from a release's chunks, with a known
-   search (names) and get. Web UI later.
+   password in its meta; Newznab search (names, inner names) and get; a web
+   interface. Categories (Newznab's) by a trigger on releases (migration
+   5200): from the name, else the inner name the prober found, else a spot's
+   category (spot_category, set by the spot inspector), else the groups
+   (erotica, sounds, ebooks, …). `releases:recategorize` recomputes old rows
+   in batches.
 
 ## Group list
 
@@ -238,6 +247,14 @@ files the set describes, into the one with the most files (chunks,
 candidates, inspections, counts; the par2's shared name when none was
 readable); `release_merges` keeps what went where. Obfuscated uploads whose
 files were each a release of their own (`lone` candidates) become one.
+It reads what's new an hour (`created_at`) at a time, its place kept after
+each, and merges only the sets held or matched by two releases or more; a
+file whose first 16 KiB and size more than 20 files share is a common one and
+matches nothing (LOG.md 2026-10-20 (85)).
+A set's file list is dropped once it can't merge anything more: complete (all
+its files in its holder), judged (every holder junk), or final (every holder
+inside a run of done ranges, those ranges' candidates probed, its hashes seen
+by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).
 
 **Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
 voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
@@ -252,7 +269,7 @@ with 68,032 releases with a password):
 | Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
 | SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
 | Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
-| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
+| 7z listings | D19 | yes (2026-10-23): the prober reads the `7z-tail` (the last segment of the highest volume, or of an unsplit multi-segment .7z) and `lib/sevenzip` lists it from the archive's end header (LZMA-packed, decoded in TypeScript); an encrypted header (`-mhe`) means passworded without names. One release keeps only the tails of its highest volume (`seven_zip_tails`). ZIP: the first segment's local headers (first entries, encryption) |
 | Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
 | Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |
 
@@ -260,9 +277,11 @@ Live indexing (the tail) runs since 2026-10-09.
 
 **Junk** (voyager 1's second filter, D42). After the header pass's rules
 (garbage signatures, executables, per-article and obfuscated subjects), the
-judge (`releases:judge`, service `judge`) judges releases hourly by what the
+judge (`releases:judge`, service `judge`) judges releases by what the
-prober, the merges and the spots found, in this order (the first rule
+prober, the merges and the spots found: every 2 minutes the releases whose
-wins): *dead* (every probed first article gone), *fake* (a film, episode
+facts changed (noted by triggers in `judge_dirty`), daily every release (for
+what time changes: a day after settling, ranges read beside it). In this
+order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
 or music name whose content is a program: archive entries all
 executables, or the file itself one by name or by its first bytes, sniffed
 `exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
@@ -326,3 +345,18 @@ f. Random-named releases (no file with a readable name) keep no chunk in
    phase 2 names them (bodies) and fetches their segments again. The full
    history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
    kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
+g. The account's NNTP connections shared by priority (`nntp_shares`): the
+   coordinator (headers) before the prober, each between its minimum and
+   maximum, 5 kept for the spotter, the spot inspector, the plain
+   connection and reconnects. A process grows only into connections the
+   others closed, so the account never goes over its limit. Replaces the
+   fixed split (38 + 8), under which the prober stayed at 8 while history
+   was paused and the coordinator's connections idle (LOG.md 2026-10-18
+   (75)). **Accepted** 2026-10-18.
+h. Idle NNTP connections closed: one with nothing to do for
+   `NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
+   (the coordinator's pool on its next sync, the plain one too; a dry
+   prober lane; the spotter between rounds; the spot inspector with no
+   spots). Eweka dropped the quiet ones and the next command failed
+   ("Spot inspection failed: NNTP connection closed"), and idle ones took
+   the account's connections from downloads. **Accepted** 2026-10-22.

app/services/batch_store.ts

@@ -1,8 +1,20 @@
 import db from '@adonisjs/lucid/services/db'
+import { pruneSevenZipTails } from '#services/seven_zip_tails'
 import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
 import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
 import ChunkPartitions from '#services/chunk_partitions'
 import { decodeChunk } from '#lib/chunks/index'
+import { SIDECAR } from '#services/sidecar_folder'
+
+/**
+ * A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
+ * fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
+ */
+function sidecarOnly(r: ReleaseChunk): boolean | null {
+  if (r.files > 3 || r.bytes >= 1_000_000) return null
+  const files = decodeChunk(r.payload)
+  return files.length > 0 && files.every((f) => SIDECAR.test(f.name))
+}
 
 export class StaleBatch extends Error {}
 
@@ -225,8 +237,9 @@ class BatchStore {
              last_article = greatest(r.last_article, a.last_article),
              readable = r.readable OR a.readable,
              has_par2 = r.has_par2 OR a.has_par2,
-             -- As it stands (the counter's files all found); settling makes it final.
+             -- As it stands (the counter's files all found); settling makes it final. Without a counter, what its par2
-             complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN NULL ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
+             -- set decided stays (app/services/par2_completeness.ts).
+             complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN r.complete ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
              last_seen_at = now()
            FROM a WHERE r.id = a.id`,
           [JSON.stringify(updates)]
@@ -235,13 +248,21 @@ class BatchStore {
       }
       const fresh = part.filter((_, idx) => !joined.has(idx))
       if (fresh.length > 0) {
-        // The first upload of a key keeps the key; later ones are key:2, key:3, …
+        // The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
-        const { rows: counts } = await trx.rawQuery(`SELECT identity_key, count(*) AS n FROM releases WHERE identity_key = ANY(?) GROUP BY 1`, [[...new Set(fresh.map((r) => r.key))]])
+        // delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
+        // taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
+        const { rows: counts } = await trx.rawQuery(
+          `SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
+                                         WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
+                                         THEN substr(release_key, length(identity_key) + 2)::int END) AS n
+           FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
+          [[...new Set(fresh.map((r) => r.key))]]
+        )
         const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
         const numbered = (r: ReleaseChunk) => {
           const n = (taken.get(r.key) ?? 0) + 1
           taken.set(r.key, n)
-          return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r) }
+          return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
         }
         // Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
         const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
@@ -252,10 +273,10 @@ class BatchStore {
         for (let attempt = 0; inserts.length > 0; attempt++) {
           if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
           const { rows } = await trx.rawQuery(
-            `INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete)
+            `INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
              SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
-                    v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END
+                    v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
-             FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)
+             FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
              ON CONFLICT (release_key) DO NOTHING
              RETURNING id, release_key`,
             [groupId, JSON.stringify(inserts)]
@@ -305,6 +326,18 @@ class BatchStore {
         .ignore()
         .returning('id')
       out.candidates += added.length
+      // A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
+      const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
+      if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
+      // Files added to a settled release (a later range of its upload): the settler made its candidates claimable
+      // already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
+      if (added.length > 0) {
+        await trx.rawQuery(
+          `UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
+           WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
+          [added.map((a: { id: string | number }) => String(a.id))]
+        )
+      }
     }
 
     return out

app/services/disk_space.ts

@@ -0,0 +1,40 @@
+import { statfs } from 'node:fs/promises'
+import logger from '@adonisjs/core/services/logger'
+
+/** How often free space is measured. */
+const CHECK_MS = 60_000
+/** Resumed only this much above the floor: not paused and resumed at every flush. */
+const MARGIN_GB = 1
+
+/**
+ * Free space on the filesystem holding `path`, against a floor: low below it, again fine MARGIN_GB above it. In a
+ * container `/` is an overlay on the host's Docker root, which on homelab is the NVMe the catalog and the WAL are on.
+ * The prober filled it ~0.8–1.1 GB an hour at ~200 articles/s, and nothing stopped it at the 10 GB floor (LOG.md
+ * 2026-10-18 (77)). Off when the floor is 0.
+ */
+export class DiskSpace {
+  private checkedAt = 0
+  private low = false
+
+  constructor(
+    private what: string,
+    private path: string,
+    private floorGb: number,
+    private free: (path: string) => Promise<number> = async (p) => {
+      const s = await statfs(p)
+      return s.bavail * s.bsize
+    },
+    private checkMs = CHECK_MS
+  ) {}
+
+  async short(): Promise<boolean> {
+    if (!this.floorGb) return false
+    if (Date.now() - this.checkedAt < this.checkMs) return this.low
+    this.checkedAt = Date.now()
+    const gb = (await this.free(this.path)) / 1e9
+    const was = this.low
+    this.low = gb < this.floorGb + (was ? MARGIN_GB : 0)
+    if (this.low !== was) logger.info(`${gb.toFixed(1)} GB free (floor ${this.floorGb} GB): ${this.what} ${this.low ? 'paused' : 'resumed'}.`)
+    return this.low
+  }
+}

app/services/leases.ts

@@ -52,6 +52,22 @@ class Leases {
     return { range: { id: Number(row.id), group: group.name, start: Number(row.start_id), end: Number(row.end_id), next: Number(row.next_id) }, rules }
   }
 
+  /**
+   * Ranges read to their end whose worker went quiet before finishing them: done. The lease query skips a range past
+   * its end, so one whose finish never came stayed leased for good (a gap range since 2026-10-16 14:03, during the lock
+   * pile-up; LOG.md 2026-10-17 (72)). Its batches are all stored; its spooled chunks go.
+   */
+  async finishAbandoned(): Promise<number> {
+    const { rows } = await db.rawQuery(
+      `UPDATE ranges SET status = 'done', done_at = now()
+       WHERE status = 'leased' AND next_id > end_id AND heartbeat_at < now() - make_interval(secs => ?)
+       RETURNING id`,
+      [coordinatorConfig.leaseTtlSeconds]
+    )
+    for (const { id } of rows) await Spool.remove(Number(id))
+    return rows.length
+  }
+
   async leased(rangeId: number, nodeId: number) {
     const range = await db.from('ranges').where('id', rangeId).first()
     if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId) throw new LeaseLost(`Range ${rangeId} is not leased to this node`)

app/services/nntp_pool.ts

@@ -2,12 +2,18 @@ import env from '#start/env'
 import logger from '@adonisjs/core/services/logger'
 import coordinatorConfig from '#config/coordinator'
 import { XoverClient } from '#lib/nntp/index'
+import type { NntpShare } from '#services/nntp_shares'
+
+/** Connections wanted beyond the most in use at once since the last sync (the planner's, a hedge's). */
+const SHARE_MARGIN = 2
 
 /**
  * The coordinator's header connections (REBUILD.md, decision e: workers
  * have no NNTP account). At most NNTP_CONNECTIONS, reused; a connection
  * that failed, or is older than NNTP_CONNECTION_MAX_MINUTES, is closed and
- * replaced on the next use.
+ * replaced on the next use. Sharing the account (share()), at most its
+ * share: over it, connections close as they come free. One left free for
+ * NNTP_IDLE_CLOSE_SECONDS is closed on the next sync.
  */
 class NntpPool {
   private idle: XoverClient[] = []
@@ -17,9 +23,43 @@ class NntpPool {
   private openedAt = new Map<XoverClient, number>()
   /** When a connection gets closed once free: its max age ±25%, so they don't all reconnect at once. */
   private retireAt = new Map<XoverClient, number>()
+  /** When an idle connection came free. */
+  private idleSince = new Map<XoverClient, number>()
+  private limit = coordinatorConfig.nntpConnections
+  /** Since the last sync: the most connections in use at once, and whether work waited for one (starved()). */
+  private peak = 0
+  private short = false
 
   get size() {
-    return coordinatorConfig.nntpConnections
+    return this.limit
+  }
+
+  /** Work that could use another connection didn't get one (the prefetcher with all its loops running). */
+  starved() {
+    this.short = true
+  }
+
+  /**
+   * Tells the share what the pool could use and has open, and takes the share's grant as its size. After work waited:
+   * twice its size (history going on gets its maximum in a few syncs; a tail round, done in seconds, doesn't take
+   * the prober's connections for one); else the most in use at once since the last sync and a margin.
+   */
+  async share(share: NntpShare) {
+    const want = this.short || this.waiting.length > 0 ? Math.max(this.limit * 2, this.limit + SHARE_MARGIN) : this.peak + SHARE_MARGIN
+    this.short = false
+    this.peak = this.busy.size
+    const granted = await share.sync(want, this.open)
+    const was = this.limit
+    this.limit = granted
+    // Over the share: the idle ones close now, the busy ones once free (release()).
+    while (this.open > this.limit && this.idle.length > 0) this.retire(this.idle.pop()!)
+    this.closeIdle()
+    // Under it with work waiting: connections for it.
+    while (this.waiting.length > 0 && this.open < this.limit) {
+      const next = this.waiting.shift()!
+      this.acquire().then(next, () => this.waiting.unshift(next))
+    }
+    return { was, granted }
   }
 
   async use<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
@@ -44,6 +84,7 @@ class NntpPool {
 
   private async run<T>(client: XoverClient, work: (client: XoverClient) => Promise<T>): Promise<T> {
     this.busy.add(client)
+    this.peak = Math.max(this.peak, this.busy.size)
     let broken = false
     try {
       return await work(client)
@@ -56,9 +97,28 @@ class NntpPool {
     }
   }
 
+  /** Closes the connections free for nntpIdleCloseMs, the plain one too. */
+  private closeIdle() {
+    const idleMs = coordinatorConfig.nntpIdleCloseMs
+    if (idleMs <= 0) return
+    const before = Date.now() - idleMs
+    for (const client of this.idle.filter((c) => (this.idleSince.get(c) ?? Infinity) <= before)) {
+      this.idle.splice(this.idle.indexOf(client), 1)
+      this.retire(client)
+    }
+    if (this.plain && this.plainPending === 0 && this.plainUsedAt <= before) {
+      void this.plain.then((c) => c.close()).catch(() => {})
+      this.plain = null
+    }
+  }
+
   private async acquire(): Promise<XoverClient> {
+    // The one used last: the others stay free long enough to close.
     const ready = this.idle.pop()
-    if (ready) return ready
+    if (ready) {
+      this.idleSince.delete(ready)
+      return ready
+    }
     if (this.open < this.size) {
       this.open++
       try {
@@ -84,15 +144,12 @@ class NntpPool {
   }
 
   private release(client: XoverClient, broken: boolean) {
+    if (this.open > this.limit) {
+      this.retire(client)
+      return
+    }
     if (broken || Date.now() >= (this.retireAt.get(client) ?? Infinity)) {
-      this.open--
+      this.retire(client)
-      this.openedAt.delete(client)
-      this.retireAt.delete(client)
-      try {
-        client.close()
-      } catch (error: any) {
-        logger.debug(`Closing a failed NNTP connection: ${error.message}`)
-      }
       // Someone waiting gets a fresh connection.
       const next = this.waiting.shift()
       if (next) this.acquire().then(next, () => this.waiting.unshift(next))
@@ -100,12 +157,30 @@ class NntpPool {
     }
     const next = this.waiting.shift()
     if (next) next(client)
-    else this.idle.push(client)
+    else {
+      this.idle.push(client)
+      this.idleSince.set(client, Date.now())
+    }
+  }
+
+  private retire(client: XoverClient) {
+    this.open--
+    this.openedAt.delete(client)
+    this.retireAt.delete(client)
+    this.idleSince.delete(client)
+    try {
+      client.close()
+    } catch (error: any) {
+      logger.debug(`Closing an NNTP connection: ${error.message}`)
+    }
   }
 
   /** Closes every connection, also those in use: their commands fail. */
   private plain: Promise<XoverClient> | null = null
   private plainQueue: Promise<unknown> = Promise.resolve()
+  /** Commands queued or running on the plain connection, and when the last one ended. */
+  private plainPending = 0
+  private plainUsedAt = 0
 
   /**
    * One connection without XFEATURE COMPRESS GZIP, opened when first needed,
@@ -113,6 +188,7 @@ class NntpPool {
    * cuts short (Leases.corrupt). Its replies are deflated here.
    */
   async usePlain<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
+    this.plainPending++
     const run = this.plainQueue.then(async () => {
       this.plain ??= XoverClient.connect({
         host: env.get('NNTP_HOST')!,
@@ -132,7 +208,10 @@ class NntpPool {
         throw error
       }
     })
-    this.plainQueue = run.catch(() => {})
+    this.plainQueue = run.catch(() => {}).finally(() => {
+      this.plainPending--
+      this.plainUsedAt = Date.now()
+    })
     return run
   }
 
@@ -143,6 +222,7 @@ class NntpPool {
     this.busy.clear()
     this.openedAt.clear()
     this.retireAt.clear()
+    this.idleSince.clear()
     this.open = 0
   }
 }

app/services/nntp_shares.ts

@@ -0,0 +1,90 @@
+import db from '@adonisjs/lucid/services/db'
+import coordinatorConfig from '#config/coordinator'
+
+/** A share whose process hasn't synced for this long is gone: its connections closed with it. */
+const STALE_SECONDS = 60
+/** Syncs one at a time (pg_advisory_xact_lock). */
+const LOCK = 2_026_101_875
+
+export interface Share {
+  name: string
+  priority: number
+  /** Kept whatever the others want (as far as the total goes, by priority). */
+  min: number
+  /** What it could use now. */
+  want: number
+  /** Connections it has open (or may have: a lane without one yet counts). */
+  open: number
+}
+
+/** What each share should have: its minimum first, then by priority what it wants, up to the total. */
+export function targets(total: number, shares: Share[]): Map<string, number> {
+  const ordered = [...shares].sort((a, b) => b.priority - a.priority || a.name.localeCompare(b.name))
+  const target = new Map<string, number>()
+  let left = total
+  for (const s of ordered) {
+    const t = Math.min(s.min, left)
+    target.set(s.name, t)
+    left -= t
+  }
+  for (const s of ordered) {
+    const more = Math.min(Math.max(s.want - target.get(s.name)!, 0), left)
+    target.set(s.name, target.get(s.name)! + more)
+    left -= more
+  }
+  return target
+}
+
+/**
+ * The connections `self` may have: its target, but no more than the others leave over. Another share holds what it
+ * has open or was granted, whichever is more (it may open up to its grant until its next sync tells it otherwise),
+ * so the account never goes over the total: a share grows only into what the others closed or gave up.
+ */
+export function grant(total: number, self: Share, others: (Share & { granted: number })[]): number {
+  const target = targets(total, [self, ...others]).get(self.name)!
+  const held = others.reduce((sum, o) => sum + Math.max(o.open, o.granted), 0)
+  return Math.max(0, Math.min(target, total - held))
+}
+
+/**
+ * A process's share of the NNTP account's connections (LOG.md 2026-10-18 (75)): the coordinator and the prober had a
+ * fixed split, and with history paused the coordinator's 38 stood idle while the prober's 8 were its limit. Each
+ * syncs every nntpShares.everyMs (what it wants and has open) and gets what it may open; one wanting less closes
+ * connections as they come free, and the other grows as those show up closed.
+ */
+export class NntpShare {
+  granted = 0
+
+  constructor(
+    readonly name: string,
+    private priority: number,
+    readonly min: number,
+    readonly max: number
+  ) {}
+
+  async sync(want: number, open: number): Promise<number> {
+    const self: Share = { name: this.name, priority: this.priority, min: this.min, want: Math.min(Math.max(want, this.min), this.max), open }
+    const total = coordinatorConfig.nntpShares.total
+    this.granted = await db.transaction(async (trx) => {
+      await trx.rawQuery(`SELECT pg_advisory_xact_lock(?)`, [LOCK])
+      const { rows } = await trx.rawQuery(
+        `SELECT name, priority, min, want, open, granted FROM nntp_shares WHERE name <> ? AND heartbeat_at > now() - make_interval(secs => ?)`,
+        [this.name, STALE_SECONDS]
+      )
+      const granted = grant(total, self, rows)
+      await trx.rawQuery(
+        `INSERT INTO nntp_shares (name, priority, min, want, open, granted, heartbeat_at) VALUES (?, ?, ?, ?, ?, ?, now())
+         ON CONFLICT (name) DO UPDATE SET priority = excluded.priority, min = excluded.min, want = excluded.want,
+           open = excluded.open, granted = excluded.granted, heartbeat_at = now()`,
+        [this.name, this.priority, this.min, self.want, open, granted]
+      )
+      return granted
+    })
+    return this.granted
+  }
+
+  /** Stopping: its connections are closed, the others needn't wait for the row to go stale. */
+  async leave() {
+    await db.from('nntp_shares').where('name', this.name).delete()
+  }
+}

app/services/nzb_service.ts

@@ -3,15 +3,26 @@ import db from '@adonisjs/lucid/services/db'
 import { decodeChunk, type ChunkFile } from '#lib/chunks/index'
 import { buildNzb } from '#lib/nzb/index'
 
+/** Where a known password came from, the most trusted first: the spot's NZB meta, its title, its text. */
+const PASSWORD_SOURCES = ['nzb', 'spot-title', 'spot-text']
+
 /**
  * A release's files and NZB (REBUILD.md, part 9), from its chunks in the
  * chunk store: every chunk decoded, its files in name order. A file stored
  * twice (the same name and first article, e.g. a range read again) is taken
- * once.
+ * once. A known password (release_passwords) goes into the NZB's meta, where
+ * SABnzbd and NZBGet take it for unpacking.
  */
 class NzbService {
-  async files(releaseId: number): Promise<ChunkFile[]> {
+  /** The release's known passwords, the most trusted first. */
-    const chunks = await db.from('release_chunks').where('release_id', releaseId).orderBy('id').select('payload')
+  async passwords(releaseId: number): Promise<{ password: string; source: string }[]> {
+    const rows = await db.from('release_passwords').where('release_id', releaseId).orderBy('created_at').select('password', 'source')
+    const rank = (source: string) => (PASSWORD_SOURCES.indexOf(source) + PASSWORD_SOURCES.length + 1) % (PASSWORD_SOURCES.length + 1)
+    return rows.sort((a, b) => rank(a.source) - rank(b.source)).map((r) => ({ password: r.password, source: r.source }))
+  }
+
+  async files(releaseId: number, client: any = db): Promise<ChunkFile[]> {
+    const chunks = await client.from('release_chunks').where('release_id', releaseId).orderBy('id').select('payload')
     const files = new Map<string, ChunkFile>()
     for (const chunk of chunks) {
       for (const file of decodeChunk(chunk.payload)) files.set(`${file.name}\n${file.segments[0]?.[1] ?? ''}`, file)
@@ -22,10 +33,11 @@ class NzbService {
   async build(releaseId: number): Promise<{ name: string; xml: string } | null> {
     const release = await db.from('releases').where('id', releaseId).select('name', 'source').first()
     if (!release) return null
+    const password = (await this.passwords(releaseId))[0]?.password
     // A release a spot made of its own (no files matched the header pass's): the spot's NZB as read.
     if (release.source === 'spot') {
       const spot = await db.from('spots').where('release_id', releaseId).whereNotNull('nzb').select('nzb').first()
-      if (spot) return { name: release.name, xml: gunzipSync(spot.nzb).toString('utf8') }
+      if (spot) return { name: release.name, xml: withPassword(gunzipSync(spot.nzb).toString('utf8'), password) }
     }
     const xml = buildNzb(
       (await this.files(releaseId)).map((f) => ({
@@ -35,10 +47,18 @@ class NzbService {
         groups: f.groups,
         segments: f.segments.map(([number, id, size]) => ({ number, id, size })),
       })),
-      { title: release.name }
+      { title: release.name, password }
     )
     return { name: release.name, xml }
   }
 }
 
+/** A read NZB with the password in its meta, when it has none (the spot's title or text named it, not its NZB). */
+export function withPassword(xml: string, password: string | undefined): string {
+  if (!password || /<meta\s+type=["']password["']/i.test(xml)) return xml
+  const meta = `<meta type="password">${password.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;')}</meta>`
+  if (/<head\b[^>]*>/i.test(xml)) return xml.replace(/<head\b[^>]*>/i, (head) => `${head}${meta}`)
+  return xml.replace(/<nzb\b[^>]*>/i, (nzb) => `${nzb}<head>${meta}</head>`)
+}
+
 export default new NzbService()

app/services/par2_completeness.ts

@@ -0,0 +1,53 @@
+import NzbService from '#services/nzb_service'
+
+/** A file a par2 set describes (par2_files). */
+export interface SetFile {
+  name: string
+  length: number
+  hash16k: string
+}
+
+const base = (name: string) => name.replace(/^.*[\\/]/, '').toLowerCase()
+
+/**
+ * Whether a release holds every file its par2 sets describe (REBUILD.md, part 8): for one without an [i/N] counter,
+ * whose completeness the settler leaves unknown. A described file is there when a file of the release has its name,
+ * or a probed file of it its first 16 KiB and size (file_hashes). An upload renamed after its par2 was made (no
+ * described name among its files) is complete when it has as many files besides par2 as the set describes, and as
+ * many bytes (its articles' bytes, a little over the files' with yEnc).
+ */
+export function par2Complete(set: SetFile[], files: { name: string; size: number }[], hashes: { hash16k: string; size: number }[]): boolean {
+  const names = new Set(files.map((f) => base(f.name)))
+  const hashed = new Set(hashes.map((h) => `${h.hash16k}:${h.size}`))
+  const byName = set.filter((f) => names.has(base(f.name))).length
+  if (set.every((f) => names.has(base(f.name)) || hashed.has(`${f.hash16k}:${f.length}`))) return true
+  if (byName > 0) return false
+  const data = files.filter((f) => !/\.par2$/i.test(f.name))
+  return data.length >= set.length && data.reduce((n, f) => n + f.size, 0) >= set.reduce((n, f) => n + f.length, 0)
+}
+
+/**
+ * Decides `complete` for the settled releases among `releaseIds` that have no counter and a par2 set with its file
+ * list (par2_files; the pruner drops it once the set is done, so it decides before). How many it decided. `client`:
+ * a transaction that moved chunks (a merge) reads them as moved.
+ */
+export async function decidePar2Completeness(client: any, releaseIds: (number | string)[]): Promise<number> {
+  if (releaseIds.length === 0) return 0
+  const { rows: releases } = await client.rawQuery(
+    `SELECT id FROM releases WHERE id = ANY(?) AND file_total IS NULL AND settled_at IS NOT NULL`,
+    [releaseIds.map(String)]
+  )
+  let decided = 0
+  for (const { id } of releases) {
+    const { rows: set } = await client.rawQuery(
+      `SELECT DISTINCT f.file_id, f.name, f.length::float8 AS length, f.hash16k FROM par2_sets s JOIN par2_files f ON f.set_id = s.set_id WHERE s.release_id = ?`,
+      [id]
+    )
+    if (set.length === 0) continue
+    const { rows: hashes } = await client.rawQuery(`SELECT hash16k, size::float8 AS size FROM file_hashes WHERE release_id = ?`, [id])
+    const files = (await NzbService.files(Number(id), client)).map((f) => ({ name: f.name, size: f.segments.reduce((n, [, , bytes]) => n + bytes, 0) }))
+    await client.rawQuery(`UPDATE releases SET complete = ? WHERE id = ?`, [par2Complete(set, files, hashes), id])
+    decided++
+  }
+  return decided
+}

app/services/par2_merger.ts

@@ -4,6 +4,8 @@ import { par2ReleaseName } from '#lib/release/index'
 import { looksRandom } from '#lib/release/random'
 import { SidecarFolder } from '#services/sidecar_folder'
 import { Par2Pruner } from '#services/par2_pruner'
+import { pruneSevenZipTails } from '#services/seven_zip_tails'
+import { decidePar2Completeness } from '#services/par2_completeness'
 
 /** Releases of a set further apart than this (posted) aren't one upload: a repost is merged separately (decision d). */
 const MAX_SPREAD_MS = 48 * 3600 * 1000
@@ -15,15 +17,28 @@ const MAX_HOLDERS = 4
 /** Members with more files than this many times the set's, plus SLACK_FILES (its par2 volumes), are more than one upload. */
 const MAX_FILES_PER_SET_FILE = 3
 const SLACK_FILES = 10
+/**
+ * A par2 file held by more file rows than this (the same first 16 KiB and size) is a common file, not one upload's:
+ * it matches nothing. One such file was in 190,380 uploads, and a set naming it loaded every one of them; the merger
+ * ran out of heap every few minutes (LOG.md 2026-10-20 (85)). The judge calls 20 releases repeated.
+ */
+const COMMON_FILE = 20
 /** Between rounds. */
 const ROUND_MS = 60_000
-/** A round looks back this far past the previous one's start (rows committed while it ran). */
+/** A window looks back this far past the previous one's end (rows committed after it was read). */
 const OVERLAP_MS = 60_000
+/**
+ * A round goes through what's new in windows of this much (created_at), its place kept after each: a round behind by
+ * two days of the probe backlog (3.5M sets) never ended, and every restart started it over (LOG.md 2026-10-20 (85)).
+ */
+const WINDOW_MS = 60 * 60_000
 /** The sidecar fold also runs this often inside a long par2 pass: its queue grows by the settler's pace. */
 const FOLD_EVERY_MS = 5 * 60_000
 /** Between par2_files prunes (complete and judged sets' file lists). */
 const PRUNE_EVERY_MS = 60 * 60_000
-/** The merger's row in worker_cursors: the start of its last whole round, kept over a restart. */
+/** A prune's share of the merger's time: the rest goes on in the next one. */
+const PRUNE_BUDGET_MS = 2 * 60_000
+/** The merger's row in worker_cursors: the end of its last whole window, kept over a restart. */
 const CURSOR = 'merger'
 
 export interface MergeRound {
@@ -44,6 +59,12 @@ export interface MergeRound {
  */
 const INNER_STEM = `substring(i.result->>'yencName' from '^([A-Za-z0-9]{16,64})\\.')`
 
+/** The releases holding a par2 file f (its first 16 KiB and size), none when it's a common file (COMMON_FILE). */
+const HOLDERS = `SELECT c.release_id FROM (
+    SELECT l.release_id, count(*) OVER () AS holders
+    FROM (SELECT h.release_id FROM file_hashes h WHERE h.hash16k = f.hash16k AND h.size = f.length LIMIT ${COMMON_FILE + 1}) l
+  ) c WHERE c.holders <= ${COMMON_FILE}`
+
 interface Member {
   id: number
   release_key: string
@@ -95,8 +116,8 @@ export class Par2Merger {
         // First, at start too: a prune needn't wait for a long round.
         if (!this.stopped && Date.now() - prunedAt >= PRUNE_EVERY_MS) {
           prunedAt = Date.now()
-          const pruned = await new Par2Pruner().round()
+          const pruned = await new Par2Pruner().round(PRUNE_BUDGET_MS)
-          if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged.`)
+          if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged, ${pruned.final} final.`)
         }
         const round = await this.round()
         if (round.merges > 0) logger.info(`Merged ${round.releases} releases into ${round.merges} by par2 set (${round.sets} sets checked).`)
@@ -109,14 +130,18 @@ export class Par2Merger {
     }
   }
 
-  /** The sets with something new since the last round: a par2 found, or a hash of one of their files. */
+  /**
+   * The sets and inner stems with something new since the last round (a par2 found, a hash of one of their files, a
+   * lone inspection), a window at a time.
+   */
   async round(): Promise<MergeRound> {
-    const { rows: now } = await db.rawQuery(`SELECT now() AS at`)
+    // A Date keeps milliseconds, now() microseconds: rounded up, a row made at now() is before the end.
+    const { rows: now } = await db.rawQuery(`SELECT now() + interval '1 millisecond' AS at`)
+    const end = new Date(now[0].at)
     if (!this.since) {
       const saved = await db.from('worker_cursors').where('name', CURSOR).first()
       this.since = saved ? new Date(saved.at) : new Date(0)
     }
-    const since = new Date(this.since.getTime() - OVERLAP_MS)
     // First: sidecars (.nfo, .sfv, .nzb) apart from their upload join it. A queue, not since: it needn't wait for the
     // par2 pass (hours when that starts over).
     const folded = await this.folder.round()
@@ -130,52 +155,96 @@ export class Par2Merger {
       foldedAt = Date.now()
       if (more.folded) logger.info(`Folded ${more.folded} sidecars into their uploads (${more.sidecars} sidecar-only of ${more.looked} small releases).`)
     }
-    const { rows } = await db.rawQuery(
+    const round: MergeRound = { sets: 0, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
-      `SELECT set_id FROM par2_sets WHERE created_at >= ?
+    while (this.since < end) {
-       UNION
-       SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size WHERE h.created_at >= ?`,
-      [since, since]
-    )
-    const round: MergeRound = { sets: rows.length, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
-    for (const { set_id: setId } of rows) {
       if (this.stopped) return round
-      await foldAgain()
+      const from = await this.firstNew(new Date(this.since.getTime() - OVERLAP_MS), end)
-      // One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
+      const until = from ? new Date(Math.min(from.getTime() + WINDOW_MS, end.getTime())) : end
-      const merged = await this.mergeSet(setId.trim()).catch((error) => {
+      if (from) {
-        logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
+        const { touched, sets } = await this.setsToMerge(from, until)
-        return 0
+        round.sets += touched
-      })
+        for (const setId of sets) {
-      if (merged > 0) {
+          if (this.stopped) return round
-        round.merges++
+          await foldAgain()
-        round.releases += merged + 1
+          // One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
+          const merged = await this.mergeSet(setId.trim()).catch((error) => {
+            logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
+            return 0
+          })
+          if (merged > 0) {
+            round.merges++
+            round.releases += merged + 1
+          }
+        }
+        const { rows: stems } = await db.rawQuery(
+          `SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i
+           WHERE i.kind = 'lone' AND i.created_at >= ? AND i.created_at <= ? AND ${INNER_STEM} IS NOT NULL`,
+          [from, until]
+        )
+        for (const { stem } of stems) {
+          if (this.stopped) return round
+          if (!looksRandom(stem)) continue
+          round.stems++
+          const merged = await this.mergeStem(stem).catch((error) => {
+            logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
+            return 0
+          })
+          if (merged > 0) {
+            round.stemMerges++
+            round.stemReleases += merged + 1
+          }
+        }
       }
+      // Only after a whole window: a restart halfway goes on from the window before.
+      this.since = until
+      await db.rawQuery(
+        `INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
+        [CURSOR, this.since]
+      )
     }
-    const { rows: stems } = await db.rawQuery(
-      `SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i WHERE i.kind = 'lone' AND i.created_at >= ? AND ${INNER_STEM} IS NOT NULL`,
-      [since]
-    )
-    for (const { stem } of stems) {
-      if (this.stopped) return round
-      if (!looksRandom(stem)) continue
-      round.stems++
-      const merged = await this.mergeStem(stem).catch((error) => {
-        logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
-        return 0
-      })
-      if (merged > 0) {
-        round.stemMerges++
-        round.stemReleases += merged + 1
-      }
-    }
-    // Only after a whole round: a restart halfway goes on from the round before.
-    this.since = now[0].at
-    await db.rawQuery(
-      `INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
-      [CURSOR, this.since]
-    )
     return round
   }
 
+  /** When the first row a window reads was made, from `from` up to `end`; null when none (a gap, the nightly pause). */
+  private async firstNew(from: Date, end: Date): Promise<Date | null> {
+    const { rows } = await db.rawQuery(
+      `SELECT least(
+         (SELECT min(created_at) FROM par2_sets WHERE created_at >= ?),
+         (SELECT min(created_at) FROM file_hashes WHERE created_at >= ?),
+         (SELECT min(created_at) FROM release_inspections WHERE created_at >= ?)) AS at`,
+      [from, from, from]
+    )
+    const at = rows[0].at ? new Date(rows[0].at) : null
+    return at && at <= end ? at : null
+  }
+
+  /**
+   * The sets a window touched (a par2 found, a hash of one of their files), and those among them a merge could
+   * change: held or matched by two releases or more. Most have every file in one release already (96% of a backlog
+   * hour's 147k): mergeSet needn't look at them.
+   */
+  private async setsToMerge(from: Date, until: Date): Promise<{ touched: number; sets: string[] }> {
+    const { rows } = await db.rawQuery(
+      `WITH touched AS MATERIALIZED (
+         SELECT set_id FROM par2_sets WHERE created_at >= ? AND created_at <= ?
+         UNION
+         SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size
+         WHERE h.created_at >= ? AND h.created_at <= ?
+       )
+       SELECT (SELECT count(*) FROM touched) AS touched, array(
+         SELECT t.set_id FROM touched t
+         WHERE EXISTS (
+           SELECT 1 FROM (
+             SELECT s.release_id FROM par2_sets s WHERE s.set_id = t.set_id
+             UNION ALL
+             SELECT m.release_id FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) m WHERE f.set_id = t.set_id
+           ) u HAVING count(DISTINCT u.release_id) >= 2)
+       ) AS sets`,
+      [from, until, from, until]
+    )
+    return { touched: Number(rows[0].touched), sets: rows[0].sets }
+  }
+
   /** Merges the releases whose lone files' inner names share a random stem; how many went into another. */
   async mergeStem(stem: string): Promise<number> {
     return db.transaction(async (trx) => {
@@ -220,7 +289,7 @@ export class Par2Merger {
       const near = (r: Member) => r.group_id === anchor.group_id && (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
       const { rows: matches } = await trx.rawQuery(
         `SELECT f.file_id, r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable
-         FROM par2_files f JOIN file_hashes h ON h.hash16k = f.hash16k AND h.size = f.length JOIN releases r ON r.id = h.release_id
+         FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) h JOIN releases r ON r.id = h.release_id
          WHERE f.set_id = ?`,
         [setId]
       )
@@ -268,6 +337,14 @@ export class Par2Merger {
       for (const table of ['release_chunks', 'probe_candidates', 'release_inspections', 'file_hashes']) {
         await trx.rawQuery(`UPDATE ${table} SET release_id = ? WHERE release_id = ANY(?)`, [target.id, others])
       }
+      // An unsettled member's candidates into a settled target: the settler won't come by again to make them claimable.
+      await trx.rawQuery(
+        `UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
+         WHERE r.id = ? AND r.settled_at IS NOT NULL AND pc.release_id = r.id AND pc.status = 'waiting' AND NOT pc.ready`,
+        [target.id]
+      )
+      // The members' 7z tails together: those below the highest volume go.
+      await pruneSevenZipTails(trx, [target.id])
       // A set held by the target already, or by two of the others (a par2 and its repost), isn't held twice (every
       // round failed on the primary key from 20:28 UTC on 2026-10-12; LOG.md 2026-10-12 (30)).
       await trx.rawQuery(
@@ -295,12 +372,13 @@ export class Par2Merger {
          WHERE t.id = ?`,
         [name ?? '', others, target.id]
       )
-      // Completeness as the settler decides it, once settled.
+      // Completeness as the settler decides it, once settled; without a counter, by the par2 sets it now holds.
       await trx.rawQuery(
-        `UPDATE releases SET complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END WHERE id = ? AND settled_at IS NOT NULL`,
+        `UPDATE releases SET complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END WHERE id = ? AND settled_at IS NOT NULL`,
         [target.id]
       )
       await trx.rawQuery(`DELETE FROM releases WHERE id = ANY(?)`, [others])
+      await decidePar2Completeness(trx, [target.id])
       return others.length
     }
   }

app/services/par2_pruner.ts

@@ -1,75 +1,107 @@
 import db from '@adonisjs/lucid/services/db'
+import { decidePar2Completeness } from '#services/par2_completeness'
 
 /** A set this long past its last holder's par2 has had its merges: one posted later is a repost (decision d). */
 const SETTLED_MS = 2 * 86400_000
 /** Sets per statement. */
 const BATCH = 20_000
-/** The complete pass's row in worker_cursors: the newest holder's created_at of the last set it looked at. */
+/** The sweep's row in worker_cursors: the last set_id it looked at (key); past the last set it starts over. */
 const CURSOR = 'par2-prune'
+/** The merger's row in worker_cursors: the end of its last whole window. */
+const MERGER = 'merger'
+/** The merger's windows look back this far (Par2Merger's OVERLAP_MS). */
+const MERGER_OVERLAP_MS = 60_000
 /** Verdicts whose releases needn't merge any more; locked isn't one (a password found takes it back). */
 const JUNK = ['dead', 'fake', 'repeated']
+/** Probe candidates whose results a merge reads: a file's hash (lone), a set's file list (par2). */
+const MERGE_KINDS = ['lone', 'par2']
 
 export interface PruneRound {
   complete: number
   judged: number
+  final: number
   rows: number
 }
 
 /**
- * Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60)): the table only grew, 35M rows and
+ * Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60), 2026-10-20 (87)). A set's file list
- * 13.6 GB at 1.8M sets. A set's file list matches file hashes to releases and names an unreadable upload once; it's
+ * matches file hashes to releases and names an unreadable upload once; it's done when
- * done when
  * - complete: SETTLED_MS after its last holder's par2, every file it describes has its hash in a release holding the
  *   set (all of it merged into one);
- * - judged: every release holding it has a junk verdict (dead, fake, repeated).
+ * - judged: every release holding it has a junk verdict (dead, fake, repeated);
+ * - final: SETTLED_MS after its last holder's par2, every holder is final (release_final: a run of done ranges holds
+ *   it with hours to spare, so no more of its upload will be read), the candidates of the ranges around it are probed,
+ *   and the merger has seen every hash of its files. A file still missing then is one no probe will hash: most sets
+ *   describe a readable upload's files, which aren't lone candidates (9 in 10 sets were kept for that, 19 GB).
  * par2_sets stays (the judge's unusable rule, the merge's holders). A set comes up again only with a new holder: its
  * par2 reposted, whose probe writes the rows again; without rows its merge finds no files and the count check holds.
+ * The sweep goes through the sets with rows by set_id and starts over at the end: a set kept now (a holder at the edge
+ * of a run, its ranges' probes not done yet) is looked at again on the next pass.
  */
 export class Par2Pruner {
-  async round(): Promise<PruneRound> {
+  /** Sets per batch (BATCH; smaller in the tests). */
-    const round: PruneRound = { complete: 0, judged: 0, rows: 0 }
+  constructor(private batch = BATCH) {}
-    const saved = await db.from('worker_cursors').where('name', CURSOR).first()
+
-    let since: Date = saved ? new Date(saved.at) : new Date(0)
+  /**
-    const until = new Date(Date.now() - SETTLED_MS)
+   * Batches until a pass is done or `budgetMs` is spent (the cursor is saved per batch, the next round goes on from
+   * it): run from the merger's loop, a first round over two days of sets held the merges back for hours (LOG.md
+   * 2026-10-17 (72)).
+   */
+  async round(budgetMs = Infinity): Promise<PruneRound> {
+    const deadline = Date.now() + budgetMs
+    const round: PruneRound = { complete: 0, judged: 0, final: 0, rows: 0 }
+    const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
+    let after: string = cursors[0]?.key ?? ''
+    // Ranges done since the oldest merge-kind candidate still unprobed: their releases' hashes and file lists may be
+    // still to come. Hashes since the merger's last window: it hasn't matched them yet.
+    const { rows: marks } = await db.rawQuery(
+      `SELECT coalesce((SELECT min(created_at) FROM probe_candidates WHERE status IN ('waiting', 'taken') AND kind = ANY(?)), now()) AS probed,
+              coalesce((SELECT at FROM worker_cursors WHERE name = ?), to_timestamp(0)) - make_interval(secs => ?) AS merged,
+              now() - make_interval(secs => ?) AS settled`,
+      [MERGE_KINDS, MERGER, MERGER_OVERLAP_MS / 1000, SETTLED_MS / 1000]
+    )
+    const { probed, merged, settled } = marks[0]
     for (;;) {
       const { rows: sets } = await db.rawQuery(
-        `SELECT set_id, max(created_at) AS at FROM par2_sets GROUP BY set_id
+        `WITH batch AS MATERIALIZED (SELECT DISTINCT set_id FROM par2_files WHERE set_id > ? ORDER BY set_id LIMIT ?),
-         HAVING max(created_at) > ? AND max(created_at) <= ? ORDER BY 2 LIMIT ?`,
+         recent AS MATERIALIZED (SELECT group_id, posted_low, posted_high FROM ranges WHERE status = 'done' AND done_at >= ?),
-        [since, until, BATCH]
+         unseen AS MATERIALIZED (SELECT DISTINCT hash16k, size FROM file_hashes WHERE created_at >= ?)
+         SELECT b.set_id, CASE
+           WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
+            AND NOT EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = b.set_id AND NOT EXISTS (
+                  SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
+                  WHERE h.hash16k = f.hash16k AND h.size = f.length)) THEN 'complete'
+           WHEN NOT EXISTS (SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts v ON v.release_id = o.release_id AND v.rule = ANY(?)
+                  WHERE o.set_id = b.set_id AND v.release_id IS NULL) THEN 'judged'
+           WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
+            AND NOT EXISTS (SELECT 1 FROM par2_sets s JOIN releases r ON r.id = s.release_id WHERE s.set_id = b.set_id AND (
+                  release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) IS NOT TRUE
+                  OR EXISTS (SELECT 1 FROM recent x WHERE x.group_id = r.group_id
+                    AND x.posted_low <= coalesce(r.last_posted_at, r.posted_at) + interval '6 hours' AND x.posted_high >= r.posted_at - interval '6 hours')))
+            AND NOT EXISTS (SELECT 1 FROM par2_files f JOIN unseen u ON u.hash16k = f.hash16k AND u.size = f.length WHERE f.set_id = b.set_id)
+           THEN 'final' END AS reason
+         FROM batch b ORDER BY b.set_id`,
+        [after, this.batch, probed, merged, settled, JUNK, settled]
       )
-      if (sets.length === 0) break
+      const done = (reason: keyof PruneRound) => sets.filter((s: { reason: string }) => s.reason === reason).map((s: { set_id: string }) => s.set_id)
-      const ids = sets.map((s: { set_id: string }) => s.set_id)
+      for (const reason of ['complete', 'judged', 'final'] as const) {
-      const { rows: done } = await db.rawQuery(
+        const ids = done(reason)
-        `SELECT c.set_id FROM unnest(?::char(32)[]) AS c(set_id)
+        round[reason] += ids.length
-         WHERE EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id)
+        // The holders' completeness while the file lists are there (app/services/par2_completeness.ts).
-           AND NOT EXISTS (
+        if (reason !== 'judged' && ids.length > 0) {
-             SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id AND NOT EXISTS (
+          const { rows: holders } = await db.rawQuery(`SELECT DISTINCT release_id FROM par2_sets WHERE set_id = ANY(?::char(32)[])`, [ids])
-               SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
+          await decidePar2Completeness(db, holders.map((h: { release_id: string }) => h.release_id))
-               WHERE h.hash16k = f.hash16k AND h.size = f.length))`,
+        }
-        [ids]
+        round.rows += await this.drop(ids)
-      )
+      }
-      round.complete += done.length
+      // Past the last set: the next batch starts the next pass.
-      round.rows += await this.drop(done.map((s: { set_id: string }) => s.set_id))
+      after = sets.length < this.batch ? '' : sets.at(-1).set_id
-      since = sets.at(-1).at
       await db.rawQuery(
-        `INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
+        `INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
-        [CURSOR, since]
+        [CURSOR, after]
       )
-      if (sets.length < BATCH) break
+      if (after === '' || Date.now() > deadline) break
     }
-    // Judged: no cursor (verdicts come and go); only sets with rows left.
-    const { rows: judged } = await db.rawQuery(
-      `SELECT DISTINCT s.set_id FROM release_verdicts v JOIN par2_sets s ON s.release_id = v.release_id
-       WHERE v.rule = ANY(?)
-         AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
-         AND NOT EXISTS (
-           SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts ov ON ov.release_id = o.release_id AND ov.rule = ANY(?)
-           WHERE o.set_id = s.set_id AND ov.release_id IS NULL)`,
-      [JUNK, JUNK]
-    )
-    round.judged = judged.length
-    const judgedIds = judged.map((s: { set_id: string }) => s.set_id)
-    for (let i = 0; i < judgedIds.length; i += BATCH) round.rows += await this.drop(judgedIds.slice(i, i + BATCH))
     return round
   }
 

app/services/prefetcher.ts

@@ -131,6 +131,8 @@ export class Prefetcher {
         .catch((error) => logger.warn(`Prefetching: ${error.message}`))
         .finally(() => this.running--)
     }
+    // Work left with every loop running: the pool's share could be bigger.
+    if (!this.stopped && this.running >= this.loops && this.peek()) NntpPool.starved()
   }
 
   stop() {
@@ -145,6 +147,9 @@ export class Prefetcher {
   /** One connection's loop: chunks until there is nothing left to fetch. */
   private async work() {
     while (!this.stopped) {
+      // The pool's share got smaller: fewer loops (waiting on the pool, they'd count as work short of a connection).
+      // Two leaving at once is one too many for a tick (TICK_MS), whose loop comes back.
+      if (this.running > this.loops) return
       const next = this.take()
       if (!next) return
       const { target, span } = next

app/services/prober.ts

@@ -6,6 +6,9 @@ import coordinatorConfig from '#config/coordinator'
 import { XoverClient } from '#lib/nntp/index'
 import { YencError } from '#lib/yenc/index'
 import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/index'
+import { NntpShare } from '#services/nntp_shares'
+import { DiskSpace } from '#services/disk_space'
+import { decidePar2Completeness } from '#services/par2_completeness'
 
 interface Candidate {
   id: number
@@ -18,7 +21,8 @@ type Outcome = { candidate: Candidate; status: 'done' | 'missing' | 'error' | 'w
 
 /**
  * The order candidates are probed in: what the par2 merge needs first (a random-named file's own start, then the
- * par2 sets and archives of releases without a readable name), then the rest (names and passwords of readable ones).
+ * par2 sets and archives of releases without a readable name), then the rest (names and passwords of readable ones),
+ * then 7z listings (the end of a split 7z's last volume; queued from 2026-10-13, probed from 2026-10-23).
  */
 const TIERS: { kind: string; unreadable: boolean }[] = [
   { kind: 'lone', unreadable: false },
@@ -26,8 +30,19 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
   { kind: 'archive', unreadable: true },
   { kind: 'par2', unreadable: false },
   { kind: 'archive', unreadable: false },
+  { kind: '7z-tail', unreadable: true },
+  { kind: '7z-tail', unreadable: false },
 ]
 const FLUSH_MS = 2000
+
+/**
+ * JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
+ * and the whole flush failed with it (LOG.md 2026-10-18 (78)).
+ */
+function jsonb(value: unknown): string {
+  return JSON.stringify(value, (_key, v) => (typeof v === 'string' ? v.replaceAll('\0', '') : v))
+}
+
 /** Slices a claim takes from random points of the queue. */
 const SLICES = 20
 const REPORT_MS = 60_000
@@ -46,6 +61,14 @@ export class Prober {
   private refilling: Promise<void> | null = null
   private stopped = false
   private stats = Prober.freshStats()
+  /** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
+  private lanes = new Set<Promise<void>>()
+  private running = 0
+  /** The last lane to look found nothing to claim: the prober wants its minimum only. */
+  private dry = false
+  /** What it stores (inspections, par2 file lists, the WAL) fills the NVMe: no claims below PROBE_MIN_FREE_GB. */
+  private space = new DiskSpace('probing', '/', coordinatorConfig.probe.minFreeGb)
+  private share = new NntpShare('prober', coordinatorConfig.nntpShares.proberPriority, coordinatorConfig.probe.minConnections, coordinatorConfig.probe.connections)
 
   /** Per minute: what was read, and where the lanes' time went (ms summed over lanes; flushes ms on their own). */
   private static freshStats() {
@@ -56,8 +79,7 @@ export class Prober {
     // A previous run's claims: one prober at a time.
     const { rowCount } = await db.rawQuery(`UPDATE probe_candidates SET status = 'waiting' WHERE status = 'taken'`)
     if (rowCount) logger.info(`Probing: ${rowCount} candidates from a previous run back in the queue.`)
-    const connections = coordinatorConfig.probe.connections
+    logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
-    logger.info(`Probing with ${connections} connections.`)
     const flusher = (async () => {
       while (!this.stopped || this.outcomes.length > 0) {
         await sleep(FLUSH_MS)
@@ -67,7 +89,26 @@ export class Prober {
         this.report()
       }
     })()
-    await Promise.all(Array.from({ length: connections }, () => this.lane()))
+    // Lanes as the share allows: the coordinator's history comes first (app/services/nntp_shares.ts).
+    while (!this.stopped) {
+      try {
+        const was = this.share.granted
+        const granted = await this.share.sync(this.dry ? 0 : this.share.max, this.running)
+        if (granted !== was) logger.info(`Probing: ${granted} connections (was ${was}).`)
+        while (this.running < granted) {
+          this.running++
+          const lane: Promise<void> = this.lane()
+            .catch((error) => logger.error(`Probing: a lane failed: ${error.message}`))
+            .finally(() => this.lanes.delete(lane))
+          this.lanes.add(lane)
+        }
+      } catch (error: any) {
+        logger.error(`Probing: NNTP share failed: ${error.message}`)
+      }
+      for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !this.stopped; waited += 1000) await sleep(1000)
+    }
+    await Promise.all(this.lanes)
+    await this.share.leave().catch(() => {})
     // What was claimed and not probed goes back.
     for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
     this.stopped = true
@@ -78,68 +119,90 @@ export class Prober {
     this.stopped = true
   }
 
-  /** One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure. */
+  /**
+   * One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure; done when
+   * there are more lanes than the share (its connection closed, the coordinator takes it on its next sync).
+   */
   private async lane() {
     let client: XoverClient | null = null
-    while (!this.stopped) {
+    let usedAt = Date.now()
-      const batch: Candidate[] = []
+    try {
-      const waited = Date.now()
+      while (!this.stopped) {
-      while (batch.length < coordinatorConfig.probe.pipeline) {
+        if (this.running > this.share.granted) break
-        const candidate = await this.next()
+        const batch: Candidate[] = []
-        if (!candidate) break
+        const waited = Date.now()
-        batch.push(candidate)
+        while (batch.length < coordinatorConfig.probe.pipeline) {
-      }
+          const candidate = await this.next()
-      this.stats.waitMs += Date.now() - waited
+          if (!candidate) break
-      if (batch.length === 0) {
+          batch.push(candidate)
-        this.stats.idleMs += 30_000
+        }
-        await sleep(30_000)
+        this.stats.waitMs += Date.now() - waited
-        continue
+        this.dry = batch.length === 0
-      }
+        if (batch.length === 0) {
-      try {
+          // Nothing to probe for a while: the connection closes (nntpIdleCloseMs), and opens again with work.
-        client ??= await XoverClient.connect({
+          const idleMs = coordinatorConfig.nntpIdleCloseMs
-          host: env.get('NNTP_HOST')!,
+          if (client && idleMs > 0 && Date.now() - usedAt >= idleMs) {
-          port: env.get('NNTP_PORT', 563),
+            client.close()
-          user: env.get('NNTP_USER'),
+            client = null
-          password: env.get('NNTP_PASSWORD'),
-          secure: env.get('NNTP_SECURE', true),
-          compress: false,
-          timeoutMs: coordinatorConfig.nntpTimeoutMs,
-        })
-        const fetched = Date.now()
-        const bodies = await client.bodies(batch.map((c) => c.messageId))
-        this.stats.fetchMs += Date.now() - fetched
-        const inspected = Date.now()
-        for (const [i, candidate] of batch.entries()) {
-          const body = bodies[i]
-          this.stats.articles++
-          if (body === null) {
-            // Taken down or expired: final.
-            this.outcomes.push({ candidate, status: 'missing' })
-            continue
           }
-          this.stats.bytes += body.length
+          this.stats.idleMs += 30_000
-          try {
+          await sleep(30_000)
-            this.outcomes.push({ candidate, status: 'done', inspection: inspect(body) })
+          continue
-          } catch (error: any) {
+        }
-            // Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
+        try {
-            if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
+          client ??= await XoverClient.connect({
-            this.outcomes.push({ candidate, status: 'error' })
+            host: env.get('NNTP_HOST')!,
+            port: env.get('NNTP_PORT', 563),
+            user: env.get('NNTP_USER'),
+            password: env.get('NNTP_PASSWORD'),
+            secure: env.get('NNTP_SECURE', true),
+            compress: false,
+            timeoutMs: coordinatorConfig.nntpTimeoutMs,
+          })
+          const fetched = Date.now()
+          const bodies = await client.bodies(batch.map((c) => c.messageId))
+          this.stats.fetchMs += Date.now() - fetched
+          const inspected = Date.now()
+          for (const [i, candidate] of batch.entries()) {
+            const body = bodies[i]
+            this.stats.articles++
+            if (body === null) {
+              // Taken down or expired: final.
+              this.outcomes.push({ candidate, status: 'missing' })
+              continue
+            }
+            this.stats.bytes += body.length
+            try {
+              this.outcomes.push({ candidate, status: 'done', inspection: inspect(body, candidate.kind) })
+            } catch (error: any) {
+              // Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
+              if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
+              this.outcomes.push({ candidate, status: 'error' })
+            }
           }
+          this.stats.inspectMs += Date.now() - inspected
+          usedAt = Date.now()
+        } catch (error: any) {
+          logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
+          for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
+          client?.close()
+          client = null
+          await sleep(5000)
         }
-        this.stats.inspectMs += Date.now() - inspected
-      } catch (error: any) {
-        logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
-        for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
-        client?.close()
-        client = null
-        await sleep(5000)
       }
+    } finally {
+      client?.close()
+      this.running--
     }
-    client?.close()
+  }
+
+  /** Candidates claimed at a time: a few rounds of every lane's pipeline (a claim takes seconds; 400 lasted ~2 s at 37 lanes). */
+  private get batch() {
+    return Math.max(coordinatorConfig.probe.batch, this.running * coordinatorConfig.probe.pipeline * 4)
   }
 
   private async next(): Promise<Candidate | undefined> {
-    if (this.queue.length < coordinatorConfig.probe.batch / 2) {
+    if (this.queue.length < this.batch / 2) {
       this.refilling ??= this.claim().finally(() => (this.refilling = null))
       if (this.queue.length === 0) await this.refilling
     }
@@ -162,13 +225,24 @@ export class Prober {
   }
 
   private async claimTiers() {
+    // The queue empties, the lanes go dry and the share shrinks to its minimum until there's room again.
+    if (await this.space.short()) return
     for (const { kind, unreadable } of TIERS) {
-      const want = coordinatorConfig.probe.batch - this.queue.length
+      const want = this.batch - this.queue.length
       if (want <= 0) return
       // The starting points chosen here: in the query Postgres drew random() once for all slices.
       // Over the claimable ones of this tier only: the rest of the queue isn't settled yet.
+      // Per value of unreadable, each an end of the index: over `unreadable >= false` Postgres walked every waiting
+      // entry of the kind, 2.5 minutes a claim at 28M archive candidates (LOG.md 2026-10-18 (74)).
+      const values = unreadable ? [true] : [false, true]
+      const ends = values
+        .map(() => `SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable = ?`)
+        .join(' UNION ALL ')
       const [{ lo, hi }] = (
-        await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable >= ?`, [kind, unreadable])
+        await db.rawQuery(
+          `SELECT min(lo) AS lo, max(hi) AS hi FROM (${ends}) e`,
+          values.flatMap((v) => [kind, v])
+        )
       ).rows
       if (lo === null) continue
       const starts = Array.from({ length: SLICES }, () => Number(lo) + Math.floor(Math.random() * (Number(hi) - Number(lo) + 1)))
@@ -209,16 +283,18 @@ export class Prober {
     await db.transaction(async (trx) => {
       await trx.rawQuery(
         `UPDATE probe_candidates pc SET status = v.status FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, status text) WHERE pc.id = v.id`,
-        [JSON.stringify(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
+        [jsonb(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
       )
       // Final answers read from the news server, kept by message-id for the next catalog (a reset).
-      const fresh = outcomes.filter((o) => !o.cached && o.status !== 'waiting')
+      // One per message-id: the same article is a candidate of more than one release, and ON CONFLICT can't take a
+      // row twice in one statement.
+      const fresh = [...new Map(outcomes.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
       if (fresh.length > 0) {
         await trx.rawQuery(
           `INSERT INTO probe_cache (message_id, status, result, version)
            SELECT v.message_id, v.status, v.result, ? FROM jsonb_to_recordset(?::jsonb) AS v(message_id text, status text, result jsonb)
            ON CONFLICT (message_id) DO UPDATE SET status = excluded.status, result = excluded.result, version = excluded.version, created_at = now()`,
-          [PROBE_VERSION, JSON.stringify(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
+          [PROBE_VERSION, jsonb(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
         )
       }
       const done = outcomes.filter((o) => o.inspection)
@@ -228,9 +304,11 @@ export class Prober {
          SELECT v.release_id, v.candidate_id, v.kind, v.result FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, candidate_id bigint, kind text, result jsonb)
          ON CONFLICT (release_id, candidate_id) DO UPDATE SET result = excluded.result, created_at = now()`,
         // A par2 set's file list goes to par2_files, not into every inspection.
-        [JSON.stringify(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
+        [jsonb(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
       )
       await this.storeForMerge(trx, done)
+      // A par2 index read: the release's completeness by its set, when it has no counter.
+      await decidePar2Completeness(trx, [...new Set(done.filter((o) => o.inspection!.par2).map((o) => o.candidate.releaseId))])
       // Each touched release from all its inspections (earlier rounds may have stored others).
       const ids = [...new Set(done.map((o) => o.candidate.releaseId))]
       const { rows } = await trx.rawQuery(
@@ -252,7 +330,7 @@ export class Prober {
       await trx.rawQuery(
         `UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
          FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, passworded boolean, inner_name text) WHERE r.id = v.id`,
-        [JSON.stringify(updates)]
+        [jsonb(updates)]
       )
     })
   }
@@ -265,7 +343,7 @@ export class Prober {
         `INSERT INTO file_hashes (candidate_id, release_id, hash16k, size)
          SELECT v.candidate_id, v.release_id, v.hash16k, v.size FROM jsonb_to_recordset(?::jsonb) AS v(candidate_id bigint, release_id bigint, hash16k text, size bigint)
          ON CONFLICT (candidate_id) DO NOTHING`,
-        [JSON.stringify(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
+        [jsonb(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
       )
     }
     for (const o of done) {
@@ -275,7 +353,7 @@ export class Prober {
         `INSERT INTO par2_files (set_id, file_id, hash16k, length, name)
          SELECT ?, v.file_id, v.hash16k, v.length, v.name FROM jsonb_to_recordset(?::jsonb) AS v(file_id text, hash16k text, length bigint, name text)
          ON CONFLICT DO NOTHING`,
-        [set.setId, JSON.stringify(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
+        [set.setId, jsonb(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
       )
       await trx.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files) VALUES (?, ?, ?) ON CONFLICT DO NOTHING`, [set.setId, o.candidate.releaseId, set.files.length])
     }

app/services/release_judge.ts

@@ -1,10 +1,18 @@
 import db from '@adonisjs/lucid/services/db'
 import logger from '@adonisjs/core/services/logger'
 
-/** Raised when a rule changes: verdicts of an older version are judged again. */
+/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
 export const JUDGE_VERSION = 1
-/** Between rounds: each rule is a scan of its table. */
+/** Between rounds of the changed releases (judge_dirty, noted by triggers; migration 4900). */
-const ROUND_MS = 60 * 60 * 1000
+const ROUND_MS = 2 * 60 * 1000
+/**
+ * A full round (every release, each rule a scan of its table: 35–38 min) this often: for what no trigger notes. Time
+ * (unusable a day after settling, a release final once the ranges beside it are read) and fewer releases with a file
+ * after a merge (repeated).
+ */
+const FULL_MS = 24 * 60 * 60 * 1000
+/** Changed releases judged at a time. */
+const CHUNK = 50_000
 /** A file's first 16 KiB and size in this many releases is one file posted again and again (test files, spam). */
 const REPEATED_RELEASES = 20
 
@@ -24,10 +32,15 @@ export interface Rule {
   name: string
   /** Why, in a few words (the log, the web interface). */
   says: string
-  /** SELECT of the release ids the rule judges. */
+  /** SELECT of the release ids the rule judges; scoped, only those in judge_scope. */
-  sql: string
+  sql: (scoped: boolean) => string
 }
 
+/** A join to the releases in scope (an incremental round), on the given release id column. */
+const only = (scoped: boolean, column: string) => (scoped ? `JOIN judge_scope s ON s.release_id = ${column}` : '')
+
+
+
 /**
  * The junk judge's rules, in order: a release takes the first that matches (REBUILD.md, junk). Each runs on what
  * the header pass, the prober, the merges and the spots know by now.
@@ -36,26 +49,33 @@ export const RULES: Rule[] = [
   {
     name: 'dead',
     says: 'every probed first article is gone from the server (taken down or expired)',
-    sql: `SELECT release_id FROM probe_candidates WHERE status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(status = 'missing')`,
+    sql: (scoped) =>
+      `SELECT c.release_id FROM probe_candidates c ${only(scoped, 'c.release_id')}
+       WHERE c.status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(c.status = 'missing')`,
   },
   {
     name: 'fake',
     says: 'named as a film, episode or music, but the content is a program',
     // An archive whose entries are all executables, under a media name (its own or its entries'); or a file that is
     // an executable by name or by its first bytes, under a media name.
-    sql: `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
+    sql: (scoped) => `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
-          WHERE i.kind = 'archive' AND jsonb_array_length(i.result->'names') > 0
+          WHERE i.kind IN ('archive', '7z-tail') AND jsonb_array_length(i.result->'names') > 0
             AND NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(i.result->'names') n WHERE n !~* '${EXECUTABLE}')
             AND (r.name ~* '${MEDIA}' OR (i.result->>'names') ~* '${MEDIA}')
           UNION
-          SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
+          SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
           WHERE i.kind = 'lone' AND ((i.result->>'yencName') ~* '${EXECUTABLE}' OR i.result->>'type' = 'exe')
             AND (r.name ~* '${MEDIA}' OR (i.result->>'yencName') ~* '${MEDIA}')`,
   },
   {
     name: 'repeated',
     says: `the same file content in ${REPEATED_RELEASES}+ releases (test files, spam)`,
-    sql: `SELECT DISTINCT h.release_id FROM file_hashes h
+    // Scoped: by the files of the releases in scope that are in REPEATED_RELEASES+ releases, counted once a chunk
+    // (judge_tipped: ReleaseJudge.changed, which first adds the other releases of a file that just tipped over).
+    sql: (scoped) =>
+      scoped
+        ? `SELECT DISTINCT h.release_id FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id JOIN judge_tipped USING (hash16k, size)`
+        : `SELECT DISTINCT h.release_id FROM file_hashes h
           JOIN (SELECT hash16k, size FROM file_hashes GROUP BY 1, 2 HAVING count(DISTINCT release_id) >= ${REPEATED_RELEASES}) x USING (hash16k, size)`,
   },
   {
@@ -64,19 +84,26 @@ export const RULES: Rule[] = [
     // "@_CelebMovie_@", "NudeCelebs@CelebMovie.com", "@_CelebWow_@", "@_CelebFX_@", "@_iLuvCelebs_@", "@nude-celebs", …:
     // a celebrity's scraped pictures as a folder under a typed subject, the same files again and again under other
     // titles (LOG.md 2026-10-16 (64)). Not Celeborn, CelebTA or other posters with "celeb" in their name.
-    sql: `SELECT r.id FROM releases r
+    // By the posters table's trigram index, each poster's releases then by releases_poster_newest_idx: 7 s, where the
-          WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
+    // regex on all of releases was 45 s, every round (LOG.md 2026-10-19 (80)).
+    sql: (scoped) =>
+      scoped
+        ? `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
+          WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`
+        : `SELECT r.id FROM posters p JOIN releases r ON left(r.poster, 200) = p.poster
+          WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
   },
   {
     name: 'locked',
     says: 'passworded, and no spot, NZB or title gives the password',
-    sql: `SELECT r.id FROM releases r WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
+    // releases_passworded_idx: a scan of all of releases was 30 s a round.
+    sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
   },
   {
     name: 'unusable',
     says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
     // Final (release_final): next to an unread range its par2 or the rest of its files may still come (LOG.md 2026-10-16 (67)).
-    sql: `SELECT r.id FROM releases r
+    sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
           WHERE r.settled_at < now() - interval '1 day' AND NOT r.readable AND r.inner_name IS NULL
             AND release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete)
             AND NOT EXISTS (SELECT 1 FROM release_merges m WHERE m.into_id = r.id)
@@ -86,15 +113,18 @@ export const RULES: Rule[] = [
 ]
 
 export interface JudgeRound {
-  /** Per rule: releases it judges now, verdicts added, verdicts taken back. */
+  /** Releases judged: every one (a full round), or the changed ones and those a hash of theirs tipped over. */
+  releases?: number
+  /** Per rule: releases it judges now (in scope), verdicts it gained, verdicts it lost. */
   rules: Record<string, { matched: number; added: number; removed: number }>
 }
 
 /**
  * The junk judge (REBUILD.md, junk; voyager 1's second filter, D42): after the header pass's rules (garbage
- * signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Each
+ * signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Every few
- * round recomputes every rule: a release a rule matches gets a verdict (the first rule in order, over a later one's), one it no longer
+ * minutes the releases whose facts changed (judge_dirty) are judged by every rule; a full round judges every release
- * matches loses its verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
+ * daily. A release gets the verdict of the first rule in order that matches it; one that no longer matches loses its
+ * verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
  */
 export class ReleaseJudge {
   private stopped = false
@@ -107,9 +137,13 @@ export class ReleaseJudge {
     while (!this.stopped) {
       const started = Date.now()
       try {
-        const round = await this.round()
+        if (await this.fullDue()) {
-        const parts = Object.entries(round.rules).map(([n, r]) => `${n} ${r.matched} (+${r.added} −${r.removed})`)
+          const round = await this.full()
-        logger.info(`Judged: ${parts.join(', ')}.`)
+          if (!this.stopped) logger.info(`Judged all releases: ${describe(round)}.`)
+        } else {
+          const round = await this.changed()
+          if (round.releases! > 0) logger.info(`Judged ${round.releases!.toLocaleString('en')} changed releases: ${describe(round)}.`)
+        }
       } catch (error: any) {
         logger.error(`Judge round failed: ${error.message}`)
       }
@@ -117,43 +151,139 @@ export class ReleaseJudge {
     }
   }
 
-  /** One round over every rule; with dry, only counts. */
+  /** A full round is due: a day after the last, or the last was by another rule set. */
-  async round(dry = false): Promise<JudgeRound> {
+  async fullDue(): Promise<boolean> {
+    const state = await db.from('judge_state').first()
+    return !state || state.version !== JUDGE_VERSION || Date.now() - new Date(state.full_at).getTime() >= FULL_MS
+  }
+
+  /**
+   * Every release by every rule, the matches gathered in judge_full (each rule its own statement: no transaction of
+   * half an hour), then applied at once. With dry, only counts. Stopped halfway, nothing is applied.
+   */
+  async full(dry = false): Promise<JudgeRound> {
+    const started = new Date()
     const round: JudgeRound = { rules: {} }
     // Verdicts of an older rule set are judged again from scratch.
     if (!dry) await db.from('release_verdicts').where('rule_version', '<>', JUDGE_VERSION).delete()
-    const judged = new Set<string>()
+    await db.rawQuery(`TRUNCATE judge_full`)
     for (const rule of RULES) {
-      if (this.stopped) break
+      if (this.stopped) return round
-      const { rows } = await db.rawQuery(rule.sql)
+      const { rowCount } = await db.rawQuery(
-      const ids = rows.map((r: Record<string, unknown>) => String(Object.values(r)[0])).filter((id: string) => !judged.has(id))
+        `INSERT INTO judge_full (release_id, rule) SELECT id, ? FROM (${rule.sql(false)}) AS x(id) ON CONFLICT DO NOTHING`,
-      for (const id of ids) judged.add(id)
+        [rule.name]
-      const stats = { matched: ids.length, added: 0, removed: 0 }
+      )
-      if (!dry) {
+      round.rules[rule.name] = { matched: rowCount ?? 0, added: 0, removed: 0 }
-        await db.transaction(async (trx) => {
+    }
-          // Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
+    if (dry || this.stopped) return round
-          await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
+    await db.rawQuery(`ANALYZE judge_full`)
-          await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
+    await db.transaction((trx) => this.apply(trx, 'judge_full', false, round))
-          for (let i = 0; i < ids.length; i += 50_000) {
+    await db.rawQuery(
-            await trx.rawQuery(`INSERT INTO judged SELECT unnest(?::bigint[]) ON CONFLICT DO NOTHING`, [ids.slice(i, i + 50_000)])
+      `INSERT INTO judge_state (id, full_at, version) VALUES (1, ?, ?) ON CONFLICT (id) DO UPDATE SET full_at = excluded.full_at, version = excluded.version`,
-          }
+      [started, JUDGE_VERSION]
-          const removed = await trx.rawQuery(
+    )
-            `DELETE FROM release_verdicts v WHERE v.rule = ? AND NOT EXISTS (SELECT 1 FROM judged j WHERE j.release_id = v.release_id)`,
+    return round
+  }
+
+  /**
+   * The releases noted in judge_dirty since, CHUNK at a time, each chunk in a transaction (a failed one stays noted).
+   * A changed release can bring a file to REPEATED_RELEASES releases: the others with it are judged too (all but those
+   * already repeated, or dead or fake, which come first). Only when it's the chunk's releases that bring it there: a
+   * file in that many without them had its releases judged before.
+   */
+  async changed(): Promise<JudgeRound> {
+    const round: JudgeRound = { releases: 0, rules: {} }
+    for (const rule of RULES) round.rules[rule.name] = { matched: 0, added: 0, removed: 0 }
+    let more = true
+    while (more && !this.stopped) {
+      await db.transaction(async (trx) => {
+        // Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
+        await trx.rawQuery(`DROP TABLE IF EXISTS judge_scope`)
+        await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
+        await trx.rawQuery(`CREATE TEMP TABLE judge_scope (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
+        const { rows } = await trx.rawQuery(
+          `WITH taken AS (DELETE FROM judge_dirty WHERE ctid = ANY(ARRAY(SELECT ctid FROM judge_dirty LIMIT ${CHUNK})) RETURNING release_id),
+             scoped AS (INSERT INTO judge_scope SELECT DISTINCT release_id FROM taken ON CONFLICT DO NOTHING RETURNING 1)
+           SELECT (SELECT count(*) FROM taken) AS taken`
+        )
+        const taken = Number(rows[0].taken)
+        more = taken === CHUNK
+        if (taken === 0) return
+        // The chunk's files in REPEATED_RELEASES+ releases, from file_hashes_hash_size_release_idx alone (migration 5000).
+        await trx.rawQuery(`DROP TABLE IF EXISTS judge_tipped`)
+        await trx.rawQuery(
+          `CREATE TEMP TABLE judge_tipped (hash16k char(32), size bigint, just boolean NOT NULL, PRIMARY KEY (hash16k, size)) ON COMMIT DROP`
+        )
+        await trx.rawQuery(
+          `INSERT INTO judge_tipped
+           SELECT x.hash16k, x.size, c.outside < ${REPEATED_RELEASES}
+           FROM (SELECT DISTINCT h.hash16k, h.size FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id) x
+           CROSS JOIN LATERAL (
+             SELECT count(DISTINCT f.release_id) AS total,
+               count(DISTINCT f.release_id) FILTER (WHERE NOT EXISTS (SELECT 1 FROM judge_scope s WHERE s.release_id = f.release_id)) AS outside
+             FROM file_hashes f WHERE f.hash16k = x.hash16k AND f.size = x.size) c
+           WHERE c.total >= ${REPEATED_RELEASES}`
+        )
+        await trx.rawQuery(
+          `INSERT INTO judge_scope SELECT DISTINCT f.release_id FROM file_hashes f JOIN judge_tipped t USING (hash16k, size)
+           WHERE t.just AND NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = f.release_id AND v.rule IN ('dead', 'fake', 'repeated'))
+           ON CONFLICT DO NOTHING`
+        )
+        await trx.rawQuery(`ANALYZE judge_scope`)
+        const { rows: scope } = await trx.rawQuery(`SELECT count(*) AS n FROM judge_scope`)
+        round.releases! += Number(scope[0].n)
+        await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL) ON COMMIT DROP`)
+        for (const rule of RULES) {
+          const { rowCount } = await trx.rawQuery(
+            `INSERT INTO judged (release_id, rule) SELECT id, ? FROM (${rule.sql(true)}) AS x(id) ON CONFLICT DO NOTHING`,
             [rule.name]
           )
-          const added = await trx.rawQuery(
+          round.rules[rule.name].matched += rowCount ?? 0
-            `INSERT INTO release_verdicts (release_id, rule, rule_version)
+        }
-             SELECT j.release_id, ?, ? FROM judged j JOIN releases r ON r.id = j.release_id
+        await trx.rawQuery(`ANALYZE judged`)
-             ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
+        await this.apply(trx, 'judged', true, round)
-             WHERE release_verdicts.rule <> excluded.rule`,
+      })
-            [rule.name, JUDGE_VERSION]
-          )
-          stats.removed = removed.rowCount ?? 0
-          stats.added = added.rowCount ?? 0
-        })
-      }
-      round.rules[rule.name] = stats
     }
     return round
   }
+
+  /**
+   * The matches (judged: release_id, rule; the first rule's each) into release_verdicts: verdicts no rule gives any
+   * more removed (scoped: of the releases in judge_scope only), changed ones written. Only the changes go through the
+   * insert: each match through its conflict check took 60–150 s a rule (LOG.md 2026-10-19 (80)).
+   */
+  private async apply(trx: any, judged: string, scoped: boolean, round: JudgeRound) {
+    const tally = (rows: { rule: string; n: string }[], field: 'added' | 'removed') => {
+      for (const { rule, n } of rows) {
+        round.rules[rule] ??= { matched: 0, added: 0, removed: 0 }
+        round.rules[rule][field] += Number(n)
+      }
+    }
+    const { rows: removed } = await trx.rawQuery(
+      `WITH gone AS (DELETE FROM release_verdicts v
+         WHERE ${scoped ? 'v.release_id IN (SELECT release_id FROM judge_scope) AND' : ''}
+           NOT EXISTS (SELECT 1 FROM ${judged} j WHERE j.release_id = v.release_id)
+         RETURNING v.rule)
+       SELECT rule, count(*) AS n FROM gone GROUP BY 1`
+    )
+    tally(removed, 'removed')
+    const { rows: added } = await trx.rawQuery(
+      `WITH written AS (INSERT INTO release_verdicts (release_id, rule, rule_version)
+         SELECT j.release_id, j.rule, ? FROM ${judged} j LEFT JOIN release_verdicts v ON v.release_id = j.release_id
+         WHERE v.rule IS DISTINCT FROM j.rule AND EXISTS (SELECT 1 FROM releases r WHERE r.id = j.release_id)
+         ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
+         WHERE release_verdicts.rule <> excluded.rule
+         RETURNING rule)
+       SELECT rule, count(*) AS n FROM written GROUP BY 1`,
+      [JUDGE_VERSION]
+    )
+    tally(added, 'added')
+  }
+}
+
+/** "dead 1,157,018 (+1,136 −712), …" for the log. */
+function describe(round: JudgeRound): string {
+  return Object.entries(round.rules)
+    .map(([name, r]) => `${name} ${r.matched.toLocaleString('en')} (+${r.added.toLocaleString('en')} −${r.removed.toLocaleString('en')})`)
+    .join(', ')
 }

app/services/search_service.ts

@@ -57,6 +57,8 @@ export interface ReleaseSummary {
   settled: boolean
   /** The prober found encrypted archive entries or headers; null: not probed (yet). */
   passworded: boolean | null
+  /** A password is known (release_passwords): it's in the NZB. */
+  passwordKnown: boolean
   /** What the prober found inside (an archive entry, a par2 set's file, a yEnc name), when it says more than the name. */
   innerName: string | null
 }
@@ -163,6 +165,7 @@ class SearchService {
     const [{ rows }, { rows: counted }] = await Promise.all([
       db.rawQuery(
         `SELECT r.id, r.name, r.source, r.poster, r.groups, r.posted_at, r.file_total, r.files, r.bytes, r.readable, r.complete, r.category, r.settled_at, r.passworded, r.inner_name,
+                EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id) AS password_known,
                 release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) AS final
          FROM ${source.from} WHERE ${source.where} ORDER BY ${order} LIMIT ? OFFSET ?`,
         [...source.bindings, ...bindings, perPage, offset]
@@ -192,6 +195,7 @@ class SearchService {
         category: r.category,
         settled: r.settled_at !== null,
         passworded: r.passworded,
+        passwordKnown: r.password_known,
         innerName: r.inner_name,
       })),
     }
@@ -201,10 +205,11 @@ class SearchService {
   async release(id: number) {
     const r = await db.from('releases').where('id', id).first()
     if (!r) return null
-    const [files, { rows: inspections }, { rows: finality }] = await Promise.all([
+    const [files, { rows: inspections }, { rows: finality }, passwords] = await Promise.all([
       NzbService.files(id),
       db.rawQuery(`SELECT kind, result, created_at FROM release_inspections WHERE release_id = ? ORDER BY candidate_id`, [id]),
       db.rawQuery(`SELECT release_final(?, ?, ?, ?, ?, ?) AS final`, [r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete]),
+      NzbService.passwords(id),
     ])
     return {
       id: Number(r.id),
@@ -226,6 +231,8 @@ class SearchService {
       lastSeenAt: r.last_seen_at,
       settledAt: r.settled_at,
       passworded: r.passworded,
+      /** Known passwords (spots, their NZBs), the most trusted first; the first is in the NZB. */
+      passwords,
       innerName: r.inner_name,
       inspectedAt: r.inspected_at,
       /** What the prober read from the release's articles (lib/probe Inspection), by probe kind. */

app/services/settler.ts

@@ -15,7 +15,8 @@ const BATCH = 50_000
  * Settles releases (REBUILD.md, part 8): once nothing has been added for
  * QUIET_HOURS (QUIET_HISTORY_MINUTES once its last post is that old), its completeness is decided: complete when every file the
  * [i/N] counter announced was found, unknown without a counter (phase 2
- * reads par2 sets for those). Phase 2 picks its work from settled releases.
+ * decides those by their par2 sets: app/services/par2_completeness.ts).
+ * Phase 2 picks its work from settled releases.
  */
 class Settler {
   /** Up to `batches` batches of BATCH (all there are without). */
@@ -25,13 +26,14 @@ class Settler {
       // Their probe candidates become claimable (Prober.claim), the unreadable ones first.
       const { rows } = await db.rawQuery(
         `WITH settled AS (
-           UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
+           UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END
            WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
              AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
-           RETURNING id, readable, files, bytes, poster),
+           RETURNING id, readable, files, bytes, poster, sidecar_only),
          marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
-         -- Small ones may be an upload's .nfo or .nzb that came apart from it: the merger's sidecar fold looks at them.
+         -- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
-         queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
+         -- from it: the merger's sidecar fold looks at them.
+         queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
            ON CONFLICT DO NOTHING)
          SELECT count(*)::int AS n FROM settled`,
         [QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]

app/services/seven_zip_tails.ts

@@ -0,0 +1,22 @@
+/**
+ * 7z keeps its listing at the end of the last volume, so a split 7z gets a `7z-tail` probe candidate: the last
+ * segment of its highest volume (lib/edge/range_indexer.ts). RAR lists its files in every volume's header, and
+ * its first volume's start (one `archive` candidate) is enough. The live tail reads an upload in pieces while
+ * it's posted, and each piece queues a tail for the last volume it saw so far (TsIgNBewuFbuwWy: 7, for
+ * .002–.008; LOG.md 2026-10-23 (94)); merges bring their members' tails together. Of the given releases'
+ * waiting tails, those below the release's highest volume go. Those of the highest volume stay, all of them:
+ * a piece that saw only part of it queued a segment that isn't its last.
+ */
+export async function pruneSevenZipTails(trx: any, releaseIds: (number | string)[]): Promise<number> {
+  if (releaseIds.length === 0) return 0
+  const result = await trx.rawQuery(
+    `DELETE FROM probe_candidates pc USING (
+       SELECT id, status, volume, max(volume) OVER (PARTITION BY release_id) AS highest
+       FROM (SELECT id, release_id, status, substring(file from '\\.7z\\.(\\d{1,4})$')::int AS volume
+             FROM probe_candidates WHERE release_id = ANY(?) AND kind = '7z-tail') t
+     ) s
+     WHERE pc.id = s.id AND s.status = 'waiting' AND s.volume < s.highest`,
+    [releaseIds.map(String)]
+  )
+  return result.rowCount ?? 0
+}

app/services/sidecar_folder.ts

@@ -6,7 +6,7 @@ const BATCH = 2000
 /** An upload's sidecar is posted with it: its release is looked for this far from it. */
 const WINDOW_MINUTES = 10
 /** The files an upload carries beside its content: an index, a checksum list, a scene release's own files. */
-const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
+export const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
 
 /** The words of a name worth comparing: 4+ letters or digits. */
 function words(text: string): Set<string> {
@@ -53,19 +53,17 @@ export class SidecarFolder {
 
   async round(): Promise<FoldRound> {
     const round: FoldRound = { looked: 0, sidecars: 0, folded: 0 }
+    // A candidate whose release went (merged, deleted) leaves with its batch: a separate scan for those over the whole
+    // queue took 5+ min a round at 13M (LOG.md 2026-10-17 (72)).
     const { rows: queued } = await db.rawQuery(
-      `SELECT r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
+      `SELECT f.release_id AS queued_id, r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
-       FROM fold_candidates f JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
+       FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
       [BATCH]
     )
-    const { rows: stale } = await db.rawQuery(
+    const done: number[] = queued.map((r: { queued_id: string }) => Number(r.queued_id))
-      `SELECT f.release_id FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id WHERE r.id IS NULL LIMIT ?`,
-      [BATCH]
-    )
-    const done: number[] = stale.map((r: { release_id: string }) => Number(r.release_id))
     for (const c of queued) {
+      if (c.id === null) continue
       round.looked++
-      done.push(Number(c.id))
       const { rows: chunks } = await db.rawQuery(`SELECT payload FROM release_chunks WHERE release_id = ?`, [c.id])
       const files: ChunkFile[] = chunks.flatMap((x: { payload: Buffer }) => decodeChunk(x.payload))
       if (files.length === 0 || !files.every((f) => SIDECAR.test(f.name))) continue

app/services/spot_inspector.ts

@@ -102,6 +102,12 @@ export class SpotInspector {
         this.client = undefined
       }
       if (!round || round.inspected === 0) {
+        // Nothing to inspect for a while: closed (nntpIdleCloseMs), opened again with work.
+        const idleMs = coordinatorConfig.nntpIdleCloseMs
+        if (idleMs > 0 && IDLE_MS >= idleMs) {
+          this.client?.close()
+          this.client = undefined
+        }
         for (let waited = 0; waited < IDLE_MS && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
       }
     }
@@ -288,6 +294,12 @@ export class SpotInspector {
           .ignore()
         await trx.from('releases').whereIn('id', [...perRelease.keys()]).update({ passworded: true })
       }
+      // The spot's category for the ones whose names say nothing (migration 5200: spot_category).
+      await trx.rawQuery(`UPDATE releases SET category = c FROM (SELECT spot_category(?::integer, ?::jsonb) AS c) s WHERE id = ANY(?) AND category = 8010 AND c IS NOT NULL`, [
+        spot.category,
+        JSON.stringify(spot.subcats ?? {}),
+        [...perRelease.keys()],
+      ])
       const top = [...perRelease.entries()].sort((a, b) => b[1] - a[1])[0]
       // The NZB is only kept for a release of the spot's own: the header pass's releases have their chunks.
       await trx.from('spots').where('message_id', spotId).update({ matched: fileRelease.size, release_id: top ? top[0] : null, nzb: null })
@@ -341,6 +353,8 @@ export class SpotInspector {
         readable: true,
         passworded: spot.password ? true : null,
         settled_at: new Date(),
+        // Kept when its title says nothing (the category trigger, migration 5200).
+        category: db.raw('coalesce(spot_category(?::integer, ?::jsonb), 8010)', [spot.category, JSON.stringify(spot.subcats ?? {})]),
       })
       .onConflict('release_key')
       .ignore()
@@ -349,6 +363,8 @@ export class SpotInspector {
     const releaseId = Number(row.id)
     await db.transaction(async (trx) => {
       await trx.table('release_spots').insert({ release_id: releaseId, spot_id: spotId, files: files.length }).onConflict(['release_id', 'spot_id']).ignore()
+      // Its poster for poster search, as the batch store adds its releases' (migration 4000).
+      if (spot.poster) await trx.rawQuery(`INSERT INTO posters (poster) VALUES (left(?, 200)) ON CONFLICT DO NOTHING`, [spot.poster])
       if (spot.password) {
         await trx.table('release_passwords').insert({ release_id: releaseId, password: spot.password, source: spot.password_source ?? 'spot-text', spot_id: spotId }).onConflict(['release_id', 'password']).ignore()
       }

app/services/spotter.ts

@@ -60,6 +60,12 @@ export class Spotter {
         this.client = undefined
       }
       const wait = coordinatorConfig.spots.everySeconds * 1000
+      // Quiet until the next round: closed (nntpIdleCloseMs), opened again then.
+      const idleMs = coordinatorConfig.nntpIdleCloseMs
+      if (idleMs > 0 && wait - (Date.now() - started) >= idleMs) {
+        this.client?.close()
+        this.client = undefined
+      }
       for (let waited = Date.now() - started; waited < wait && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
     }
   }

commands/coordinator_run.ts

@@ -6,7 +6,9 @@ import Planner from '#services/planner'
 import FragmentMerger from '#services/fragment_merger'
 import Settler from '#services/settler'
 import RangeBlocks from '#services/range_blocks'
+import Leases from '#services/leases'
 import NntpPool from '#services/nntp_pool'
+import { NntpShare } from '#services/nntp_shares'
 import coordinatorConfig from '#config/coordinator'
 
 const TICK_MS = 2000
@@ -42,6 +44,7 @@ export default class CoordinatorRun extends BaseCommand {
     const [index, count] = (this.shard ?? '1/1').split('/').map(Number)
     if (!(count >= 1 && index >= 1 && index <= count)) throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
     const prefetcher = new Prefetcher({ index: index - 1, count })
+    const share = new NntpShare(count > 1 ? `coordinator ${index}/${count}` : 'coordinator', coordinatorConfig.nntpShares.coordinatorPriority, coordinatorConfig.nntpMinConnections, coordinatorConfig.nntpConnections)
     const main = index === 1
     let stopping = false
     let stopped: () => void
@@ -54,6 +57,21 @@ export default class CoordinatorRun extends BaseCommand {
       await done
     })
     this.logger.info(count > 1 ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).` : 'Coordinator started.')
+    // The pool's size: its share of the account's connections, the prober the rest (app/services/nntp_shares.ts).
+    // Before the first connection: until then it's NNTP_CONNECTIONS.
+    const first = await NntpPool.share(share)
+    this.logger.info(`NNTP share: ${first.granted} connections.`)
+    const sharing = (async () => {
+      while (!stopping) {
+        try {
+          const { was, granted } = await NntpPool.share(share)
+          if (granted !== was) this.logger.info(`NNTP share: ${granted} connections (was ${was}).`)
+        } catch (error: any) {
+          this.logger.error(`NNTP share failed: ${error.message}`)
+        }
+        for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !stopping; waited += TICK_MS) await sleep(TICK_MS)
+      }
+    })()
     // Two loops: the prefetcher's ticks don't wait for merging or planning (a merge round
     // of up to MERGE_BUDGET_MS left the connections idle once their targets were fetched).
     const prefetching = (async () => {
@@ -70,6 +88,8 @@ export default class CoordinatorRun extends BaseCommand {
     const blocks = (async () => {
       while (main && !stopping) {
         try {
+          const abandoned = await Leases.finishAbandoned()
+          if (abandoned > 0) this.logger.info(`Finished ${abandoned} ranges read to their end whose worker went quiet.`)
           await RangeBlocks.rebuild()
         } catch (error: any) {
           this.logger.error(`Range blocks failed: ${error.message}`)
@@ -123,12 +143,13 @@ export default class CoordinatorRun extends BaseCommand {
       }
       await sleep(TICK_MS)
     }
-    await Promise.all([prefetching, blocks])
+    await Promise.all([prefetching, blocks, sharing])
     // The chunks in flight, for at most STOP_WAIT_MS (a connection the server holds back can take minutes); then closed.
     const waitUntil = Date.now() + STOP_WAIT_MS
     while (prefetcher.active > 0 && Date.now() < waitUntil) await sleep(200)
     if (prefetcher.active > 0) this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
     await NntpPool.close()
+    await share.leave().catch(() => {})
     this.logger.info('Coordinator stopped.')
     stopped!()
     await this.terminate()

commands/par2_prune.ts

@@ -0,0 +1,17 @@
+import { BaseCommand } from '@adonisjs/core/ace'
+import type { CommandOptions } from '@adonisjs/core/types/ace'
+import db from '@adonisjs/lucid/services/db'
+import { Par2Pruner } from '#services/par2_pruner'
+
+/** The par2 prune (Par2Pruner) in one go: a whole pass from the first set, instead of the merger's 2 minutes an hour. */
+export default class Par2Prune extends BaseCommand {
+  static commandName = 'par2:prune'
+  static description = 'Drops the par2 file lists the merge can no longer use: one whole pass over every set, then exits'
+  static options: CommandOptions = { startApp: true }
+
+  async run() {
+    await db.from('worker_cursors').where('name', 'par2-prune').update({ key: '' })
+    const round = await new Par2Pruner().round()
+    this.logger.info(`Pruned ${round.rows} par2 file rows: ${round.complete} complete sets, ${round.judged} judged, ${round.final} final.`)
+  }
+}

commands/releases_judge.ts

@@ -5,10 +5,10 @@ import { ReleaseJudge, RULES } from '#services/release_judge'
 /** The junk judge (REBUILD.md, junk): releases judged after their probes; verdicts hide them from search. */
 export default class ReleasesJudge extends BaseCommand {
   static commandName = 'releases:judge'
-  static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable), hourly'
+  static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable): the changed ones every few minutes, all daily'
   static options: CommandOptions = { startApp: true, staysAlive: true }
 
-  @flags.boolean({ description: 'One round, then exit' })
+  @flags.boolean({ description: 'One full round, then exit' })
   declare once: boolean
 
   @flags.boolean({ description: 'With --once: count only, write no verdicts' })
@@ -17,7 +17,7 @@ export default class ReleasesJudge extends BaseCommand {
   async run() {
     const judge = new ReleaseJudge()
     if (this.once) {
-      const round = await judge.round(this.dry)
+      const round = await judge.full(this.dry)
       for (const rule of RULES) {
         const r = round.rules[rule.name]
         if (r) this.logger.info(`${rule.name}: ${r.matched} (+${r.added} −${r.removed}) — ${rule.says}`)

commands/releases_par2_complete.ts

@@ -0,0 +1,38 @@
+import { BaseCommand } from '@adonisjs/core/ace'
+import type { CommandOptions } from '@adonisjs/core/types/ace'
+import db from '@adonisjs/lucid/services/db'
+import { decidePar2Completeness } from '#services/par2_completeness'
+
+/** Releases per batch. */
+const BATCH = 1000
+
+/**
+ * Decides completeness by par2 set (app/services/par2_completeness.ts) for the releases that had their par2 read
+ * before it was built (2026-10-23): every holder of a set whose file list is still there, by release id. Once.
+ */
+export default class ReleasesPar2Complete extends BaseCommand {
+  static commandName = 'releases:par2-complete'
+  static description = 'Decides completeness by their par2 sets for settled releases without a counter, then exits'
+  static options: CommandOptions = { startApp: true }
+
+  async run() {
+    let after = 0
+    let looked = 0
+    let decided = 0
+    for (;;) {
+      const { rows } = await db.rawQuery(
+        `SELECT DISTINCT s.release_id FROM par2_sets s
+         WHERE s.release_id > ? AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
+         ORDER BY s.release_id LIMIT ?`,
+        [after, BATCH]
+      )
+      if (rows.length === 0) break
+      const ids = rows.map((r: { release_id: string }) => Number(r.release_id))
+      decided += await decidePar2Completeness(db, ids)
+      looked += ids.length
+      after = ids.at(-1)!
+      if (looked % 50_000 < BATCH) this.logger.info(`${looked} releases with a par2 set looked at, ${decided} decided (up to id ${after}).`)
+    }
+    this.logger.info(`Done: ${looked} releases with a par2 set looked at, ${decided} decided.`)
+  }
+}

commands/releases_recategorize.ts

@@ -0,0 +1,92 @@
+import { BaseCommand, flags } from '@adonisjs/core/ace'
+import type { CommandOptions } from '@adonisjs/core/types/ace'
+import db from '@adonisjs/lucid/services/db'
+import { setTimeout as sleep } from 'node:timers/promises'
+import { DiskSpace } from '#services/disk_space'
+
+/** Its place in worker_cursors (the last release id done), for a run that was stopped. */
+const CURSOR = 'recategorize'
+
+/**
+ * The categories of releases from before migration 5200 (release_category_for: the fixed name rules, the inner
+ * name, the groups; then the spots'), in batches by id: only rows whose category changes are written, each a new row
+ * version in a 70 GB table on the NVMe, so it waits while the filesystem is under --floor-gb free (autovacuum makes
+ * the dead versions reusable) and goes on from its cursor when run again.
+ */
+export default class ReleasesRecategorize extends BaseCommand {
+  static commandName = 'releases:recategorize'
+  static description = 'Recomputes release categories (name, inner name, groups, spots) in batches; --dry-run counts only'
+  static options: CommandOptions = { startApp: true }
+
+  @flags.number({ description: 'Release ids per batch' })
+  declare batch: number
+
+  @flags.number({ description: 'Stop after this id (a sample with --dry-run)' })
+  declare toId: number
+
+  @flags.number({ description: 'Wait while the NVMe has less free (GB)' })
+  declare floorGb: number
+
+  @flags.boolean({ description: 'Count what would change, write nothing' })
+  declare dryRun: boolean
+
+  @flags.boolean({ description: 'Start from the first id, not the cursor' })
+  declare restart: boolean
+
+  async run() {
+    const batch = this.batch ?? 20_000
+    const space = new DiskSpace('recategorizing', '/', this.floorGb ?? 14)
+    const { rows: ends } = await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM releases`)
+    const hi = Math.min(Number(ends[0].hi), this.toId ?? Infinity)
+    const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
+    let after = this.restart || this.dryRun || !cursors[0] ? Number(ends[0].lo) - 1 : Number(cursors[0].key)
+    const changes = new Map<string, number>()
+    let changed = 0
+    let looked = 0
+    const started = Date.now()
+    while (after < hi) {
+      while (!this.dryRun && (await space.short())) await sleep(60_000)
+      const to = Math.min(after + batch, hi)
+      const fresh = `release_category_for(name, inner_name, groups, NULL)`
+      if (this.dryRun) {
+        const { rows } = await db.rawQuery(
+          `SELECT category AS was, ${fresh} AS now, count(*)::int AS n FROM releases WHERE id > ? AND id <= ? GROUP BY 1, 2`,
+          [after, to]
+        )
+        for (const r of rows) {
+          looked += r.n
+          if (r.was === r.now) continue
+          changed += r.n
+          changes.set(`${r.was} → ${r.now}`, (changes.get(`${r.was} → ${r.now}`) ?? 0) + r.n)
+        }
+      } else {
+        const result = await db.rawQuery(
+          `UPDATE releases r SET category = x.c FROM (SELECT id, ${fresh} AS c FROM releases WHERE id > ? AND id <= ?) x
+           WHERE r.id = x.id AND r.category IS DISTINCT FROM x.c`,
+          [after, to]
+        )
+        changed += result.rowCount ?? 0
+        await db.rawQuery(
+          `INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
+          [CURSOR, String(to)]
+        )
+      }
+      after = to
+      if (Math.floor(after / 5_000_000) !== Math.floor((after - batch) / 5_000_000)) {
+        this.logger.info(`Up to id ${after} of ${hi}: ${changed} changed${this.dryRun ? ` of ${looked}` : ''} (${Math.round((Date.now() - started) / 1000)} s).`)
+      }
+    }
+    if (this.dryRun) {
+      this.logger.info(`Would change ${changed} of ${looked}:`)
+      for (const [change, n] of [...changes].sort((a, b) => b[1] - a[1]).slice(0, 30)) this.logger.info(`  ${change}: ${n}`)
+      return
+    }
+    // The spots' categories for the linked releases whose names say nothing.
+    const spots = await db.rawQuery(
+      `UPDATE releases r SET category = spot_category(s.category, s.subcats)
+       FROM release_spots rs JOIN spots s ON s.message_id = rs.spot_id
+       WHERE r.id = rs.release_id AND r.category = 8010 AND spot_category(s.category, s.subcats) IS NOT NULL`
+    )
+    this.logger.info(`Done: ${changed} releases recategorized, ${spots.rowCount ?? 0} by their spots.`)
+  }
+}

config/coordinator.ts

@@ -31,8 +31,21 @@ export default {
   },
   /** A leased range without a heartbeat for this long goes back to the queue. */
   leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
-  /** NNTP connections the coordinator fetches headers with. */
+  /** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
   nntpConnections: env.get('NNTP_CONNECTIONS', 4),
+  nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
+  /**
+   * The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
+   * all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
+   * and 2 for a hedged request's or a reconnect's new connection while the old one closes).
+   */
+  nntpShares: {
+    total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
+    everyMs: 15_000,
+    /** Headers before probes: history takes what it can use, the prober the rest. */
+    coordinatorPriority: 2,
+    proberPriority: 1,
+  },
   /**
    * A reply with no data for this long fails, and its connection is replaced.
    * Eweka sometimes stops answering on long-lived connections (all 40 quiet,
@@ -68,13 +81,24 @@ export default {
    * under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
    */
   nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
+  /**
+   * A connection with nothing to do for this long is closed, and opened again
+   * when there's work. 0: kept. Eweka drops connections that stay quiet, and
+   * the next command on one failed ("Spot inspection failed: NNTP connection
+   * closed"); idle ones also take the account's connections from downloads.
+   */
+  nntpIdleCloseMs: env.get('NNTP_IDLE_CLOSE_SECONDS', 60) * 1000,
   probe: {
     /**
-     * NNTP connections the prober reads articles with, besides the
+     * At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
      * coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
      * couple stay free for downloads. ~7 MB/s each (~10 articles/s).
      */
     connections: env.get('PROBE_CONNECTIONS', 6),
+    /** Of them, always (its share's minimum; nntpShares). */
+    minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
+    /** No claims while the filesystem under `/` (the NVMe, in the container) has less free (app/services/disk_space.ts). 0: off. */
+    minFreeGb: env.get('PROBE_MIN_FREE_GB', 0),
     /** Candidates claimed at a time. */
     batch: 400,
     /** BODY commands in flight per connection (pipelined; XoverClient.bodies). */

database/migrations/1770000004200_releases_sidecar_only.ts

@@ -0,0 +1,16 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * releases.sidecar_only: set by the batch store when it inserts a small release (at most 3 files, under 1 MB) whose
+ * files are all sidecars (.nfo, .sfv, .nzb, …). The settler queues only those for the sidecar fold: it queued every
+ * small release, 13.5M (single images mostly), and the fold's queue scans held the merger back (LOG.md 2026-10-17 (72)).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS sidecar_only boolean`)
+  }
+
+  async down() {
+    this.schema.raw(`ALTER TABLE releases DROP COLUMN IF EXISTS sidecar_only`)
+  }
+}

database/migrations/1770000004300_worker_cursors_key.ts

@@ -0,0 +1,16 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * worker_cursors.key: a tie-breaker after `at`. The par2 pruner's cursor was a timestamp alone, read back to the
+ * millisecond: 20,000 sets of one prober flush share created_at 2026-10-15 14:38:39.883984, and every round took the
+ * same batch again (LOG.md 2026-10-17 (73)). Now (at to the microsecond, set_id).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`ALTER TABLE worker_cursors ADD COLUMN IF NOT EXISTS key text`)
+  }
+
+  async down() {
+    this.schema.raw(`ALTER TABLE worker_cursors DROP COLUMN IF EXISTS key`)
+  }
+}

database/migrations/1770000004400_nntp_shares.ts

@@ -0,0 +1,26 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * nntp_shares: the account's connections shared by the processes that read articles (app/services/nntp_shares.ts).
+ * Each process's row says what it could use and has open, and what it may open; a row without a heartbeat for a
+ * minute is a process gone (its connections closed with it). The split was fixed per process: with history paused
+ * the coordinator's 38 stood idle while the prober's 8 were the limit (LOG.md 2026-10-18 (75)).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`
+      CREATE TABLE IF NOT EXISTS nntp_shares (
+        name text PRIMARY KEY,
+        priority integer NOT NULL,
+        min integer NOT NULL,
+        want integer NOT NULL,
+        open integer NOT NULL,
+        granted integer NOT NULL,
+        heartbeat_at timestamptz NOT NULL DEFAULT now()
+      )`)
+  }
+
+  async down() {
+    this.schema.raw(`DROP TABLE IF EXISTS nntp_shares`)
+  }
+}

database/migrations/1770000004500_probe_cache_pkey_on_nvme.ts

@@ -0,0 +1,26 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * probe_cache's primary key in the catalog's tablespace (default_tablespace: nvme on homelab); the table stays on the
+ * HDD. Each claim looks its candidates up by message_id (`message_id = ANY(…)`): with the index on the HDD that took
+ * 1.27 s a claim, 48 s of each minute, the prober's lanes waiting on claims (LOG.md 2026-10-18 (76)). ~97% aren't in it,
+ * so only the index is read. Homelab moved it by hand (prober stopped); a no-op where it's there already, or without a
+ * default tablespace (tests).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.defer(async (db) => {
+      const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
+      const target = setting[0].target
+      if (!target) return
+      const { rows } = await db.rawQuery(
+        `SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
+         WHERE c.relname = 'probe_cache_pkey' AND coalesce(t.spcname, 'pg_default') <> ?`,
+        [target]
+      )
+      if (rows.length > 0) await db.rawQuery(`ALTER INDEX probe_cache_pkey SET TABLESPACE ${target}`)
+    })
+  }
+
+  async down() {}
+}

database/migrations/1770000004600_release_inspections_on_hdd.ts

@@ -0,0 +1,12 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * Was: release_inspections' rows on the HDD (pg_default), for the NVMe's room during the probe backlog (LOG.md
+ * 2026-10-18 (79)). The judge's fake rule reads all of it every round: on the HDD that took 17–18 min, was ~3
+ * (LOG.md 2026-10-19 (81)). Migration 4800 moves it back; nothing here now.
+ */
+export default class extends BaseSchema {
+  async up() {}
+
+  async down() {}
+}

database/migrations/1770000004700_judge_indexes.ts

@@ -0,0 +1,22 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * The junk judge's locked rule by an index of the passworded releases (1.2M of 74M), not a scan of all of releases
+ * every round (30 s; LOG.md 2026-10-19 (80)). And the posters of the spot inspector's own releases (source `spot`),
+ * which it didn't add to posters: poster search, and the judge's scraped rule by posters, missed them.
+ */
+export default class extends BaseSchema {
+  static disableTransactions = true
+
+  async up() {
+    this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS releases_passworded_idx ON releases (id) WHERE passworded`)
+    this.schema.raw(
+      `INSERT INTO posters (poster) SELECT DISTINCT left(s.poster, 200) FROM spots s
+       JOIN releases r ON r.id = s.release_id AND r.source = 'spot' WHERE s.poster IS NOT NULL ON CONFLICT DO NOTHING`
+    )
+  }
+
+  async down() {
+    this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS releases_passworded_idx`)
+  }
+}

database/migrations/1770000004800_release_inspections_on_nvme.ts

@@ -0,0 +1,25 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * release_inspections back in the catalog's tablespace (default_tablespace: nvme on homelab), where migration 4600 had
+ * moved it to the HDD: the judge's fake rule reads all of it every round, 17–18 min on the HDD, ~3 on the NVMe (LOG.md
+ * 2026-10-19 (81)). Homelab moved it by hand (prober, merger and judge stopped); a no-op where it's there already, or
+ * without a default tablespace (tests).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.defer(async (db) => {
+      const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
+      const target = setting[0].target
+      if (!target) return
+      const { rows } = await db.rawQuery(
+        `SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
+         WHERE c.relname = 'release_inspections' AND c.relkind = 'r' AND coalesce(t.spcname, 'pg_default') <> ?`,
+        [target]
+      )
+      if (rows.length > 0) await db.rawQuery(`ALTER TABLE release_inspections SET TABLESPACE ${target}`)
+    })
+  }
+
+  async down() {}
+}

database/migrations/1770000004900_judge_dirty.ts

@@ -0,0 +1,119 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * The junk judge by what changed (LOG.md 2026-10-19 (83)): a full round over every release took 35–38 min of each
+ * hour, longer with every inspection the prober added. Triggers note each release whose facts a rule reads changed
+ * (judge_dirty: its probes' statuses, inspections, file hashes, passwords, spots, par2 sets, merges, its own judged
+ * columns); the judge judges those every few minutes, and every release in a daily full round (judge_state: when the
+ * last one was, by which rule set), its matches gathered in judge_full. judge_dirty is unlogged: what a crash loses
+ * the next full round judges.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_dirty (release_id bigint NOT NULL)`)
+    this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_full (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL)`)
+    this.schema.raw(
+      `CREATE TABLE IF NOT EXISTS judge_state (id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1), full_at timestamptz NOT NULL, version integer NOT NULL)`
+    )
+    // The verdicts are an hourly full round's: the next full one a day from now.
+    this.schema.raw(`INSERT INTO judge_state (full_at, version) VALUES (now(), 1) ON CONFLICT DO NOTHING`)
+
+    // Rows of a table with release_id: the releases of the rows inserted, deleted, or updated (both, when moved).
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION judge_dirty_by_release_id() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        IF TG_OP IN ('INSERT', 'UPDATE') THEN
+          INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM new_rows WHERE release_id IS NOT NULL;
+        END IF;
+        IF TG_OP IN ('UPDATE', 'DELETE') THEN
+          INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM old_rows WHERE release_id IS NOT NULL;
+        END IF;
+        RETURN NULL;
+      END $$`)
+    // probe_candidates: only a status that counts for the dead rule, or a candidate moved by a merge (not the claims'
+    // 'taken', the settler's ready).
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION judge_dirty_probes() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        INSERT INTO judge_dirty (release_id)
+        SELECT DISTINCT unnest(ARRAY[n.release_id, o.release_id]) FROM new_rows n JOIN old_rows o ON o.id = n.id
+        WHERE n.release_id IS DISTINCT FROM o.release_id
+           OR (n.status IS DISTINCT FROM o.status AND (n.status IN ('done', 'missing', 'error') OR o.status IN ('done', 'missing', 'error')));
+        RETURN NULL;
+      END $$`)
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION judge_dirty_merges() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        INSERT INTO judge_dirty (release_id) SELECT DISTINCT into_id FROM new_rows;
+        RETURN NULL;
+      END $$`)
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION judge_dirty_releases() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        IF TG_OP = 'INSERT' THEN
+          INSERT INTO judge_dirty (release_id) SELECT id FROM new_rows;
+        ELSIF TG_OP = 'DELETE' THEN
+          INSERT INTO judge_dirty (release_id) SELECT id FROM old_rows;
+        ELSE
+          INSERT INTO judge_dirty (release_id) VALUES (NEW.id);
+        END IF;
+        RETURN NULL;
+      END $$`)
+
+    for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets']) {
+      this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON ${table}`)
+      this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON ${table}`)
+      this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON ${table}`)
+      this.schema.raw(
+        `CREATE TRIGGER judge_dirty_insert AFTER INSERT ON ${table} REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+      )
+      this.schema.raw(
+        `CREATE TRIGGER judge_dirty_update AFTER UPDATE ON ${table} REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+      )
+      this.schema.raw(
+        `CREATE TRIGGER judge_dirty_delete AFTER DELETE ON ${table} REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+      )
+    }
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON probe_candidates`)
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON probe_candidates`)
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_update AFTER UPDATE ON probe_candidates REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_probes()`
+    )
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_delete AFTER DELETE ON probe_candidates REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+    )
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON release_merges`)
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_insert AFTER INSERT ON release_merges REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_merges()`
+    )
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON releases`)
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON releases`)
+    this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON releases`)
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_insert AFTER INSERT ON releases REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
+    )
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_delete AFTER DELETE ON releases REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
+    )
+    // Per row, and only when a column a rule reads changed: the tail and the settler update releases all the time.
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_update AFTER UPDATE ON releases FOR EACH ROW
+       WHEN (OLD.name IS DISTINCT FROM NEW.name OR OLD.poster IS DISTINCT FROM NEW.poster OR OLD.source IS DISTINCT FROM NEW.source
+          OR OLD.file_total IS DISTINCT FROM NEW.file_total OR OLD.passworded IS DISTINCT FROM NEW.passworded
+          OR OLD.settled_at IS DISTINCT FROM NEW.settled_at OR OLD.readable IS DISTINCT FROM NEW.readable
+          OR OLD.inner_name IS DISTINCT FROM NEW.inner_name OR OLD.group_id IS DISTINCT FROM NEW.group_id
+          OR OLD.first_article IS DISTINCT FROM NEW.first_article OR OLD.last_article IS DISTINCT FROM NEW.last_article
+          OR OLD.posted_at IS DISTINCT FROM NEW.posted_at OR OLD.last_posted_at IS DISTINCT FROM NEW.last_posted_at
+          OR OLD.complete IS DISTINCT FROM NEW.complete)
+       EXECUTE FUNCTION judge_dirty_releases()`
+    )
+  }
+
+  async down() {
+    for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets', 'probe_candidates', 'release_merges', 'releases']) {
+      for (const event of ['insert', 'update', 'delete']) this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_${event} ON ${table}`)
+    }
+    this.schema.raw(`DROP FUNCTION IF EXISTS judge_dirty_by_release_id(), judge_dirty_probes(), judge_dirty_merges(), judge_dirty_releases()`)
+    this.schema.raw(`DROP TABLE IF EXISTS judge_dirty, judge_full, judge_state`)
+  }
+}

database/migrations/1770000005000_file_hashes_hash_size_release.ts

@@ -0,0 +1,21 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * file_hashes by (hash16k, size, release_id), in place of (hash16k): how many releases have a file, from the index
+ * alone. The judge counts that for every file of each changed release (repeated); with the row fetched for size and
+ * release_id per entry, a file posted in 100k releases cost 100k heap reads every round (24–32 s a chunk; LOG.md
+ * 2026-10-19 (84)). The merger and pruner join on (hash16k, size) too.
+ */
+export default class extends BaseSchema {
+  static disableTransactions = true
+
+  async up() {
+    this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_size_release_idx ON file_hashes (hash16k, size, release_id)`)
+    this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_idx`)
+  }
+
+  async down() {
+    this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_idx ON file_hashes (hash16k)`)
+    this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_size_release_idx`)
+  }
+}

database/migrations/1770000005100_late_candidates_ready.ts

@@ -0,0 +1,17 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * Probe candidates added to a release after it settled (a later range of its upload, a merge into it) stayed not
+ * ready: the settler marks a release's candidates only when it settles it (LOG.md 2026-10-20 (87)). 93,642 waited
+ * since 2026-10-13 (17k lone, 31k par2, 45k archive). The batch store and the merge mark them now; these are made
+ * claimable once.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`
+      UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
+      WHERE pc.status = 'waiting' AND NOT pc.ready AND r.id = pc.release_id AND r.settled_at IS NOT NULL`)
+  }
+
+  async down() {}
+}

database/migrations/1770000005200_release_category_trigger.ts

@@ -0,0 +1,128 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * A release's category from more than its name (REBUILD.md, part 9; LOG.md 2026-10-23 (95)). Migration 1200 made it
+ * a generated column of release_category(name), and knex took the `?` in its regexes for bindings: the function held
+ * `blu-$1ray`, `web-$2dl`, `h\.$326[45]`, so BluRay, WEB-DL and H.264 names without a resolution were Misc. Now:
+ * - release_category(name), fixed (no `?` in any regex here: `{0,1}`) and with XXX, audiobooks, console and PC games,
+ *   and TV codes the old one missed (`S01e07[`, `S04E01E02`);
+ * - the inner name the prober found (an obfuscated release's real name) when the name says nothing;
+ * - else a category given before (a spot's, below), else the groups' (erotica, sounds, ebooks, …);
+ * - a spot's category (spot_category: its head category and subcategories), set by the spot inspector.
+ * `category` becomes a plain column (DROP EXPRESSION keeps the values, no rewrite) set by a trigger; existing rows are
+ * recategorized by `releases:recategorize` in batches, not here (a rewrite of the 70 GB table).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(String.raw`
+      CREATE OR REPLACE FUNCTION release_category(name text) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
+        SELECT CASE
+          -- XXX first: its names carry years, resolutions and episode-like codes too.
+          WHEN name ~* '(^|[^a-z0-9])xxx([^a-z0-9]|$)' THEN
+            CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 6045
+                 WHEN name ~* '720p|1080p|x26[45]|h\.{0,1}26[45]|hevc' THEN 6040
+                 WHEN name ~* 'imageset|picset|(^|[._ -])pics([._ -]|$)' THEN 6060 ELSE 6070 END
+          -- TV: S01E02 (also S01e02[, S04E01E02), 1x02, or a season pack (S01).
+          WHEN name ~* '(^|[^a-z0-9])(s[0-9]{1,2}[ ._-]{0,1}e[0-9]{1,3}|[0-9]{1,2}x[0-9]{2})([^0-9]|$)' OR name ~* '(^|[._ -])s[0-9]{1,2}([._ -]|$)' THEN
+            CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 5045
+                 WHEN name ~* '720p|1080p' THEN 5040 ELSE 5030 END
+          -- Movies: a year plus a source or quality marker.
+          WHEN name ~* '(^|[._ (-])(19|20)[0-9]{2}([._ )-]|$)'
+               AND name ~* '(480p|576p|720p|1080p|2160p|blu-{0,1}ray|web-{0,1}dl|webrip|bdrip|brrip|dvdrip|hdtv|x26[45]|h\.{0,1}26[45]|hevc|remux)' THEN
+            CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 2045
+                 WHEN name ~* '720p|1080p|blu-{0,1}ray' THEN 2040 ELSE 2030 END
+          -- Audio: audiobooks, lossless, mp3, or scene music names (-WEB-2026-GROUP).
+          WHEN name ~* '(audiobook|audioboek|luisterboek|h(oe|ö)rbuch)' THEN 3030
+          WHEN name ~* '(^|[._ -])flac([._ -]|$)' THEN 3040
+          WHEN name ~* '(^|[._ -])mp3([._ -]|$)' OR name ~* '-(WEB|CD|CDM|CDS|EP|VINYL|VLS|SAT|FM|DAB)-(19|20)[0-9]{2}-[A-Za-z0-9]+$' THEN 3010
+          WHEN name ~* '(^|[._ -])(epub|ebook|e-book|mobi|azw3)([._ -]|$)' THEN 7020
+          -- Games: console tags, PC scene groups and repackers.
+          WHEN name ~* '(^|[._ -])(nsw|ps4|ps5|xboxone|xbox360|wiiu|3ds|nds)([._ -]|$)' THEN 1000
+          WHEN name ~* '-(codex|plaza|skidrow|reloaded|flt|doge|tenoke|rune|empress|razor1911|tinyiso|darksiders|simplex)$' OR name ~* '(fitgirl|dodi|elamigos)[._ -]{0,1}repack' THEN 4050
+          ELSE 8010
+        END
+      $$`)
+    this.schema.raw(String.raw`
+      CREATE OR REPLACE FUNCTION release_group_category(groups jsonb) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
+        SELECT CASE
+          WHEN g ~* '(erotica|xxx|porn|erotic|nude|naked)' THEN CASE WHEN g ~* '(pictures|pics|images|photos)' THEN 6060 ELSE 6070 END
+          WHEN g ~* 'anime' THEN 5070
+          WHEN g ~* 'audiobook' THEN 3030
+          WHEN g ~* '(sounds|mp3|music)' THEN CASE WHEN g ~* '(lossless|flac)' THEN 3040 ELSE 3010 END
+          WHEN g ~* 'comics' THEN 7030
+          WHEN g ~* 'e-{0,1}books{0,1}' THEN 7020
+          WHEN g ~* '(games|nintendo|xbox|playstation|wii)' THEN 1000
+          WHEN g ~* '(^|[. ])(tv|hdtv|teevee|tvseries)([. ]|$)' THEN 5000
+          WHEN g ~* '(^|[. ])(movies|dvd|bluray|dvdr)([. ]|$)' THEN 2000
+          ELSE 8010
+        END
+        FROM (SELECT coalesce(string_agg(value, ' '), '') AS g FROM jsonb_array_elements_text(CASE WHEN jsonb_typeof(groups) = 'array' THEN groups ELSE '[]' END)) x
+      $$`)
+    // The name's, else the inner name's, else one given before (a spot's), else the groups'.
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION release_category_for(name text, inner_name text, groups jsonb, given integer) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
+        SELECT CASE WHEN n <> 8010 THEN n WHEN i <> 8010 THEN i WHEN coalesce(given, 8010) <> 8010 THEN given ELSE release_group_category(groups) END
+        FROM (SELECT release_category(name) AS n, coalesce(release_category(inner_name), 8010) AS i) x
+      $$`)
+    // Whether a spot's subcategories of one type letter ({"a": [9], "z": [0]}) hold one of `vals`.
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION spot_has(subcats jsonb, letter text, vals integer[]) RETURNS boolean IMMUTABLE LANGUAGE sql AS $$
+        SELECT coalesce((SELECT bool_or((e #>> '{}')::integer = ANY(vals)) FROM jsonb_array_elements(
+          CASE WHEN jsonb_typeof(subcats->letter) = 'array' THEN subcats->letter ELSE '[]' END) e), false)
+      $$`)
+    // A spot's category (lib/spotnet/categories.ts) in Newznab's; null for pictures and the unknown.
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION spot_category(category integer, subcats jsonb) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
+        SELECT CASE category
+          WHEN 0 THEN CASE
+            WHEN spot_has(subcats, 'z', '{3}') THEN CASE WHEN spot_has(subcats, 'a', '{15}') THEN 6045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 6040
+                                                       WHEN spot_has(subcats, 'a', '{12,13}') THEN 6060 ELSE 6070 END
+            WHEN spot_has(subcats, 'z', '{1}') THEN CASE WHEN spot_has(subcats, 'a', '{15}') THEN 5045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 5040 ELSE 5030 END
+            WHEN spot_has(subcats, 'z', '{2}') OR spot_has(subcats, 'a', '{5,11}') THEN 7020
+            WHEN spot_has(subcats, 'z', '{4}') THEN NULL
+            ELSE CASE WHEN spot_has(subcats, 'a', '{15}') THEN 2045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 2040 ELSE 2030 END
+          END
+          WHEN 1 THEN CASE WHEN spot_has(subcats, 'z', '{3}') THEN 3030 WHEN spot_has(subcats, 'a', '{2,4,7,8}') OR spot_has(subcats, 'c', '{8}') THEN 3040 ELSE 3010 END
+          WHEN 2 THEN CASE WHEN spot_has(subcats, 'a', '{0,1,2}') THEN 4050 ELSE 1000 END
+          WHEN 3 THEN 4000
+        END
+      $$`)
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION releases_set_category() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        NEW.category := release_category_for(NEW.name, NEW.inner_name, NEW.groups, CASE WHEN TG_OP = 'UPDATE' THEN OLD.category ELSE NEW.category END);
+        RETURN NEW;
+      END
+      $$`)
+    // ACCESS EXCLUSIVE for a moment, while the workers write: a few tries instead of a queue behind them.
+    this.schema.raw(`
+      DO $$
+      BEGIN
+        FOR attempt IN 1..30 LOOP
+          BEGIN
+            PERFORM set_config('lock_timeout', '5s', true);
+            IF EXISTS (SELECT 1 FROM information_schema.columns WHERE table_name = 'releases' AND column_name = 'category' AND is_generated = 'ALWAYS') THEN
+              ALTER TABLE releases ALTER COLUMN category DROP EXPRESSION;
+            END IF;
+            DROP TRIGGER IF EXISTS releases_category ON releases;
+            CREATE TRIGGER releases_category BEFORE INSERT OR UPDATE OF name, inner_name, groups ON releases
+              FOR EACH ROW EXECUTE FUNCTION releases_set_category();
+            RETURN;
+          EXCEPTION WHEN lock_not_available THEN
+            PERFORM pg_sleep(2);
+          END;
+        END LOOP;
+        RAISE EXCEPTION 'releases: no lock for the category trigger in 30 tries';
+      END
+      $$`)
+  }
+
+  async down() {
+    this.schema.raw(`DROP TRIGGER IF EXISTS releases_category ON releases`)
+    this.schema.raw(`DROP FUNCTION IF EXISTS releases_set_category()`)
+    this.schema.raw(`DROP FUNCTION IF EXISTS spot_category(integer, jsonb)`)
+    this.schema.raw(`DROP FUNCTION IF EXISTS spot_has(jsonb, text, integer[])`)
+    this.schema.raw(`DROP FUNCTION IF EXISTS release_category_for(text, text, jsonb, integer)`)
+    this.schema.raw(`DROP FUNCTION IF EXISTS release_group_category(jsonb)`)
+  }
+}

deploy/compose.yaml

@@ -115,11 +115,14 @@ services:
       <<: *app-env
       NNTP_CONNECTIONS: ${NNTP_CONNECTIONS:-38}
 
-  # Phase 2's prober: reads the probe queue's articles over PROBE_CONNECTIONS
+  # Phase 2's prober: reads the probe queue's articles over connections of its
-  # connections of its own. The account has 50: the coordinator's 38 (+ its
+  # own. The account has 50 (NNTP_ACCOUNT_CONNECTIONS): 5 reserved (the
-  # plain one), the prober's 8, the spotter's and the spot inspector's 1 each,
+  # coordinator's plain one, the spotter's and the spot inspector's 1 each, 2
-  # 2 to spare (a hedged request closes its connection and opens another)
+  # to spare: a hedged request closes its connection and opens another; with
-  # (with 10 the spot inspector got 502 Too many connections).
+  # 10 the spot inspector got 502 Too many connections), the other 45 shared
+  # by the coordinator (NNTP_MIN_CONNECTIONS–NNTP_CONNECTIONS) and the prober
+  # (PROBE_MIN_CONNECTIONS–PROBE_CONNECTIONS), headers first
+  # (app/services/nntp_shares.ts).
   prober:
     <<: *app
     command: ['node', 'ace', 'probe:run']
@@ -127,6 +130,9 @@ services:
     environment:
       <<: *app-env
       PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-8}
+      # No claims under this much free on the NVMe (its overlay root): 2 GB above the 10 GB floor, for what the
+      # tail, the merger and the judge write meanwhile.
+      PROBE_MIN_FREE_GB: ${PROBE_MIN_FREE_GB:-12}
 
   # The par2 merge: releases whose files one par2 set describes (what the prober read) become one. No NNTP.
   merger:
@@ -134,8 +140,8 @@ services:
     command: ['node', 'ace', 'releases:merge']
     volumes: []
 
-  # The junk judge: releases judged hourly by what their probes showed (dead, fake, repeated, locked, unusable);
+  # The junk judge: releases judged by what their probes showed (dead, fake, repeated, locked, unusable), the changed
-  # verdicts hide them from search. No NNTP.
+  # ones every 2 minutes, all daily; verdicts hide them from search. No NNTP.
   judge:
     <<: *app
     command: ['node', 'ace', 'releases:judge']

edge/src/indexer.rs

@@ -915,12 +915,17 @@ fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate>
     let c = |kind, s: &Seg| Candidate { release_key: key.to_string(), upload, kind, message_id: s.message_id.clone(), file: s.file.name.clone() };
     for f in files {
         let name = &f[0].file.name;
-        if let Some(volume) = names::seven_zip_volume(name).filter(|&v| v > 1) {
+        let volume = names::seven_zip_volume(name);
+        if let Some(volume) = volume.filter(|&v| v > 1) {
             if tail.is_none_or(|(v, _)| volume > v) {
                 tail = Some((volume, f));
             }
             continue;
         }
+        // An unsplit 7z of more than one segment: its listing is in its last one.
+        if volume == Some(0) && f.len() > 1 && tail.is_none() {
+            tail = Some((0, f));
+        }
         if names::is_archive_start(name) {
             out.push(c("archive", &f[0]));
         } else if names::is_par2(name) && !names::is_later_volume(name) {
@@ -965,6 +970,40 @@ mod tests {
         assert_eq!(batch.releases[0].key, batch.releases[1].key);
     }
 
+    #[test]
+    fn archive_candidates_as_in_the_typescript() {
+        let mut priorities = FxHashMap::default();
+        priorities.insert(GROUP.to_string(), 1);
+        let mut ix = RangeIndexer::new(GROUP, 1, 10_000, 1, Rules::defaults(priorities), 5);
+        let file = |at: i64, release: &str, name: &str, n: usize, parts: i64| -> Vec<String> {
+            (0..parts).map(|i| line(at + i, &format!(r#"{release} [{n}/3] - "{name}" yEnc ({}/{parts})"#, i + 1), 0)).collect()
+        };
+        let mut lines = Vec::new();
+        // A split 7z: its start, and the end of its highest volume.
+        lines.extend(file(100, "Split.Seven", "Split.Seven.7z.001", 1, 2));
+        lines.extend(file(110, "Split.Seven", "Split.Seven.7z.003", 3, 2));
+        lines.extend(file(120, "Split.Seven", "Split.Seven.7z.002", 2, 2));
+        // An unsplit 7z of 3 segments: its start and its end.
+        lines.extend(file(200, "Whole.Seven", "Whole.Seven.7z", 1, 3));
+        // A ZIP: its start.
+        lines.extend(file(300, "Some.Zip", "Some.Zip.zip", 1, 2));
+        ix.add_text(&lines.join("\r\n"), 10_001);
+        ix.finish();
+        let batch = ix.take();
+        let mut got: Vec<(&str, &str, &str)> = batch.candidates.iter().map(|c| (c.kind, c.file.as_str(), c.message_id.as_str())).collect();
+        got.sort();
+        assert_eq!(
+            got,
+            vec![
+                ("7z-tail", "Split.Seven.7z.003", "u111@test"),
+                ("7z-tail", "Whole.Seven.7z", "u202@test"),
+                ("archive", "Some.Zip.zip", "u300@test"),
+                ("archive", "Split.Seven.7z.001", "u100@test"),
+                ("archive", "Whole.Seven.7z", "u200@test"),
+            ]
+        );
+    }
+
     #[test]
     fn copies_and_fills_as_in_the_typescript() {
         let hour = 3_600_000i64;

edge/src/names.rs

@@ -1030,7 +1030,7 @@ static SEVEN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.7z$"));
 static PART1: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.part0*1\.rar$"));
 static RAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.rar$"));
 static PART: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.part([0-9]+)\.rar$"));
-static BIN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.bin$"));
+static BIN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(bin|zip)$"));
 static OLD_RAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.r[0-9]{2,3}$"));
 static PAR2_VOL: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.vol[0-9]+[+-][0-9]+\.par2$"));
 pub static PAR2: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.par2$"));

lib/edge/range_indexer.ts

@@ -434,6 +434,8 @@ export function candidatesOf(key: string, files: AggregatedFile<EdgeSegment>[],
       if (!tail || volume > tail.volume) tail = { volume, file: f }
       continue
     }
+    // An unsplit 7z of more than one segment: its listing is in its last one.
+    if (volume === 0 && f.segments.length > 1 && !tail) tail = { volume, file: f }
     if (isArchiveStart(name)) out.push({ releaseKey: key, upload, kind: 'archive', messageId: f.segments[0].messageId, file: name })
     else if (/\.par2$/i.test(name) && !isLaterVolume(name)) out.push({ releaseKey: key, upload, kind: 'par2', messageId: f.segments[0].messageId, file: name })
   }

lib/files/volumes.ts

@@ -1,10 +1,11 @@
 /**
  * Files whose first segment says what an archive holds: named first RAR
- * volumes (.part01.rar, .rar), 7z (.7z, .7z.001) and `.bin`, an extension
+ * volumes (.part01.rar, .rar), 7z (.7z, .7z.001: its type; the listing is at
+ * its end, the 7z-tail), ZIP (its first entries) and `.bin`, an extension
  * uploaders use for archive volumes too (sniffed by content).
  */
 export function isArchiveStart(name: string): boolean {
-  if (/\.bin$/i.test(name)) return true
+  if (/\.(bin|zip)$/i.test(name)) return true
   const volume = sevenZipVolume(name)
   if (volume === 0 || volume === 1) return true
   return /\.part0*1\.rar$/i.test(name) || (/\.rar$/i.test(name) && !/\.part\d+\.rar$/i.test(name))

lib/newznab/categories.ts

@@ -1,10 +1,12 @@
-/** Newznab categories this indexer assigns (release_category() in SQL, migration …1200). */
+/** Newznab categories this indexer assigns (release_category_for() and spot_category() in SQL, migration …5200). */
 export const CATEGORIES: { id: number; name: string; subcats: { id: number; name: string }[] }[] = [
+  { id: 1000, name: 'Console', subcats: [] },
   { id: 2000, name: 'Movies', subcats: [{ id: 2030, name: 'SD' }, { id: 2040, name: 'HD' }, { id: 2045, name: 'UHD' }] },
-  { id: 3000, name: 'Audio', subcats: [{ id: 3010, name: 'MP3' }, { id: 3040, name: 'Lossless' }] },
+  { id: 3000, name: 'Audio', subcats: [{ id: 3010, name: 'MP3' }, { id: 3030, name: 'Audiobook' }, { id: 3040, name: 'Lossless' }] },
-  { id: 4000, name: 'PC', subcats: [] },
+  { id: 4000, name: 'PC', subcats: [{ id: 4050, name: 'Games' }] },
-  { id: 5000, name: 'TV', subcats: [{ id: 5030, name: 'SD' }, { id: 5040, name: 'HD' }, { id: 5045, name: 'UHD' }] },
+  { id: 5000, name: 'TV', subcats: [{ id: 5030, name: 'SD' }, { id: 5040, name: 'HD' }, { id: 5045, name: 'UHD' }, { id: 5070, name: 'Anime' }] },
-  { id: 7000, name: 'Books', subcats: [{ id: 7020, name: 'Ebook' }] },
+  { id: 6000, name: 'XXX', subcats: [{ id: 6040, name: 'x264' }, { id: 6045, name: 'UHD' }, { id: 6060, name: 'ImgSet' }, { id: 6070, name: 'Other' }] },
+  { id: 7000, name: 'Books', subcats: [{ id: 7020, name: 'Ebook' }, { id: 7030, name: 'Comics' }] },
   { id: 8000, name: 'Other', subcats: [{ id: 8010, name: 'Misc' }] },
 ]
 

lib/probe/inspect.ts

@@ -2,6 +2,8 @@ import { createHash } from 'node:crypto'
 import { decodeSegment, YencCrcError, YencError, type DecodedSegment } from '#lib/yenc/index'
 import { isPar2, parsePar2 } from '#lib/par2/index'
 import { parseRarHeaders } from '#lib/rar/index'
+import { parseSevenZipTail } from '#lib/sevenzip/index'
+import { parseZipStart } from '#lib/zip/index'
 import { sniffContentType, type ContentType } from '#lib/content/index'
 
 /** What one probed article showed (REBUILD.md, part 6; stored per release and kind). */
@@ -55,8 +57,11 @@ function largestFirst(items: { name: string; size: number }[]): string[] {
     .map((i) => i.name.slice(0, MAX_NAME_LENGTH))
 }
 
-/** One article's body: its yEnc header, what the first bytes are, and for par2 and RAR what's inside. Throws when it isn't yEnc. */
+/**
-export function inspect(body: Buffer): Inspection {
+ * One article's body: its yEnc header, what the first bytes are, and for par2, RAR, ZIP and 7z what's inside.
+ * Throws when it isn't yEnc. `kind` is the probe candidate's: a 7z-tail's article ends the archive.
+ */
+export function inspect(body: Buffer, kind?: string): Inspection {
   const segment = decodeBody(body)
   const result: Inspection = { yencName: segment.meta.begin.name.slice(0, MAX_NAME_LENGTH), size: segment.meta.begin.size, type: sniffContentType(segment.data) }
   // The file's first 16 KiB, or all of a smaller file.
@@ -85,6 +90,23 @@ export function inspect(body: Buffer): Inspection {
       result.passworded = info.headersEncrypted || files.some((e) => e.encrypted)
     }
   }
+  if (result.type === 'zip') {
+    const start = parseZipStart(segment.data)
+    if (start) {
+      result.entries = start.files.length
+      result.names = largestFirst(start.files)
+      result.passworded = start.encrypted
+    }
+  }
+  // 7z keeps its listing at the end: the last segment of the last volume (a 7z-tail), or a small archive whole.
+  if ((kind === '7z-tail' || result.type === '7z') && segment.offset + segment.data.length === result.size) {
+    const listing = parseSevenZipTail(segment.data)
+    if (listing) {
+      result.entries = listing.files.length
+      if (listing.files.length > 0) result.names = largestFirst(listing.files)
+      result.passworded = listing.encrypted
+    }
+  }
   return result
 }
 
@@ -97,7 +119,7 @@ export function inspect(body: Buffer): Inspection {
 export function innerName(inspections: { kind: string; result: Inspection }[], releaseName = ''): string | null {
   const of = (kind: string) => inspections.filter((i) => i.kind === kind).map((i) => i.result)
   const content = (kind: string) => of(kind).flatMap((i) => i.names ?? []).find((n) => !PACKAGING.test(n))
-  const pick = content('archive') ?? content('par2') ?? of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
+  const pick = content('archive') ?? content('7z-tail') ?? content('par2') ?? of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
   const name = pick?.replace(/^.*[\\/]/, '').slice(0, 255)
   // The release's own name with an extension says nothing new (a random-named file's yEnc name is mostly its random name).
   return name && !name.toLowerCase().startsWith(releaseName.toLowerCase() || '\0') ? name : null

lib/sevenzip/index.ts

@@ -0,0 +1,2 @@
+export { parseSevenZipTail, type SevenZipListing } from '#lib/sevenzip/parse'
+export { lzmaDecode, LzmaError } from '#lib/sevenzip/lzma'

lib/sevenzip/lzma.ts

@@ -0,0 +1,221 @@
+/**
+ * An LZMA decoder (raw stream, as a 7z coder stores it: the properties apart, the unpacked size known), after the
+ * LZMA SDK's reference decoder (LzmaSpec.cpp). For 7z headers: those are small, so the whole output is one buffer.
+ */
+
+const MODEL_BITS = 11
+const BIT_MODEL_TOTAL = 1 << MODEL_BITS
+const MOVE_BITS = 5
+const TOP = 1 << 24
+const POS_BITS_MAX = 4
+const STATES = 12
+const LEN_TO_POS_STATES = 4
+const ALIGN_BITS = 4
+const END_POS_MODEL_INDEX = 14
+const FULL_DISTANCES = 1 << (END_POS_MODEL_INDEX >> 1)
+const MATCH_MIN_LEN = 2
+
+export class LzmaError extends Error {}
+
+const probs = (n: number) => new Uint16Array(n).fill(BIT_MODEL_TOTAL >>> 1)
+
+class RangeDecoder {
+  private range = 0xffffffff
+  private code = 0
+
+  constructor(
+    private input: Uint8Array,
+    private pos: number
+  ) {
+    if (this.byte() !== 0) throw new LzmaError('LZMA: bad stream start')
+    for (let i = 0; i < 4; i++) this.code = ((this.code << 8) | this.byte()) >>> 0
+    if (this.code === this.range) throw new LzmaError('LZMA: corrupt stream')
+  }
+
+  private byte(): number {
+    if (this.pos >= this.input.length) throw new LzmaError('LZMA: input ended')
+    return this.input[this.pos++]
+  }
+
+  private normalize() {
+    if (this.range < TOP) {
+      this.range = (this.range << 8) >>> 0
+      this.code = ((this.code << 8) | this.byte()) >>> 0
+    }
+  }
+
+  direct(bits: number): number {
+    let result = 0
+    do {
+      this.range >>>= 1
+      this.code = (this.code - this.range) >>> 0
+      const t = this.code >>> 31 ? 0xffffffff : 0
+      this.code = (this.code + (this.range & t)) >>> 0
+      this.normalize()
+      result = ((result << 1) | (t ? 0 : 1)) >>> 0
+    } while (--bits)
+    return result
+  }
+
+  bit(p: Uint16Array, i: number): number {
+    const v = p[i]
+    const bound = ((this.range >>> MODEL_BITS) * v) >>> 0
+    let symbol: number
+    if (this.code < bound) {
+      p[i] = v + ((BIT_MODEL_TOTAL - v) >>> MOVE_BITS)
+      this.range = bound
+      symbol = 0
+    } else {
+      p[i] = v - (v >>> MOVE_BITS)
+      this.code = (this.code - bound) >>> 0
+      this.range = (this.range - bound) >>> 0
+      symbol = 1
+    }
+    this.normalize()
+    return symbol
+  }
+
+  tree(p: Uint16Array, offset: number, bits: number): number {
+    let m = 1
+    for (let i = 0; i < bits; i++) m = (m << 1) + this.bit(p, offset + m)
+    return m - (1 << bits)
+  }
+
+  reverseTree(p: Uint16Array, offset: number, bits: number): number {
+    let m = 1
+    let symbol = 0
+    for (let i = 0; i < bits; i++) {
+      const bit = this.bit(p, offset + m)
+      m = (m << 1) + bit
+      symbol |= bit << i
+    }
+    return symbol
+  }
+}
+
+class LenDecoder {
+  private choice = probs(2)
+  private low = probs((1 << POS_BITS_MAX) << 3)
+  private mid = probs((1 << POS_BITS_MAX) << 3)
+  private high = probs(1 << 8)
+
+  decode(rc: RangeDecoder, posState: number): number {
+    if (rc.bit(this.choice, 0) === 0) return rc.tree(this.low, posState << 3, 3)
+    if (rc.bit(this.choice, 1) === 0) return 8 + rc.tree(this.mid, posState << 3, 3)
+    return 16 + rc.tree(this.high, 0, 8)
+  }
+}
+
+/** Decodes `size` bytes of a raw LZMA stream with its 5 property bytes (lc/lp/pb, dictionary size). */
+export function lzmaDecode(properties: Uint8Array, input: Uint8Array, size: number): Buffer {
+  if (properties.length < 5) throw new LzmaError('LZMA: short properties')
+  let d = properties[0]
+  if (d >= 9 * 5 * 5) throw new LzmaError('LZMA: bad properties')
+  const lc = d % 9
+  d = Math.floor(d / 9)
+  const lp = d % 5
+  const pb = Math.floor(d / 5)
+  const dictSize = Math.max(properties[1] | (properties[2] << 8) | (properties[3] << 16) | (properties[4] << 24), 1 << 12) >>> 0
+
+  const out = Buffer.alloc(size)
+  let pos = 0
+  const rc = new RangeDecoder(input, 0)
+  const literals = probs(0x300 << (lc + lp))
+  const posSlot = probs(LEN_TO_POS_STATES << 6)
+  const align = probs(1 << ALIGN_BITS)
+  const posDecoders = probs(1 + FULL_DISTANCES - END_POS_MODEL_INDEX)
+  const isMatch = probs(STATES << POS_BITS_MAX)
+  const isRep = probs(STATES)
+  const isRepG0 = probs(STATES)
+  const isRepG1 = probs(STATES)
+  const isRepG2 = probs(STATES)
+  const isRep0Long = probs(STATES << POS_BITS_MAX)
+  const lenDecoder = new LenDecoder()
+  const repLenDecoder = new LenDecoder()
+
+  const distance = (len: number): number => {
+    const lenState = Math.min(len, LEN_TO_POS_STATES - 1)
+    const slot = rc.tree(posSlot, lenState << 6, 6)
+    if (slot < 4) return slot
+    const directBits = (slot >>> 1) - 1
+    let dist = ((2 | (slot & 1)) * 2 ** directBits) >>> 0
+    if (slot < END_POS_MODEL_INDEX) {
+      dist = (dist + rc.reverseTree(posDecoders, dist - slot, directBits)) >>> 0
+    } else {
+      dist = (dist + rc.direct(directBits - ALIGN_BITS) * 2 ** ALIGN_BITS) >>> 0
+      dist = (dist + rc.reverseTree(align, 0, ALIGN_BITS)) >>> 0
+    }
+    return dist
+  }
+
+  let state = 0
+  let rep0 = 0
+  let rep1 = 0
+  let rep2 = 0
+  let rep3 = 0
+  while (pos < size) {
+    const posState = pos & ((1 << pb) - 1)
+    if (rc.bit(isMatch, (state << POS_BITS_MAX) + posState) === 0) {
+      const prev = pos > 0 ? out[pos - 1] : 0
+      const base = 0x300 * (((pos & ((1 << lp) - 1)) << lc) + (prev >>> (8 - lc)))
+      let symbol = 1
+      if (state >= 7) {
+        let match = out[pos - rep0 - 1]
+        do {
+          const matchBit = (match >>> 7) & 1
+          match <<= 1
+          const bit = rc.bit(literals, base + ((1 + matchBit) << 8) + symbol)
+          symbol = (symbol << 1) | bit
+          if (matchBit !== bit) break
+        } while (symbol < 0x100)
+      }
+      while (symbol < 0x100) symbol = (symbol << 1) | rc.bit(literals, base + symbol)
+      out[pos++] = symbol - 0x100
+      state = state < 4 ? 0 : state < 10 ? state - 3 : state - 6
+      continue
+    }
+    let len: number
+    if (rc.bit(isRep, state) !== 0) {
+      if (pos === 0) throw new LzmaError('LZMA: repeat before any output')
+      if (rc.bit(isRepG0, state) === 0) {
+        if (rc.bit(isRep0Long, (state << POS_BITS_MAX) + posState) === 0) {
+          state = state < 7 ? 9 : 11
+          out[pos] = out[pos - rep0 - 1]
+          pos++
+          continue
+        }
+      } else {
+        let dist: number
+        if (rc.bit(isRepG1, state) === 0) {
+          dist = rep1
+        } else {
+          if (rc.bit(isRepG2, state) === 0) {
+            dist = rep2
+          } else {
+            dist = rep3
+            rep3 = rep2
+          }
+          rep2 = rep1
+        }
+        rep1 = rep0
+        rep0 = dist
+      }
+      len = repLenDecoder.decode(rc, posState)
+      state = state < 7 ? 8 : 11
+    } else {
+      rep3 = rep2
+      rep2 = rep1
+      rep1 = rep0
+      len = lenDecoder.decode(rc, posState)
+      state = state < 7 ? 7 : 10
+      rep0 = distance(len)
+      // The end marker: allowed only where the output is complete, which the loop already ended at.
+      if (rep0 === 0xffffffff) throw new LzmaError('LZMA: end marker before the end')
+      if (rep0 >= dictSize || rep0 >= pos) throw new LzmaError('LZMA: distance out of range')
+    }
+    len += MATCH_MIN_LEN
+    if (len > size - pos) throw new LzmaError('LZMA: match past the end')
+    for (let i = 0; i < len; i++, pos++) out[pos] = out[pos - rep0 - 1]
+  }
+  return out
+}

lib/sevenzip/parse.ts

@@ -0,0 +1,350 @@
+import { crc32 } from 'node:zlib'
+import { lzmaDecode } from '#lib/sevenzip/lzma'
+
+/**
+ * What a 7z archive's header says (its listing): read from the archive's end, where 7z keeps it, so the last
+ * segment of the last volume is enough (the `7z-tail` probe; REBUILD.md, phase 2). The header there is mostly
+ * an encoded one: a pointer to the real header, LZMA-packed just before it, or AES-encrypted (`7z -mhe`), which
+ * hides the names and means a password.
+ */
+export interface SevenZipListing {
+  /** Files, not directories, with their unpacked size (0 for an empty one). */
+  files: { name: string; size: number }[]
+  /** Entries or the header encrypted (AES). */
+  encrypted: boolean
+  /** The header itself encrypted: no names known. */
+  headerEncrypted: boolean
+}
+
+const AES = '06f10701'
+const LZMA = '030101'
+/** A header bigger than this isn't unpacked (headers list names; 64 MiB is ~a million files). */
+const MAX_HEADER = 64 * 1024 * 1024
+
+const ID = {
+  end: 0x00,
+  header: 0x01,
+  archiveProperties: 0x02,
+  additionalStreamsInfo: 0x03,
+  mainStreamsInfo: 0x04,
+  filesInfo: 0x05,
+  packInfo: 0x06,
+  unpackInfo: 0x07,
+  subStreamsInfo: 0x08,
+  size: 0x09,
+  crc: 0x0a,
+  folder: 0x0b,
+  codersUnpackSize: 0x0c,
+  numUnpackStream: 0x0d,
+  emptyStream: 0x0e,
+  emptyFile: 0x0f,
+  name: 0x11,
+  encodedHeader: 0x17,
+  dummy: 0x19,
+}
+
+class Malformed extends Error {}
+
+interface Coder {
+  id: string
+  properties: Uint8Array
+  inStreams: number
+  outStreams: number
+}
+interface Folder {
+  coders: Coder[]
+  bindOut: Set<number>
+  unpackSizes: number[]
+  crc?: number
+}
+interface StreamsInfo {
+  packPos: number
+  packSizes: number[]
+  folders: Folder[]
+  /** Per folder, how many files' streams it holds, and every stream's size in order. */
+  streamsPerFolder: number[]
+  streamSizes: number[]
+}
+
+class Reader {
+  constructor(
+    private buf: Uint8Array,
+    public pos = 0
+  ) {}
+
+  get done() {
+    return this.pos >= this.buf.length
+  }
+
+  byte(): number {
+    if (this.pos >= this.buf.length) throw new Malformed('7z: header ended')
+    return this.buf[this.pos++]
+  }
+
+  bytes(n: number): Uint8Array {
+    if (n < 0 || this.pos + n > this.buf.length) throw new Malformed('7z: header ended')
+    const out = this.buf.subarray(this.pos, this.pos + n)
+    this.pos += n
+    return out
+  }
+
+  /** 7z's variable-length number: the first byte's high bits say how many bytes follow. */
+  number(): number {
+    const first = this.byte()
+    let mask = 0x80
+    let value = 0
+    for (let i = 0; i < 8; i++) {
+      if ((first & mask) === 0) {
+        value += (first & (mask - 1)) * 2 ** (8 * i)
+        break
+      }
+      value += this.byte() * 2 ** (8 * i)
+      mask >>>= 1
+    }
+    if (value > Number.MAX_SAFE_INTEGER) throw new Malformed('7z: number too large')
+    return value
+  }
+
+  uint32(): number {
+    const b = this.bytes(4)
+    return (b[0] | (b[1] << 8) | (b[2] << 16) | (b[3] << 24)) >>> 0
+  }
+
+  bits(n: number): boolean[] {
+    const out: boolean[] = []
+    let byte = 0
+    for (let i = 0; i < n; i++) {
+      if (i % 8 === 0) byte = this.byte()
+      out.push((byte & (0x80 >>> i % 8)) !== 0)
+    }
+    return out
+  }
+
+  /** "All defined" or a bit vector, then a CRC per defined item. */
+  digests(n: number): (number | undefined)[] {
+    const allDefined = this.byte() !== 0
+    const defined = allDefined ? Array(n).fill(true) : this.bits(n)
+    return defined.map((d) => (d ? this.uint32() : undefined))
+  }
+
+  expect(id: number) {
+    if (this.byte() !== id) throw new Malformed(`7z: expected property ${id}`)
+  }
+}
+
+/** Counts that come from the data: a garbage position makes huge ones, which shouldn't allocate. */
+function count(n: number, max = 1_000_000): number {
+  if (n > max) throw new Malformed('7z: implausible count')
+  return n
+}
+
+function readFolder(r: Reader): Folder {
+  const coders: Coder[] = []
+  const numCoders = count(r.number(), 64)
+  if (numCoders === 0) throw new Malformed('7z: folder without coders')
+  for (let i = 0; i < numCoders; i++) {
+    const flags = r.byte()
+    if (flags & 0x80) throw new Malformed('7z: alternative coder methods')
+    const id = Buffer.from(r.bytes(flags & 0x0f)).toString('hex')
+    const complex = (flags & 0x10) !== 0
+    const inStreams = complex ? count(r.number(), 64) : 1
+    const outStreams = complex ? count(r.number(), 64) : 1
+    const properties = flags & 0x20 ? r.bytes(count(r.number(), 1 << 16)) : new Uint8Array(0)
+    coders.push({ id, properties, inStreams, outStreams })
+  }
+  const totalOut = coders.reduce((n, c) => n + c.outStreams, 0)
+  const totalIn = coders.reduce((n, c) => n + c.inStreams, 0)
+  const bindOut = new Set<number>()
+  for (let i = 0; i < totalOut - 1; i++) {
+    r.number()
+    bindOut.add(r.number())
+  }
+  const packed = totalIn - (totalOut - 1)
+  if (packed < 1) throw new Malformed('7z: folder without packed streams')
+  if (packed > 1) for (let i = 0; i < packed; i++) r.number()
+  return { coders, bindOut, unpackSizes: [] }
+}
+
+/** The folder's output: the one out stream no bind pair consumes. */
+function unpackSize(folder: Folder): number {
+  for (let i = folder.unpackSizes.length - 1; i >= 0; i--) if (!folder.bindOut.has(i)) return folder.unpackSizes[i]
+  return 0
+}
+
+function readStreamsInfo(r: Reader): StreamsInfo {
+  const info: StreamsInfo = { packPos: 0, packSizes: [], folders: [], streamsPerFolder: [], streamSizes: [] }
+  let id = r.byte()
+  if (id === ID.packInfo) {
+    info.packPos = r.number()
+    const n = count(r.number())
+    for (id = r.byte(); id !== ID.end; id = r.byte()) {
+      if (id === ID.size) for (let i = 0; i < n; i++) info.packSizes.push(r.number())
+      else if (id === ID.crc) r.digests(n)
+      else throw new Malformed('7z: unexpected property in pack info')
+    }
+    id = r.byte()
+  }
+  if (id === ID.unpackInfo) {
+    r.expect(ID.folder)
+    const n = count(r.number())
+    if (r.byte() !== 0) throw new Malformed('7z: external folders')
+    for (let i = 0; i < n; i++) info.folders.push(readFolder(r))
+    r.expect(ID.codersUnpackSize)
+    for (const f of info.folders) for (const c of f.coders) for (let i = 0; i < c.outStreams; i++) f.unpackSizes.push(r.number())
+    for (id = r.byte(); id !== ID.end; id = r.byte()) {
+      if (id === ID.crc) r.digests(n).forEach((crc, i) => (info.folders[i].crc = crc))
+      else throw new Malformed('7z: unexpected property in unpack info')
+    }
+    id = r.byte()
+  }
+  info.streamsPerFolder = info.folders.map(() => 1)
+  if (id === ID.subStreamsInfo) {
+    id = r.byte()
+    if (id === ID.numUnpackStream) {
+      info.streamsPerFolder = info.folders.map(() => count(r.number()))
+      id = r.byte()
+    }
+    const hasSizes = id === ID.size
+    info.folders.forEach((f, i) => {
+      const n = info.streamsPerFolder[i]
+      if (n === 0) return
+      let sum = 0
+      for (let j = 0; j < n - 1 && hasSizes; j++) {
+        const size = r.number()
+        info.streamSizes.push(size)
+        sum += size
+      }
+      info.streamSizes.push(unpackSize(f) - sum)
+    })
+    if (hasSizes) id = r.byte()
+    while (id !== ID.end) {
+      if (id === ID.crc) {
+        // CRCs for the streams whose folder had none (a folder of one stream with a CRC has its own).
+        const unknown = info.folders.reduce((n, f, i) => n + (info.streamsPerFolder[i] === 1 && f.crc !== undefined ? 0 : info.streamsPerFolder[i]), 0)
+        r.digests(unknown)
+      } else {
+        throw new Malformed('7z: unexpected property in substreams info')
+      }
+      id = r.byte()
+    }
+    id = r.byte()
+  } else {
+    info.streamSizes = info.folders.map(unpackSize)
+  }
+  if (id !== ID.end) throw new Malformed('7z: streams info not ended')
+  return info
+}
+
+function readFilesInfo(r: Reader, streams: StreamsInfo | null): { name: string; size: number }[] {
+  const n = count(r.number())
+  let emptyStream: boolean[] = Array(n).fill(false)
+  let emptyFile: boolean[] = []
+  let names: string[] = []
+  for (let id = r.byte(); id !== ID.end; id = r.byte()) {
+    const size = r.number()
+    const end = r.pos + size
+    if (id === ID.emptyStream) {
+      emptyStream = r.bits(n)
+    } else if (id === ID.emptyFile) {
+      emptyFile = r.bits(emptyStream.filter(Boolean).length)
+    } else if (id === ID.name) {
+      if (r.byte() !== 0) throw new Malformed('7z: external names')
+      const text = Buffer.from(r.bytes(size - 1)).toString('utf16le')
+      names = text.split('\0').slice(0, n)
+    } else if (id !== ID.dummy && id > 0x19) {
+      throw new Malformed('7z: unknown file property')
+    }
+    r.pos = end
+  }
+  if (names.length !== n) throw new Malformed('7z: names missing')
+  const out: { name: string; size: number }[] = []
+  let stream = 0
+  let empty = 0
+  for (let i = 0; i < n; i++) {
+    if (!emptyStream[i]) {
+      out.push({ name: names[i], size: streams?.streamSizes[stream++] ?? 0 })
+      continue
+    }
+    // An empty stream is a directory unless it's marked an empty file.
+    if (emptyFile[empty++]) out.push({ name: names[i], size: 0 })
+  }
+  return out
+}
+
+function readHeader(r: Reader): SevenZipListing {
+  let id = r.byte()
+  if (id === ID.archiveProperties) {
+    for (let type = r.byte(); type !== 0; type = r.byte()) r.bytes(r.number())
+    id = r.byte()
+  }
+  if (id === ID.additionalStreamsInfo) {
+    readStreamsInfo(r)
+    id = r.byte()
+  }
+  let streams: StreamsInfo | null = null
+  if (id === ID.mainStreamsInfo) {
+    streams = readStreamsInfo(r)
+    id = r.byte()
+  }
+  let files: { name: string; size: number }[] = []
+  if (id === ID.filesInfo) {
+    files = readFilesInfo(r, streams)
+    id = r.byte()
+  }
+  if (id !== ID.end) throw new Malformed('7z: header not ended')
+  const encrypted = (streams?.folders ?? []).some((f) => f.coders.some((c) => c.id === AES))
+  return { files, encrypted, headerEncrypted: false }
+}
+
+/**
+ * The header that ends exactly at `end` in `data`, starting at `start`: a plain one, or an encoded one whose packed
+ * data lies just before it (where 7z writes it). Throws when it isn't one.
+ */
+function headerAt(data: Uint8Array, start: number, end: number): SevenZipListing {
+  const r = new Reader(data.subarray(0, end), start)
+  const id = r.byte()
+  if (id === ID.header) {
+    const listing = readHeader(r)
+    if (!r.done) throw new Malformed('7z: bytes after the header')
+    return listing
+  }
+  if (id !== ID.encodedHeader) throw new Malformed('7z: not a header')
+  const info = readStreamsInfo(r)
+  if (!r.done) throw new Malformed('7z: bytes after the encoded header')
+  if (info.folders.length !== 1 || info.packSizes.length < 1) throw new Malformed('7z: unexpected encoded header')
+  const folder = info.folders[0]
+  if (folder.coders.some((c) => c.id === AES)) return { files: [], encrypted: true, headerEncrypted: true }
+  if (folder.coders.length !== 1 || folder.coders[0].id !== LZMA) throw new Malformed(`7z: header coder ${folder.coders[0].id} not supported`)
+  const packSize = info.packSizes[0]
+  const size = unpackSize(folder)
+  if (packSize > start || size > MAX_HEADER) throw new Malformed('7z: packed header not in the data')
+  const header = lzmaDecode(folder.coders[0].properties, data.subarray(start - packSize, start), size)
+  if (folder.crc !== undefined && crc32(header) >>> 0 !== folder.crc) throw new Malformed('7z: header CRC mismatch')
+  return headerAt(header, 0, header.length)
+}
+
+/**
+ * The listing of the 7z archive that `data` is the end of (the last segment of its last volume, or a whole small
+ * archive); null when no header ends there.
+ */
+export function parseSevenZipTail(data: Uint8Array): SevenZipListing | null {
+  const end = data.length
+  // An encoded header is a few dozen bytes; a plain one (7z -mhc=off) can be large.
+  for (let start = end - 2; start >= Math.max(0, end - 1024); start--) {
+    if (data[start] !== ID.encodedHeader || data[start + 1] !== ID.packInfo) continue
+    try {
+      return headerAt(data, start, end)
+    } catch {
+      // Not a header here (any byte can look like the start of one): the next position.
+    }
+  }
+  for (let start = end - 2; start >= 0; start--) {
+    if (data[start] !== ID.header || ![ID.archiveProperties, ID.additionalStreamsInfo, ID.mainStreamsInfo, ID.filesInfo, ID.end].includes(data[start + 1])) continue
+    try {
+      return headerAt(data, start, end)
+    } catch {
+      // Not a header here (any byte can look like the start of one): the next position.
+    }
+  }
+  return null
+}

lib/zip/index.ts

@@ -0,0 +1 @@
+export { parseZipStart, type ZipStart } from '#lib/zip/parse'

lib/zip/parse.ts

@@ -0,0 +1,50 @@
+/**
+ * The entries a ZIP's start shows: its local file headers, one before each entry's data, as far as the data
+ * holds them (the first segment of a .zip; REBUILD.md, phase 2). The full list is in the central directory at
+ * the end; the start names the first entries and says whether they're encrypted, as a RAR's first volume does.
+ */
+export interface ZipStart {
+  files: { name: string; size: number }[]
+  encrypted: boolean
+}
+
+const LOCAL = 0x04034b50
+/** Traditional PKWARE encryption (bit 0 of the flags), or WinZip AES (method 99). */
+const ENCRYPTED_FLAG = 0x1
+const DATA_DESCRIPTOR_FLAG = 0x8
+const AES_METHOD = 99
+const MAX_ENTRIES = 10_000
+
+export function parseZipStart(data: Buffer): ZipStart | null {
+  if (data.length < 30 || data.readUInt32LE(0) !== LOCAL) return null
+  const result: ZipStart = { files: [], encrypted: false }
+  let pos = 0
+  while (pos + 30 <= data.length && data.readUInt32LE(pos) === LOCAL && result.files.length < MAX_ENTRIES) {
+    const flags = data.readUInt16LE(pos + 6)
+    const method = data.readUInt16LE(pos + 8)
+    let compressed = data.readUInt32LE(pos + 18)
+    let size = data.readUInt32LE(pos + 22)
+    const nameLength = data.readUInt16LE(pos + 26)
+    const extraLength = data.readUInt16LE(pos + 28)
+    const nameEnd = pos + 30 + nameLength
+    if (nameEnd > data.length) break
+    // Bit 11: UTF-8 names; others are mostly CP437, read as latin1.
+    const name = data.toString(flags & 0x800 ? 'utf8' : 'latin1', pos + 30, nameEnd)
+    // Zip64: the real sizes in extra field 0x0001.
+    for (let e = nameEnd; e + 4 <= Math.min(nameEnd + extraLength, data.length); ) {
+      const id = data.readUInt16LE(e)
+      const length = data.readUInt16LE(e + 2)
+      if (id === 0x0001 && e + 4 + length <= data.length && length >= 16) {
+        size = Number(data.readBigUInt64LE(e + 4))
+        compressed = Number(data.readBigUInt64LE(e + 12))
+      }
+      e += 4 + length
+    }
+    if (flags & ENCRYPTED_FLAG || method === AES_METHOD) result.encrypted = true
+    if (!name.endsWith('/')) result.files.push({ name, size })
+    // Sizes after the data (a streamed zip): where the next header starts isn't known.
+    if (flags & DATA_DESCRIPTOR_FLAG) break
+    pos = nameEnd + extraLength + compressed
+  }
+  return result.files.length > 0 || result.encrypted ? result : null
+}

public/app.js

@@ -17,12 +17,21 @@ const SOURCE_LABELS = {
   subject: 'Subject',
   post: 'Post id',
 }
+/** Where a known password came from (release_passwords.source). */
+const PASSWORD_SOURCES = {
+  nzb: "from the spot's NZB",
+  'spot-title': "from the spot's title",
+  'spot-text': "from the spot's text",
+}
 /** Newznab categories (lib/newznab/categories.ts). */
 const CATEGORIES = [
+  [1000, 'Console', []],
   [2000, 'Movies', [[2030, 'SD'], [2040, 'HD'], [2045, 'UHD']]],
-  [3000, 'Audio', [[3010, 'MP3'], [3040, 'Lossless']]],
+  [3000, 'Audio', [[3010, 'MP3'], [3030, 'Audiobook'], [3040, 'Lossless']]],
-  [5000, 'TV', [[5030, 'SD'], [5040, 'HD'], [5045, 'UHD']]],
+  [4000, 'PC', [[4050, 'Games']]],
-  [7000, 'Books', [[7020, 'Ebook']]],
+  [5000, 'TV', [[5030, 'SD'], [5040, 'HD'], [5045, 'UHD'], [5070, 'Anime']]],
+  [6000, 'XXX', [[6040, 'x264'], [6045, 'UHD'], [6060, 'ImgSet'], [6070, 'Other']]],
+  [7000, 'Books', [[7020, 'Ebook'], [7030, 'Comics']]],
   [8000, 'Other', [[8010, 'Misc']]],
 ]
 function categoryLabel(id) {
@@ -220,9 +229,10 @@ function releaseRow(release) {
   if (release.complete) badges.append(badge(files, 'good', 'Complete'))
   else if (release.fileTotal) badges.append(badge(`${release.fileCount}/${release.fileTotal}`, 'warn', 'Files present / announced'))
   else badges.append(badge(files))
-  if (release.passworded) badges.append(badge('Password', 'bad', 'Its archive is encrypted: a password is needed'))
+  if (release.passworded && release.passwordKnown) badges.append(badge('Password known', 'warn', 'Its archive is encrypted; the password is in the NZB'))
+  else if (release.passworded) badges.append(badge('Password', 'bad', 'Its archive is encrypted: a password is needed'))
   if (!release.settled) badges.append(badge('Collecting', 'accent', 'Recently seen: more files may still arrive'))
-  if (release.category && release.category !== 8010) badges.append(badge(categoryLabel(release.category), '', 'Category (from the name)'))
+  if (release.category && release.category !== 8010) badges.append(badge(categoryLabel(release.category), '', 'Category (from the name, the inner name, a spot or the groups)'))
   if (release.source && release.source !== 'stem' && release.source !== 'subject') {
     badges.append(badge(SOURCE_LABELS[release.source] ?? release.source, 'accent', 'How this release was recognised'))
   }
@@ -347,7 +357,12 @@ function renderDetail(release) {
   if (!release.settledAt) {
     notices.push(el('div', { class: 'notice notice-warn', text: 'Recently seen: more files of this release may still arrive. The NZB has what is indexed so far.' }))
   }
-  if (release.passworded) {
+  const passwords = release.passwords ?? []
+  if (passwords.length) {
+    notices.push(el('div', { class: 'notice notice-warn' },
+      el('span', { text: `${passwords.length === 1 ? 'Password' : 'Passwords'} (the first is in the NZB): ` }),
+      ...passwords.map((p, i) => el('span', {}, i ? ', ' : '', el('code', { text: p.password }), ` (${PASSWORD_SOURCES[p.source] ?? p.source})`))))
+  } else if (release.passworded) {
     notices.push(el('div', { class: 'notice notice-bad', text: 'Passworded: its archive has encrypted entries or headers. The password is not in the index.' }))
   }
   if (!release.readable && !release.innerName) {

start/env.ts

@@ -43,10 +43,14 @@ export default await Env.create(new URL('../', import.meta.url), {
   NNTP_PASSWORD: Env.schema.string.optional(),
   NNTP_SECURE: Env.schema.boolean.optional(),
   NNTP_CONNECTIONS: Env.schema.number.optional(),
+  NNTP_MIN_CONNECTIONS: Env.schema.number.optional(),
+  NNTP_ACCOUNT_CONNECTIONS: Env.schema.number.optional(),
+  NNTP_RESERVED_CONNECTIONS: Env.schema.number.optional(),
   NNTP_TIMEOUT_MS: Env.schema.number.optional(),
   NNTP_HEDGE_AFTER_SECONDS: Env.schema.number.optional(),
   NNTP_HEDGE_CONNECTIONS: Env.schema.number.optional(),
   NNTP_CONNECTION_MAX_MINUTES: Env.schema.number.optional(),
+  NNTP_IDLE_CLOSE_SECONDS: Env.schema.number.optional(),
   NNTP_STALL_KB_PER_MINUTE: Env.schema.number.optional(),
 
   // Newznab API key (empty: no key needed)
@@ -74,6 +78,8 @@ export default await Env.create(new URL('../', import.meta.url), {
   SPOOL_MAX_GB: Env.schema.number.optional(),
   PREFETCH_PER_RANGE: Env.schema.number.optional(),
   PROBE_CONNECTIONS: Env.schema.number.optional(),
+  PROBE_MIN_CONNECTIONS: Env.schema.number.optional(),
+  PROBE_MIN_FREE_GB: Env.schema.number.optional(),
   HTTP_ROUTES: Env.schema.enum.optional(['all', 'node', 'public'] as const),
   PREFETCH_PER_GROUP: Env.schema.number.optional(),
 })

tests/functional/batch_store.spec.ts

@@ -120,6 +120,40 @@ test.group('BatchStore', (group) => {
     assert.equal(Number(chunks!.n), 4)
   })
 
+  test('candidates added to a settled release (a later range of its upload) are claimable at once', async ({ assert }) => {
+    await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E02.1080p.mkv', 2, 'Some.Show.S01E02.1080p [1/2]', '2026-10-05T10:00:00Z'))
+    // Settled before the next range came (the settler marks only what it settles).
+    await db.from('releases').update({ settled_at: new Date() })
+    await db.from('probe_candidates').update({ ready: true })
+    await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E02.1080p.par2', 1, 'Some.Show.S01E02.1080p [2/2]', '2026-10-05T10:20:00Z'))
+    const candidates = await db.from('probe_candidates').orderBy('id').select('kind', 'ready')
+    assert.deepEqual(candidates.map((c) => c.kind), ['par2'])
+    assert.isTrue(candidates.every((c) => c.ready))
+    assert.lengthOf(await releasesOf(), 1)
+  })
+
+  test('a split 7z read in pieces keeps the tail of its highest volume only; RAR gets none', async ({ assert }) => {
+    // As the live tail reads it while posted: each piece queues a tail for the last volume it saw so far.
+    await storeIn(GROUP, 1, [
+      ...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
+      ...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
+    ])
+    await storeIn(GROUP, 100_001, upload(100_010, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'))
+    // A lower volume posted late: its tail goes at once.
+    await storeIn(GROUP, 200_001, upload(200_010, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'))
+    await storeIn(GROUP, 300_001, upload(300_010, 'Some.Show.S01E03.part01.rar', 2, 'Some.Show.S01E03 [1/2]', '2026-10-23T09:00:00Z'))
+    await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E03.part02.rar', 2, 'Some.Show.S01E03 [2/2]', '2026-10-23T09:02:00Z'))
+    const candidates = await db.from('probe_candidates').orderBy('file').select('kind', 'file')
+    assert.deepEqual(
+      candidates.map((c) => [c.kind, c.file]),
+      [
+        ['archive', 'Some.Show.S01E03.part01.rar'],
+        ['archive', 'TsIgNBewuFbuwWy.7z.001'],
+        ['7z-tail', 'TsIgNBewuFbuwWy.7z.004'],
+      ]
+    )
+  })
+
   test('the same upload in another group (multi-posted) is a release of its own', async ({ assert }) => {
     await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
     await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
@@ -146,4 +180,29 @@ test.group('BatchStore', (group) => {
     const keys = (await releasesOf()).filter((r) => r.identity_key === first.identity_key).map((r) => r.release_key)
     assert.deepEqual(keys, [first.identity_key, `${first.identity_key}:3`])
   })
+
+  test('a re-upload is numbered after the highest number, not into the holes merges left', async ({ assert }) => {
+    for (let day = 0; day < 4; day++) {
+      await storeIn(GROUP, 1 + day * 100_000, upload(10 + day * 100_000, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', `2026-10-0${day * 2 + 1}T10:00:00Z`))
+    }
+    const [first] = await releasesOf()
+    // Merged away: key:2 and key:3 gone, key and key:4 left.
+    await db.from('releases').whereIn('release_key', [`${first.identity_key}:2`, `${first.identity_key}:3`]).delete()
+    await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-09T10:00:00Z'))
+    const keys = (await releasesOf()).map((r) => r.release_key.split(':')[1] ?? '1')
+    assert.deepEqual(keys, ['1', '4', '5'])
+  })
+
+  test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
+    await storeIn(GROUP, 1_000_000, [
+      ...upload(1_000_000, 'Murdoch Mysteries S8D1.nzb', 1, '[00/80]', '2026-10-05T10:00:00Z'),
+      ...upload(1_000_010, 'Some.Tool.v1.zip', 1, '[01/01]', '2026-10-05T10:00:00Z'),
+      ...upload(1_000_020, 'Some.Show.S01E01.part01.rar', 2, '[02/30]', '2026-10-05T10:00:00Z'),
+    ])
+    const flags = new Map((await db.from('releases').select('name', 'sidecar_only')).map((r) => [r.name, r.sidecar_only]))
+    assert.isTrue(flags.get('Murdoch Mysteries S8D1'))
+    assert.isFalse(flags.get('Some.Tool.v1'))
+    // Over 1 MB: not looked at.
+    assert.isNull(flags.get('Some.Show.S01E01'))
+  })
 })

tests/functional/leases.spec.ts

@@ -31,5 +31,12 @@ test.group('Leases', (group) => {
     // a1's worker went quiet: its lease is taken over (a2 alone counts).
     await db.from('ranges').where('id', a1).update({ heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
     assert.equal(await leased(5), a1)
+
+    // Read to its end, its worker gone before finishing: done. One still reading, or quiet but not at its end, stays.
+    await db.from('ranges').where('id', a2).update({ next_id: 201, heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
+    await db.from('ranges').where('id', b1).update({ next_id: 101 })
+    assert.equal(await Leases.finishAbandoned(), 1)
+    assert.equal((await db.from('ranges').where('id', a2).firstOrFail()).status, 'done')
+    assert.equal((await db.from('ranges').where('id', b1).firstOrFail()).status, 'leased')
   })
 })

tests/functional/nntp_shares.spec.ts

@@ -0,0 +1,53 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import coordinatorConfig from '#config/coordinator'
+import { grant, targets, NntpShare } from '#services/nntp_shares'
+
+test.group('NNTP shares', (group) => {
+  group.each.setup(() => testUtils.db().withGlobalTransaction())
+
+  test('targets: minimums first, then by priority what each wants', ({ assert }) => {
+    const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 0 }
+    const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 0 }
+    // History running: the coordinator all it wants, the prober the rest.
+    assert.deepEqual(Object.fromEntries(targets(45, [prober, coordinator])), { coordinator: 38, prober: 7 })
+    // History paused: the coordinator its minimum (or what it uses), the prober the rest.
+    assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 8 }])), { coordinator: 8, prober: 37 })
+    // Wanting more than its minimum doesn't take another's minimum.
+    assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 60 }])), { coordinator: 43, prober: 2 })
+  })
+
+  test('grant: growing only into connections the others closed or gave up', ({ assert }) => {
+    const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 37, granted: 37 }
+    const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 8 }
+    // History came back: the prober still holds 37, the coordinator keeps its 8 for now.
+    assert.equal(grant(45, coordinator, [prober]), 8)
+    // The prober syncs: its target is 7, its grant goes down at once.
+    assert.equal(grant(45, { ...prober, open: 37 }, [{ ...coordinator, granted: 8 }]), 7)
+    // Its connections not closed yet: still held.
+    assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 30 }]), 15)
+    // Closed: the coordinator gets all it wants.
+    assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 7 }]), 38)
+  })
+
+  test('sync: each process its row; a process gone (no heartbeat) holds nothing', async ({ assert }) => {
+    const total = coordinatorConfig.nntpShares.total
+    const coordinator = new NntpShare('coordinator', 2, 8, 38)
+    const prober = new NntpShare('prober', 1, 2, total)
+    assert.equal(await coordinator.sync(8, 0), 8)
+    assert.equal(await prober.sync(total, 0), total - 8)
+    // The coordinator wants all: the prober holds its grant until it has synced and closed them.
+    assert.equal(await coordinator.sync(38, 8), 8)
+    assert.equal(await prober.sync(total, total - 8), total - 38)
+    assert.equal(await coordinator.sync(38, 8), 8)
+    assert.equal(await prober.sync(total, total - 38), total - 38)
+    assert.equal(await coordinator.sync(38, 8), 38)
+
+    // The prober's process died: after a minute without a sync its share is free.
+    await db.from('nntp_shares').where('name', 'prober').update({ open: total, granted: total, heartbeat_at: new Date(Date.now() - 120_000) })
+    assert.equal(await coordinator.sync(38, 38), 38)
+    await coordinator.leave()
+    assert.equal(await prober.sync(total, 0), total)
+  })
+})

tests/functional/par2_completeness.spec.ts

@@ -0,0 +1,67 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import BatchStore from '#services/batch_store'
+import { decidePar2Completeness, par2Complete } from '#services/par2_completeness'
+import { RangeIndexer } from '#lib/edge/index'
+import { DEFAULT_RULES } from '#lib/rules/index'
+import type { Overview } from '#lib/nntp/index'
+
+const GROUP = 'alt.binaries.test'
+
+/** A file posted without an [i/N] counter: the release gets no file_total. */
+function file(name: string, parts: number, at: number): Overview[] {
+  return Array.from({ length: parts }, (_, i) => [at + i, {
+    'subject': `"${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
+    'message-id': `<${name}-${i + 1}@test>`, ':bytes': '700000', 'xref': `news.example ${GROUP}:${at + i}`,
+  }] as Overview)
+}
+
+async function store(start: number, overviews: Overview[]) {
+  const g = (await db.from('groups').where('name', GROUP).first()) ?? (await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('*'))[0]
+  const [n] = await db.table('nodes').insert({ name: `n${start}` }).returning('id')
+  const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: start, end_id: start + 999, next_id: start, status: 'leased', node_id: n.id }).returning('id')
+  const indexer = new RangeIndexer({ group: GROUP, start, end: start + 999, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
+  indexer.add(overviews, start + 1000)
+  indexer.finish()
+  await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
+}
+
+const setFile = (name: string, length: number, hash16k = name.padEnd(32, '0').slice(0, 32)) => ({ name, length, hash16k })
+
+test.group('par2 completeness', (group) => {
+  group.each.setup(() => testUtils.db().withGlobalTransaction())
+
+  test('every described file by name or by hash; a renamed upload by its files and bytes', ({ assert }) => {
+    const set = [setFile('a.mkv', 1000), setFile('a.nfo', 10)]
+    const files = [{ name: 'a.mkv', size: 1100 }, { name: 'a.nfo', size: 12 }, { name: 'a.par2', size: 50 }]
+    assert.isTrue(par2Complete(set, files, []))
+    assert.isFalse(par2Complete(set, files.slice(0, 1), []))
+    // The nfo renamed, but its probe hashed it.
+    assert.isTrue(par2Complete(set, [files[0], { name: 'x.nfo', size: 12 }], [{ hash16k: set[1].hash16k, size: 10 }]))
+    // Renamed after the par2 was made (random names): two data files and enough bytes; one missing, or too small.
+    assert.isTrue(par2Complete(set, [{ name: 'Qx81.bin', size: 1100 }, { name: 'Zr02.bin', size: 12 }, { name: 'Qx.par2', size: 50 }], []))
+    assert.isFalse(par2Complete(set, [{ name: 'Qx81.bin', size: 1100 }, { name: 'Qx.par2', size: 50 }], []))
+    assert.isFalse(par2Complete(set, [{ name: 'Qx81.bin', size: 500 }, { name: 'Zr02.bin', size: 12 }], []))
+  })
+
+  test('decided once settled with a par2 set, and kept when more files come', async ({ assert }) => {
+    await store(1, [...file('Some.Upload.mkv', 3, 10), ...file('Some.Upload.par2', 1, 20)])
+    const release = await db.from('releases').firstOrFail()
+    assert.isNull(release.file_total)
+    await db.table('par2_files').multiInsert([setFile('Some.Upload.mkv', 2_000_000), setFile('Some.Upload.nfo', 100)].map((f, i) => ({ set_id: 's'.repeat(32), file_id: String(i).repeat(32), ...f })))
+    await db.table('par2_sets').insert({ set_id: 's'.repeat(32), release_id: release.id, files: 2 })
+    // Not settled yet: left alone.
+    assert.equal(await decidePar2Completeness(db, [release.id]), 0)
+    await db.from('releases').where('id', release.id).update({ settled_at: new Date() })
+    assert.equal(await decidePar2Completeness(db, [release.id]), 1)
+    assert.isFalse((await db.from('releases').where('id', release.id).firstOrFail()).complete)
+    // The nfo comes in a later range: complete, and the batch store's own update keeps it.
+    await store(1001, file('Some.Upload.nfo', 1, 1010))
+    await decidePar2Completeness(db, [release.id])
+    await store(2001, file('Some.Upload.sfv', 1, 2010))
+    const after = await db.from('releases').where('id', release.id).firstOrFail()
+    assert.equal(after.files, 4)
+    assert.isTrue(after.complete)
+  })
+})

tests/functional/par2_merger.spec.ts

@@ -78,6 +78,44 @@ test.group('Par2Merger', (group) => {
     assert.equal((await new Par2Merger().round()).sets, 1)
   })
 
+  test('a round behind by hours goes through them a window at a time, past the gaps', async ({ assert }) => {
+    const ago = (hours: number) => new Date(Date.now() - hours * 3600_000)
+    await db.table('par2_sets').multiInsert([
+      { set_id: 'a1'.repeat(16), release_id: 1, files: 1, created_at: ago(30) },
+      { set_id: 'a2'.repeat(16), release_id: 2, files: 1, created_at: ago(2.5) },
+      { set_id: 'a3'.repeat(16), release_id: 3, files: 1, created_at: ago(0.5) },
+    ])
+    await db.table('worker_cursors').insert({ name: 'merger', at: ago(48) })
+    assert.equal((await new Par2Merger().round()).sets, 3)
+    const saved = await db.from('worker_cursors').where('name', 'merger').firstOrFail()
+    assert.isAbove(new Date(saved.at).getTime(), ago(0.5).getTime())
+  })
+
+  test('a par2 file held by more than 20 files is a common one: it merges nothing', async ({ assert }) => {
+    const at = new Date('2022-07-08T03:46:00Z')
+    const ids: number[] = []
+    for (const key of ['dd000001', 'dd000002']) {
+      const [row] = await db
+        .table('releases')
+        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
+        .returning('id')
+      ids.push(Number(row.id))
+    }
+    // The par2 is held by the first; the second holds the file, and so do 20 releases elsewhere and years apart.
+    const setId = hash('1')
+    await db.table('par2_sets').insert({ set_id: setId, release_id: ids[0], files: 1 })
+    await db.table('par2_files').insert({ set_id: setId, file_id: hash('0'), hash16k: hash('9'), length: 30214, name: 'sample.nfo' })
+    await db.table('file_hashes').insert({ candidate_id: 4000, release_id: ids[1], hash16k: hash('9'), size: 30214 })
+    for (let i = 1; i <= 20; i++) await db.table('file_hashes').insert({ candidate_id: 4000 + i, release_id: 900000 + i, hash16k: hash('9'), size: 30214 })
+    const round = await new Par2Merger().round()
+    assert.deepEqual([round.sets, round.merges], [1, 0])
+    assert.equal(await new Par2Merger().mergeSet(setId), 0)
+    assert.equal((await db.from('releases').whereIn('id', ids).select('id')).length, 2)
+    // One holder fewer: the file is one upload's again.
+    await db.from('file_hashes').where('candidate_id', 4020).delete()
+    assert.equal(await new Par2Merger().mergeSet(setId), 1)
+  })
+
   test('a set held by two of the releases merged is held once by the target', async ({ assert }) => {
     const at = new Date('2022-07-08T03:46:00Z')
     const ids: number[] = []
@@ -220,6 +258,8 @@ test.group('Par2Merger', (group) => {
     // A picture isn't a sidecar; an nfo an hour later belongs to nothing here.
     const picture = await release('cover', 1, null, at, ['cover.jpg'])
     const late = await release('Murdoch Mysteries S8D1', 1, null, new Date(at.getTime() + 3600_000), ['Murdoch Mysteries S8D1.nfo'])
+    // Queued, its release gone meanwhile (merged): leaves the queue with its batch.
+    await db.table('fold_candidates').insert({ release_id: 987654321 })
 
     const round = await new Par2Merger().round()
     assert.deepEqual(round.folded, { looked: 5, sidecars: 2, folded: 1 })

tests/functional/par2_pruner.spec.ts

@@ -53,16 +53,72 @@ test.group('Par2Pruner', (group) => {
     const locked = await set('e', e, 2, null)
     await db.table('release_verdicts').insert({ release_id: e, rule: 'locked', rule_version: 1 })
 
-    const round = await new Par2Pruner().round()
+    // Final: its holder held by a run of done ranges with hours to spare, nothing hashed (a readable upload's files).
-    assert.deepEqual(round, { complete: 1, judged: 1, rows: 4 })
+    const f = await release()
+    const final = await set('f', f, 2, null)
+    const posted = new Date('2024-03-01T12:00:00Z')
+    const [group] = await db.table('groups').insert({ name: 'alt.binaries.pruned', priority: 1 }).returning('id')
+    const groupId = Number(group.id)
+    await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100 })
+    const [range] = await db
+      .table('ranges')
+      .insert({ group_id: groupId, start_id: 1, end_id: 10000, next_id: 10001, status: 'done', done_at: old, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z') })
+      .returning('id')
+    await db.table('range_blocks').insert({ group_id: groupId, start_id: 1, end_id: 10000, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z'), at_group_start: false })
+    // A lone candidate unprobed since before the range was done: its hash may still match. Kept.
+    await db.table('probe_candidates').insert({ id: 9001, release_id: b, kind: 'lone', message_id: '<9001@x>', file: 'x', status: 'waiting', created_at: new Date(old.getTime() - 3600_000) })
+
+    // Out of budget: one batch of two sets (a, b), the next round goes on.
+    assert.deepEqual(await new Par2Pruner(2).round(0), { complete: 1, judged: 0, final: 0, rows: 2 })
+    assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 1, final: 0, rows: 2 })
     assert.equal(await count(complete), 0)
     assert.equal(await count(apart), 2)
     assert.equal(await count(recent), 2)
     assert.equal(await count(dead), 0)
     assert.equal(await count(locked), 2)
-    assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '5')
+    assert.equal(await count(final), 2)
 
-    // Again: the cursor is past them, nothing more.
+    // Probed: final. The next pass starts over from the first set.
-    assert.deepEqual(await new Par2Pruner().round(), { complete: 0, judged: 0, rows: 0 })
+    await db.from('probe_candidates').where('id', 9001).update({ status: 'done' })
+    assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 0, final: 1, rows: 2 })
+    assert.equal(await count(final), 0)
+    assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '6')
+
+    // A hash of its files since the merger's last window: the merger hasn't matched it yet. Kept.
+    const g = await release()
+    const unseen = await set('g', g, 1, null)
+    await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100 })
+    await db.table('worker_cursors').insert({ name: 'merger', at: new Date(Date.now() - 3600_000) })
+    await db.table('probe_candidates').insert({ id: 9002, release_id: await release(), kind: 'lone', message_id: '<9002@x>', file: 'x', status: 'done' })
+    await db.table('file_hashes').insert({ candidate_id: 9002, release_id: 1, hash16k: 'g0'.padEnd(32, '0'), size: 1000 })
+    assert.equal((await new Par2Pruner().round()).final, 0)
+    // Past the hash by more than the merger's overlap.
+    await db.from('worker_cursors').where('name', 'merger').update({ at: new Date(Date.now() + 120_000) })
+    assert.equal((await new Par2Pruner().round()).final, 1)
+    assert.equal(await count(unseen), 0)
+    assert.isNotNull(range)
+  })
+
+  test('sets of one created_at, a batch each: every one looked at', async ({ assert }) => {
+    // One prober flush gives thousands of sets the same created_at (to the microsecond).
+    const flush = new Date('2026-10-15T14:38:39.883Z')
+    let candidate = 7000
+    for (const letter of ['p', 'q', 'r']) {
+      const key = `pr${letter}`.padEnd(40, '0')
+      const [row] = await db.table('releases').insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }).returning('id')
+      const id = Number(row.id)
+      const setId = letter.repeat(32)
+      const h = `${letter}0`.padEnd(32, '0')
+      await db.table('par2_files').insert({ set_id: setId, file_id: h, hash16k: h, length: 1000, name: 'f.rar' })
+      candidate++
+      await db.table('probe_candidates').insert({ id: candidate, release_id: id, kind: 'lone', message_id: `<${candidate}@x>`, file: 'x', status: 'done' })
+      await db.table('file_hashes').insert({ candidate_id: candidate, release_id: id, hash16k: h, size: 1000 })
+      await db.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files, created_at) VALUES (?, ?, 1, ?::timestamptz + interval '984 microseconds')`, [setId, id, flush])
+    }
+    const round = await new Par2Pruner(1).round()
+    assert.equal(round.complete, 3)
+    assert.equal((await db.from('par2_files').whereIn('set_id', ['p'.repeat(32), 'q'.repeat(32), 'r'.repeat(32)]).count('* as n'))[0].n, '0')
+    // The pass ended: the next one starts from the first set.
+    assert.equal((await db.from('worker_cursors').where('name', 'par2-prune').firstOrFail()).key, '')
   })
 })

tests/functional/prober.spec.ts

@@ -0,0 +1,45 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import { Prober } from '#services/prober'
+
+test.group('Prober', (group) => {
+  group.each.setup(() => testUtils.db().withGlobalTransaction())
+
+  test('a claim takes each tier: the readable tiers also take unreadable candidates, a kind with only one of them too', async ({ assert }) => {
+    const [row] = await db
+      .table('releases')
+      .insert({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 })
+      .returning('id')
+    const candidate = (kind: string, unreadable: boolean, ready = true) => ({
+      release_id: row.id, kind, message_id: `<${kind}-${unreadable}-${ready}@x>`, file: kind, ready, unreadable,
+    })
+    // One per kind, so each tier's id range is a single id (the slices' starts aren't random then).
+    await db.table('probe_candidates').multiInsert([candidate('lone', true), candidate('par2', true), candidate('archive', false), candidate('7z-tail', false, false)])
+
+    const prober = new Prober() as any
+    await prober.claimTiers()
+    assert.sameMembers(prober.queue.map((c: { kind: string }) => c.kind), ['lone', 'par2', 'archive'])
+    const status = Object.fromEntries((await db.from('probe_candidates').select('kind', 'status')).map((c) => [c.kind, c.status]))
+    assert.deepEqual(status, { 'lone': 'taken', 'par2': 'taken', 'archive': 'taken', '7z-tail': 'waiting' })
+  })
+
+  test('a flush stores a name with a NUL in it, and one article that two releases have as a candidate', async ({ assert }) => {
+    const release = async (key: string) =>
+      Number((await db.table('releases').insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 }).returning('id'))[0].id)
+    const a = await release('fa')
+    const b = await release('fb')
+    const candidate = async (releaseId: number) =>
+      Number((await db.table('probe_candidates').insert({ release_id: releaseId, kind: 'archive', message_id: '<same@x>', file: 'x.rar', status: 'taken' }).returning('id'))[0].id)
+    const inspection = { yencName: 'x.rar', size: 1000, type: null, names: ['Movie\u0000.mkv'], entries: 1, passworded: false }
+    const prober = new Prober() as any
+    for (const releaseId of [a, b]) {
+      prober.outcomes.push({ candidate: { id: await candidate(releaseId), releaseId, kind: 'archive', messageId: '<same@x>' }, status: 'done', inspection })
+    }
+    await prober.flush()
+    assert.deepEqual((await db.from('probe_candidates').select('status')).map((c) => c.status), ['done', 'done'])
+    const cached = await db.from('probe_cache').where('message_id', '<same@x>').firstOrFail()
+    assert.deepEqual(cached.result.names, ['Movie.mkv'])
+    assert.equal((await db.from('release_inspections').count('* as n'))[0].n, '2')
+  })
+})

tests/functional/release_category.spec.ts

@@ -0,0 +1,65 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+
+const categoryOf = async (name: string) => Number((await db.rawQuery(`SELECT release_category(?) AS c`, [name])).rows[0].c)
+
+test.group('Release categories', (group) => {
+  group.each.setup(() => testUtils.db().withGlobalTransaction())
+
+  test('by name: the sources and codes the first function lost or missed, XXX, audio, books, games', async ({ assert }) => {
+    const expected: [string, number][] = [
+      // Lost to knex's bindings in migration 1200 (`blu-$1ray`, `web-$2dl`, `h\.$326[45]`).
+      ['Some.Movie.2020.BluRay.DTS-X', 2040],
+      ['Some.Movie.2020.WEB-DL.DDP5', 2030],
+      ['Some.Movie.2020.H.264', 2030],
+      ['Some.Movie.2020.2160p.WEB-DL', 2045],
+      // TV codes it missed.
+      ['Rizzoli & Isles S01e07[Mux XviD', 5030],
+      ['Rush S04E01E02 WS PDTVXviD', 5030],
+      ['Show.S01E02.1080p.WEB', 5040],
+      ['Show.1x02.HDTV', 5030],
+      ['Resolution 1920x1080 wallpapers', 8010],
+      ['Studio.24.10.22.Some.Name.XXX.1080p.MP4-GRP', 6040],
+      ['Studio.Some.Name.XXX.2160p', 6045],
+      ['Some.Name.XXX.IMAGESET-GRP', 6060],
+      ['Artist-Album-WEB-2026-GRP', 3010],
+      ['Artist - Album (2020) FLAC', 3040],
+      ['Author - Title (Audiobook)', 3030],
+      ['Book.Title.EPUB', 7020],
+      ['Game.Title-CODEX', 4050],
+      ['Game.Title.NSW-VENOM', 1000],
+      ['TsIgNBewuFbuwWy', 8010],
+    ]
+    const got: [string, number][] = []
+    for (const [name] of expected) got.push([name, await categoryOf(name)])
+    assert.deepEqual(got, expected)
+  })
+
+  test("the name's, else the inner name's, else a given one (a spot's), else the groups'", async ({ assert }) => {
+    const row = (key: string, extra: Record<string, unknown>) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), source: 'stem', files: 1, groups: '["alt.binaries.misc"]', ...extra })
+    await db.table('releases').multiInsert([
+      row('a', { name: 'Show.S01E02.1080p.WEB' }),
+      row('b', { name: 'TsIgNBewuFbuwWy' }),
+      row('c', { name: 'TsIgNBewuFbuwWy', groups: '["alt.binaries.pictures.erotica.puffies"]' }),
+      // A spot's release of its own, inserted with the spot's category.
+      row('d', { name: 'Some spot title', category: 2040 }),
+    ])
+    const byName = async () => Object.fromEntries((await db.from('releases').select('release_key', 'category')).map((r) => [r.release_key[0], r.category]))
+    assert.deepEqual(await byName(), { a: 5040, b: 8010, c: 6060, d: 2040 })
+    // The prober finds an inner name; a spot links a release (the spot inspector sets a Misc one's category).
+    await db.from('releases').where('release_key', 'b'.padEnd(40, '0')).update({ inner_name: 'Some.Movie.2020.1080p.BluRay.x264-GRP.mkv' })
+    await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ category: 5030 })
+    await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ inner_name: 'nothing.bin' })
+    assert.deepEqual(await byName(), { a: 5040, b: 2040, c: 5030, d: 2040 })
+  })
+
+  test("a spot's category in Newznab's", async ({ assert }) => {
+    const { rows } = await db.rawQuery(
+      `SELECT spot_category(0, '{"a":[9],"z":[0]}') AS movie_hd, spot_category(0, '{"a":[15],"z":[1]}') AS series_uhd, spot_category(0, '{"a":[0],"z":[3]}') AS erotica,
+              spot_category(0, '{"a":[5]}') AS epub, spot_category(0, '{"z":[4]}') AS picture, spot_category(1, '{"a":[8]}') AS flac, spot_category(1, '{"a":[0],"z":[3]}') AS audiobook,
+              spot_category(2, '{"a":[0]}') AS pc_game, spot_category(2, '{"a":[12]}') AS console, spot_category(3, '{}') AS app`
+    )
+    assert.deepEqual(rows[0], { movie_hd: 2040, series_uhd: 5045, erotica: 6070, epub: 7020, picture: null, flac: 3040, audiobook: 3030, pc_game: 4050, console: 1000, app: 4000 })
+  })
+})

tests/functional/release_judge.spec.ts

@@ -59,6 +59,8 @@ test.group('ReleaseJudge', (group) => {
 
     const scraped = await release('Jamie Lee Curtis Nude Celebrity Pictures', { poster: '@_CelebMovie_@', source: 'subject' })
     const celeborn = await release('Some.Album.2010', { poster: 'celeborn@not.here (Celeborn)', source: 'subject' })
+    // Scraped posters are found through posters (the batch store adds each release's).
+    await db.table('posters').multiInsert([{ poster: '@_CelebMovie_@' }, { poster: 'celeborn@not.here (Celeborn)' }])
     const locked = await release('Locked.Thing', { passworded: true })
 
     const unlocked = await release('Known.Password', { passworded: true })
@@ -78,7 +80,7 @@ test.group('ReleaseJudge', (group) => {
     const spotted = await release('yy12CD34ef56GH78ij90', { readable: false, ...placed })
     await db.table('release_spots').insert({ release_id: spotted, spot_id: 's@x', files: 1 })
 
-    const round = await new ReleaseJudge().round()
+    const round = await new ReleaseJudge().full()
     const verdicts = new Map((await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [Number(v.release_id), v.rule]))
     assert.equal(verdicts.get(dead), 'dead')
     assert.isUndefined(verdicts.get(halfDead))
@@ -105,8 +107,88 @@ test.group('ReleaseJudge', (group) => {
 
     // A password found: the next round takes the verdict back.
     await db.table('release_passwords').insert({ release_id: locked, password: 'found', source: 'nzb' })
-    const again = await new ReleaseJudge().round()
+    const again = await new ReleaseJudge().full()
     assert.equal(again.rules.locked.removed, 1)
+    // Verdicts that stay are not written again.
+    assert.equal(again.rules.repeated.added, 0)
+    assert.equal(again.rules.fake.added, 0)
     assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
   })
+
+  test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({ assert }) => {
+    let n = 0
+    const settled = new Date(Date.now() - 3 * 86400_000)
+    const release = async (name: string, extra: Record<string, unknown> = {}) => {
+      n++
+      const key = `jc${String(n).padStart(6, '0')}`.padEnd(40, '0')
+      const [row] = await db
+        .table('releases')
+        .insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', files: 3, posted_at: settled, readable: true, settled_at: settled, ...extra })
+        .returning('id')
+      return Number(row.id)
+    }
+    const judge = new ReleaseJudge()
+    const verdict = async (id: number) => (await db.from('release_verdicts').where('release_id', id).first())?.rule
+    const dirty = async () => Number((await db.from('judge_dirty').count('* as n'))[0].n)
+
+    // New releases are noted; a passworded one is locked, until a password turns up.
+    const locked = await release('Locked.Thing', { passworded: true })
+    assert.isAbove(await dirty(), 0)
+    let round = await judge.changed()
+    assert.equal(await verdict(locked), 'locked')
+    assert.equal(round.rules.locked.added, 1)
+    assert.equal(await dirty(), 0)
+    await db.table('release_passwords').insert({ release_id: locked, password: 'pw', source: 'nzb' })
+    round = await judge.changed()
+    assert.isUndefined(await verdict(locked))
+    assert.equal(round.rules.locked.removed, 1)
+
+    // Probes: a release whose probed articles are all missing is dead once the prober's statuses come in.
+    const dead = await release('Gone.Movie.2020.1080p')
+    const ids = await db.table('probe_candidates').multiInsert([
+      { release_id: dead, kind: 'lone', message_id: '<jc1@x>', file: 'a', status: 'waiting' },
+      { release_id: dead, kind: 'lone', message_id: '<jc2@x>', file: 'b', status: 'waiting' },
+    ]).returning('id')
+    await judge.changed()
+    assert.isUndefined(await verdict(dead))
+    await db.from('probe_candidates').whereIn('id', ids.map((r: { id: string }) => r.id)).update({ status: 'missing' })
+    await judge.changed()
+    assert.equal(await verdict(dead), 'dead')
+
+    // One file in 19 releases, one of them locked: the 20th makes them all repeated, the locked one too (repeated
+    // comes first), though only the 20th changed.
+    const hash = 'b'.repeat(32)
+    const holders: number[] = []
+    for (let i = 0; i < 19; i++) holders.push(await release(`holder${i}`, i === 0 ? { passworded: true } : {}))
+    for (const [i, id] of holders.entries()) await db.table('file_hashes').insert({ candidate_id: 30000 + i, release_id: id, hash16k: hash, size: 5000 })
+    await judge.changed()
+    assert.equal(await verdict(holders[0]), 'locked')
+    assert.isUndefined(await verdict(holders[1]))
+    const twentieth = await release('holder19')
+    await db.table('file_hashes').insert({ candidate_id: 30019, release_id: twentieth, hash16k: hash, size: 5000 })
+    round = await judge.changed()
+    assert.equal(round.releases, 20)
+    const rules = new Set()
+    for (const id of [...holders, twentieth]) rules.add(await verdict(id))
+    assert.deepEqual([...rules], ['repeated'])
+    // A 21st: the file was in 20 releases without it, so only it is judged.
+    const twentyFirst = await release('holder20')
+    await db.table('file_hashes').insert({ candidate_id: 30020, release_id: twentyFirst, hash16k: hash, size: 5000 })
+    round = await judge.changed()
+    assert.equal(round.releases, 1)
+    assert.equal(await verdict(twentyFirst), 'repeated')
+
+    // A verdict nothing noted stays until the full round; it's due a day after the last one.
+    const stale = await release('Fine.Release')
+    await judge.changed()
+    await db.table('release_verdicts').insert({ release_id: stale, rule: 'fake', rule_version: 1 })
+    await judge.changed()
+    assert.equal(await verdict(stale), 'fake')
+    await db.from('judge_state').update({ full_at: new Date(Date.now() - 2 * 86400_000) })
+    assert.isTrue(await judge.fullDue())
+    round = await judge.full()
+    assert.isUndefined(await verdict(stale))
+    assert.equal(await verdict(dead), 'dead')
+    assert.isFalse(await judge.fullDue())
+  })
 })

tests/functional/releases_api.spec.ts

@@ -113,4 +113,27 @@ test.group('Releases API', (group) => {
     assert.include(feed, '<title>Hidden.Gem.2023</title>')
     assert.match(feed, /Other Movie 2024[^]*?name="password" value="1"/)
   })
+
+  test('a known password: in the NZB (the most trusted source first), the details and the list', async ({ client, assert }) => {
+    const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
+    const [n] = await db.table('nodes').insert({ name: 'api-node' }).returning('id')
+    const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: n.id }).returning('id')
+    const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
+    indexer.add(file('Locked.Movie.2024.1080p.BluRay.x264.part1.rar', 'Locked.Movie [1/1]', 2, 100), 1001)
+    indexer.finish()
+    await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
+    const movie = await db.from('releases').firstOrFail()
+    await db.from('releases').where('id', movie.id).update({ passworded: true })
+    await db.table('release_passwords').multiInsert([
+      { release_id: movie.id, password: 'from-text', source: 'spot-text' },
+      { release_id: movie.id, password: 'from-nzb', source: 'nzb' },
+    ])
+
+    const nzb = parseNzb((await client.get(`/api/v1/releases/${movie.id}/nzb`)).text())!
+    assert.equal(nzb.meta.password, 'from-nzb')
+    const detail = (await client.get(`/api/v1/releases/${movie.id}`)).body() as any
+    assert.deepEqual(detail.passwords, [{ password: 'from-nzb', source: 'nzb' }, { password: 'from-text', source: 'spot-text' }])
+    const list = (await client.get('/api/v1/releases?q=locked movie')).body() as { releases: { passwordKnown: boolean }[] }
+    assert.isTrue(list.releases[0].passwordKnown)
+  })
 })

tests/functional/seven_zip_tails.spec.ts

@@ -0,0 +1,30 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import { pruneSevenZipTails } from '#services/seven_zip_tails'
+
+test.group('7z tails', (group) => {
+  group.each.setup(() => testUtils.db().withGlobalTransaction())
+
+  test("of a release's waiting tails, those below its highest volume go; the highest volume's all stay", async ({ assert }) => {
+    const row = (key: string) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', files: 4 })
+    const [merged, other] = (await db.table('releases').multiInsert([row('a'), row('b')]).returning('id')).map((r) => r.id)
+    const tail = (releaseId: number, volume: string, mid: string, status = 'waiting') => ({ release_id: releaseId, kind: '7z-tail', message_id: mid, file: `x.7z.${volume}`, status })
+    await db.table('probe_candidates').multiInsert([
+      // Two merged members' tails, one of the highest volume probed already.
+      tail(merged, '002', '<1@x>'),
+      tail(merged, '005', '<2@x>'),
+      tail(merged, '003', '<3@x>', 'done'),
+      tail(merged, '009', '<4@x>', 'taken'),
+      // Two pieces that each saw part of volume 009.
+      tail(merged, '009', '<5@x>'),
+      { release_id: merged, kind: 'archive', message_id: '<6@x>', file: 'x.7z.001' },
+      // Another release: not asked for.
+      tail(other, '002', '<7@x>'),
+      tail(other, '004', '<8@x>'),
+    ])
+    assert.equal(await pruneSevenZipTails(db, [merged]), 2)
+    const left = await db.from('probe_candidates').orderBy('message_id').select('message_id')
+    assert.deepEqual(left.map((c) => c.message_id), ['<3@x>', '<4@x>', '<5@x>', '<6@x>', '<7@x>', '<8@x>'])
+  })
+})

tests/functional/spots.spec.ts

@@ -102,7 +102,11 @@ test.group('Spots', (group) => {
     assert.isTrue(await inspector.standalone(SPOT))
     const release = await db.from('releases').where('source', 'spot').firstOrFail()
     assert.equal(release.name, 'Lonely File')
+    // Its title says nothing: the spot's category (01, a09: a movie, x264).
+    assert.equal(release.category, 2040)
     assert.equal(release.files, 1)
+    assert.isNotNull(release.poster)
+    assert.isNotNull(await db.from('posters').where('poster', release.poster).first())
     const { default: NzbService } = await import('#services/nzb_service')
     assert.include((await NzbService.build(Number(release.id)))!.xml, 'lonely.file.mkv')
     // Not again while it has one; once the header pass builds the file's release, the spot moves there.

tests/unit/disk_space.spec.ts

@@ -0,0 +1,22 @@
+import { test } from '@japa/runner'
+import { DiskSpace } from '#services/disk_space'
+
+test.group('DiskSpace', () => {
+  test('short below the floor, fine again a margin above it; off without a floor', async ({ assert }) => {
+    let gb = 15
+    const space = new DiskSpace('probing', '/', 12, async () => gb * 1e9, 0)
+    assert.isFalse(await space.short())
+    gb = 11.5
+    assert.isTrue(await space.short())
+    // Back over the floor, not yet over the margin: still short.
+    gb = 12.5
+    assert.isTrue(await space.short())
+    gb = 13.1
+    assert.isFalse(await space.short())
+    assert.isFalse(await new DiskSpace('probing', '/', 0, async () => 0, 0).short())
+  })
+
+  test('measures a real filesystem', async ({ assert }) => {
+    assert.isFalse(await new DiskSpace('probing', '/', 0.001).short())
+  })
+})

tests/unit/edge/range_indexer.spec.ts

@@ -55,6 +55,20 @@ test.group('edge / RangeIndexer', () => {
     assert.equal(batch.next, 1_000_001)
   })
 
+  test("archive candidates: a split 7z's start and its highest volume's end, an unsplit 7z's start and end, a ZIP's start", ({ assert }) => {
+    const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1_000_000, rules, edgeArticles: 10 })
+    indexer.add([
+      ...post(100, 'Split.Seven.7z.001', 2, { counter: 'Split.Seven [1/3]' }),
+      ...post(110, 'Split.Seven.7z.003', 2, { counter: 'Split.Seven [3/3]' }),
+      ...post(120, 'Split.Seven.7z.002', 2, { counter: 'Split.Seven [2/3]' }),
+      ...post(200, 'Whole.Seven.7z', 3, { counter: 'Whole.Seven [1/1]' }),
+      ...post(300, 'Some.Zip.zip', 2, { counter: 'Some.Zip [1/1]' }),
+    ], 200_000)
+    indexer.finish()
+    const got = indexer.take().candidates.map((c) => `${c.kind} ${c.file}`).sort()
+    assert.deepEqual(got, ['7z-tail Split.Seven.7z.003', '7z-tail Whole.Seven.7z', 'archive Some.Zip.zip', 'archive Split.Seven.7z.001', 'archive Whole.Seven.7z'])
+  })
+
   test('files the edges cut leave as fragments; one missing parts in the middle is only counted', ({ assert }) => {
     const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1_000_000, rules, edgeArticles: 1000 })
     indexer.add([

tests/unit/nntp/pool.spec.ts

@@ -0,0 +1,69 @@
+import { test } from '@japa/runner'
+import net from 'node:net'
+import env from '#start/env'
+import coordinatorConfig from '#config/coordinator'
+import pool from '#services/nntp_pool'
+import type { NntpShare } from '#services/nntp_shares'
+
+/** A tiny NNTP server that only logs in, counting the connections open. */
+function server(): Promise<{ port: number; open: () => number; close: () => void }> {
+  return new Promise((resolve) => {
+    let open = 0
+    const srv = net.createServer((sock) => {
+      open++
+      sock.on('close', () => open--)
+      sock.write('200 welcome\r\n')
+      sock.on('data', (chunk) => {
+        for (const line of chunk.toString().split('\r\n')) {
+          if (line.startsWith('AUTHINFO USER')) sock.write('381 more\r\n')
+          else if (line.startsWith('AUTHINFO PASS')) sock.write('281 ok\r\n')
+          else if (line === 'XFEATURE COMPRESS GZIP') sock.write('290 Feature Enabled\r\n')
+          else if (line === 'QUIT') sock.end('205 bye\r\n')
+        }
+      })
+    })
+    srv.listen(0, '127.0.0.1', () =>
+      resolve({
+        port: (srv.address() as net.AddressInfo).port,
+        open: () => open,
+        close: () => srv.close(),
+      })
+    )
+  })
+}
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
+const share = { sync: async () => 4 } as unknown as NntpShare
+
+test.group('nntp / pool', () => {
+  test('a connection free for nntpIdleCloseMs closes on the next sync; one used since stays', async ({
+    assert,
+    cleanup,
+  }) => {
+    const { port, open, close } = await server()
+    const idleCloseMs = coordinatorConfig.nntpIdleCloseMs
+    env.set('NNTP_HOST', '127.0.0.1')
+    env.set('NNTP_PORT', port)
+    env.set('NNTP_SECURE', false)
+    coordinatorConfig.nntpIdleCloseMs = 100
+    cleanup(async () => {
+      coordinatorConfig.nntpIdleCloseMs = idleCloseMs
+      await pool.close()
+      close()
+    })
+    await pool.share(share)
+    // Two at once: two connections.
+    await Promise.all([pool.use(() => sleep(20)), pool.use(() => sleep(20))])
+    assert.equal(open(), 2)
+    await sleep(150)
+    // One used again: the other has been free for longer than 100 ms.
+    await pool.use(async () => {})
+    await pool.share(share)
+    await sleep(50)
+    assert.equal(open(), 1)
+    await sleep(150)
+    await pool.share(share)
+    await sleep(50)
+    assert.equal(open(), 0)
+  })
+})

tests/unit/nzb/with_password.spec.ts

@@ -0,0 +1,15 @@
+import { test } from '@japa/runner'
+import { withPassword } from '#services/nzb_service'
+import { parseNzb } from '#lib/nzb/index'
+
+const NZB = (head: string) =>
+  `<?xml version="1.0" encoding="UTF-8"?><nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">${head}<file poster="p" date="1" subject="&quot;a.rar&quot; yEnc (1/1)"><groups><group>a.b.x</group></groups><segments><segment bytes="1" number="1">a@x</segment></segments></file></nzb>`
+
+test.group('nzb / a read NZB with the password', () => {
+  test('added into its head, or a head made; one it has already is kept', ({ assert }) => {
+    assert.equal(parseNzb(withPassword(NZB('<head><meta type="name">x</meta></head>'), 'p<&>w'))!.meta.password, 'p<&>w')
+    assert.equal(parseNzb(withPassword(NZB(''), 'pw'))!.meta.password, 'pw')
+    assert.equal(parseNzb(withPassword(NZB('<head><meta type="password">own</meta></head>'), 'pw'))!.meta.password, 'own')
+    assert.equal(withPassword(NZB(''), undefined), NZB(''))
+  })
+})

tests/unit/probe/inspect.spec.ts

@@ -1,5 +1,6 @@
 import { test } from '@japa/runner'
 import { createHash } from 'node:crypto'
+import { readFileSync } from 'node:fs'
 import { crc32 } from 'node:zlib'
 import yencode from 'yencode'
 import { crc32Hex } from '#lib/yenc/index'
@@ -148,4 +149,45 @@ test.group('probe / innerName', () => {
     // Nor does packaging: a lone par2 volume.
     assert.isNull(innerName([{ kind: 'lone', result: { yencName: 'a1b2.vol007+08.par2', size: 1, type: 'par2' } }], 'x9y8'))
   })
+
+  test("a 7z-tail: the listing from the last segment of the last volume; one that isn't the last segment: none", ({ assert }) => {
+    const tail = readFileSync(new URL('../../fixtures/sevenzip/split-last-volume.tail', import.meta.url))
+    // The volume's last 8,000 bytes as its third part, of a volume 1,000,000 bytes long.
+    const last = tail.subarray(tail.length - 8000)
+    const part = (data: Buffer, begin: number) =>
+      Buffer.from(
+        [`=ybegin part=3 total=3 line=128 size=1000000 name=x.7z.004`, `=ypart begin=${begin} end=${begin + data.length - 1}`, yencode.encode(data).toString('latin1'), `=yend size=${data.length} part=3 pcrc32=${crc32Hex(data)}`].join('\r\n'),
+        'latin1'
+      )
+    const result = inspect(part(last, 1_000_000 - 8000 + 1), '7z-tail')
+    assert.equal(result.entries, 4)
+    assert.deepEqual(result.names?.[0], 'Some.Movie.2020.1080p.BluRay.x264-GRP/some.movie.2020.1080p.bluray.x264-grp.mkv')
+    assert.isFalse(result.passworded)
+    assert.isUndefined(inspect(part(last, 1), '7z-tail').names)
+    const encrypted = readFileSync(new URL('../../fixtures/sevenzip/header-encrypted.tail', import.meta.url))
+    const hidden = inspect(part(encrypted.subarray(encrypted.length - 8000), 1_000_000 - 8000 + 1), '7z-tail')
+    assert.isTrue(hidden.passworded)
+    assert.equal(hidden.entries, 0)
+    assert.isUndefined(hidden.names)
+    // A small 7z whole in one article, probed as an archive start.
+    assert.deepEqual(inspect(article(readFileSync(new URL('../../fixtures/sevenzip/tiny.7z', import.meta.url)), 'x.7z'), 'archive').names, ['b.nfo', 'a.txt'])
+  })
+
+  test('a ZIP start: the names of its first entries, and whether they are encrypted', ({ assert }) => {
+    const local = (name: string, data: Buffer, flags = 0) => {
+      const header = Buffer.alloc(30)
+      header.writeUInt32LE(0x04034b50, 0)
+      header.writeUInt16LE(flags, 6)
+      header.writeUInt32LE(data.length, 18)
+      header.writeUInt32LE(data.length, 22)
+      header.writeUInt16LE(Buffer.byteLength(name), 26)
+      return Buffer.concat([header, Buffer.from(name), data])
+    }
+    const zip = Buffer.concat([local('Album/', Buffer.alloc(0)), local('Album/01 - Track.flac', Buffer.alloc(3000, 1)), local('Album/cover.jpg', Buffer.alloc(500, 2))])
+    const plain = inspect(article(zip, 'album.zip'))
+    assert.equal(plain.type, 'zip')
+    assert.deepEqual(plain.names, ['Album/01 - Track.flac', 'Album/cover.jpg'])
+    assert.isFalse(plain.passworded)
+    assert.isTrue(inspect(article(local('secret.mkv', Buffer.alloc(100), 0x1), 'x.zip')).passworded)
+  })
 })

tests/unit/probe/lzma.spec.ts

@@ -0,0 +1,20 @@
+import { test } from '@japa/runner'
+import { readFileSync } from 'node:fs'
+import { lzmaDecode, LzmaError } from '#lib/sevenzip/lzma'
+
+const fixture = (name: string) => readFileSync(new URL(`../../fixtures/sevenzip/${name}`, import.meta.url))
+
+test.group('probe / LZMA', () => {
+  test('decodes a stream Python made (.lzma: 5 property bytes, the size, the raw stream)', ({ assert }) => {
+    const alone = fixture('sample.lzma')
+    const expected = fixture('sample.bin')
+    const decoded = lzmaDecode(alone.subarray(0, 5), alone.subarray(13), expected.length)
+    assert.isTrue(decoded.equals(expected))
+  })
+
+  test('a cut-off or damaged stream throws LzmaError', ({ assert }) => {
+    const alone = fixture('sample.lzma')
+    const size = fixture('sample.bin').length
+    assert.throws(() => lzmaDecode(alone.subarray(0, 5), alone.subarray(13, 200), size), LzmaError)
+  })
+})

tests/unit/probe/sevenzip.spec.ts

@@ -0,0 +1,45 @@
+import { test } from '@japa/runner'
+import { readFileSync } from 'node:fs'
+import { parseSevenZipTail } from '#lib/sevenzip/index'
+
+const fixture = (name: string) => readFileSync(new URL(`../../fixtures/sevenzip/${name}`, import.meta.url))
+const DIR = 'Some.Movie.2020.1080p.BluRay.x264-GRP'
+
+test.group('probe / 7z listing from the archive end', () => {
+  test('an LZMA-packed header (7z default): files with sizes, directories left out', ({ assert }) => {
+    const listing = parseSevenZipTail(fixture('plain.tail'))
+    assert.deepEqual(listing, {
+      files: [
+        { name: `${DIR}/empty.txt`, size: 0 },
+        { name: `${DIR}/Sample/sample.mkv`, size: 20000 },
+        { name: `${DIR}/grp.nfo`, size: 4 },
+        { name: `${DIR}/some.movie.2020.1080p.bluray.x264-grp.mkv`, size: 300000 },
+      ],
+      encrypted: false,
+      headerEncrypted: false,
+    })
+  })
+
+  test('an uncompressed header (7z -mhc=off) and the last volume of a split archive', ({ assert }) => {
+    for (const name of ['uncompressed-header.tail', 'split-last-volume.tail']) {
+      const listing = parseSevenZipTail(fixture(name))!
+      assert.lengthOf(listing.files, 4, name)
+      assert.isFalse(listing.encrypted, name)
+    }
+  })
+
+  test('encrypted entries: names listed, encrypted; an encrypted header (7z -mhe): no names, encrypted', ({ assert }) => {
+    const files = parseSevenZipTail(fixture('files-encrypted.tail'))!
+    assert.lengthOf(files.files, 4)
+    assert.isTrue(files.encrypted)
+    assert.isFalse(files.headerEncrypted)
+    assert.deepEqual(parseSevenZipTail(fixture('header-encrypted.tail')), { files: [], encrypted: true, headerEncrypted: true })
+  })
+
+  test('a whole small archive; not the end of an archive: null', ({ assert }) => {
+    assert.deepEqual(parseSevenZipTail(fixture('tiny.7z'))?.files.map((f) => f.name), ['a.txt', 'b.nfo'])
+    // The end cut off, or data that isn't 7z at all.
+    assert.isNull(parseSevenZipTail(fixture('plain.tail').subarray(0, 30000)))
+    assert.isNull(parseSevenZipTail(fixture('sample.bin')))
+  })
+})