Content inspection and archive walks on seeding nodes (relay)

Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.

A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

seed-node/README.md

@@ -5,8 +5,14 @@ D38). It asks the seed API for a range over HTTPS, gets that range's headers
 (by default through the seed API, which fetches them from the news server and
 passes them on gzip-compressed), groups them into files, and posts the
 results back. It only makes outgoing connections: no port forwarding,
-database, Redis or news server account. Only a token and
-`SEED_CONNECTIONS` need setting.
+database, Redis or news server account. Only the token needs setting, plus
+how much work to take on.
+
+A node can also do content inspection and archive walks (CONCEPT.md D40,
+D41) with `SEED_INSPECTORS`: it takes files from the indexer's inspection
+queue, downloads their first segment(s) through the seed API, works out
+what they are (par2 sets, RAR and 7z listings, NZBs, passwords) and posts
+the findings back; the indexer only stores them.
 
 Headers through the seed API (`SEED_FETCH=relay`, the default) use the
 indexer's news server account: `SEED_NNTP_CONNECTIONS` in the indexer's
@@ -22,8 +28,9 @@ the NNTP lines in `node.env`).
   server (port 563 for TLS).
 - Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
   gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
-  the image.
-- An NNTP account. Every connection a node opens counts against that
+  the image. Per inspector: up to a core while decoding, and a few MB/s
+  download (whole articles, ~700 KB each).
+- Only with `SEED_FETCH=direct`: an NNTP account. Every connection a node opens counts against that
   account's connection limit, next to the ~20 the indexer itself uses if it's
   the same account. Too many gives `481`/`502` errors at login. Some
   providers also limit the number of IP addresses at once (Eweka: 2, as
@@ -55,13 +62,12 @@ for the same name replaces the token; `--disable` revokes it.
 |---|---|
 | `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
 | `SEED_TOKEN` | the token from step 1 |
-| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. |
-| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) |
-| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account |
-| `NNTP_TLS_SERVERNAME` | only through a tunnel: the news server's real hostname, for the TLS certificate check |
+| `SEED_CONNECTIONS` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
+| `SEED_INSPECTORS` | inspections and archive walks at once (default `0`). Try 2–4 on a machine with spare cores. |
+| `SEED_FETCH=direct` and `NNTP_*` | optional: seed with this node's own NNTP account (see the commented lines). `NNTP_TLS_SERVERNAME` only through a tunnel: the news server's real hostname, for the TLS certificate check. |
 
-Leave the block below "Required by the app's configuration, but unused" as
-it is: the app checks these settings at startup, but a node never uses them.
+Nothing else: the database, Redis and web server settings the indexer
+needs are filled in for a node automatically.
 
 ## 4. Start it
 
@@ -109,3 +115,4 @@ articles reached, and per node its leases and headers.
 | `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
 | `Range … failed: …` | the range is given back and retried later, possibly by another node |
 | `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |
+| `[inspect n] "…" failed: …` | an inspection failed (often a missing article); the indexer retries it later |

seed-node/node.env.example

@@ -1,34 +1,18 @@
-# --- Seed API: the orchestrator, and this node's token (`node ace seed:token <name>` on the indexer).
+# The seed API (the orchestrator) and this node's token (`node ace seed:token <name>` on the indexer).
 SEED_URL=https://seed.exp.sforum.nl
 SEED_TOKEN=sn_replace_me
-# Ranges worked on at once (about 0.4 CPU core each).
-SEED_CONNECTIONS=4
-# relay: headers come through the seed API, no news server account needed (the default).
-# direct: the node fetches with its own NNTP account; then fill in the NNTP_* lines below.
-SEED_FETCH=relay
 
-# --- Only for SEED_FETCH=direct: the NNTP account (counts against its connection and IP limits).
-NNTP_HOST=localhost
-NNTP_PORT=563
-NNTP_SECURE=true
-NNTP_USER=unused
-NNTP_PASSWORD=unused
+# Ranges seeded at once (about 0.4 CPU core each). 0: no seeding.
+SEED_CONNECTIONS=4
+# Content inspections and archive walks at once (CPU for decoding and parsing; articles come through the seed API).
+SEED_INSPECTORS=0
+
+# Optional: SEED_FETCH=direct seeds with this node's own NNTP account instead of through the seed API.
+# SEED_FETCH=direct
+# NNTP_HOST=news.eweka.nl
+# NNTP_PORT=563
+# NNTP_SECURE=true
+# NNTP_USER=
+# NNTP_PASSWORD=
 # Through a tunnel or proxy: the hostname the server's TLS certificate is for.
 # NNTP_TLS_SERVERNAME=news.eweka.nl
-
-# --- Required by the app's configuration, but unused on a seeding node (no database, Redis or web server).
-NODE_ENV=production
-TZ=UTC
-PORT=3333
-HOST=0.0.0.0
-LOG_LEVEL=info
-APP_KEY=unused-on-seed-nodes-unused-on-seed-nodes
-APP_URL=http://localhost:3333
-SESSION_DRIVER=memory
-DB_CONNECTION=pg
-DB_HOST=localhost
-DB_PORT=5432
-DB_USER=unused
-DB_DATABASE=unused
-REDIS_HOST=localhost
-REDIS_PORT=6379