Content inspection and archive walks on seeding nodes (relay)
Test and deploy / test (push) Successful in 1m49s
Test and deploy / deploy (push) Successful in 2m15s

Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.

A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-06 11:17:02 +02:00
1 parent 51975fe603
commit 2dbca9b152
22 files changed
+1198 -540

No files matched your search

+17 -10
View File
@@ -5,8 +5,14 @@ D38). It asks the seed API for a range over HTTPS, gets that range's headers
(by default through the seed API, which fetches them from the news server and
passes them on gzip-compressed), groups them into files, and posts the
results back. It only makes outgoing connections: no port forwarding,
database, Redis or news server account. Only a token and
`SEED_CONNECTIONS` need setting.
database, Redis or news server account. Only the token needs setting, plus
how much work to take on.
A node can also do content inspection and archive walks (CONCEPT.md D40,
D41) with `SEED_INSPECTORS`: it takes files from the indexer's inspection
queue, downloads their first segment(s) through the seed API, works out
what they are (par2 sets, RAR and 7z listings, NZBs, passwords) and posts
the findings back; the indexer only stores them.
Headers through the seed API (`SEED_FETCH=relay`, the default) use the
indexer's news server account: `SEED_NNTP_CONNECTIONS` in the indexer's
@@ -22,8 +28,9 @@ the NNTP lines in `node.env`).
server (port 563 for TLS).
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
the image.
- An NNTP account. Every connection a node opens counts against that
the image. Per inspector: up to a core while decoding, and a few MB/s
download (whole articles, ~700 KB each).
- Only with `SEED_FETCH=direct`: an NNTP account. Every connection a node opens counts against that
account's connection limit, next to the ~20 the indexer itself uses if it's
the same account. Too many gives `481`/`502` errors at login. Some
providers also limit the number of IP addresses at once (Eweka: 2, as
@@ -55,13 +62,12 @@ for the same name replaces the token; `--disable` revokes it.
|---|---|
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
| `SEED_TOKEN` | the token from step 1 |
| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. |
| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) |
| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account |
| `NNTP_TLS_SERVERNAME` | only through a tunnel: the news server's real hostname, for the TLS certificate check |
| `SEED_CONNECTIONS` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
| `SEED_INSPECTORS` | inspections and archive walks at once (default `0`). Try 2–4 on a machine with spare cores. |
| `SEED_FETCH=direct` and `NNTP_*` | optional: seed with this node's own NNTP account (see the commented lines). `NNTP_TLS_SERVERNAME` only through a tunnel: the news server's real hostname, for the TLS certificate check. |
Leave the block below "Required by the app's configuration, but unused" as
it is: the app checks these settings at startup, but a node never uses them.
Nothing else: the database, Redis and web server settings the indexer
needs are filled in for a node automatically.
## 4. Start it
@@ -109,3 +115,4 @@ articles reached, and per node its leases and headers.
| `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |
| `[inspect n] "…" failed: …` | an inspection failed (often a missing article); the indexer retries it later |