Content inspection and archive walks on seeding nodes (relay)
Inspection is split into prepare (database), probe (lib/content/probe: downloads, decoding and parsing, no database) and apply (database). worker:collection runs all three; seed:node with SEED_INSPECTORS leases jobs from the same queues through the seed API (BullMQ manual leases), fetches articles through the seed API and posts the probes back. A node now needs only SEED_URL and SEED_TOKEN: the settings the app requires but a node never uses get placeholders for seed:node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51975fe603
commit
2dbca9b152
22 files changed
+1198
-540
No files matched your search
+17
-10
@@ -5,8 +5,14 @@ D38). It asks the seed API for a range over HTTPS, gets that range's headers
|
||||
(by default through the seed API, which fetches them from the news server and
|
||||
passes them on gzip-compressed), groups them into files, and posts the
|
||||
results back. It only makes outgoing connections: no port forwarding,
|
||||
database, Redis or news server account. Only a token and
|
||||
`SEED_CONNECTIONS` need setting.
|
||||
database, Redis or news server account. Only the token needs setting, plus
|
||||
how much work to take on.
|
||||
|
||||
A node can also do content inspection and archive walks (CONCEPT.md D40,
|
||||
D41) with `SEED_INSPECTORS`: it takes files from the indexer's inspection
|
||||
queue, downloads their first segment(s) through the seed API, works out
|
||||
what they are (par2 sets, RAR and 7z listings, NZBs, passwords) and posts
|
||||
the findings back; the indexer only stores them.
|
||||
|
||||
Headers through the seed API (`SEED_FETCH=relay`, the default) use the
|
||||
indexer's news server account: `SEED_NNTP_CONNECTIONS` in the indexer's
|
||||
@@ -22,8 +28,9 @@ the NNTP lines in `node.env`).
|
||||
server (port 563 for TLS).
|
||||
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
|
||||
gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
|
||||
the image.
|
||||
- An NNTP account. Every connection a node opens counts against that
|
||||
the image. Per inspector: up to a core while decoding, and a few MB/s
|
||||
download (whole articles, ~700 KB each).
|
||||
- Only with `SEED_FETCH=direct`: an NNTP account. Every connection a node opens counts against that
|
||||
account's connection limit, next to the ~20 the indexer itself uses if it's
|
||||
the same account. Too many gives `481`/`502` errors at login. Some
|
||||
providers also limit the number of IP addresses at once (Eweka: 2, as
|
||||
@@ -55,13 +62,12 @@ for the same name replaces the token; `--disable` revokes it.
|
||||
|---|---|
|
||||
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
|
||||
| `SEED_TOKEN` | the token from step 1 |
|
||||
| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. |
|
||||
| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) |
|
||||
| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account |
|
||||
| `NNTP_TLS_SERVERNAME` | only through a tunnel: the news server's real hostname, for the TLS certificate check |
|
||||
| `SEED_CONNECTIONS` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
|
||||
| `SEED_INSPECTORS` | inspections and archive walks at once (default `0`). Try 2–4 on a machine with spare cores. |
|
||||
| `SEED_FETCH=direct` and `NNTP_*` | optional: seed with this node's own NNTP account (see the commented lines). `NNTP_TLS_SERVERNAME` only through a tunnel: the news server's real hostname, for the TLS certificate check. |
|
||||
|
||||
Leave the block below "Required by the app's configuration, but unused" as
|
||||
it is: the app checks these settings at startup, but a node never uses them.
|
||||
Nothing else: the database, Redis and web server settings the indexer
|
||||
needs are filled in for a node automatically.
|
||||
|
||||
## 4. Start it
|
||||
|
||||
@@ -109,3 +115,4 @@ articles reached, and per node its leases and headers.
|
||||
| `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
|
||||
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
|
||||
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |
|
||||
| `[inspect n] "…" failed: …` | an inspection failed (often a missing article); the indexer retries it later |
|
||||
Reference in new issue
Block a user