Daan MeijerandClaude Opus 5.5
2dbca9b152
Inspection is split into prepare (database), probe (lib/content/probe: downloads, decoding and parsing, no database) and apply (database). worker:collection runs all three; seed:node with SEED_INSPECTORS leases jobs from the same queues through the seed API (BullMQ manual leases), fetches articles through the seed API and posts the probes back. A node now needs only SEED_URL and SEED_TOKEN: the settings the app requires but a node never uses get placeholders for seed:node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
119 lines
6.1 KiB
Markdown
119 lines
6.1 KiB
Markdown
# Seeding node
|
||
|
||
A seeding node indexes old article ranges for the indexer (CONCEPT.md D37,
|
||
D38). It asks the seed API for a range over HTTPS, gets that range's headers
|
||
(by default through the seed API, which fetches them from the news server and
|
||
passes them on gzip-compressed), groups them into files, and posts the
|
||
results back. It only makes outgoing connections: no port forwarding,
|
||
database, Redis or news server account. Only the token needs setting, plus
|
||
how much work to take on.
|
||
|
||
A node can also do content inspection and archive walks (CONCEPT.md D40,
|
||
D41) with `SEED_INSPECTORS`: it takes files from the indexer's inspection
|
||
queue, downloads their first segment(s) through the seed API, works out
|
||
what they are (par2 sets, RAR and 7z listings, NZBs, passwords) and posts
|
||
the findings back; the indexer only stores them.
|
||
|
||
Headers through the seed API (`SEED_FETCH=relay`, the default) use the
|
||
indexer's news server account: `SEED_NNTP_CONNECTIONS` in the indexer's
|
||
`.env` sets how many connections it fetches with for all nodes together.
|
||
With `SEED_FETCH=direct` a node uses its own NNTP account instead (fill in
|
||
the NNTP lines in `node.env`).
|
||
|
||
## What a node needs
|
||
|
||
- Docker with the compose plugin (`docker compose version` works).
|
||
- Outgoing HTTPS to `seed.exp.sforum.nl` (the seed API) and
|
||
`gitea.exp.sforum.nl` (to build the image), and outgoing NNTP to the news
|
||
server (port 563 for TLS).
|
||
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
|
||
gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
|
||
the image. Per inspector: up to a core while decoding, and a few MB/s
|
||
download (whole articles, ~700 KB each).
|
||
- Only with `SEED_FETCH=direct`: an NNTP account. Every connection a node opens counts against that
|
||
account's connection limit, next to the ~20 the indexer itself uses if it's
|
||
the same account. Too many gives `481`/`502` errors at login. Some
|
||
providers also limit the number of IP addresses at once (Eweka: 2, as
|
||
`502 (max number of simultaneous IP addresses reached: 2)`): a node behind
|
||
the same public IP as the indexer doesn't count extra, others do. A node
|
||
can share an IP by tunnelling its NNTP traffic (e.g. SSH
|
||
`-L 1563:news.eweka.nl:563` to the indexer's machine); then set
|
||
`NNTP_HOST` to the tunnel and `NNTP_TLS_SERVERNAME=news.eweka.nl`.
|
||
|
||
## 1. On the indexer (homelab): a token for the node
|
||
|
||
cd ~/dev/usenet-indexer
|
||
docker compose exec web node ace seed:token my-server
|
||
|
||
This prints a token starting with `sn_` once. Treat it like a password: it
|
||
lets the holder lease ranges and submit results. Running the command again
|
||
for the same name replaces the token; `--disable` revokes it.
|
||
|
||
## 2. On the server: get the files
|
||
|
||
mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node
|
||
curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml
|
||
curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example
|
||
chmod 600 node.env
|
||
|
||
## 3. Configure `node.env`
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
|
||
| `SEED_TOKEN` | the token from step 1 |
|
||
| `SEED_CONNECTIONS` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
|
||
| `SEED_INSPECTORS` | inspections and archive walks at once (default `0`). Try 2–4 on a machine with spare cores. |
|
||
| `SEED_FETCH=direct` and `NNTP_*` | optional: seed with this node's own NNTP account (see the commented lines). `NNTP_TLS_SERVERNAME` only through a tunnel: the news server's real hostname, for the TLS certificate check. |
|
||
|
||
Nothing else: the database, Redis and web server settings the indexer
|
||
needs are filled in for a node automatically.
|
||
|
||
## 4. Start it
|
||
|
||
docker compose up -d --build
|
||
docker compose logs -f
|
||
|
||
The first build takes a few minutes. A working node logs:
|
||
|
||
[ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s).
|
||
[ info ] [1] alt.binaries.ath 49834000000–49835999999
|
||
[ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)
|
||
|
||
With nothing planned it waits and asks again every minute (no log lines).
|
||
|
||
## 5. Plan work and follow it (on homelab)
|
||
|
||
docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
|
||
docker compose exec web node ace seed:status
|
||
|
||
`seed:plan` queues ranges of 2 million articles from below what is already
|
||
indexed down to `--days` back (or everything the server has, without
|
||
`--days`), newest first and interleaved across groups. It takes over the
|
||
rest of those groups' backfill. `seed:status` shows ranges per status,
|
||
articles reached, and per node its leases and headers.
|
||
|
||
## Day to day
|
||
|
||
- **Update** after a new version of the indexer:
|
||
`docker compose build --pull && docker compose up -d`
|
||
- **Stop**: `docker compose stop` gives the node's ranges back; another node
|
||
resumes them after the last batch that was stored.
|
||
- **A node that disappears** (crash, network) loses its ranges to other
|
||
nodes after 15 minutes without a batch. A range that fails 5 times is set
|
||
aside as `failed` (see `seed:status`).
|
||
- **Remove a node**: `docker compose down` on the server, then
|
||
`docker compose exec web node ace seed:token my-server --disable` on homelab.
|
||
|
||
## Troubleshooting
|
||
|
||
| Log line | Meaning |
|
||
|---|---|
|
||
| `The seed API rejected this node token` | wrong or revoked `SEED_TOKEN`; the node stops |
|
||
| `Seed API …: 404 …; retrying` | the seed API is restarting (a deploy on homelab); the node retries |
|
||
| `NNTP login failed: …` | the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); `481`/`502`: wrong account, too many connections (lower `SEED_CONNECTIONS`) or too many IP addresses |
|
||
| `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
|
||
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
|
||
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |
|
||
| `[inspect n] "…" failed: …` | an inspection failed (often a missing article); the indexer retries it later |
|