usenet-indexer/seed-node/README.md

Seeding node

A seeding node indexes old article ranges for the indexer (CONCEPT.md D37, D38). It asks the seed API for a range over HTTPS, gets that range's headers (by default through the seed API, which fetches them from the news server and passes them on gzip-compressed), groups them into files, and posts the results back. It only makes outgoing connections: no port forwarding, database, Redis or news server account. Only a token and SEED_CONNECTIONS need setting.

Headers through the seed API (SEED_FETCH=relay, the default) use the indexer's news server account: SEED_NNTP_CONNECTIONS in the indexer's .env sets how many connections it fetches with for all nodes together. With SEED_FETCH=direct a node uses its own NNTP account instead (fill in the NNTP lines in node.env).

What a node needs

• Docker with the compose plugin (docker compose version works).
• Outgoing HTTPS to seed.exp.sforum.nl (the seed API) and gitea.exp.sforum.nl (to build the image), and outgoing NNTP to the news server (port 563 for TLS).
• Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for the image.
• An NNTP account. Every connection a node opens counts against that account's connection limit, next to the ~20 the indexer itself uses if it's the same account. Too many gives 481/502 errors at login. Some providers also limit the number of IP addresses at once (Eweka: 2, as 502 (max number of simultaneous IP addresses reached: 2)): a node behind the same public IP as the indexer doesn't count extra, others do. A node can share an IP by tunnelling its NNTP traffic (e.g. SSH -L 1563:news.eweka.nl:563 to the indexer's machine); then set NNTP_HOST to the tunnel and NNTP_TLS_SERVERNAME=news.eweka.nl.

1. On the indexer (homelab): a token for the node

cd ~/dev/usenet-indexer
docker compose exec web node ace seed:token my-server

This prints a token starting with sn_ once. Treat it like a password: it lets the holder lease ranges and submit results. Running the command again for the same name replaces the token; --disable revokes it.

2. On the server: get the files

mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node
curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml
curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example
chmod 600 node.env

3. Configure node.env

Setting	Value
SEED_URL	https://seed.exp.sforum.nl (leave as is)
SEED_TOKEN	the token from step 1
SEED_CONNECTIONS	ranges worked on at once, one NNTP connection each. Start with 2–4.
NNTP_HOST, NNTP_PORT, NNTP_SECURE	the news server; over the internet use TLS: port 563, NNTP_SECURE=true (for Eweka: news.eweka.nl)
NNTP_USER, NNTP_PASSWORD	the NNTP account
NNTP_TLS_SERVERNAME	only through a tunnel: the news server's real hostname, for the TLS certificate check

Leave the block below "Required by the app's configuration, but unused" as it is: the app checks these settings at startup, but a node never uses them.

4. Start it

docker compose up -d --build
docker compose logs -f

The first build takes a few minutes. A working node logs:

[ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s).
[ info ] [1] alt.binaries.ath 49834000000–49835999999
[ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)

With nothing planned it waits and asks again every minute (no log lines).

5. Plan work and follow it (on homelab)

docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
docker compose exec web node ace seed:status

seed:plan queues ranges of 2 million articles from below what is already indexed down to --days back (or everything the server has, without --days), newest first and interleaved across groups. It takes over the rest of those groups' backfill. seed:status shows ranges per status, articles reached, and per node its leases and headers.

Day to day

• Update after a new version of the indexer: docker compose build --pull && docker compose up -d
• Stop: docker compose stop gives the node's ranges back; another node resumes them after the last batch that was stored.
• A node that disappears (crash, network) loses its ranges to other nodes after 15 minutes without a batch. A range that fails 5 times is set aside as failed (see seed:status).
• Remove a node: docker compose down on the server, then docker compose exec web node ace seed:token my-server --disable on homelab.

Troubleshooting

Log line	Meaning
The seed API rejected this node token	wrong or revoked SEED_TOKEN; the node stops
Seed API …: 404 …; retrying	the seed API is restarting (a deploy on homelab); the node retries
NNTP login failed: …	the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); 481/502: wrong account, too many connections (lower SEED_CONNECTIONS) or too many IP addresses
Hostname/IP does not match certificate's altnames: Host: localhost	TLS through a tunnel: set NNTP_TLS_SERVERNAME to the news server's hostname
Range … failed: …	the range is given back and retried later, possibly by another node
Range … passed to another node; dropping it.	this node was too slow to report (15 minutes); harmless