usenet-indexer/seed-node

Seeding node

A seeding node indexes old article ranges for the indexer (CONCEPT.md D37, D38). It asks the seed API for a range over HTTPS, gets that range's headers (by default through the seed API, which fetches them from the news server and passes them on gzip-compressed), groups them into files, and posts the results back. It only makes outgoing connections: no port forwarding, database, Redis or news server account. Only the token needs setting, plus how much work to take on.

A node can also do content inspection and archive walks (CONCEPT.md D40, D41) with SEED_INSPECTORS: it takes files from the indexer's inspection queue, downloads their first segment(s) through the seed API, works out what they are (par2 sets, RAR and 7z listings, NZBs, passwords) and posts the findings back; the indexer only stores them.

Headers through the seed API (SEED_FETCH=relay, the default) use the indexer's news server account: SEED_NNTP_CONNECTIONS in the indexer's .env sets how many connections it fetches with for all nodes together. With SEED_FETCH=direct a node uses its own NNTP account instead (fill in the NNTP lines in node.env).

What a node needs

• Docker with the compose plugin (docker compose version works).
• Outgoing HTTPS to seed.exp.sforum.nl (the seed API) and gitea.exp.sforum.nl (to build the image), and outgoing NNTP to the news server (port 563 for TLS).
• Per range seeded at once: roughly 0.4 CPU core and 1–3 MB/s download (headers come gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for the image. Per inspector: up to a core while decoding, and a few MB/s download (whole articles, ~700 KB each).
• Only with SEED_FETCH=direct: an NNTP account. Every connection a node opens counts against that account's connection limit, next to the ~20 the indexer itself uses if it's the same account. Too many gives 481/502 errors at login. Some providers also limit the number of IP addresses at once (Eweka: 2, as 502 (max number of simultaneous IP addresses reached: 2)): a node behind the same public IP as the indexer doesn't count extra, others do. A node can share an IP by tunnelling its NNTP traffic (e.g. SSH -L 1563:news.eweka.nl:563 to the indexer's machine); then set NNTP_HOST to the tunnel and NNTP_TLS_SERVERNAME=news.eweka.nl.

1. On the indexer (homelab): a token for the node

cd ~/dev/usenet-indexer
docker compose exec web node ace seed:token my-server

This prints a token starting with sn_ once. Treat it like a password: it lets the holder lease ranges and submit results. Running the command again for the same name replaces the token; --disable revokes it.

2. On the server: get the files

mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node
curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml
curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example
chmod 600 node.env

3. Configure node.env

Setting	Value
SEED_URL	https://seed.exp.sforum.nl (leave as is)
SEED_TOKEN	the token from step 1
SEED_RANGES	ranges seeded at once. Start with 2–4; 0 for no seeding.
SEED_INSPECTORS	inspections and archive walks at once (default 0). Try 2–4 on a machine with spare cores.
SEED_FETCH=direct and NNTP_*	optional: seed with this node's own NNTP account (see the commented lines). NNTP_TLS_SERVERNAME only through a tunnel: the news server's real hostname, for the TLS certificate check.

Nothing else: the database, Redis and web server settings the indexer needs are filled in for a node automatically.

4. Start it

docker compose up -d --build
docker compose logs -f

The first build takes a few minutes. A working node logs:

[ info ] Seeding node for https://seed.exp.sforum.nl: 2 range(s) at once, headers through the seed API; 2 inspection(s) at once, articles through the seed API.
[ info ] [1] alt.binaries.ath 49834000000–49835999999
[ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)

With nothing planned it waits and asks again every minute (no log lines).

5. Plan work and follow it (on homelab)

docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
docker compose exec web node ace seed:status

seed:plan queues ranges of 2 million articles from below what is already indexed down to --days back (or everything the server has, without --days), newest first and interleaved across groups. It takes over the rest of those groups' backfill. seed:status shows ranges per status, articles reached, and per node its leases and headers.

Day to day

• Update after a new version of the indexer: docker compose build --pull && docker compose up -d
• Stop: docker compose stop gives the node's ranges back; another node resumes them after the last batch that was stored.
• A node that disappears (crash, network) loses its ranges to other nodes after 15 minutes without a batch. A range that fails 5 times is set aside as failed (see seed:status).
• Remove a node: docker compose down on the server, then docker compose exec web node ace seed:token my-server --disable on homelab.

Troubleshooting

Log line	Meaning
The seed API rejected this node token	wrong or revoked SEED_TOKEN; the node stops
Seed API …: 404 …; retrying	the seed API is restarting (a deploy on homelab); the node retries
NNTP login failed: …	the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); 481/502: wrong account, too many connections (lower SEED_RANGES) or too many IP addresses
Hostname/IP does not match certificate's altnames: Host: localhost	TLS through a tunnel: set NNTP_TLS_SERVERNAME to the news server's hostname
Range … failed: …	the range is given back and retried later, possibly by another node
Range … passed to another node; dropping it.	this node was too slow to report (15 minutes); harmless
[inspect n] "…" failed: …	an inspection failed (often a missing article); the indexer retries it later