usenet-indexer/seed-node/README.md

# Seeding node

A seeding node indexes old article ranges for the indexer (CONCEPT.md D37,
D38). It asks the seed API for a range over HTTPS, fetches that range's
headers over its own NNTP connections, groups them into files, and posts the
results back. It only makes outgoing connections: no port forwarding,
database or Redis.

## What a node needs

- Docker with the compose plugin (`docker compose version` works).
- Outgoing HTTPS to `seed.exp.sforum.nl` (the seed API) and
  `gitea.exp.sforum.nl` (to build the image), and outgoing NNTP to the news
  server (port 563 for TLS).
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
  gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
  the image.
- An NNTP account. Every connection a node opens counts against that
  account's connection limit, next to the ~20 the indexer itself uses if it's
  the same account. Too many gives `481`/`502` errors at login. Some
  providers also limit the number of IP addresses at once (Eweka: 2, as
  `502 (max number of simultaneous IP addresses reached: 2)`): a node behind
  the same public IP as the indexer doesn't count extra, others do. A node
  can share an IP by tunnelling its NNTP traffic (e.g. SSH
  `-L 1563:news.eweka.nl:563` to the indexer's machine); then set
  `NNTP_HOST` to the tunnel and `NNTP_TLS_SERVERNAME=news.eweka.nl`.

## 1. On the indexer (homelab): a token for the node

    cd ~/dev/usenet-indexer
    docker compose exec web node ace seed:token my-server

This prints a token starting with `sn_` once. Treat it like a password: it
lets the holder lease ranges and submit results. Running the command again
for the same name replaces the token; `--disable` revokes it.

## 2. On the server: get the files

    mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node
    curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml
    curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example
    chmod 600 node.env

## 3. Configure `node.env`

| Setting | Value |
|---|---|
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
| `SEED_TOKEN` | the token from step 1 |
| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. |
| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) |
| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account |
| `NNTP_TLS_SERVERNAME` | only through a tunnel: the news server's real hostname, for the TLS certificate check |

Leave the block below "Required by the app's configuration, but unused" as
it is: the app checks these settings at startup, but a node never uses them.

## 4. Start it

    docker compose up -d --build
    docker compose logs -f

The first build takes a few minutes. A working node logs:

    [ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s).
    [ info ] [1] alt.binaries.ath 49834000000–49835999999
    [ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)

With nothing planned it waits and asks again every minute (no log lines).

## 5. Plan work and follow it (on homelab)

    docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
    docker compose exec web node ace seed:status

`seed:plan` queues ranges of 2 million articles from below what is already
indexed down to `--days` back (or everything the server has, without
`--days`), newest first and interleaved across groups. It takes over the
rest of those groups' backfill. `seed:status` shows ranges per status,
articles reached, and per node its leases and headers.

## Day to day

- **Update** after a new version of the indexer:
  `docker compose build --pull && docker compose up -d`
- **Stop**: `docker compose stop` gives the node's ranges back; another node
  resumes them after the last batch that was stored.
- **A node that disappears** (crash, network) loses its ranges to other
  nodes after 15 minutes without a batch. A range that fails 5 times is set
  aside as `failed` (see `seed:status`).
- **Remove a node**: `docker compose down` on the server, then
  `docker compose exec web node ace seed:token my-server --disable` on homelab.

## Troubleshooting

| Log line | Meaning |
|---|---|
| `The seed API rejected this node token` | wrong or revoked `SEED_TOKEN`; the node stops |
| `Seed API …: 404 …; retrying` | the seed API is restarting (a deploy on homelab); the node retries |
| `NNTP login failed: …` | the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); `481`/`502`: wrong account, too many connections (lower `SEED_CONNECTIONS`) or too many IP addresses |
| `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |