diff --git a/seed-node/README.md b/seed-node/README.md index 83b47ea..e77f480 100644 --- a/seed-node/README.md +++ b/seed-node/README.md @@ -3,24 +3,93 @@ A seeding node indexes old article ranges for the indexer (CONCEPT.md D37, D38). It asks the seed API for a range over HTTPS, fetches that range's headers over its own NNTP connections, groups them into files, and posts the -result back. It only makes outgoing connections: no port forwarding, database -or Redis. +results back. It only makes outgoing connections: no port forwarding, +database or Redis. -On the indexer (homelab): +## What a node needs - docker compose exec web node ace seed:token my-server # prints the node's token once - docker compose exec web node ace seed:plan alt.binaries.ath --days 365 - docker compose exec web node ace seed:status +- Docker with the compose plugin (`docker compose version` works). +- Outgoing HTTPS to `seed.exp.sforum.nl` (the seed API) and + `gitea.exp.sforum.nl` (to build the image), and outgoing NNTP to the news + server (port 563 for TLS). +- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come + gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for + the image. +- An NNTP account. Every connection a node opens counts against that + account's connection limit, next to the ~40 the indexer itself uses if it's + the same account. Too many gives `481`/`502` errors at login. -On the node (any machine with Docker): +## 1. On the indexer (homelab): a token for the node + + cd ~/dev/usenet-indexer + docker compose exec web node ace seed:token my-server + +This prints a token starting with `sn_` once. Treat it like a password: it +lets the holder lease ranges and submit results. Running the command again +for the same name replaces the token; `--disable` revokes it. + +## 2. On the server: get the files + + mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node + curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml + curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example + chmod 600 node.env + +## 3. Configure `node.env` + +| Setting | Value | +|---|---| +| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) | +| `SEED_TOKEN` | the token from step 1 | +| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. | +| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) | +| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account | + +Leave the block below "Required by the app's configuration, but unused" as +it is: the app checks these settings at startup, but a node never uses them. + +## 4. Start it - cp node.env.example node.env # SEED_TOKEN, NNTP account, SEED_CONNECTIONS docker compose up -d --build docker compose logs -f -`docker compose stop` gives the node's ranges back; a node that disappears -loses its ranges to other nodes after 15 minutes without a batch. Updating: -`docker compose up -d --build` again. +The first build takes a few minutes. A working node logs: -Mind the NNTP account's connection limit: every node's SEED_CONNECTIONS -count against it, next to the indexer's own connections. + [ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s). + [ info ] [1] alt.binaries.ath 49834000000–49835999999 + [ info ] [1] done: 2,000,000 headers in 237 s (8,450/s) + +With nothing planned it waits and asks again every minute (no log lines). + +## 5. Plan work and follow it (on homelab) + + docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365 + docker compose exec web node ace seed:status + +`seed:plan` queues ranges of 2 million articles from below what is already +indexed down to `--days` back (or everything the server has, without +`--days`), newest first and interleaved across groups. It takes over the +rest of those groups' backfill. `seed:status` shows ranges per status, +articles reached, and per node its leases and headers. + +## Day to day + +- **Update** after a new version of the indexer: + `docker compose build --pull && docker compose up -d` +- **Stop**: `docker compose stop` gives the node's ranges back; another node + resumes them after the last batch that was stored. +- **A node that disappears** (crash, network) loses its ranges to other + nodes after 15 minutes without a batch. A range that fails 5 times is set + aside as `failed` (see `seed:status`). +- **Remove a node**: `docker compose down` on the server, then + `docker compose exec web node ace seed:token my-server --disable` on homelab. + +## Troubleshooting + +| Log line | Meaning | +|---|---| +| `The seed API rejected this node token` | wrong or revoked `SEED_TOKEN`; the node stops | +| `Seed API …: 404 …; retrying` | the seed API is restarting (a deploy on homelab); the node retries | +| `NNTP: unexpected reply "481 …"` or `"502 …"` | NNTP login refused: wrong account, or too many connections on it; lower `SEED_CONNECTIONS` | +| `Range … failed: …` | the range is given back and retried later, possibly by another node | +| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |