96 lines
4.3 KiB
Markdown
96 lines
4.3 KiB
Markdown
# Seeding node
|
||
|
||
A seeding node indexes old article ranges for the indexer (CONCEPT.md D37,
|
||
D38). It asks the seed API for a range over HTTPS, fetches that range's
|
||
headers over its own NNTP connections, groups them into files, and posts the
|
||
results back. It only makes outgoing connections: no port forwarding,
|
||
database or Redis.
|
||
|
||
## What a node needs
|
||
|
||
- Docker with the compose plugin (`docker compose version` works).
|
||
- Outgoing HTTPS to `seed.exp.sforum.nl` (the seed API) and
|
||
`gitea.exp.sforum.nl` (to build the image), and outgoing NNTP to the news
|
||
server (port 563 for TLS).
|
||
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
|
||
gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
|
||
the image.
|
||
- An NNTP account. Every connection a node opens counts against that
|
||
account's connection limit, next to the ~40 the indexer itself uses if it's
|
||
the same account. Too many gives `481`/`502` errors at login.
|
||
|
||
## 1. On the indexer (homelab): a token for the node
|
||
|
||
cd ~/dev/usenet-indexer
|
||
docker compose exec web node ace seed:token my-server
|
||
|
||
This prints a token starting with `sn_` once. Treat it like a password: it
|
||
lets the holder lease ranges and submit results. Running the command again
|
||
for the same name replaces the token; `--disable` revokes it.
|
||
|
||
## 2. On the server: get the files
|
||
|
||
mkdir -p ~/usenet-seed-node && cd ~/usenet-seed-node
|
||
curl -fsSLO https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/compose.yaml
|
||
curl -fsSL -o node.env https://gitea.exp.sforum.nl/daan/usenet-indexer/raw/branch/master/seed-node/node.env.example
|
||
chmod 600 node.env
|
||
|
||
## 3. Configure `node.env`
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
|
||
| `SEED_TOKEN` | the token from step 1 |
|
||
| `SEED_CONNECTIONS` | ranges worked on at once, one NNTP connection each. Start with 2–4. |
|
||
| `NNTP_HOST`, `NNTP_PORT`, `NNTP_SECURE` | the news server; over the internet use TLS: port `563`, `NNTP_SECURE=true` (for Eweka: `news.eweka.nl`) |
|
||
| `NNTP_USER`, `NNTP_PASSWORD` | the NNTP account |
|
||
|
||
Leave the block below "Required by the app's configuration, but unused" as
|
||
it is: the app checks these settings at startup, but a node never uses them.
|
||
|
||
## 4. Start it
|
||
|
||
docker compose up -d --build
|
||
docker compose logs -f
|
||
|
||
The first build takes a few minutes. A working node logs:
|
||
|
||
[ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s).
|
||
[ info ] [1] alt.binaries.ath 49834000000–49835999999
|
||
[ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)
|
||
|
||
With nothing planned it waits and asks again every minute (no log lines).
|
||
|
||
## 5. Plan work and follow it (on homelab)
|
||
|
||
docker compose exec web node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
|
||
docker compose exec web node ace seed:status
|
||
|
||
`seed:plan` queues ranges of 2 million articles from below what is already
|
||
indexed down to `--days` back (or everything the server has, without
|
||
`--days`), newest first and interleaved across groups. It takes over the
|
||
rest of those groups' backfill. `seed:status` shows ranges per status,
|
||
articles reached, and per node its leases and headers.
|
||
|
||
## Day to day
|
||
|
||
- **Update** after a new version of the indexer:
|
||
`docker compose build --pull && docker compose up -d`
|
||
- **Stop**: `docker compose stop` gives the node's ranges back; another node
|
||
resumes them after the last batch that was stored.
|
||
- **A node that disappears** (crash, network) loses its ranges to other
|
||
nodes after 15 minutes without a batch. A range that fails 5 times is set
|
||
aside as `failed` (see `seed:status`).
|
||
- **Remove a node**: `docker compose down` on the server, then
|
||
`docker compose exec web node ace seed:token my-server --disable` on homelab.
|
||
|
||
## Troubleshooting
|
||
|
||
| Log line | Meaning |
|
||
|---|---|
|
||
| `The seed API rejected this node token` | wrong or revoked `SEED_TOKEN`; the node stops |
|
||
| `Seed API …: 404 …; retrying` | the seed API is restarting (a deploy on homelab); the node retries |
|
||
| `NNTP: unexpected reply "481 …"` or `"502 …"` | NNTP login refused: wrong account, or too many connections on it; lower `SEED_CONNECTIONS` |
|
||
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
|
||
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |
|