daan/usenet-indexer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release completeness check without reading every binary; SEED_RANGES
Test and deploy / test (push) Successful in 2m15s
Details
Test and deploy / deploy (push) Successful in 2m26s
Details

Browse Source 
release_complete() matched each par2-listed file through a correlated
subquery that scanned all binaries per listed file, run twice per
inserted release_files row: one insert of a few thousand rows ran for
20+ minutes. It now reads the release's files once by release id, and
release_files refreshes once per INSERT statement.

seed:node's SEED_CONNECTIONS/--connections is now SEED_RANGES/--ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-06 12:10:57 +02:00
1 parent 2dbca9b152
commit 4d17c5a8e8
5 files changed
+135 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

seed-node/README.md
+4 -4
View File
@@ -26,7 +26,7 @@ the NNTP lines in `node.env`).
- Outgoing HTTPS to `seed.exp.sforum.nl` (the seed API) and
`gitea.exp.sforum.nl` (to build the image), and outgoing NNTP to the news
server (port 563 for TLS).
- Per connection: roughly 0.4 CPU core and 1–3 MB/s download (headers come
- Per range seeded at once: roughly 0.4 CPU core and 1–3 MB/s download (headers come
gzip-compressed); a few hundred MB of RAM per node. Disk: about 1.5 GB for
the image. Per inspector: up to a core while decoding, and a few MB/s
download (whole articles, ~700 KB each).
@@ -62,7 +62,7 @@ for the same name replaces the token; `--disable` revokes it.
|---|---|
| `SEED_URL` | `https://seed.exp.sforum.nl` (leave as is) |
| `SEED_TOKEN` | the token from step 1 |
| `SEED_CONNECTIONS` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
| `SEED_RANGES` | ranges seeded at once. Start with 2–4; `0` for no seeding. |
| `SEED_INSPECTORS` | inspections and archive walks at once (default `0`). Try 2–4 on a machine with spare cores. |
| `SEED_FETCH=direct` and `NNTP_*` | optional: seed with this node's own NNTP account (see the commented lines). `NNTP_TLS_SERVERNAME` only through a tunnel: the news server's real hostname, for the TLS certificate check. |
@@ -76,7 +76,7 @@ needs are filled in for a node automatically.
The first build takes a few minutes. A working node logs:
[ info ] Seeding node for https://seed.exp.sforum.nl with 2 connection(s).
[ info ] Seeding node for https://seed.exp.sforum.nl: 2 range(s) at once, headers through the seed API; 2 inspection(s) at once, articles through the seed API.
[ info ] [1] alt.binaries.ath 49834000000–49835999999
[ info ] [1] done: 2,000,000 headers in 237 s (8,450/s)
@@ -111,7 +111,7 @@ articles reached, and per node its leases and headers.
|---|---|
| `The seed API rejected this node token` | wrong or revoked `SEED_TOKEN`; the node stops |
| `Seed API …: 404 …; retrying` | the seed API is restarting (a deploy on homelab); the node retries |
| `NNTP login failed: …` | the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); `481`/`502`: wrong account, too many connections (lower `SEED_CONNECTIONS`) or too many IP addresses |
| `NNTP login failed: …` | the node didn't take a range and tries again in a minute (5 minutes for the IP-address limit); `481`/`502`: wrong account, too many connections (lower `SEED_RANGES`) or too many IP addresses |
| `Hostname/IP does not match certificate's altnames: Host: localhost` | TLS through a tunnel: set `NNTP_TLS_SERVERNAME` to the news server's hostname |
| `Range … failed: …` | the range is given back and retried later, possibly by another node |
| `Range … passed to another node; dropping it.` | this node was too slow to report (15 minutes); harmless |