daan/usenet-indexer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rebuild: the edge workers' API in its own service
Test and deploy / test (push) Successful in 49s
Details
Test and deploy / deploy (push) Successful in 1m16s
Details

Browse Source 
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-11 19:49:56 +02:00
1 parent 34fea342f5
commit 48f4e982f9
5 files changed
+87 -52
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

deploy/README.md
+4 -3
View File
@@ -15,7 +15,8 @@ homelab with the deploy key, whose forced command
|---|---|---|
| `postgres` | postgres:17 | the catalog (tablespace `nvme`) and the chunks (HDD) |
| `migrate` | `usenet-rebuild` (target `app`) | `migration:run`, before every deploy |
| `web` | `usenet-rebuild` | node API, catalog API, Newznab, web interface; port 3334 |
| `web` | `usenet-rebuild` | catalog API, Newznab, web interface; port 3334 (`HTTP_ROUTES=public`) |
| `node-api` | `usenet-rebuild` | the edge workers' API (leases, streams, batches); port 3335 on the compose network only (`HTTP_ROUTES=node`) |
| `coordinator` | `usenet-rebuild` | prefetching (all NNTP connections), planning, the tail, merging, settling |
| `prober` | `usenet-rebuild` | phase 2: reads the probe queue's articles over `PROBE_CONNECTIONS` (6) connections, stores what they show |
| `edge` | `usenet-rebuild-edge` (target `edge`) | the Rust worker, `--threads $EDGE_THREADS` (8) |
@@ -60,8 +61,8 @@ docker compose up -d --no-deps edge coordinator
- Recreate app services only, with `--no-deps`; Postgres only on purpose,
with the worker and coordinator stopped first.
- Stop `edge` before `web`: a worker holding a stream keeps the server from
exiting.
- Stop `edge` before `node-api`: a worker holding a stream keeps the server
from exiting.
- The NNTP account is shared with voyager 1 and downloads: `NNTP_CONNECTIONS`
(40) plus one plain connection, plus the prober's `PROBE_CONNECTIONS` (6),
out of 50.