Daan MeijerandClaude Opus 5.5
8c36744088
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with lease, resume point and counts); migration …140. - SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a lease without a batch for 15 min passes to the next node and resumes at next_id), batch (stores files and fragments, moves next_id), finish, release, plan (from below live indexing / the backfill / earlier plans down to N days back; takes over the group's backfill), status. - Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release, status) behind a node-token middleware; JSON bodies up to 64 MB after gzip. - seed:node runs a node: no database or Redis, outgoing HTTPS only, N connections, gzip batches, retries for network errors and 5xx, drops a range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose file that builds the image from this repository. - seed:token, seed:plan, seed:status for the operator. Header parsing moved to lib/ingest and the range loop to lib/seed/runner, shared by seed:range and seed:node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27 lines
1.1 KiB
Markdown
27 lines
1.1 KiB
Markdown
# Seeding node
|
|
|
|
A seeding node indexes old article ranges for the indexer (CONCEPT.md D37,
|
|
D38). It asks the seed API for a range over HTTPS, fetches that range's
|
|
headers over its own NNTP connections, groups them into files, and posts the
|
|
result back. It only makes outgoing connections: no port forwarding, database
|
|
or Redis.
|
|
|
|
On the indexer (homelab):
|
|
|
|
docker compose exec web node ace seed:token my-server # prints the node's token once
|
|
docker compose exec web node ace seed:plan alt.binaries.ath --days 365
|
|
docker compose exec web node ace seed:status
|
|
|
|
On the node (any machine with Docker):
|
|
|
|
cp node.env.example node.env # SEED_TOKEN, NNTP account, SEED_CONNECTIONS
|
|
docker compose up -d --build
|
|
docker compose logs -f
|
|
|
|
`docker compose stop` gives the node's ranges back; a node that disappears
|
|
loses its ranges to other nodes after 15 minutes without a batch. Updating:
|
|
`docker compose up -d --build` again.
|
|
|
|
Mind the NNTP account's connection limit: every node's SEED_CONNECTIONS
|
|
count against it, next to the indexer's own connections.
|