Daan MeijerandClaude Opus 5.5
8c36744088
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with lease, resume point and counts); migration …140. - SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a lease without a batch for 15 min passes to the next node and resumes at next_id), batch (stores files and fragments, moves next_id), finish, release, plan (from below live indexing / the backfill / earlier plans down to N days back; takes over the group's backfill), status. - Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release, status) behind a node-token middleware; JSON bodies up to 64 MB after gzip. - seed:node runs a node: no database or Redis, outgoing HTTPS only, N connections, gzip batches, retries for network errors and 5xx, drops a range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose file that builds the image from this repository. - seed:token, seed:plan, seed:status for the operator. Header parsing moved to lib/ingest and the range loop to lib/seed/runner, shared by seed:range and seed:node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
30 lines
859 B
Bash
30 lines
859 B
Bash
# --- Seed API: the orchestrator, and this node's token (`node ace seed:token <name>` on the indexer).
|
|
SEED_URL=https://seed.exp.sforum.nl
|
|
SEED_TOKEN=sn_replace_me
|
|
# Ranges worked on at once; each uses one NNTP connection of the account below.
|
|
SEED_CONNECTIONS=2
|
|
|
|
# --- NNTP account this node fetches headers with (mind its connection limit across all nodes).
|
|
NNTP_HOST=news.example.com
|
|
NNTP_PORT=563
|
|
NNTP_SECURE=true
|
|
NNTP_USER=replace_me
|
|
NNTP_PASSWORD=replace_me
|
|
|
|
# --- Required by the app's configuration, but unused on a seeding node (no database, Redis or web server).
|
|
NODE_ENV=production
|
|
TZ=UTC
|
|
PORT=3333
|
|
HOST=0.0.0.0
|
|
LOG_LEVEL=info
|
|
APP_KEY=unused-on-seed-nodes-unused-on-seed-nodes
|
|
APP_URL=http://localhost:3333
|
|
SESSION_DRIVER=memory
|
|
DB_CONNECTION=pg
|
|
DB_HOST=localhost
|
|
DB_PORT=5432
|
|
DB_USER=unused
|
|
DB_DATABASE=unused
|
|
REDIS_HOST=localhost
|
|
REDIS_PORT=6379
|