Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka answers 'no such article' after ~450 ms; a quarter of inspection
requests were for missing articles and each failed job was retried up
to three times. Files get content_type 'missing' instead; homelab also
records it from failure reports of nodes not rebuilt yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Half the live leases go to higher-priority ranges, the rest to the others,
so a slow recent gap runs beside older history. With no article
connections the article relay answers 503 instead of waiting forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:plan plans every part of a group's history no range covers (it
started below the lowest known range and left a.b.ath's 13 most recent
months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
About 5% of seeded headers are fragments. They went through staging,
where they waited 30 minutes for parts that were never coming (the
neighbouring range is worked on by another node, often hours later) and
expired: the staging table grew to 22 GB and the NVMe part of the database
reached its 60 GB limit, which stopped leasing and the backfill. Files cut
by a range edge can still be completed by the neighbouring range, so
fragments within 50k articles of an edge go straight into binaries and
parts; the rest, files with articles missing for good, are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A range reported to its end but not finished (its node went away at
the very end) was leased again with start past end; nodes asked for
"from 49208000000 to 49207999999", got 400, and the range failed after
five leases. The orchestrator now marks such ranges done before
leasing, and runRange reports them done without fetching.
- Some old ranges come so slowly that one XOVER of 20k headers takes over
two minutes: five ranges failed on "NNTP reply timed out". The relay
pool's timeout is SEED_NNTP_TIMEOUT_MS (default 10 minutes, the node's
HTTP timeout).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed NNTP login gives the range back uncounted (no attempt used), and a
node with nothing to lease holds no connection. Replaces log-in-before-lease,
which kept idle connections and logged in again every minute while idle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No new range while the database's NVMe part or the hdd tablespace is over
BACKFILL_MAX_NVME_GB / BACKFILL_MAX_HDD_GB; nodes ask again a minute later.
seed:status says when leasing is paused. The NVMe part was at 47 GB of 50
with a plan for all of a.b.ath queued.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
lease without a batch for 15 min passes to the next node and resumes at
next_id), batch (stores files and fragments, moves next_id), finish,
release, plan (from below live indexing / the backfill / earlier plans
down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
status) behind a node-token middleware; JSON bodies up to 64 MB after
gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
connections, gzip batches, retries for network errors and 5xx, drops a
range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
and seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
