A worker thread leases a group and takes its ranges in article order through
one carried window; the coordinator keeps one cursor a group (group_leases,
migration 6500) and hands out the next range, the rules and nothing about
where the chunks are: the range stream serves a stored range's chunks from the
corpus file and a live one's from the spool. The window's snapshot (bincode:
the frontier, the open files, releases and albums, the upload numbers, the
batch-counter table) goes to the coordinator every EDGE_SNAPSHOT_MINUTES and
when the group is given back, and the next holder restores it; a takeover
after LEASE_TTL_SECONDS plans the ranges from the snapshot's article again.
The prefetcher runs ahead of held groups' cursors and skips chunks the corpus
has. LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY bound what is leased. edge-node
holds groups by default; --ranges keeps the old per-range path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node API (leases, hour-long streams, batches of up to 256 MB JSON
parsed in-process) shared the web process with the catalog API, Newznab
and the web interface; voyager 1 had to make the same split on
2026-10-06. HTTP_ROUTES picks a process's routes (node, public, all by
default): compose service node-api (same image, port 3335 on the compose
network) serves the workers, web the rest. The workers' COORDINATOR_URL
points at node-api; deploy.sh stops edge before recreating node-api.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api (Newznab: caps, search, tvsearch, movie, music, book, get) for
Prowlarr, Sonarr and Radarr: voyager 1's controller and XML on the new
catalog; readable releases not known to miss files; NEWZNAB_API_KEY.
- /api/v1/releases (q, cat, readable, complete, source, poster, size,
sort, paging), /api/v1/releases/:id with its files (decoded from the
chunks; NzbService.files, shared with the NZB), /api/v1/stats
(estimates, plus history and tail progress), /health. Open, as voyager
1's catalog; the web interface uses it.
- release_category(name): voyager 1's categorizer without the archive
contents, a generated column with a (category, posted_at) index.
- @adonisjs/static for the web interface (next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.
".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nzb (buildNzb, parseNzb) from the current app; NzbService builds a
release's NZB from its chunks; GET /api/v1/releases?q= (every word of the
name) and GET /api/v1/releases/:id/nzb behind the api guard. A test stores
a batch, finds the release and reads its NZB back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health checks of web, seed-api and seed-relay called /api/v1/stats
every 30 s, whose release statistics query kept about a core busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
lease without a batch for 15 min passes to the next node and resumes at
next_id), batch (stores files and fragments, moves next_id), finish,
release, plan (from below live indexing / the backfill / earlier plans
down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
status) behind a node-token middleware; JSON bodies up to 64 MB after
gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
connections, gzip batches, retries for network errors and 5xx, drops a
range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
and seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
