XoverClient joined all received pieces into one buffer on every arrival
(quadratic for a 2.6 MB reply), and turned the inflated check into text
the raw path doesn't use. The coordinator under load (3 workers, 4
connections): busy 40% → 13% of a core, Buffer.concat gone from the
profile. (master's relay uses the same client.)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nntp (XOVER client, Xref, GROUP), lib/headers (subjects, file keys,
segments from overview lines; the yEnc path stays behind for phase 2),
lib/release (identity, random names, now in lib), lib/files (aggregator,
volume names), lib/rules (posting tools, owner by priority, release
verdicts), lib/chunks (release chunk codec). Their unit tests came along;
new ones for codec, rules and volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For 2026 a.b.ath articles Eweka's gzip replies have a long tail (p90 12.4 s
per 20k vs 3.9 s plain; 6.1k vs 10.5k headers/s). SEED_NNTP_COMPRESS turns
compression back on; plain replies are deflated for nodes asynchronously.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five ranges "never answered" in 10 minutes, yet their chunk came back in
0.3 s on a fresh connection: the news server drops idle connections
without telling, and the pool's next request on one waited for the
10-minute inactivity timeout. Now: SEED_NNTP_TIMEOUT_MS 90 s (inactivity;
slow ranges keep streaming), one retry on a fresh connection, and pooled
connections idle over 60 s are replaced before use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:node logs in to NNTP before taking a range and keeps the connection
across ranges (replaced after 2 minutes idle or a failed range). A node
that can't reach its news server takes no range: two ranges were set
aside as failed while an office node's tunnel was being set up
(ECONNREFUSED 127.0.0.1:1563, five leases each). After an IP-address
limit refusal it waits 5 minutes.
- NNTP_TLS_SERVERNAME: the hostname the TLS certificate is checked
against when NNTP_HOST is a tunnel ("Hostname/IP does not match
certificate's altnames: Host: localhost").
- seed-node/README: the IP-address limit and tunnelling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seeding node that waited over two minutes for the seed API (restarting)
had its idle NNTP connection closed by its own timeout, and gave its range
back. Now the timeout runs only while a reply is awaited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lib/nntp/xover_client: a minimal one-connection client for seed:range
with XFEATURE COMPRESS GZIP (a zlib stream after "224 … [COMPRESS=GZIP]",
then a "." line; 420 for an empty range). About 6x less header traffic.
Overviews come out with the field names nntp-js uses.
- SfvService.claimAll: one sfv_entries lookup per stored batch.
- Fragments go through the regular ingest path beside fetching (at most
two batches in flight); a failure fails the run at the end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
