probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A picture group's 500k articles made batches over the server's 256 MB
body limit: HTTP 413, the range given back and leased again in a loop
(148597: 39 times in 30 minutes). The worker now also posts once
100,000 completed or edge-cut files wait (RangeIndexer.pending).
A stats refresh that failed kept the previous totals silently; it
logs the error now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)
A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picture groups made a release per image: 501k single-file releases in
the first 24 ranges of the bottom 500 groups (a.b.models 385k), ~120M
for all of them. Counters written out ('File 027 of 132', '[12of42]',
'104 of 472', '[2239-6389]', '02/17', '0097-1574'; only when the first
number is at most the second, so '[1948-12]' stays) and per-file sizes
('76.88 MB', '335555 Bytes') kept every subject unique.
- subjectRemainder removes those counters and sizes, and empty brackets.
- A subject-named release is bounded by its counter's total
(counterTotal: the first form found): uploads a poster names alike
stay apart, an upload's files (par2, nfo) together.
- Images and audio tracks don't take the poster-tag rule: the subject
names the set or album. An image set without a counter: the image's
name up to its last number, without numbers.
- Documents and comics (pdf, epub, mobi, azw3, djvu, cbr, cbz, cb7, chm,
lit, doc, docx) are a release each: collections post hundreds under
one subject ('cw-horror&mystery').
releases:regroup --group re-keys a group's indexed releases together:
cheerleaders 55,741 -> 3,165 releases, models 385,298 -> 40,313, German
audiobooks 4,106 -> 1,724, playstation 3,309 -> 2,345; comics and
e-books a release per item; boneless unchanged. Ported to edge-bench-rs;
check.ts: identical on all captures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release 145637 held four albums: the poster puts '[KiLL.dA.FuCkIn.
EaStErRaBbIt]----' before every upload, and the text outside the quotes
(when it names a release) was the key with the poster. The same made
one release of a poster's albums under '(PowBeertje80 Posted)' (a post
id, with the same [i/M]), of masterposts' films, and of one volume of
every set in a DVD9 post.
Now, when the file's stem describes something itself (12+ letters and
digits, two words of 3+ letters) and the text outside the quotes shares
no word with it (4+ letters as part of the stem, 3-letter words and
numbers whole; boilerplate like www/net/posted not counted; no 5+ digit
per-post id), that text is taken for the poster's tag and the stem
says which release the file is. Albums of track files ('FTA … Ketchup'),
'DIE_ZEIT_35' with '35_China_dl', 'FTD769997' posts, scene names with
abbreviated file names ('grp-sm') and generic sidecars ('file_id.diz')
keep their grouping.
releases:regroup re-keys a sample of the catalog with the current rule:
on homelab's test data 145637 → 4 albums, 111566 → 17 albums of 27
files, the rest of the subject releases that change mostly lose an ad
image ('Flyer-download-2-Day.gif'); stem releases unchanged. Ported to
edge-bench-rs: check.ts still finds all captures identical (ath 2013:
16 → 76 releases on both sides).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rules:signatures publishes a rule set version whose dropped signatures
come from a file of tool<TAB>group lines (--add to add to the current
ones). database/data/voyager1_signatures.tsv: voyager 1's dropped ones
(>=100 probes, <=0.5% named), with their probe counts; the two builds
name the posting tools the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.
".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed
XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so
one process tops out around 60 MB/s: less than 50 connections bring
(~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with
id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges
and settles. Two shards of 5 connections fetched what one of 10 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It checked only after a data frame, so a worker waiting for its range's
spool (keep-alive frames only) never stopped on SIGTERM.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It waited for every chunk in flight, and a connection the server held
back (after killed coordinators left sessions open) waits up to the
10-minute reply timeout: it never seemed to stop. It now waits at most
15 s, then closes all connections, busy ones too; the greeting and login
time out after 30 s. And run() ends with terminate(), as edge:node does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator ran the prefetch tick in the same loop as the merger
(up to 30 s a minute): its connections went idle once their targets
were fetched, and the first leases after a start found nothing spooled.
The prefetcher now ticks in a loop of its own.
The prefetcher took the first planned ranges with nothing spooled, even
ones with nothing left to fetch (next_id past the end): two such ranges
(edge:bench's, set back to planned) kept it from fetching anything. It
and the lease query now skip those, and edge:bench marks its range done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Waiting files counted as work, so the merge loop re-read them for its
whole 30 s budget every minute: Postgres used ~a core even without
workers. Now ~4% at rest, and storing a boneless range costs Postgres
~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads
are stored EXTERNAL (zstd already; no pglz attempt).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of a chunk's text keeps the whole text alive (tens of MB) for as
long as the file it belongs to stays open: message-ids, file details and
cache keys are now copies. edge:bench on boneless: peak RSS 1,137 → ~660
MB, external memory 570 → 40–75 MB, the same speed (~190k headers/s);
the same results on 800k real lines. edge:bench prints the breakdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hourly in coordinator:run: a release unchanged for 6 hours is settled,
complete when every file its [i/N] counter announced was found, unknown
without a counter (phase 2 reads par2 sets for those). Partial index on
unsettled releases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nyuu's 12 capital+small letter pairs in 15 bytes (10 bits a pair), its
timestamp as a delta within the file, the suffix once per file;
ngPost-style 32 hex digits in 16 bytes; other ids as zstd text as before.
Recent boneless: 25.2 → 20.3 B per segment (and 25.0 → 19.5 on the next
range); a 2013 ath range stays at 17.9 (JBinUp ids are text). Every chunk
of three real ranges decodes to itself (edge:bench --verify). Encoding is
no slower than v1 (less text for zstd). Version 1 chunks stay readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The neighbour check was three queries per file (~2,000 files/minute); now
one per round of 20k files, with a cursor so waiting files don't take
every round's places, and rounds until nothing is left or 30 s. On the
dev backlog (old-style fragments): ~18k files/minute in the 30 s budget;
at full throughput the new fragment rule leaves ~1,200 files/minute.
A group that fails GROUP (gone from the server) no longer stops planning
the others; edge:bench's groups are inactive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the 4-core workstation (shared with coordinator, web server and
database): 3 processes indexed 6 boneless ranges (12M headers) in 37 s,
~324k headers/s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:
- before storing (seeding, ingest): random-named uploads of a dropped
signature (posting tool from the Message-ID, first group) or of a
release dropped after its probe aren't stored; 2% of a dropped
signature's releases are still kept and probed. Stored files of dropped
signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
password, names from files or spot, with all its probes done, and
encrypted, a container or missing its first article, is deleted and
tombstoned in dropped_releases.
Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating a partition deadlocked with file inserts 11 times in an hour, and
each failure ended the GC sweep before its deletes. The parent is now
locked first (lock_timeout 10 s), four partitions are kept ahead, and an
error is logged while the sweep goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
index-scheduler's loop ended when Postgres restarted (the active-groups
query was outside its try), leaving live indexing stopped for ~3 hours
while the process stayed up. Each step's error is now caught and retried
next round; worker:spots and worker:deferred wait 30 s and claim again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SeedService looked up every file of a batch across all files partitions
after inserting it (15% of Postgres' active time); the merge job read
every release on each GC sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 naming needs the fingerprint inspection stores on a binary, and the
GC deleted seeded binaries (6 h unfingerprinted, 48 h stale) long before
the inspection queue reached them. binaries.awaiting_inspection protects
them for up to 7 days; the GC deletes use partial indexes so they don't
read past waiting rows. Seeding no longer creates binaries for files that
aren't inspected (~85% of a.b.ath), which only churned the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dense recent a.b.ath ranges produced batches over the 64 MB body limit
(413). The node now posts such a batch in halves; only the second moves
the range's resume point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- seed:plan plans every part of a group's history no range covers (it
started below the lowest known range and left a.b.ath's 13 most recent
months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
matchPar2Releases checked every same-size release_files row's set and
date (18 s per batch of body-derived binaries); now an index on
release_files (release_id, size) drops the repeated sizes first: 0.16 s.
File inserts updated their release in four row triggers; one statement
trigger now updates each release once per INSERT. Completeness is noted
in release_dirty and recomputed by gc:scheduler every 30 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One process ran everything on one core. SEED_PROCESSES (default: the
number of cores) worker processes now each run seed:node with a share of
the slots; stop signals are passed on and crashed workers restarted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_complete() matched each par2-listed file through a correlated
subquery that scanned all binaries per listed file, run twice per
inserted release_files row: one insert of a few thousand rows ran for
20+ minutes. It now reads the release's files once by release id, and
release_files refreshes once per INSERT statement.
seed:node's SEED_CONNECTIONS/--connections is now SEED_RANGES/--ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.
A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One poster adds the same 22 GB hidden '.1917-….mkv' decoy to every 7z
upload; the largest-entry rule named 840 releases after it. Entries with
a hidden path part are skipped; release:rename --hidden renames the
existing ones from their stored listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inspection lists only the entries whose headers are in the first segment
of a first volume. A walk hops from header to header, fetching one segment
per entry and volume, in its own queue (one job per collection worker) on
NNTP connections inspection isn't waiting for. content:walk-archives queues
releases inspected earlier (CONCEPT D40).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
About 5% of seeded headers are fragments. They went through staging,
where they waited 30 minutes for parts that were never coming (the
neighbouring range is worked on by another node, often hours later) and
expired: the staging table grew to 22 GB and the NVMe part of the database
reached its 60 GB limit, which stopped leasing and the backfill. Files cut
by a range edge can still be completed by the neighbouring range, so
fragments within 50k articles of an edge go straight into binaries and
parts; the rest, files with articles missing for good, are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New rows went into free space scattered through the 9.6 GB heap on the HDD
(left behind by inspection's updates): one random HDD read per insert, and
seed batches took a median 13 s to store. The existing table becomes the
first partition as it is (a CHECK scan, no rewrite; indexes, its foreign
key and the triggers are taken over by the partitioned parent); new
partitions of 4M ids go in the same tablespace and stay in memory while
they fill.
- Every partition has a unique file_key index; FileStore.insert looks keys
up in all partitions first (a key can't be unique across partitions).
- FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead.
- Stats count files from the partitions; seed:node waits ~20 minutes for
the API before giving a range up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed NNTP login gives the range back uncounted (no attempt used), and a
node with nothing to lease holds no connection. Replaces log-in-before-lease,
which kept idle connections and logged in again every minute while idle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
