The coordinator logs a line a minute: chunks fetched, articles/s,
seconds a chunk, how busy its connections were. It showed the
workstation's runs are bound by its 100 Mbit line now (10 connections
99% busy, ~89 Mbit/s: compressed XOVER is ~69 B an article on recent
boneless), not by the code.
The check that a compressed reply is complete (inflating it) ran on the
coordinator's event loop, ~60 ms a chunk; now on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lease was only granted for a range with spooled chunks, and the
prefetcher kept ranges ahead per active lease: once the spool ran dry a
worker got none and slept 30 s, fewer ranges were leased, fewer
prefetched. Three workers held one lease between them (~50k headers/s).
A lease now takes a spooled range first, else the next planned one
(leased ranges are fetched first; the stream waits).
crypto.hash instead of createHash().update().digest() for file and
release keys: same digests, ~2.5x faster (~4.6% of a worker's time).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator ran the prefetch tick in the same loop as the merger
(up to 30 s a minute): its connections went idle once their targets
were fetched, and the first leases after a start found nothing spooled.
The prefetcher now ticks in a loop of its own.
The prefetcher took the first planned ranges with nothing spooled, even
ones with nothing left to fetch (next_id past the end): two such ranges
(edge:bench's, set back to planned) kept it from fetching anything. It
and the lease query now skip those, and edge:bench marks its range done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merger's neighbour query matched "end_id = ? OR start_id = ?", which
no index serves: it read every range of the group per file, 5.5 s a
round and most of Postgres's time in an end-to-end run. Now the range
holding the article before and after, by (group_id, start_id): 0.08 ms.
The prefetcher's leased and started ranges (every 2 s) get partial
indexes: 15-25 ms -> <0.1 ms with 145k planned ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Waiting files counted as work, so the merge loop re-read them for its
whole 30 s budget every minute: Postgres used ~a core even without
workers. Now ~4% at rest, and storing a boneless range costs Postgres
~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads
are stored EXTERNAL (zstd already; no pglz attempt).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of a chunk's text keeps the whole text alive (tens of MB) for as
long as the file it belongs to stays open: message-ids, file details and
cache keys are now copies. edge:bench on boneless: peak RSS 1,137 → ~660
MB, external memory 570 → 40–75 MB, the same speed (~190k headers/s);
the same results on 800k real lines. edge:bench prints the breakdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hourly in coordinator:run: a release unchanged for 6 hours is settled,
complete when every file its [i/N] counter announced was found, unknown
without a counter (phase 2 reads par2 sets for those). Partial index on
unsettled releases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nzb (buildNzb, parseNzb) from the current app; NzbService builds a
release's NZB from its chunks; GET /api/v1/releases?q= (every word of the
name) and GET /api/v1/releases/:id/nzb behind the api guard. A test stores
a batch, finds the release and reads its NZB back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nyuu's 12 capital+small letter pairs in 15 bytes (10 bits a pair), its
timestamp as a delta within the file, the suffix once per file;
ngPost-style 32 hex digits in 16 bytes; other ids as zstd text as before.
Recent boneless: 25.2 → 20.3 B per segment (and 25.0 → 19.5 on the next
range); a 2013 ath range stays at 17.9 (JBinUp ids are text). Every chunk
of three real ranges decodes to itself (edge:bench --verify). Encoding is
no slower than v1 (less text for zstd). Version 1 chunks stay readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The neighbour check was three queries per file (~2,000 files/minute); now
one per round of 20k files, with a cursor so waiting files don't take
every round's places, and rounds until nothing is left or 30 s. On the
dev backlog (old-style fragments): ~18k files/minute in the 30 s budget;
at full throughput the new fragment rule leaves ~1,200 files/minute.
A group that fails GROUP (gone from the server) no longer stops planning
the others; edge:bench's groups are inactive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XoverClient joined all received pieces into one buffer on every arrival
(quadratic for a 2.6 MB reply), and turned the inflated check into text
the raw path doesn't use. The coordinator under load (3 workers, 4
connections): busy 40% → 13% of a core, Buffer.concat gone from the
profile. (master's relay uses the same client.)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the 4-core workstation (shared with coordinator, web server and
database): 3 processes indexed 6 boneless ranges (12M headers) in 37 s,
~324k headers/s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge:bench (one core): boneless 193k → 215k headers/s, ath 303k → 343k.
Same results on 800k real lines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file key and release identity (two SHA-1 hashes and the release rules)
were ~25% of the edge worker's time. edge:bench: boneless 100k → 137k
headers/s on one core, ath (2013) 70k → 143k.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview chunks in, per batch out: releases as encoded chunks, probe
candidates, counts per rule (crossposts per owner group, obfuscated,
garbage, per-article, incomplete), a sample of what was dropped, and
fragments near the range's edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nntp (XOVER client, Xref, GROUP), lib/headers (subjects, file keys,
segments from overview lines; the yEnc path stays behind for phase 2),
lib/release (identity, random names, now in lib), lib/files (aggregator,
volume names), lib/rules (posting tools, owner by priority, release
verdicts), lib/chunks (release chunk codec). Their unit tests came along;
new ones for codec, rules and volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two probes of one release can be applied at once; when one deletes the
release as garbage, the other's inserts (e.g. a par2 listing) violated
the foreign key and the node got a 500.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:
- before storing (seeding, ingest): random-named uploads of a dropped
signature (posting tool from the Message-ID, first group) or of a
release dropped after its probe aren't stored; 2% of a dropped
signature's releases are still kept and probed. Stored files of dropped
signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
password, names from files or spot, with all its probes done, and
encrypted, a container or missing its first article, is deleted and
tombstoned in dropped_releases.
Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep checked 430k noted releases in 51 minutes, nearly all without a
par2 listing or file total, where completeness can't change. Listed names
are now matched against the files themselves rather than through
binaries, which seeding mostly no longer creates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health checks of web, seed-api and seed-relay called /api/v1/stats
every 30 s, whose release statistics query kept about a core busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating a partition deadlocked with file inserts 11 times in an hour, and
each failure ended the GC sweep before its deletes. The parent is now
locked first (lock_timeout 10 s), four partitions are kept ahead, and an
error is logged while the sweep goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
index-scheduler's loop ended when Postgres restarted (the active-groups
query was outside its try), leaving live indexing stopped for ~3 hours
while the process stayed up. Each step's error is now caught and retried
next round; worker:spots and worker:deferred wait 30 s and claim again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eweka answers 'no such article' after ~450 ms; a quarter of inspection
requests were for missing articles and each failed job was retried up
to three times. Files get content_type 'missing' instead; homelab also
records it from failure reports of nodes not rebuilt yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Par2 sets, NZBs and SFVs were sniffed from their first segment and then
read whole, downloading that segment a second time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
