For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lease was only granted for a range with spooled chunks, and the
prefetcher kept ranges ahead per active lease: once the spool ran dry a
worker got none and slept 30 s, fewer ranges were leased, fewer
prefetched. Three workers held one lease between them (~50k headers/s).
A lease now takes a spooled range first, else the next planned one
(leased ranges are fetched first; the stream waits).
crypto.hash instead of createHash().update().digest() for file and
release keys: same digests, ~2.5x faster (~4.6% of a worker's time).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of a chunk's text keeps the whole text alive (tens of MB) for as
long as the file it belongs to stays open: message-ids, file details and
cache keys are now copies. edge:bench on boneless: peak RSS 1,137 → ~660
MB, external memory 570 → 40–75 MB, the same speed (~190k headers/s);
the same results on 800k real lines. edge:bench prints the breakdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge:bench (one core): boneless 193k → 215k headers/s, ath 303k → 343k.
Same results on 800k real lines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file key and release identity (two SHA-1 hashes and the release rules)
were ~25% of the edge worker's time. edge:bench: boneless 100k → 137k
headers/s on one core, ath (2013) 70k → 143k.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview chunks in, per batch out: releases as encoded chunks, probe
candidates, counts per rule (crossposts per owner group, obfuscated,
garbage, per-article, incomplete), a sample of what was dropped, and
fragments near the range's edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
