One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or
"bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release:
looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k
non-Bob releases, and about 60% of all single-article releases in a 2% sample.
looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+
that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone
articles given up before the block's end only; at the end the strict looksRandom stays, as the
rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same.
The prefetcher's corpus fill skips groups that aren't active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pending namer's TypeScript re-index gave a file short of parts on the
server to the range path's fragment rule, so a partial release the edge
wrote (Bob's hex-named uploads in chello, ratcave, font, storage) never came
out again: 10 of 148k named releases moved. RangeIndexer's partialFiles
keeps such a file in its release, marked partial; tools/namer_check.ts
replays sample rows from the corpus (10 of 10 found with it, 1 without).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.
A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file name with a NUL ("…Brutal Erpresst.ra\u0000…") failed every
batch of range 155595 (HTTP 500, jsonb: \u0000 cannot be converted to
text), so the range looped. fromWire drops NUL from every string.
The chunk deadline was given up 258 times in 3 hours at 300 s (~1.4 a
minute, each holding a connection for 5 minutes): 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on two neighbouring boneless ranges (merging their fragments):
- 99.8% of fragment files were single articles of a random-named
'file' of >100 parts: a random subject per article, which never
completes from headers. They now count as per-article.
- A file is kept as a fragment when its first part seen is above 1 and
comes within the start window (the idle window: uploads in busy groups
spread over minutes, and a 50k-article edge zone is ~30 s of boneless),
or when it was still open at the range's end with later parts missing.
Anything else missing parts is incomplete and only counted.
Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met
complete (57,558 segments, ~8% of the range's kept segments).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).
- Prefetcher: spooled_at was set against a total counted from the range's
resume point, which shrinks as the worker advances: the last chunks were
never fetched and the worker waited. Now counted against the whole
range, only new chunks counted, and leased ranges are always checked
against the files. Planned ranges ahead are bounded (it had spooled ever
more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
(Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
(it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
current one, and doesn't wait for a post before indexing on (posts stay
in order; a failed post stops the range).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.
Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview chunks in, per batch out: releases as encoded chunks, probe
candidates, counts per rule (crossposts per owner group, obfuscated,
garbage, per-article, incomplete), a sample of what was dropped, and
fragments near the range's edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
