Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.
Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The name pass put every holder of a set into its merge: a par2 posted
again years later ("Auto Bild Klassik Februar 02-2014", held by a 2014
and a 2026 release) would have joined the repost to the old upload. Now
the newest holder anchors it and only holders in its group within 12
hours take part. Every round, three sets failed on the probe candidates'
unique key: a member held a candidate of an article the target had.
merge() now drops such a candidate (the target's, or a lower member's,
stays), as it already does for par2 sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.
Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hep, hephaestus and gyges post each file under a subject name of its own
(31,000 one-file releases), but the yEnc name in each file's first article
keeps the upload's random stem ("bX147zx8I0oEgj7nzFCiorCT0ps2jEcb.7z.001"),
one stem per counter total. The merger's second pass joins releases whose
lone inspections share a random stem: one group and poster, the same total,
within 48 h, no more files than the total plus 10. An expression index over
the stem; release_merges records the kind (via) and takes a stem.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every kantodoNNN.bin test upload carries the same par2 set: 64 releases held
it nearby and became one release of 631 files for a set of 10. A set held by
more than 4 releases nearby isn't merged, nor members with more than 3x the
set's files plus 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A par2 index and its repost both hold a set: moving both rows to the target
broke par2_sets' primary key, and the round failed on it every minute from
20:28 UTC (no merges at all). Duplicates among the others go first; a set that
fails is logged and the round goes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their
first 16 KiB and size, so all matched one set's single file and became one
release of 1,284 files. A par2 file that matches more than one release in
the anchor's group and time window is left out; a plain repost of the same
file stays apart too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
