59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merges delete releases, so the count of a key's releases fell behind its
numbers (83 releases up to key:129): the 20 numbers tried were all taken
and the range's batch failed for good. Spot releases (spot:key) are not
numbered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FTD posts one upload's files to different groups ("Knowsum-Loops_Are_My_
Religion-WEB-2026-FTD": par2 and part1 to a.b.music.misc, part2 to
a.b.sounds.music, each article in one group only; releases 81825221 and
81825412): the catalog matched a chunk only to releases of its own
group. A chunk without one there now joins a release of its key in
another group whose window is within 3 hours, when none of its file
names are in that release yet; a post of the same files to another group
(other message-ids, same names) stays a release of its own, or its NZB
would hold every file twice. Functional tests for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Writers lock their group only, so two groups' batches can number the
same identity key at once (a spam post crossposted to 67 groups:
duplicate key ...:65, HTTP 500, the batch retried). Releases are now
inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped
take the next number (up to 20 tries).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker waited on one chunk of a leased range for an hour (range
146477: five 15-minute stream timeouts, then the range took 63 s; 88
"stream: Peer disconnected" overnight). The silence timeout doesn't
catch a reply that trickles: a chunk still coming in after
NNTP_CHUNK_DEADLINE_SECONDS (300) is given up, its connection closed,
and fetched again. The report names the oldest request in flight (range,
chunk, age, bytes in) and counts deadlines.
Batches failed with "Maximum call stack size exceeded" (8 overnight):
Math.min(...segments) on a fragment with more segments than the stack
takes arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BatchStore.store held the range's row (SELECT … FOR UPDATE) for the
whole batch transaction (15-20 s on the HDD), so the prefetcher's
spooled_chunks updates and the heartbeats of that range waited on it
(0.5 s on average): the prefetcher's NNTP connections sat idle. Now an
advisory lock per range serializes its batches, and the resume point is
checked again in the final, conditional update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A picture group's batch has 30-55k release keys; an advisory lock each
ran Postgres out of shared memory (max_locks_per_transaction), and those
batches failed with HTTP 500 (the ranges went back, nothing stored).
A chunk only matches releases of its range's group, so one lock per
group serializes exactly the writers that could both start an upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.
- File assembly keeps a re-upload's copy apart: a part already in every
open copy, posted more than 3 h from them, starts another copy; a
missing part fills the nearest copy (a re-send of what was missing),
a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
more than 3 h, counted from the latest article posted so far (a file
that takes hours to post doesn't split it); each upload is judged and
chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
their upload).
- The catalog matches a chunk to the release of its key whose articles
the range's group owns and whose posting window is within 3 h, else
starts one (key, key:2, …): releases get identity_key, group_id,
last_posted_at. Keys are locked in order (advisory locks), so
concurrent batches neither deadlock nor both start one; the merger
writes in key order too. The same upload multi-posted to other groups
is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
come later (releases:upload-gaps). releases:duplicates measures the
duplicate names.
Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- releases.complete is computed with every batch (files >= the
counter's total) instead of only when a release settles after 6 quiet
hours: the 'complete' filter matched nothing in a fresh catalog.
- A release's file total comes from any counter form (counterTotal:
'(01/10)', 'File 27 of 132', …), not only [i/M]; the post rule's key
keeps the [i/M] total, so keys are unchanged. Both implementations.
- groups.tail: a group's live tail can be paused (its history and owner
priority stay), e.g. to let other groups' history go first.
- Stats: history by articles, and the current pass (history ranges
before those deferred 2000 priorities behind) as its own figure; the
header showed 0.1% of ranges counting 145k deferred ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.
- Prefetcher: every connection takes the next missing chunk (leased
ranges first, in order); one connection per range (~26k lines/s) kept
the worker waiting
- executable rule: releases of bare executables (malware floods named
after films and software) are dropped; they were 92% of the catalog rows
of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
1.3 s for a boneless range (1.24 ms per release chunk)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
