Rebuild: re-uploads are releases of their own
Test and deploy / test (push) Successful in 26s
Test and deploy / deploy (push) Skipped

A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.

- File assembly keeps a re-upload's copy apart: a part already in every
  open copy, posted more than 3 h from them, starts another copy; a
  missing part fills the nearest copy (a re-send of what was missing),
  a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
  more than 3 h, counted from the latest article posted so far (a file
  that takes hours to post doesn't split it); each upload is judged and
  chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
  their upload).
- The catalog matches a chunk to the release of its key whose articles
  the range's group owns and whose posting window is within 3 h, else
  starts one (key, key:2, …): releases get identity_key, group_id,
  last_posted_at. Keys are locked in order (advisory locks), so
  concurrent batches neither deadlock nor both start one; the merger
  writes in key order too. The same upload multi-posted to other groups
  is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
  come later (releases:upload-gaps). releases:duplicates measures the
  duplicate names.

Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 13:31:35 +02:00
1 parent d0f3497ec9
commit 0613984de9
15 files changed
+715 -140

No files matched your search

+16
View File
@@ -81,4 +81,20 @@ test.group('edge / taking over a range', () => {
indexer.add(post(500_001, 'A.Release.2020.mkv', 1), 600_000)
assert.deepEqual([indexer.take().from, indexer.take().from], [500_001, 600_000])
})
test('uploads split where posting pauses for more than 3 hours, counted from the latest article', ({ assert }) => {
const art = (n: number, subject: string, minutes: number): Overview => [n, { 'subject': subject, 'from': 'p <p@x>', 'date': new Date(Date.UTC(2026, 9, 5, 10) + minutes * 60_000).toUTCString(), 'message-id': `<u${n}@test>`, ':bytes': '1000', 'xref': `news.example ${GROUP}:${n}` }]
const ix = new RangeIndexer({ group: GROUP, start: 1, end: 10_000, rules })
ix.add([
// A big file posted over 4 hours, then its par2 right after: one upload.
...Array.from({ length: 5 }, (_, i) => art(100 + i, `Big.Release [1/2] - "Big.Release.mkv" yEnc (${i + 1}/5)`, i * 60)),
art(200, 'Big.Release [2/2] - "Big.Release.par2" yEnc (1/1)', 245),
// The same file again a day later: another.
...Array.from({ length: 5 }, (_, i) => art(300 + i, `Big.Release [1/2] - "Big.Release.mkv" yEnc (${i + 1}/5)`, 1440 + i)),
], 10_001)
ix.finish()
const batch = ix.take()
assert.deepEqual(batch.releases.map((r) => [r.upload, r.files]), [[0, 2], [1, 1]])
assert.equal(batch.releases[0].postedLast! - batch.releases[0].postedAt!, 245 * 60_000)
assert.equal(batch.releases[0].key, batch.releases[1].key)
})
})