Rebuild: re-uploads are releases of their own
A poster re-uploading under the same name made one release of all uploads: in a sample, 24% of releases with 2+ files had a file name more than once (4.3% of files extra copies), 70% of those copies posted a day or more apart; their NZBs listed each copy. - File assembly keeps a re-upload's copy apart: a part already in every open copy, posted more than 3 h from them, starts another copy; a missing part fills the nearest copy (a re-send of what was missing), a re-sent part near a copy that has it is ignored as before. - A key's files in a batch split into uploads where posting pauses for more than 3 h, counted from the latest article posted so far (a file that takes hours to post doesn't split it); each upload is judged and chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry their upload). - The catalog matches a chunk to the release of its key whose articles the range's group owns and whose posting window is within 3 h, else starts one (key, key:2, …): releases get identity_key, group_id, last_posted_at. Keys are locked in order (advisory locks), so concurrent batches neither deadlock nor both start one; the merger writes in key order too. The same upload multi-posted to other groups is a release per group. - The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies come later (releases:upload-gaps). releases:duplicates measures the duplicate names. Both implementations; check.ts compares the windows and uploads too (identical on all captures; Rust tests mirror the TypeScript cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d0f3497ec9
commit
0613984de9
15 files changed
+715
-140
No files matched your search
@@ -72,7 +72,7 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
|
||||
line.push_str(&format!("{{\"t\":\"b\",\"from\":{},\"next\":{},\"headers\":{}}}", batch.from, batch.next, batch.headers));
|
||||
emit(&mut line);
|
||||
let release = |r: &indexer::ReleaseChunk, kind: &str, line: &mut String| {
|
||||
line.push_str(&format!("{{\"t\":\"{kind}\",\"key\":"));
|
||||
line.push_str(&format!("{{\"t\":\"{kind}\",\"upload\":{},\"key\":", r.upload));
|
||||
json_str(&r.key, line);
|
||||
line.push_str(",\"name\":");
|
||||
json_str(&r.name, line);
|
||||
@@ -89,6 +89,8 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
|
||||
}
|
||||
line.push_str("],\"postedAt\":");
|
||||
opt_num(r.posted_at, line);
|
||||
line.push_str(",\"postedLast\":");
|
||||
opt_num(r.posted_last, line);
|
||||
line.push_str(&format!(
|
||||
",\"files\":{},\"segments\":{},\"bytes\":{},\"firstArticle\":{},\"lastArticle\":{},\"readable\":{},\"hasPar2\":{},\"payload\":\"{}\"",
|
||||
r.files,
|
||||
@@ -114,7 +116,7 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
|
||||
emit(&mut line);
|
||||
}
|
||||
for c in &batch.candidates {
|
||||
line.push_str("{\"t\":\"c\",\"releaseKey\":");
|
||||
line.push_str(&format!("{{\"t\":\"c\",\"upload\":{},\"releaseKey\":", c.upload));
|
||||
json_str(&c.release_key, &mut line);
|
||||
line.push_str(&format!(",\"kind\":\"{}\",\"messageId\":", c.kind));
|
||||
json_str(&c.message_id, &mut line);
|
||||
|
||||
Reference in new issue
Block a user