Rebuild: re-uploads are releases of their own
A poster re-uploading under the same name made one release of all uploads: in a sample, 24% of releases with 2+ files had a file name more than once (4.3% of files extra copies), 70% of those copies posted a day or more apart; their NZBs listed each copy. - File assembly keeps a re-upload's copy apart: a part already in every open copy, posted more than 3 h from them, starts another copy; a missing part fills the nearest copy (a re-send of what was missing), a re-sent part near a copy that has it is ignored as before. - A key's files in a batch split into uploads where posting pauses for more than 3 h, counted from the latest article posted so far (a file that takes hours to post doesn't split it); each upload is judged and chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry their upload). - The catalog matches a chunk to the release of its key whose articles the range's group owns and whose posting window is within 3 h, else starts one (key, key:2, …): releases get identity_key, group_id, last_posted_at. Keys are locked in order (advisory locks), so concurrent batches neither deadlock nor both start one; the merger writes in key order too. The same upload multi-posted to other groups is a release per group. - The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies come later (releases:upload-gaps). releases:duplicates measures the duplicate names. Both implementations; check.ts compares the windows and uploads too (identical on all captures; Rust tests mirror the TypeScript cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d0f3497ec9
commit
0613984de9
15 files changed
+715
-140
No files matched your search
@@ -0,0 +1,26 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Re-uploads as releases of their own (REBUILD.md, decision d): releases
|
||||
* keep the identity key their files gave (identity_key; release_key is that
|
||||
* for the first upload, then key:2, key:3, …), the group that owns their
|
||||
* articles, and their posting window (posted_at … last_posted_at). A batch's
|
||||
* chunk joins a release with its key and group whose window is within
|
||||
* UPLOAD_GAP_MS, else starts one.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS identity_key varchar(40)`)
|
||||
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS group_id integer`)
|
||||
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS last_posted_at timestamptz`)
|
||||
this.schema.raw(`UPDATE releases SET identity_key = release_key, last_posted_at = posted_at WHERE identity_key IS NULL`)
|
||||
this.schema.raw(`ALTER TABLE releases ALTER COLUMN identity_key SET NOT NULL`)
|
||||
this.schema.raw(`ALTER TABLE releases ALTER COLUMN release_key TYPE varchar(48)`)
|
||||
this.schema.raw(`CREATE INDEX IF NOT EXISTS releases_identity_idx ON releases (identity_key, group_id)`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP INDEX IF EXISTS releases_identity_idx`)
|
||||
this.schema.raw(`ALTER TABLE releases DROP COLUMN IF EXISTS identity_key, DROP COLUMN IF EXISTS group_id, DROP COLUMN IF EXISTS last_posted_at`)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user