Drop garbage uploads early (D42)
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 2m33s

Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:

- before storing (seeding, ingest): random-named uploads of a dropped
  signature (posting tool from the Message-ID, first group) or of a
  release dropped after its probe aren't stored; 2% of a dropped
  signature's releases are still kept and probed. Stored files of dropped
  signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
  password, names from files or spot, with all its probes done, and
  encrypted, a container or missing its first article, is deleted and
  tombstoned in dropped_releases.

Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-08 11:21:01 +02:00
1 parent 35d0b2da65
commit 211820067a
13 files changed
+683 -6

No files matched your search

+41
View File
@@ -151,6 +151,17 @@ export class DeferredRangeSchema extends BaseModel {
declare lastArticle: bigint | number
}
export class DroppedReleaseSchema extends BaseModel {
static $columns = ['droppedAt', 'reason', 'releaseKey'] as const
$columns = DroppedReleaseSchema.$columns
@column.dateTime()
declare droppedAt: DateTime
@column()
declare reason: string
@column({ isPrimary: true })
declare releaseKey: string
}
export class FetchRangeSchema extends BaseModel {
static $columns = ['backfill', 'createdAt', 'endId', 'error', 'groupId', 'id', 'requeues', 'startId', 'status', 'updatedAt'] as const
$columns = FetchRangeSchema.$columns
@@ -576,6 +587,19 @@ export class SfvEntrySchema extends BaseModel {
declare releaseId: bigint | number
}
export class SignatureProbeSchema extends BaseModel {
static $columns = ['groupName', 'probedAt', 'releaseId', 'tool'] as const
$columns = SignatureProbeSchema.$columns
@column()
declare groupName: string
@column.dateTime()
declare probedAt: DateTime
@column({ isPrimary: true })
declare releaseId: bigint | number
@column()
declare tool: string
}
export class SpotSchema extends BaseModel {
static $columns = ['articleNumber', 'category', 'createdAt', 'description', 'detailsAt', 'error', 'groupName', 'image', 'keyId', 'messageId', 'postedAt', 'poster', 'processedAt', 'releaseId', 'size', 'spotterId', 'status', 'subcats', 'tag', 'title', 'verified', 'website'] as const
$columns = SpotSchema.$columns
@@ -638,6 +662,23 @@ export class StagedSegmentSchema extends BaseModel {
declare stagedAt: DateTime
}
export class UploadSignatureSchema extends BaseModel {
static $columns = ['dropped', 'groupName', 'named', 'probed', 'tool', 'updatedAt'] as const
$columns = UploadSignatureSchema.$columns
@column()
declare dropped: boolean
@column()
declare groupName: string
@column()
declare named: number
@column()
declare probed: number
@column({ isPrimary: true })
declare tool: string
@column.dateTime({ autoCreate: true, autoUpdate: true })
declare updatedAt: DateTime
}
export class UserSchema extends BaseModel {
static $columns = ['createdAt', 'email', 'fullName', 'id', 'password', 'updatedAt'] as const
$columns = UserSchema.$columns