Drop garbage uploads early (D42)
Random-named uploads that never get a real name or password were ~90% of what seeding stores. Two filters: - before storing (seeding, ingest): random-named uploads of a dropped signature (posting tool from the Message-ID, first group) or of a release dropped after its probe aren't stored; 2% of a dropped signature's releases are still kept and probed. Stored files of dropped signatures aren't probed. - after a probe: a random-named release that is still unnamed, without password, names from files or spot, with all its probes done, and encrypted, a container or missing its first article, is deleted and tombstoned in dropped_releases. Signatures are learned by gc:scheduler from a sample of probed releases (signature_probes); garbage:learn --seed fills it from existing files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
35d0b2da65
commit
211820067a
13 files changed
+683
-6
No files matched your search
@@ -0,0 +1,64 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Dropping garbage uploads early (CONCEPT.md D42): random-named uploads that
|
||||
* never get a real name or password.
|
||||
* - upload_signatures: per posting tool (from the Message-ID) and first group,
|
||||
* how many probed random-named releases got a real name; `dropped` ones
|
||||
* aren't stored any more (GarbageFilter).
|
||||
* - signature_probes: a sample of probed random-named releases, what
|
||||
* upload_signatures is learned from (the outcome is read from the release
|
||||
* at learning time, so names that come later count too).
|
||||
* - dropped_releases: releases deleted after their probe, by release_key, so
|
||||
* seeding and ingest don't add them again.
|
||||
* - is_real_release_name(): a name someone would search for. is_readable_name
|
||||
* lets spam through (usenetszene net, VTS_01_1.VOB, Nieuwe map, 100 MiB):
|
||||
* this also wants a year, episode, resolution, codec or language tag, or
|
||||
* three words (LOG.md 2026-10-08 (5)). No `?` in its regexes: knex takes it
|
||||
* for a binding (hence {0,1}).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
await this.db.rawQuery(`
|
||||
CREATE TABLE IF NOT EXISTS upload_signatures (
|
||||
tool varchar(32) NOT NULL,
|
||||
group_name text NOT NULL,
|
||||
probed integer NOT NULL DEFAULT 0,
|
||||
named integer NOT NULL DEFAULT 0,
|
||||
dropped boolean NOT NULL DEFAULT false,
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (tool, group_name))`)
|
||||
await this.db.rawQuery(`
|
||||
CREATE TABLE IF NOT EXISTS signature_probes (
|
||||
release_id bigint PRIMARY KEY,
|
||||
tool varchar(32) NOT NULL,
|
||||
group_name text NOT NULL,
|
||||
probed_at timestamptz NOT NULL DEFAULT now())`)
|
||||
await this.db.rawQuery(`CREATE INDEX IF NOT EXISTS signature_probes_probed_at_idx ON signature_probes (probed_at)`)
|
||||
await this.db.rawQuery(`
|
||||
CREATE TABLE IF NOT EXISTS dropped_releases (
|
||||
release_key varchar(40) PRIMARY KEY,
|
||||
reason varchar(16) NOT NULL,
|
||||
dropped_at timestamptz NOT NULL DEFAULT now())`)
|
||||
await this.db.rawQuery(`
|
||||
CREATE OR REPLACE FUNCTION is_real_release_name(name text) RETURNS boolean IMMUTABLE LANGUAGE sql AS $$
|
||||
SELECT name IS NOT NULL AND is_readable_name(name)
|
||||
AND NOT (
|
||||
name ~* '(usenet|video_ts|nieuwe map|new folder|(^|[^a-z0-9])mib([^a-z0-9]|$))'
|
||||
OR name ~* '^(vts_|sample|cd[0-9])'
|
||||
OR name ~ '^\\(.*\\)$'
|
||||
OR name ~ '^[A-Za-z0-9]{8,}$')
|
||||
AND (
|
||||
name ~* '(^|[^a-z0-9])((19|20)[0-9]{2}|(480|576|720|1080|2160)[pi]|dl|nl)([^a-z0-9]|$)'
|
||||
OR name ~* '(s[0-9]{2}e[0-9]{2}|x26[45]|h\\.{0,1}26[45]|hevc|web[- .]{0,1}(dl|rip)|bluray|remux|german|dutch|multi|flac|mp3|epub|pdf|ac3|dts|xxx)'
|
||||
OR (SELECT count(*) FROM regexp_matches(name, '[A-Za-z]{3,}', 'g')) >= 3)
|
||||
$$`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
await this.db.rawQuery(`DROP FUNCTION IF EXISTS is_real_release_name(text)`)
|
||||
await this.db.rawQuery(`DROP TABLE IF EXISTS dropped_releases`)
|
||||
await this.db.rawQuery(`DROP TABLE IF EXISTS signature_probes`)
|
||||
await this.db.rawQuery(`DROP TABLE IF EXISTS upload_signatures`)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user