Drop garbage uploads early (D42)
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 2m33s

Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:

- before storing (seeding, ingest): random-named uploads of a dropped
  signature (posting tool from the Message-ID, first group) or of a
  release dropped after its probe aren't stored; 2% of a dropped
  signature's releases are still kept and probed. Stored files of dropped
  signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
  password, names from files or spot, with all its probes done, and
  encrypted, a container or missing its first article, is deleted and
  tombstoned in dropped_releases.

Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-08 11:21:01 +02:00
1 parent 35d0b2da65
commit 211820067a
13 files changed
+683 -6

No files matched your search

@@ -0,0 +1,64 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Dropping garbage uploads early (CONCEPT.md D42): random-named uploads that
* never get a real name or password.
* - upload_signatures: per posting tool (from the Message-ID) and first group,
* how many probed random-named releases got a real name; `dropped` ones
* aren't stored any more (GarbageFilter).
* - signature_probes: a sample of probed random-named releases, what
* upload_signatures is learned from (the outcome is read from the release
* at learning time, so names that come later count too).
* - dropped_releases: releases deleted after their probe, by release_key, so
* seeding and ingest don't add them again.
* - is_real_release_name(): a name someone would search for. is_readable_name
* lets spam through (usenetszene net, VTS_01_1.VOB, Nieuwe map, 100 MiB):
* this also wants a year, episode, resolution, codec or language tag, or
* three words (LOG.md 2026-10-08 (5)). No `?` in its regexes: knex takes it
* for a binding (hence {0,1}).
*/
export default class extends BaseSchema {
async up() {
await this.db.rawQuery(`
CREATE TABLE IF NOT EXISTS upload_signatures (
tool varchar(32) NOT NULL,
group_name text NOT NULL,
probed integer NOT NULL DEFAULT 0,
named integer NOT NULL DEFAULT 0,
dropped boolean NOT NULL DEFAULT false,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (tool, group_name))`)
await this.db.rawQuery(`
CREATE TABLE IF NOT EXISTS signature_probes (
release_id bigint PRIMARY KEY,
tool varchar(32) NOT NULL,
group_name text NOT NULL,
probed_at timestamptz NOT NULL DEFAULT now())`)
await this.db.rawQuery(`CREATE INDEX IF NOT EXISTS signature_probes_probed_at_idx ON signature_probes (probed_at)`)
await this.db.rawQuery(`
CREATE TABLE IF NOT EXISTS dropped_releases (
release_key varchar(40) PRIMARY KEY,
reason varchar(16) NOT NULL,
dropped_at timestamptz NOT NULL DEFAULT now())`)
await this.db.rawQuery(`
CREATE OR REPLACE FUNCTION is_real_release_name(name text) RETURNS boolean IMMUTABLE LANGUAGE sql AS $$
SELECT name IS NOT NULL AND is_readable_name(name)
AND NOT (
name ~* '(usenet|video_ts|nieuwe map|new folder|(^|[^a-z0-9])mib([^a-z0-9]|$))'
OR name ~* '^(vts_|sample|cd[0-9])'
OR name ~ '^\\(.*\\)$'
OR name ~ '^[A-Za-z0-9]{8,}$')
AND (
name ~* '(^|[^a-z0-9])((19|20)[0-9]{2}|(480|576|720|1080|2160)[pi]|dl|nl)([^a-z0-9]|$)'
OR name ~* '(s[0-9]{2}e[0-9]{2}|x26[45]|h\\.{0,1}26[45]|hevc|web[- .]{0,1}(dl|rip)|bluray|remux|german|dutch|multi|flac|mp3|epub|pdf|ac3|dts|xxx)'
OR (SELECT count(*) FROM regexp_matches(name, '[A-Za-z]{3,}', 'g')) >= 3)
$$`)
}
async down() {
await this.db.rawQuery(`DROP FUNCTION IF EXISTS is_real_release_name(text)`)
await this.db.rawQuery(`DROP TABLE IF EXISTS dropped_releases`)
await this.db.rawQuery(`DROP TABLE IF EXISTS signature_probes`)
await this.db.rawQuery(`DROP TABLE IF EXISTS upload_signatures`)
}
}