Drop garbage uploads early (D42)
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 2m33s

Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:

- before storing (seeding, ingest): random-named uploads of a dropped
  signature (posting tool from the Message-ID, first group) or of a
  release dropped after its probe aren't stored; 2% of a dropped
  signature's releases are still kept and probed. Stored files of dropped
  signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
  password, names from files or spot, with all its probes done, and
  encrypted, a container or missing its first article, is deleted and
  tombstoned in dropped_releases.

Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-08 11:21:01 +02:00
1 parent 35d0b2da65
commit 211820067a
13 files changed
+683 -6

No files matched your search

+19
View File
@@ -95,6 +95,25 @@ export default {
maxHddGb: env.get('BACKFILL_MAX_HDD_GB', 150),
},
/**
* Dropping garbage uploads early (CONCEPT.md D42). A signature (posting tool, first
* group) is dropped once at least minProbes of its probed random-named releases are
* known and at most maxNamedRate of them got a real name or password; its
* random-named files are then not stored, except explorePercent of its releases.
*/
garbage: {
/** Off: nothing is filtered or deleted (the learning still runs). */
enabled: env.get('GARBAGE_FILTER', true),
explorePercent: env.get('GARBAGE_EXPLORE_PERCENT', 2),
minProbes: 100,
maxNamedRate: 0.005,
/** Probed releases learned from: 1 in this many (by release id), for the last windowDays. */
probeSample: 4,
windowDays: 30,
/** How often gc:scheduler learns the signatures again. */
learnIntervalMs: env.get('GARBAGE_LEARN_INTERVAL_MS', 3 * 60 * 60 * 1000),
},
/** worker:deferred leaves a file alone until it has had no new articles for this long. */
deferredSettleMs: env.get('DEFERRED_SETTLE_MS', 15 * 60 * 1000),
/** Files of at most this many parts are downloaded completely (small archives, NZBs, par2 index files). */