Drop garbage uploads early (D42)
Random-named uploads that never get a real name or password were ~90% of what seeding stores. Two filters: - before storing (seeding, ingest): random-named uploads of a dropped signature (posting tool from the Message-ID, first group) or of a release dropped after its probe aren't stored; 2% of a dropped signature's releases are still kept and probed. Stored files of dropped signatures aren't probed. - after a probe: a random-named release that is still unnamed, without password, names from files or spot, with all its probes done, and encrypted, a container or missing its first article, is deleted and tombstoned in dropped_releases. Signatures are learned by gc:scheduler from a sample of probed releases (signature_probes); garbage:learn --seed fills it from existing files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
35d0b2da65
commit
211820067a
13 files changed
+683
-6
No files matched your search
@@ -95,6 +95,25 @@ export default {
|
||||
maxHddGb: env.get('BACKFILL_MAX_HDD_GB', 150),
|
||||
},
|
||||
|
||||
/**
|
||||
* Dropping garbage uploads early (CONCEPT.md D42). A signature (posting tool, first
|
||||
* group) is dropped once at least minProbes of its probed random-named releases are
|
||||
* known and at most maxNamedRate of them got a real name or password; its
|
||||
* random-named files are then not stored, except explorePercent of its releases.
|
||||
*/
|
||||
garbage: {
|
||||
/** Off: nothing is filtered or deleted (the learning still runs). */
|
||||
enabled: env.get('GARBAGE_FILTER', true),
|
||||
explorePercent: env.get('GARBAGE_EXPLORE_PERCENT', 2),
|
||||
minProbes: 100,
|
||||
maxNamedRate: 0.005,
|
||||
/** Probed releases learned from: 1 in this many (by release id), for the last windowDays. */
|
||||
probeSample: 4,
|
||||
windowDays: 30,
|
||||
/** How often gc:scheduler learns the signatures again. */
|
||||
learnIntervalMs: env.get('GARBAGE_LEARN_INTERVAL_MS', 3 * 60 * 60 * 1000),
|
||||
},
|
||||
|
||||
/** worker:deferred leaves a file alone until it has had no new articles for this long. */
|
||||
deferredSettleMs: env.get('DEFERRED_SETTLE_MS', 15 * 60 * 1000),
|
||||
/** Files of at most this many parts are downloaded completely (small archives, NZBs, par2 index files). */
|
||||
|
||||
Reference in new issue
Block a user