Lone articles under scrambled names are per-article; inactive groups get no corpus fill
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or "bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release: looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k non-Bob releases, and about 60% of all single-article releases in a 2% sample. looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+ that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone articles given up before the block's end only; at the end the strict looksRandom stays, as the rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same. The prefetcher's corpus fill skips groups that aren't active. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5132ab9083
commit
cfce8884d4
7 files changed
+81
-12
No files matched your search
+12
-8
@@ -5,7 +5,7 @@
|
||||
|
||||
use crate::codec::{encode_chunk, ChunkFile};
|
||||
use crate::js::{self, sha1_hex};
|
||||
use crate::names::{self, is_random_file, JudgedFile, Release, Rules, Verdict};
|
||||
use crate::names::{self, is_random_file, looks_scrambled, JudgedFile, Release, Rules, Verdict};
|
||||
use rustc_hash::{FxHashMap, FxHashSet};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::rc::Rc;
|
||||
@@ -1176,11 +1176,11 @@ impl RangeIndexer {
|
||||
let t = std::time::Instant::now();
|
||||
let done = self.aggregator.add(segments);
|
||||
unsafe { PROF[3] += t.elapsed().as_nanos() as u64; }
|
||||
self.files(done);
|
||||
self.files(done, false);
|
||||
self.frontier.update();
|
||||
let t = std::time::Instant::now();
|
||||
let gone = self.aggregator.evict(self.frontier.closed_before(), self.idle, false);
|
||||
self.files(gone);
|
||||
self.files(gone, false);
|
||||
self.close(false);
|
||||
unsafe { PROF[4] += t.elapsed().as_nanos() as u64; }
|
||||
self.batch.next = next;
|
||||
@@ -1191,7 +1191,7 @@ impl RangeIndexer {
|
||||
pub fn finish(&mut self) {
|
||||
self.frontier.settle();
|
||||
let gone = self.aggregator.evict(None, 0, true);
|
||||
self.files(gone);
|
||||
self.files(gone, true);
|
||||
self.close(true);
|
||||
self.batch.next = self.end + 1;
|
||||
}
|
||||
@@ -1243,12 +1243,16 @@ impl RangeIndexer {
|
||||
out
|
||||
}
|
||||
|
||||
/// Finished files (complete, or the window closed on them) into the open release of their key and upload.
|
||||
fn files(&mut self, files: Vec<Vec<Seg>>) {
|
||||
/// Finished files (complete, or the window closed on them) into the open release of their key and upload; `at_end`
|
||||
/// when the block ends on them.
|
||||
fn files(&mut self, files: Vec<Vec<Seg>>, at_end: bool) {
|
||||
let batch = self.aggregator.batch;
|
||||
for file in files {
|
||||
if file.len() == 1 && (file[0].number != 1 || file[0].file.total_parts > 1) && is_random_file(&file[0].file.name) {
|
||||
// One article of a multi-part random file (its upload per article: ngPost's, nyuu's): not a release.
|
||||
let lone = file.len() == 1 && (file[0].number != 1 || file[0].file.total_parts > 1);
|
||||
// A scrambled name only before the block's end, where the rest of a file can still be in the next block.
|
||||
if lone && (is_random_file(&file[0].file.name) || (!at_end && looks_scrambled(&file[0].file.name))) {
|
||||
// One article of a multi-part random file (its upload per article: ngPost's, nyuu's, and a random poster's
|
||||
// per article, 160k in alt.binaries.storage; LOG.md (122)): not a release.
|
||||
let tool = names::upload_tool(&file[0].message_id);
|
||||
self.count("per-article", tool.to_string(), 1, 0);
|
||||
continue;
|
||||
|
||||
Reference in new issue
Block a user