Lone articles under scrambled names are per-article; inactive groups get no corpus fill
One article of a multi-part file under a name like "nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr" or "bslXlwgPiqZadhRa" (a random subject and poster per article, with fake counters) was a release: looksRandom misses these names for their separators. 160k of alt.binaries.storage's 162k non-Bob releases, and about 60% of all single-article releases in a 2% sample. looks_scrambled / isScrambledFile: at least half the stem's letters and digits in tokens of 5+ that switch case or letter/digit 3 times or more (93% of the storage spam). Used for lone articles given up before the block's end only; at the end the strict looksRandom stays, as the rest of such a file can be in the next block. The namer's re-index (partialFiles) does the same. The prefetcher's corpus fill skips groups that aren't active. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5132ab9083
commit
cfce8884d4
7 files changed
+81
-12
No files matched your search
@@ -105,6 +105,20 @@ test.group('edge / RangeIndexer', () => {
|
||||
assert.deepEqual(batch.fragments, [])
|
||||
assert.deepEqual(batch.counts.map((c) => [c.rule, c.articles, c.files]), [['per-article', 1, 0]])
|
||||
})
|
||||
|
||||
test('partialFiles: one article of a multi-part file under a scrambled name is per-article before the end, a partial release at it', ({ assert }) => {
|
||||
const indexer = new RangeIndexer({ group: GROUP, start: 100, end: 1_000_000, rules, idleChunks: 1, edgeArticles: 0, partialFiles: true })
|
||||
// Part 8 of 67, a random poster's: given up mid-range.
|
||||
indexer.add(post(100, 'nYDeXeF49dieZ1J7flK-JFzZ3j.ZEDsr', 67, { poster: 'guAym5TqSlp SCTLKQ0R <8nGc3Rk@OrjrYqFkp.z>' }).slice(7, 8), 200)
|
||||
indexer.add([], 300)
|
||||
indexer.add([], 400)
|
||||
// The same shape still open when the block ends: the rest can be in the next one.
|
||||
indexer.add(post(999_990, 'bslXlwgPiqZadhRa', 5).slice(1, 2), 1_000_001)
|
||||
indexer.finish()
|
||||
const batch = indexer.take()
|
||||
assert.deepEqual(batch.releases.map((r) => [r.name, r.partial]), [['bslXlwgPiqZadhRa', true]])
|
||||
assert.deepEqual(batch.counts.map((c) => [c.rule, c.articles, c.files]), [['per-article', 1, 0]])
|
||||
})
|
||||
})
|
||||
|
||||
test.group('edge / taking over a range', () => {
|
||||
|
||||
Reference in new issue
Block a user