Albums in the Rust edge, and the batch-counter rule (design 3)
A release the window closed on with one file named by its stem or post, the name with a number and no archive, volume, sidecar or random name, waits in the album of its poster and template (the name's digit runs as #, the extension apart) posted within the window of the album's first file; the album closes a window after that span and is written as one release (source album, no file total) named by the members' shared numbers, a lone member as the release it was. The batch store writes an album's member names as release_texts kind 6 (member). A counter total of 100 or more under which one poster started 20 or more releases is a batch counter: those releases get no file total. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed
1 parent
219fe4fd4c
commit
4d0610ca06
7 files changed
+382
-24
No files matched your search
+21
-3
@@ -448,6 +448,23 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
|
||||
poster. A counter total shared by many stems of one poster within the
|
||||
window is a batch counter, not a file count (Yenc-PP-A&A's [n/68067]):
|
||||
`file_total` null for them, not 68,067 incomplete singles.
|
||||
**Built 2026-10-26** (edge/src/indexer.rs, LOG.md (113)): a release the
|
||||
window closed on with one file named by its stem or post, the name with
|
||||
a number and no archive, volume, sidecar or random name, waits in the
|
||||
album of its poster and template (the name's digit runs as `#`, the
|
||||
extension apart: `#_#.jpg.data`) posted within WINDOW_MS of the album's
|
||||
first file, at most ALBUM_MAX_FILES (1,000) a album; the album closes a
|
||||
window after that span and is written as one release (`source: album`,
|
||||
no file total) named by the members' shared numbers (`1481439079025_#
|
||||
.jpg.data`), a lone member as the release it was. The batch store writes
|
||||
the members' names as `release_texts` kind 6 (`member`, the first 20).
|
||||
The directory-like prefix variant is not built (no case seen in the
|
||||
corpus samples). The batch counter: a total ≥ 100 under which one poster
|
||||
started 20+ releases within 100 batches gives none of them a file
|
||||
total. On six a.b.ath ranges of 2016: 46 albums of 790 files among
|
||||
2,099 chunks, most of them a subject parser miss (`[n/60] Shogun I
|
||||
(1980) Shogun I (1980).avi.012`: the counter text taken into the name,
|
||||
a volume each) that the album rule glues back together.
|
||||
|
||||
4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` →
|
||||
`posters(id)` (the table gets an id; 33 B → 4), `group_set_id` →
|
||||
@@ -665,9 +682,10 @@ until the window is done (LOG.md 2026-10-26 (111) has the first local
|
||||
numbers, (112) the homelab run).
|
||||
|
||||
Next: the replay on homelab over the 2016 window (the NVMe bytes per
|
||||
header for real), albums in the edge (design 3), the ordered range leases
|
||||
with the carried window in the edge-node and its snapshot (design 1 and
|
||||
5), the prober writing `release_texts` kind 2. Gone from the list: block
|
||||
header for real), the ordered range leases with the carried window in
|
||||
the edge-node and its snapshot (design 1 and 5), the prober writing
|
||||
`release_texts` kind 2, random-named singles without a chunk (decision f:
|
||||
2.3 M of the first replay's 4.0 M singles are "Bob"'s 32-hex posts). Gone from the list: block
|
||||
leases, the planner and the block merger (design 1, amended).
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -8,7 +8,7 @@ import { decodeChunk } from '#lib/chunks/index'
|
||||
import { SIDECAR } from '#services/sidecar_folder'
|
||||
import { identityKey, releaseKey } from '#lib/release/keys'
|
||||
import { sourceCode } from '#lib/release/source'
|
||||
import { TEXT_KINDS, TEXT_MAX } from '#lib/release/text_kind'
|
||||
import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
|
||||
|
||||
/**
|
||||
* A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
|
||||
@@ -237,7 +237,7 @@ class BatchStore {
|
||||
]
|
||||
)
|
||||
)
|
||||
const texts: { release_id: number; text: string }[] = []
|
||||
const texts: { release_id: number; kind: number; text: string }[] = []
|
||||
for (const row of rows) {
|
||||
const w = byKey.get(row.release_key)
|
||||
if (!w) continue
|
||||
@@ -248,15 +248,21 @@ class BatchStore {
|
||||
out.finalized++
|
||||
finalized.add(w.id)
|
||||
}
|
||||
texts.push({ release_id: w.id, text: row.name.slice(0, TEXT_MAX) })
|
||||
texts.push({ release_id: w.id, kind: TEXT_KINDS.name, text: row.name.slice(0, TEXT_MAX) })
|
||||
// An album's members stay findable by their own names (the first TEXTS_PER_KIND of them).
|
||||
if (r.source === 'album' && r.files > 0) {
|
||||
for (const text of textsToStore(decodeChunk(r.payload).map((f) => f.name))) {
|
||||
texts.push({ release_id: w.id, kind: TEXT_KINDS.member, text })
|
||||
}
|
||||
}
|
||||
}
|
||||
// The name, searchable (release_texts; a name sent again is a no-op).
|
||||
// The name (and an album's members), searchable (release_texts; a text sent again is a no-op).
|
||||
if (texts.length > 0) {
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO release_texts (release_id, kind, text)
|
||||
SELECT v.release_id, ?, v.text FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, text text) WHERE v.text <> ''
|
||||
SELECT v.release_id, v.kind, v.text FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, kind int, text text) WHERE v.text <> ''
|
||||
ON CONFLICT DO NOTHING`,
|
||||
[TEXT_KINDS.name, JSON.stringify(texts)]
|
||||
[JSON.stringify(texts)]
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
+302
-14
@@ -479,6 +479,16 @@ pub const FRONTIER_ARTICLES: usize = 1024;
|
||||
pub const PLAUSIBLE_MS: i64 = 7 * 24 * 60 * 60 * 1000;
|
||||
/// The frontier is established once this many dated articles were seen (a lone bogus date doesn't set it).
|
||||
const FRONTIER_MIN: usize = 64;
|
||||
/// Albums (REBUILD.md, "Voyager 3: window indexing", 3): a poster's single-file releases whose file names differ only
|
||||
/// by number sequences, posted within WINDOW_MS of the album's first, written as one release (`source: "album"`); an
|
||||
/// album holds at most this many files (a dump of 38k pictures a day is a few albums, not one).
|
||||
pub const ALBUM_MAX_FILES: usize = 1000;
|
||||
/// A counter total this large, shared by this many releases of one poster, is a batch counter (Yenc-PP-A&A's
|
||||
/// [n/68067] over thousands of different programs), not a file count: those releases get no file total.
|
||||
pub const BATCH_COUNTER_MIN_TOTAL: i64 = 100;
|
||||
pub const BATCH_COUNTER_RELEASES: u32 = 20;
|
||||
/// Batches after which a poster's counter total is forgotten (BATCH_COUNTER_RELEASES must come within them).
|
||||
const BATCH_COUNTER_BATCHES: u64 = 100;
|
||||
|
||||
fn median(dates: &mut [i64]) -> i64 {
|
||||
let mid = dates.len() / 2;
|
||||
@@ -714,6 +724,98 @@ impl OpenRelease {
|
||||
}
|
||||
}
|
||||
|
||||
/// An album still open: the closed singles of one poster and name template (ALBUM_MAX_FILES at most), with their
|
||||
/// release keys, so a lone member is written as the release it was.
|
||||
struct OpenAlbum {
|
||||
upload: usize,
|
||||
members: Vec<(String, OpenRelease)>,
|
||||
first: Option<i64>,
|
||||
last: Option<i64>,
|
||||
touched: u64,
|
||||
}
|
||||
|
||||
/// The template a file name joins an album by: every digit run as "#" ("1481439079025_014.jpg.data" →
|
||||
/// "#_#.jpg.data"); none for a name without a number, a random one, an archive, a volume or a sidecar.
|
||||
pub fn album_template(name: &str) -> Option<String> {
|
||||
if is_random_file(name) || names::is_par2(name) || names::is_archive_start(name) || names::is_later_volume(name) || names::is_sidecar(name) {
|
||||
return None;
|
||||
}
|
||||
let (stem, ext) = split_extension(name);
|
||||
let mut out = String::with_capacity(name.len());
|
||||
let mut runs = 0;
|
||||
let mut in_digits = false;
|
||||
for c in stem.chars() {
|
||||
if c.is_ascii_digit() {
|
||||
if !in_digits {
|
||||
out.push('#');
|
||||
runs += 1;
|
||||
in_digits = true;
|
||||
}
|
||||
} else {
|
||||
in_digits = false;
|
||||
out.push(c);
|
||||
}
|
||||
}
|
||||
out.push_str(ext);
|
||||
(runs > 0).then_some(out)
|
||||
}
|
||||
|
||||
/// A name's stem and its extension (a last ".xxx" of 1–5 letters and digits, "mp4"'s 4 no number sequence).
|
||||
fn split_extension(name: &str) -> (&str, &str) {
|
||||
match name.rfind('.') {
|
||||
Some(dot) if dot > 0 && (1..=5).contains(&(name.len() - dot - 1)) && name[dot + 1..].bytes().all(|c| c.is_ascii_alphanumeric()) => (&name[..dot], &name[dot..]),
|
||||
_ => (name, ""),
|
||||
}
|
||||
}
|
||||
|
||||
/// A name's runs of digits and of other characters, in order.
|
||||
fn digit_runs(name: &str) -> Vec<(bool, &str)> {
|
||||
let mut out = Vec::new();
|
||||
let mut start = 0;
|
||||
let b = name.as_bytes();
|
||||
let mut i = 0;
|
||||
while i < b.len() {
|
||||
let digits = b[i].is_ascii_digit();
|
||||
let mut j = i + 1;
|
||||
while j < b.len() && b[j].is_ascii_digit() == digits {
|
||||
j += 1;
|
||||
}
|
||||
out.push((digits, &name[start..j]));
|
||||
start = j;
|
||||
i = j;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// An album's name: its members' shared name, with the numbers that vary among them as "#" and the ones they share
|
||||
/// kept ("1481439079025_#.jpg.data" for one timestamp's pictures).
|
||||
pub fn album_name(names: &[&str]) -> String {
|
||||
let (stem, ext) = split_extension(names[0]);
|
||||
let first = digit_runs(stem);
|
||||
let mut varies = vec![false; first.len()];
|
||||
for name in &names[1..] {
|
||||
let runs = digit_runs(split_extension(name).0);
|
||||
if runs.len() != first.len() {
|
||||
return album_template(names[0]).unwrap_or_else(|| names[0].to_string());
|
||||
}
|
||||
for (i, (digits, text)) in runs.iter().enumerate() {
|
||||
if *digits && first[i].1 != *text {
|
||||
varies[i] = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut out = String::with_capacity(names[0].len());
|
||||
for (i, (digits, text)) in first.iter().enumerate() {
|
||||
if *digits && varies[i] {
|
||||
out.push('#');
|
||||
} else {
|
||||
out.push_str(text);
|
||||
}
|
||||
}
|
||||
out.push_str(ext);
|
||||
out
|
||||
}
|
||||
|
||||
pub struct RangeIndexer {
|
||||
group: String,
|
||||
rules: Rules,
|
||||
@@ -725,8 +827,12 @@ pub struct RangeIndexer {
|
||||
frontier: Frontier,
|
||||
/// Per release key its open uploads, oldest first.
|
||||
open: FxHashMap<String, Vec<OpenRelease>>,
|
||||
/// Per album key its open albums, oldest first.
|
||||
albums: FxHashMap<String, Vec<OpenAlbum>>,
|
||||
/// The next upload number per key (distinct within a batch: the store's slot is the key and the upload).
|
||||
uploads: FxHashMap<String, usize>,
|
||||
/// Per (poster, counter total ≥ BATCH_COUNTER_MIN_TOTAL) the releases started under it and the batch of the last.
|
||||
batch_totals: FxHashMap<(String, i64), (u32, u64)>,
|
||||
batch: Batch,
|
||||
counts: FxHashMap<(&'static str, String), (u64, u64)>,
|
||||
count_order: Vec<(&'static str, String)>,
|
||||
@@ -749,7 +855,9 @@ impl RangeIndexer {
|
||||
aggregator: Aggregator::default(),
|
||||
frontier: Frontier::default(),
|
||||
open: FxHashMap::default(),
|
||||
albums: FxHashMap::default(),
|
||||
uploads: FxHashMap::default(),
|
||||
batch_totals: FxHashMap::default(),
|
||||
batch: Batch { from: resume_at, next: resume_at, ..Default::default() },
|
||||
counts: FxHashMap::default(),
|
||||
count_order: Vec::new(),
|
||||
@@ -834,7 +942,7 @@ impl RangeIndexer {
|
||||
|
||||
/// Releases the window still holds (edge-bench's progress line).
|
||||
pub fn open_releases(&self) -> usize {
|
||||
self.open.values().map(|u| u.len()).sum()
|
||||
self.open.values().map(|u| u.len()).sum::<usize>() + self.albums.values().map(|a| a.len()).sum::<usize>()
|
||||
}
|
||||
|
||||
/// Files still short of parts, or complete and not yet in a release.
|
||||
@@ -853,8 +961,10 @@ impl RangeIndexer {
|
||||
}
|
||||
self.pending_files = 0;
|
||||
// Upload numbers only have to be distinct within a batch: those of keys with nothing open go.
|
||||
let open = &self.open;
|
||||
self.uploads.retain(|key, _| open.contains_key(key));
|
||||
let (open, albums) = (&self.open, &self.albums);
|
||||
self.uploads.retain(|key, _| open.contains_key(key) || albums.contains_key(key));
|
||||
let batch = self.aggregator.batch;
|
||||
self.batch_totals.retain(|_, (_, touched)| batch - *touched < BATCH_COUNTER_BATCHES);
|
||||
out
|
||||
}
|
||||
|
||||
@@ -879,6 +989,11 @@ impl RangeIndexer {
|
||||
let n = self.uploads.entry(key).or_insert(0);
|
||||
let upload = *n;
|
||||
*n += 1;
|
||||
if let Some(total) = file[0].file.release.file_total_truthy().filter(|&t| t >= BATCH_COUNTER_MIN_TOTAL) {
|
||||
let e = self.batch_totals.entry((file[0].file.poster.clone(), total)).or_insert((0, batch));
|
||||
e.0 += 1;
|
||||
e.1 = batch;
|
||||
}
|
||||
uploads.push(OpenRelease { upload, files: Vec::new(), first: None, last: None, touched: batch });
|
||||
uploads.last_mut().unwrap()
|
||||
}
|
||||
@@ -926,29 +1041,127 @@ impl RangeIndexer {
|
||||
self.open.remove(&key);
|
||||
}
|
||||
}
|
||||
// An edge: still open at the block's end, or posted within the window of the block's start (its other
|
||||
// part may be in the neighbour block); undated ones can't be placed, so they are edges too.
|
||||
let near_start = |first: Option<i64>| match (start, first) {
|
||||
(Some(s), Some(f)) => f < s + WINDOW_MS,
|
||||
_ => true,
|
||||
};
|
||||
// In posting order, so an album's members join it from its first file on (the keys' order is a hash's).
|
||||
done.sort_by(|a, b| a.1.first.cmp(&b.1.first).then_with(|| a.0.cmp(&b.0)));
|
||||
for (key, u, closed) in done {
|
||||
// An edge: still open at the block's end, or posted within the window of the block's start (its other
|
||||
// part may be in the neighbour block); undated ones can't be placed, so they are edges too.
|
||||
let near_start = match (start, u.first) {
|
||||
(Some(s), Some(f)) => f < s + WINDOW_MS,
|
||||
_ => true,
|
||||
};
|
||||
self.emit(&key, u, closed, !closed || near_start);
|
||||
// A single the window closed on waits in its album (a lone one is written as it is when that closes).
|
||||
if closed {
|
||||
if let Some(template) = album_candidate(&u) {
|
||||
self.album(key, u, template);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
let edge = !closed || near_start(u.first);
|
||||
self.emit(&key, u, closed, edge);
|
||||
}
|
||||
// Albums: closed once every single that could join (posted within WINDOW_MS of the first) has closed too.
|
||||
let mut done_albums: Vec<(String, OpenAlbum, bool)> = Vec::new();
|
||||
let keys: Vec<String> = self.albums.keys().cloned().collect();
|
||||
for key in keys {
|
||||
let list = self.albums.get_mut(&key).unwrap();
|
||||
let mut i = 0;
|
||||
while i < list.len() {
|
||||
let a = &list[i];
|
||||
let closed = match a.first {
|
||||
Some(f) => before.is_some_and(|b| f + WINDOW_MS < b),
|
||||
None => idle > 0 && batch - a.touched >= idle,
|
||||
};
|
||||
if !all && !closed {
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
done_albums.push((key.clone(), list.remove(i), closed));
|
||||
}
|
||||
if list.is_empty() {
|
||||
self.albums.remove(&key);
|
||||
}
|
||||
}
|
||||
for (key, mut a, closed) in done_albums {
|
||||
let edge = !closed || near_start(a.first);
|
||||
if a.members.len() < 2 {
|
||||
let (release_key, u) = a.members.remove(0);
|
||||
self.emit(&release_key, u, closed, edge);
|
||||
continue;
|
||||
}
|
||||
let names: Vec<&str> = a.members.iter().map(|(_, u)| u.files[0][0].file.name.as_str()).collect();
|
||||
let name = album_name(&names);
|
||||
let files: Vec<Vec<Seg>> = a.members.into_iter().flat_map(|(_, u)| u.files).collect();
|
||||
self.write(&key, a.upload, files, closed, edge, Some(name));
|
||||
}
|
||||
}
|
||||
|
||||
/// A closed single into the open album of its poster and template posted within WINDOW_MS of its first file
|
||||
/// (with room), else a new one.
|
||||
fn album(&mut self, release_key: String, u: OpenRelease, template: String) {
|
||||
let batch = self.aggregator.batch;
|
||||
let poster = &u.files[0][0].file.poster;
|
||||
let poster = if names::random_poster(poster) { "" } else { poster.as_str() };
|
||||
let key = sha1_hex(&format!("album\n{poster}\n{template}"));
|
||||
let albums = self.albums.entry(key.clone()).or_default();
|
||||
let fits = |a: &OpenAlbum| {
|
||||
a.members.len() < ALBUM_MAX_FILES
|
||||
&& match (a.first, u.first) {
|
||||
(Some(af), Some(f)) => f >= af && f < af + WINDOW_MS,
|
||||
_ => true,
|
||||
}
|
||||
};
|
||||
let index = match albums.iter().position(fits) {
|
||||
Some(i) => i,
|
||||
None => {
|
||||
let n = self.uploads.entry(key).or_insert(0);
|
||||
let upload = *n;
|
||||
*n += 1;
|
||||
albums.push(OpenAlbum { upload, members: Vec::new(), first: None, last: None, touched: batch });
|
||||
albums.len() - 1
|
||||
}
|
||||
};
|
||||
let a = &mut albums[index];
|
||||
if let Some(f) = u.first {
|
||||
if a.first.is_none_or(|r| f < r) {
|
||||
a.first = Some(f);
|
||||
}
|
||||
}
|
||||
if let Some(l) = u.last {
|
||||
if a.last.is_none_or(|r| l > r) {
|
||||
a.last = Some(l);
|
||||
}
|
||||
}
|
||||
a.touched = batch;
|
||||
a.members.push((release_key, u));
|
||||
}
|
||||
|
||||
fn emit(&mut self, key: &str, release: OpenRelease, finished: bool, edge: bool) {
|
||||
let list = release.files;
|
||||
self.write(key, release.upload, release.files, finished, edge, None);
|
||||
}
|
||||
|
||||
/// A release (or an album, under its name) judged and written to the batch.
|
||||
fn write(&mut self, key: &str, upload: usize, list: Vec<Vec<Seg>>, finished: bool, edge: bool, album: Option<String>) {
|
||||
let judged: Vec<JudgedFile> = list
|
||||
.iter()
|
||||
.map(|f| JudgedFile { name: &f[0].file.name, total_parts: f[0].file.total_parts, groups: &f[0].file.groups, first_message_id: &f[0].message_id, poster: &f[0].file.poster })
|
||||
.collect();
|
||||
match names::judge(key, &judged, &self.rules) {
|
||||
Verdict::Keep => {
|
||||
let candidates = candidates_of(key, &list, release.upload);
|
||||
let mut chunk = chunk_of(key, &list, release.upload);
|
||||
let candidates = candidates_of(key, &list, upload);
|
||||
let mut chunk = chunk_of(key, &list, upload);
|
||||
chunk.finished = finished;
|
||||
chunk.edge = edge;
|
||||
if let Some(name) = album {
|
||||
chunk.name = name;
|
||||
chunk.source = "album";
|
||||
chunk.file_total = None;
|
||||
} else if let Some(total) = chunk.file_total.filter(|&t| t >= BATCH_COUNTER_MIN_TOTAL && (chunk.files as i64) < t) {
|
||||
// A batch counter: no file total (the release is as complete as its files are).
|
||||
if self.batch_totals.get(&(chunk.poster.clone(), total)).is_some_and(|(n, _)| *n >= BATCH_COUNTER_RELEASES) {
|
||||
chunk.file_total = None;
|
||||
}
|
||||
}
|
||||
self.pending_files += chunk.files;
|
||||
self.batch.releases.push(chunk);
|
||||
self.batch.candidates.extend(candidates);
|
||||
@@ -957,7 +1170,7 @@ impl RangeIndexer {
|
||||
let segments: usize = list.iter().map(|f| f.len()).sum();
|
||||
self.count(rule, rule_key.clone(), segments as u64, list.len() as u64);
|
||||
if sample {
|
||||
self.batch.samples.push((chunk_of(key, &list, release.upload), rule, rule_key));
|
||||
self.batch.samples.push((chunk_of(key, &list, upload), rule, rule_key));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -977,6 +1190,19 @@ impl RangeIndexer {
|
||||
}
|
||||
}
|
||||
|
||||
/// A closed release that may be an album's member: one file, named by its stem or its post (no release name in the
|
||||
/// subject), with a number in its name (album_template).
|
||||
fn album_candidate(u: &OpenRelease) -> Option<String> {
|
||||
if u.files.len() != 1 {
|
||||
return None;
|
||||
}
|
||||
let meta = &u.files[0][0].file;
|
||||
if !matches!(meta.release.source, "stem" | "post") {
|
||||
return None;
|
||||
}
|
||||
album_template(&meta.name)
|
||||
}
|
||||
|
||||
fn release_key_of(s: &Seg) -> String {
|
||||
// A named file always has a release (releaseIdentity isn't null unless weak).
|
||||
s.file.release.key.clone()
|
||||
@@ -1154,6 +1380,68 @@ mod tests {
|
||||
assert_eq!(b4.next, 1_000_001);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn singles_of_one_template_become_an_album() {
|
||||
let mut priorities = FxHashMap::default();
|
||||
priorities.insert(GROUP.to_string(), 1);
|
||||
let mut ix = RangeIndexer::new(GROUP, 1, 1_000_000, 1, Rules::defaults(priorities), 5);
|
||||
let day = 24 * 60;
|
||||
// A dump's pictures of one timestamp (a stem each: ".data" is no image), two scans, a lone document, and dated fillers.
|
||||
let mut lines: Vec<String> = (1..=14).map(|i| line(1000 + i, &format!(r#"<IMGBANK> [1/1] "1481439079025_{i:03}.jpg.data" yEnc (1/1)"#), i)).collect();
|
||||
lines.push(line(2001, r#"[1/1] "Scan_001.pdf" yEnc (1/1)"#, 20));
|
||||
lines.push(line(2002, r#"[1/1] "Scan_002.pdf" yEnc (1/1)"#, 21));
|
||||
lines.push(line(2003, r#"[1/1] "Lonely_007.pdf" yEnc (1/1)"#, 22));
|
||||
lines.extend(fillers(3000, 100, 23, 1));
|
||||
ix.add_text(&lines.join("\n"), 4000);
|
||||
let b1 = ix.take();
|
||||
assert!(b1.releases.iter().all(|r| r.name.starts_with("Filler")));
|
||||
// Three days on: the window closed on the singles, and on their albums.
|
||||
ix.add_text(&fillers(5000, 300, 3 * day, 1).join("\n"), 6000);
|
||||
let b2 = ix.take();
|
||||
let mut albums: Vec<(&str, &str, usize, bool)> = b2.releases.iter().filter(|r| !r.name.starts_with("Filler")).map(|r| (r.name.as_str(), r.source, r.files, r.finished)).collect();
|
||||
albums.sort();
|
||||
assert_eq!(albums, vec![("1481439079025_#.jpg.data", "album", 14, true), ("Lonely_007", "stem", 1, true), ("Scan_#.pdf", "album", 2, true)]);
|
||||
let album = b2.releases.iter().find(|r| r.source == "album" && r.files == 14).unwrap();
|
||||
// Posted within the window of the block's start: an edge, like any release there.
|
||||
assert!(album.file_total.is_none() && album.edge && !album.partial && album.readable);
|
||||
assert_eq!(album.segments, 14);
|
||||
assert_eq!(b2.candidates.len(), 0);
|
||||
assert!(ix.albums.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn album_templates_and_names() {
|
||||
assert_eq!(album_template("1481439079025_014.jpg.data").as_deref(), Some("#_#.jpg.data"));
|
||||
assert_eq!(album_template("ActiCalc 1.06.rar"), None);
|
||||
assert_eq!(album_template("Scan_001.vol00+01.par2"), None);
|
||||
assert_eq!(album_template("Scan_001.nfo"), None);
|
||||
assert_eq!(album_template("NoNumber.mp4"), None);
|
||||
assert_eq!(album_template("c3453cdd2a3645a8f3cf3d22319abe63"), None);
|
||||
assert_eq!(album_name(&["1481439079025_001.jpg.data", "1481439079025_014.jpg.data"]), "1481439079025_#.jpg.data");
|
||||
assert_eq!(album_name(&["1481439079025_001.jpg.data", "1481439080000_001.jpg.data"]), "#_001.jpg.data");
|
||||
assert_eq!(album_name(&["Scan_001.pdf", "Scan_002.pdf", "Scan_003.pdf"]), "Scan_#.pdf");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_batch_counter_is_no_file_total() {
|
||||
let mut priorities = FxHashMap::default();
|
||||
priorities.insert(GROUP.to_string(), 1);
|
||||
let mut ix = RangeIndexer::new(GROUP, 1, 1_000_000, 1, Rules::defaults(priorities), 5);
|
||||
let day = 24 * 60;
|
||||
// 25 different programs of one poster under one running counter, and one upload with a counter of its own.
|
||||
let mut lines: Vec<String> = (0..25).map(|i| line(100 + i, &format!(r#"[{}/68067] - "Program.Number.{i}.v1.0.rar" yEnc (1/1)"#, 1000 + i), i)).collect();
|
||||
lines.push(line(200, r#"[1/120] - "Real.Upload.part001.rar" yEnc (1/1)"#, 30));
|
||||
lines.extend(fillers(300, 100, 31, 1));
|
||||
ix.add_text(&lines.join("\n"), 1000);
|
||||
ix.add_text(&fillers(2000, 300, 3 * day, 1).join("\n"), 3000);
|
||||
let batch = ix.take();
|
||||
let programs: Vec<&ReleaseChunk> = batch.releases.iter().filter(|r| r.name.starts_with("Program")).collect();
|
||||
assert_eq!(programs.len(), 25);
|
||||
assert!(programs.iter().all(|r| r.file_total.is_none() && r.files == 1));
|
||||
let real = batch.releases.iter().find(|r| r.name.starts_with("Real")).unwrap();
|
||||
assert_eq!(real.file_total, Some(120));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_frontier_ignores_implausible_dates_and_follows_a_pause() {
|
||||
let hour = 3_600_000i64;
|
||||
|
||||
@@ -1237,6 +1237,11 @@ pub fn explored(key: &str, percent: f64) -> bool {
|
||||
}
|
||||
|
||||
static SIDECAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(par2|nfo|sfv|txt|jpe?g|png|nzb)$"));
|
||||
|
||||
/// A sidecar by its extension (the judge's SIDECAR): no release of its own, and no album's member.
|
||||
pub fn is_sidecar(name: &str) -> bool {
|
||||
test(&SIDECAR, name)
|
||||
}
|
||||
static EXECUTABLE: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(exe|scr|com|pif|bat|cmd|vbs|lnk)SP*$"));
|
||||
|
||||
pub struct JudgedFile<'a> {
|
||||
|
||||
@@ -15,6 +15,8 @@ export const TEXT_KINDS = {
|
||||
par2: 4,
|
||||
/** A title from a media container's metadata. */
|
||||
media: 5,
|
||||
/** An album's member, by its own file name (the edge's albums: one poster's files differing by a number sequence). */
|
||||
member: 6,
|
||||
} as const
|
||||
|
||||
export type TextKind = keyof typeof TEXT_KINDS
|
||||
|
||||
@@ -487,6 +487,45 @@ test.group('BatchStore', (group) => {
|
||||
assert.notEqual(rows[0].release_key, rows[1].release_key)
|
||||
})
|
||||
|
||||
test("an album's members stay findable by their own names (release_texts kind member)", async ({
|
||||
assert,
|
||||
}) => {
|
||||
const { rangeId, nodeId } = await rangeIn(GROUP, 1)
|
||||
// Two scans the TypeScript indexer puts under their subject; the Rust edge's album of them is the same chunk
|
||||
// with its name and source.
|
||||
const batch = batchOf(GROUP, 1, [
|
||||
...upload(10, 'Scan_001.dat', 1, 'Some Scans [1/2]', '2026-10-05T10:00:00Z'),
|
||||
...upload(20, 'Scan_002.dat', 1, 'Some Scans [2/2]', '2026-10-05T10:01:00Z'),
|
||||
])
|
||||
assert.lengthOf(batch.releases, 1)
|
||||
const album: ReleaseChunk = {
|
||||
...batch.releases[0],
|
||||
name: 'Scan_#.dat',
|
||||
source: 'album',
|
||||
fileTotal: null,
|
||||
}
|
||||
const stored = await BatchStore.store(rangeId, nodeId, { ...batch, releases: [album] })
|
||||
assert.include(stored, { releases: 1, newReleases: 1 })
|
||||
const release = await db.from('releases').firstOrFail()
|
||||
assert.include(release, {
|
||||
name: 'Scan_#.dat',
|
||||
source: 6,
|
||||
files: 2,
|
||||
file_total: null,
|
||||
complete: null,
|
||||
})
|
||||
assert.deepEqual(
|
||||
(
|
||||
await db.from('release_texts').where('release_id', release.id).orderBy(['kind', 'text'])
|
||||
).map((t) => [t.kind, t.text]),
|
||||
[
|
||||
[1, 'Scan_#.dat'],
|
||||
[6, 'Scan_001.dat'],
|
||||
[6, 'Scan_002.dat'],
|
||||
]
|
||||
)
|
||||
})
|
||||
|
||||
test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
|
||||
await storeIn(GROUP, 1_000_000, [
|
||||
...upload(1_000_000, 'Murdoch Mysteries S8D1.nzb', 1, '[00/80]', '2026-10-05T10:00:00Z'),
|
||||
|
||||
@@ -10,7 +10,7 @@ import {
|
||||
|
||||
test.group('release text kinds', () => {
|
||||
test('names and codes map both ways; the codes are fixed', ({ assert }) => {
|
||||
assert.deepEqual(TEXT_KINDS, { name: 1, file: 2, nfo: 3, par2: 4, media: 5 })
|
||||
assert.deepEqual(TEXT_KINDS, { name: 1, file: 2, nfo: 3, par2: 4, media: 5, member: 6 })
|
||||
for (const [name, code] of Object.entries(TEXT_KINDS)) {
|
||||
assert.equal(textKindCode(name), code)
|
||||
assert.equal(textKindName(code), name)
|
||||
|
||||
Reference in new issue
Block a user