The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).
- lib/release/classes.ts, names::pending_class, SQL pending_class():
a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
target's article; candidates carry bytes now), pending_classes with
each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
the unexplored, at most a quarter of a claim from one), the cheapest
target first within one, then the oldest of the rest; records classes
like signatures and drops them at the same thresholds; discards their
waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
dropped class out of the pending store (counted as garbage), the
explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
catalog release's names a pending release the probe didn't, and sets
the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
the finished ones; rules:classes publishes the dropped classes;
releases:readable also marks the rows' candidates unreadable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Searches with the readable filter on returned scrambles: with a random
4-letter query nearly every hit was one ("Zx5AjNaK-mh7CFg6Hf8uu",
"Description - Lt5nHjlo3RiuBb3C", "PsNdnYiugfZ", "bd5b...-addon"), and
15% of a 6,000-row sample of the catalog's readable names were. looksRandom
knows single-token shapes only; separators, spaces, a random "extension"
of any length and a packaging word hid the rest.
isUnreadableName (lib/release/readable.ts) and names::is_unreadable_name
read a name token by token: words (one case or Capitalized), scrambles
(letter/digit and case switches, lone letters between capitals, inner
capital runs, few vowels, hex) and neutral tokens (digits, codes like
S02E10 and 1080p, CamelCase, packaging words); a name is unreadable when
its scrambles outweigh its words. Scene tags' "i" for "I" and the
NZBSPLIT hash are read as such. On the evaluation sets: 554 of 559
random-search names caught, 0 of 630 real names flagged (LOG.md (129)).
The edge's readable flag and the prober's inner names use it; the drop
rules keep looksRandom, which is narrower. An unnamed release goes to the
pending store (decision f), so more of the obfuscated groups' releases
wait for a probe now. releases:readable clears the flag on the existing
catalog's rows by name (a dry run by default).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A poster's counter without its "[" ("… Facialized 002/477] - 0177.jpg")
left "… Facialized ]" as the release name (11286967). displayName (TS) and
display_name (Rust) drop unmatched brackets from subject and post names
only; the identity key keeps the remainder as it was, so no release splits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each file of these uploads was a release of its own:
- "BD_NoTHing018.s41", "…CyjI.s00": rar volumes after .r99 (as the file's
own lower-case extension only, so "show.s01.rar" keeps its season);
- "chappelle1.part02.rev": rar recovery volumes;
- "X.part1.par2", "X.part1.vol00+34.par2" beside "X.part1.rar";
- "NZPEC….180.bc": BCArchive volumes;
- "[27/70] - 2407.jpg - 28": the size in kB after an unquoted name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nntp (XOVER client, Xref, GROUP), lib/headers (subjects, file keys,
segments from overview lines; the yEnc path stays behind for phase 2),
lib/release (identity, random names, now in lib), lib/files (aggregator,
volume names), lib/rules (posting tools, owner by priority, release
verdicts), lib/chunks (release chunk codec). Their unit tests came along;
new ones for codec, rules and volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
