7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.
An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lone candidates are often par2 volumes of random-named sets
("42507096….vol007+08.par2"): their yEnc name was taken as the inner
name. Packaging names are skipped for lone files too, and a name that
starts with the release's own name says nothing new.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release can have several candidates of a kind (two RAR sets): keyed by
(release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE
command cannot affect row a second time"), and the batch was lost.
release_inspections is keyed by candidate now; the first run's results
are cleared and probed again. A random-named file's yEnc name is often
its random name: an inner name equal to the release's name is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.
Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
