Commit Graph
100 Commits
Author SHA1 Message Date
Daan Meijer cf51db047f Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m51s
Test and deploy / deploy (push) Skipped
2026-10-25 22:43:14 +01:00
Daan MeijerandClaude Fable 5.1 e779f67c77 Prefetcher: corpus candidates a range of every group at a time
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Successful in 1m24s
The 512 candidates were the lowest ids at the lowest priority; a group's
ranges have consecutive ids, so once a window's small groups were stored
they were all one group's, two targets on two connections (10 ranges an
hour). Ordered by the per-group rank now: the window spans 85 groups.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 22:43:01 +01:00
Daan MeijerandClaude Fable 5.1 f8cb17903b voyager 3: the window in the Rust edge
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped
The indexer finishes files and releases by posting time instead of the
range path: the frontier is the median Date of the last 1,024 plausible
dates (within 7 days of it; a run of 1,024 agreeing dates moves it), and
a release is finished, judged and written once the frontier is 24 h past
its last part, flagged edge when posted within a day of the block's start.
At the block's end the rest is written provisionally as edges for the
block merger. Files short of parts stay in their release (partial: the
store writes complete = false). Gone: fragments and the cut-off,
uploads_of (online now), check.ts and the fixture comparison; CI replays
the fixtures through edge-bench instead. The wire carries finished, edge
and partial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:28:35 +01:00
Daan MeijerandClaude Fable 5.1 2580f20a43 voyager 3: the batch store writes releases once, by id references and bytea keys
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Skipped
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.

Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 21:04:03 +01:00
Daan MeijerandClaude Fable 5.1 bc4e17e86f voyager 3: release_texts, the searchable strings of a release (migration 6200)
Test and deploy / test (push) Failing after 52s
Test and deploy / edge (push) Successful in 1m52s
Test and deploy / deploy (push) Skipped
One table and one trigram index for every string search matches against:
the release name (kind 1, stored once more) and what anyone learns later
(the prober's file names inside an archive, NFO titles, par2 set names,
media titles), appended by insert, never an update of the releases row.
Kinds and the per-kind cap in lib/release/text_kind.ts (textsToStore).
The judge's dirty mark follows an insert or delete. Migration 6000 loses
its two trigram indexes on releases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 20:33:28 +01:00
Daan MeijerandClaude Fable 5.1 346493b6c3 Voyager 3 schema: the slim catalog (migration 6000, the reset) and blocks (6100)
Test and deploy / test (push) Failing after 50s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Skipped
releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 13:44:54 +01:00
Daan Meijer 1a448f37f9 Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Skipped
2026-10-25 12:50:10 +01:00
Daan MeijerandClaude Fable 5.1 3cc7d9738e TruncatedReply: a compressed reply the server cut at a ".\r\n" fails in 2 s; ranges:dates samples plain
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m4s
Eweka stops a compressed XOVER reply at the first ".\r\n" in its own
output (78 01 2e 0d 0a and nothing more), which some small replies begin
with; the text path waited for more until the reply timeout. ranges:dates
uses plain connections of its own instead of the pool.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:49:53 +01:00
Daan MeijerandClaude Fable 5.1 1e01702dff nntp:probe: shows the bytes a stalled reply left
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 1m14s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:40:02 +01:00
Daan MeijerandClaude Fable 5.1 d048c6cdb5 nntp:probe: --plain and --sizes
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m6s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:31:01 +01:00
Daan Meijer 37750fa4b3 Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Skipped
2026-10-25 12:16:28 +01:00
Daan MeijerandClaude Fable 5.1 e336538d8d nntp:probe: bisects XOVER spans that hang
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m12s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 12:16:17 +01:00
Daan Meijer f4df25d774 Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Skipped
2026-10-25 10:48:35 +01:00
Daan MeijerandClaude Fable 5.1 d43716388a REBUILD.md: the 2016 and 2021 windows marked for the corpus
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 2m6s
Test and deploy / deploy (push) Successful in 47s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 10:48:34 +01:00
Daan Meijer c825dc7a8f Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Skipped
2026-10-25 10:32:39 +01:00
Daan MeijerandClaude Fable 5.1 33405eb8ca ranges:dates: a point on a connection of its own; --missing still dates every group
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 1m53s
Sixteen groups never got through on one connection (a reply timed out
after 2 minutes); now a timed-out point is skipped and logged, the
group goes on, and stops after 3 failures. estimate() ignored --missing's
filter: it dated 0 groups on the second run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 10:31:35 +01:00
Daan Meijer 562c6d02fc Merge branch 'rebuild' into voyager-3
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Skipped
2026-10-25 09:47:46 +01:00
Daan MeijerandClaude Fable 5.1 137e931e9e Posting-time windows into the corpus: ranges:dates, ranges:capture
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Successful in 2m12s
ranges:dates samples each active group's posting dates from the news
server (group_dates) and estimates every range's date (ranges.est_date).
ranges:capture --from --to marks a window's ranges of every group for the
corpus. The corpus fill takes capture ranges in priority order, done ones
too, and remembers complete ones (ranges.stored_at).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 09:47:28 +01:00
Daan MeijerandClaude Fable 5.1 0a4d863f42 REBUILD.md: voyager 3 design (window indexing, albums, slimmer catalog)
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Skipped
A 24 h posting-time window per block instead of write-early-merge-later,
albums for counterless series, poster/group-set ids and binary keys in
releases, cold tables on the HDD, the corpus test before the reset.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-25 00:54:14 +02:00
Daan MeijerandClaude Opus 5.5 0469fcd975 Prober stops claiming under 6 GB free (was 12)
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m10s
The NVMe floor is 4 GB since Oct 23; at 12 the prober had claimed nothing
since Oct 23 15:30 (7-10 GB free).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:35:35 +02:00
Daan MeijerandClaude Opus 5.5 03ca96b255 Fix findCounter infinite loop; corpus filter skips chunks past 256 MB
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 2m5s
lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "("
that isn't a counter, without a quote, looped forever. The corpus filter hit
it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU
for 10 hours and no range finished. The Rust find_counter stops at 0 already.

A chunk inflating past V8's longest string can't be filtered as text; such a
chunk is now stored unfiltered instead of throwing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:08:22 +02:00
Daan MeijerandClaude Opus 5.5 2dd5b4cd81 releases:merge-names: keyset paging on (created_at, set_id)
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 3m5s
Sets stored in one batch share created_at, so paging on created_at < last
skipped ~10k of 61k sets; the timestamp now also stays text so microseconds
survive the round trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 11:15:32 +02:00
Daan MeijerandClaude Opus 5.5 a66b0980f4 Corpus: store chunks without the lines the worker only counts
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 16:34:09 +02:00
Daan MeijerandClaude Opus 5.5 c1946792b2 Corpus prefetch: at most perGroup/perRange capture ranges per group
Test and deploy / test (push) Successful in 1m2s
Test and deploy / edge (push) Successful in 2m10s
Test and deploy / deploy (push) Successful in 1m19s
The prefetcher fetched capture ranges into the corpus 16 at a time in id
order, and ids run group by group: the 16 were one or two groups', the
per-group connection limit (4) let 4-8 loops work and left the rest
idle, ~30k articles/s for the first batch of 19,955 capture ranges. Now
at most perGroup / perRange (2) of a group are taken, so the 16 come
from 8+ groups and every loop gets work (and the pool asks for more).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 15:25:18 +02:00
Daan MeijerandClaude Opus 5.5 fb6b00f230 Name pass: only holders near the newest; merges drop duplicate candidates
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 57s
The name pass put every holder of a set into its merge: a par2 posted
again years later ("Auto Bild Klassik Februar 02-2014", held by a 2014
and a 2026 release) would have joined the repost to the old upload. Now
the newest holder anchors it and only holders in its group within 12
hours take part. Every round, three sets failed on the probe candidates'
unique key: a member held a candidate of an article the target had.
merge() now drops such a candidate (the target's, or a lower member's,
stays), as it already does for par2 sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:59:09 +02:00
Daan MeijerandClaude Opus 5.5 66d08a0e92 REBUILD.md: the merger's name, inner and sidecar passes
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 41s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:12:08 +02:00
Daan MeijerandClaude Opus 5.5 e00e4f3988 Join an upload's releases by its par2 set's file names
Test and deploy / test (push) Successful in 1m5s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 1m19s
The files of one upload often became several releases whose names
differ slightly: scene tracks apart from their album's nfo/sfv/par2,
"Vermaechtnis" beside a garbled "Vermächtnis", an index named
Die.Bergretter.S18E05... beside its video wishtv-diebergretter-s18e05.
The par2 merge joined releases only by first-16-KiB hashes, which
readable files never get. A name pass now looks, for every set the
merger finds, at the set poster's releases in its group posted within
12 hours: those whose files the set all names (without case or
punctuation, Latin-1-read UTF-8 read back; sidecars, par2 and images
aside) join the set's release, within the hash pass's file bound.
Merges are kept in release_merges as via 'name'. releases:merge-names
runs it over the sets found before; on 300 recent sets 22 merges of 147
releases, all one upload each.

Also: volumes split after .wmv, .mov, .mpg and other video extensions
share a stem ("Morning Lovers.wmv.001" to ".058" were 58 releases), in
the Rust edge and the TypeScript identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:02:51 +02:00
Daan MeijerandClaude Opus 5.5 e4044cb1d1 releases:recategorize: a camelCase name
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 1m45s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:07:56 +02:00
Daan MeijerandClaude Opus 5.5 61bdd43b34 Categories from the inner name, spots and groups; fix the name rules
Migration 1200 passed regexes with `?` through knex, which made them
bindings: the live function held blu-$1ray, web-$2dl and h\.$326[45], so
BluRay, WEB-DL and H.264 names without a resolution were Misc. 93% of
recent releases were Misc, as the category came from the name alone.

category becomes a plain column (DROP EXPRESSION, no rewrite) set by a
trigger: the name's category (fixed, no `?` anywhere; adds XXX,
audiobooks, console and PC games, and TV codes like S01e07[ and
S04E01E02), else the prober's inner name's, else one given before (a
spot's), else the groups'. The spot inspector sets a spot's category
(spot_category) on Misc releases it links and on its own releases. The
Newznab and web category lists gain Console, PC Games, Audiobook, Anime,
XXX and Comics. releases:recategorize recomputes existing rows in id
batches, writing only changed rows and waiting under a free-space floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:07:12 +02:00
Daan MeijerandClaude Opus 5.5 ebe9f8bb36 Decide completeness by par2 set for releases without a counter
59% of settled releases had complete = NULL: without an [i/N] counter
the settler left it unknown, and the "phase 2 reads par2 sets for those"
step was never built. A release is now complete when every file its par2
sets describe is there by name or by its first-16-KiB hash, or, for an
upload renamed after its par2 was made, when it has as many non-par2
files and bytes. Decided when a par2 index is probed, after a par2 merge,
and before the pruner drops a set's file list; the batch store, settler
and merger keep a decided value instead of resetting it to NULL.
releases:par2-complete decides the releases probed before this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:00:19 +02:00
Daan MeijerandClaude Opus 5.5 f75097d7ac Put known passwords in the NZB and show them
release_passwords (from spots: their NZB meta, title or text) was only
read by the judge. The NZB now carries the most trusted one as
<meta type="password">, which SABnzbd and NZBGet use to unpack; a spot's
own NZB gets it added when its meta has none. The release details list
all known passwords with their source, the list marks "Password known",
and the web interface shows them instead of "not in the index".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:56:43 +02:00
Daan MeijerandClaude Opus 5.5 d446bb57df Probe 7z listings and ZIP starts
7z keeps its file list at the end of the archive, so the 7z-tail
candidates (the last segment of the highest volume) were queued but
never probed, and the .7z.001 start only gave the type. The prober now
claims 7z-tail candidates after the archives; lib/sevenzip finds the end
header in the article, decodes the LZMA-packed header (a TypeScript port
of the LZMA SDK's reference decoder) and lists the files with their
sizes. An AES-encrypted header means passworded without names. Names
feed the inner name and the judge's fake rule.

An unsplit .7z of several segments now also gets a tail candidate, and
.zip files an archive candidate: their first segment's local headers
give the first entries and whether they are encrypted. Both in the Rust
edge and the TypeScript indexer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:55:07 +02:00
Daan MeijerandClaude Opus 5.5 3b61215f6b Keep only the 7z tails of a release's highest volume
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m47s
Test and deploy / deploy (push) Successful in 1m49s
The live tail reads an upload in pieces while it is posted, and each
piece queued a 7z-tail candidate for the last volume it had seen so far
(release 306905984: seven, for .002-.008). Only the last volume's end
holds the 7z listing. When candidates are stored, and when releases are
merged, waiting tails below the release's highest volume are deleted.
All tails of the highest volume stay: a piece that saw only part of it
queued a segment that is not its last. RAR is unaffected: it lists its
files in the first volume's header and gets no tail candidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:16:25 +02:00
Daan MeijerandClaude Opus 5.5 b60d716db0 Close NNTP connections left idle for NNTP_IDLE_CLOSE_SECONDS
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m45s
Test and deploy / deploy (push) Successful in 2m20s
Eweka drops quiet connections, so the next command on one failed
("Spot inspection failed: NNTP connection closed"), and idle connections
held account slots that downloads could use. The coordinator pool now
closes free connections (and the plain one) on its sync once idle for
60 s; dry prober lanes, the spotter between rounds and the spot inspector
with nothing to inspect close theirs too, reconnecting when work comes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 14:31:49 +02:00
Daan MeijerandClaude Opus 5.5 705f9c3897 Release keys numbered after the highest number, not the count
Test and deploy / test (push) Successful in 57s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 2m10s
Merges delete releases, so the count of a key's releases fell behind its
numbers (83 releases up to key:129): the 20 numbers tried were all taken
and the range's batch failed for good. Spot releases (spot:key) are not
numbered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 18:08:13 +02:00
Daan MeijerandClaude Opus 5.5 293d9d4b52 par2:prune: one whole prune pass in one go
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m11s
Test and deploy / deploy (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:02:18 +02:00
Daan MeijerandClaude Opus 5.5 8ad5ea162c par2 file lists dropped once their holders are final; late candidates claimable
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m46s
Test and deploy / deploy (push) Successful in 58s
Most par2 sets describe readable uploads whose files are never hashed, so they
were never complete and kept forever (19 GB). Candidates added to a release
after it settled were never marked ready (93,642 since 2026-10-13); the batch
store and merge mark them now, migration 5100 repairs the waiting ones
(LOG.md 2026-10-20 (87)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 13:56:57 +02:00
Daan MeijerandClaude Opus 5.5 4f41303482 Merger: what's new an hour at a time, only the sets two releases share; common files match nothing
Test and deploy / test (push) Successful in 1m3s
Test and deploy / edge (push) Successful in 2m14s
Test and deploy / deploy (push) Successful in 3m9s
A round behind by two days of the probe backlog read 3.5M sets and never ended;
a par2 file shared by up to 190k files loaded every release it matched and the
merger ran out of heap every few minutes (LOG.md 2026-10-20 (85)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:54:50 +02:00
Daan MeijerandClaude Opus 5.5 9725bdbd7b Judge: count a chunk's file hashes once, from a covering index
Test and deploy / test (push) Successful in 1m1s
Test and deploy / edge (push) Successful in 2m6s
Test and deploy / deploy (push) Successful in 8m54s
The incremental judge counted each changed release's files twice, each
index entry a heap fetch: 24-32 s a chunk. file_hashes by (hash16k,
size, release_id) replaces (hash16k); the other releases of a file are
judged only when the chunk brings it to 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:32:52 +02:00
Daan MeijerandClaude Opus 5.5 76e52f69ce Judge the releases whose facts changed every 2 minutes, all daily
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m5s
A full round took 35-38 min of each hour and grew with every inspection
the prober added. Triggers note changed releases in judge_dirty; the
judge judges those (and releases a file hash of theirs tipped to
repeated) by every rule, and every release in a daily full round for
what time changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:10:49 +02:00
Daan MeijerandClaude Opus 5.5 036e2a8397 release_inspections back on the NVMe
Test and deploy / test (push) Successful in 1m2s
Test and deploy / edge (push) Successful in 2m11s
Test and deploy / deploy (push) Successful in 2m3s
The judge's fake rule reads all of it every round: 17-18 min on the HDD,
~3 on the NVMe. Migration 4600 is a no-op now; 4800 moves it back where
4600 ran. Homelab moved it by hand; a no-op there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:08:35 +02:00
Daan MeijerandClaude Opus 5.5 a983cad509 Judge: write only changed verdicts; scraped and locked by index
Test and deploy / test (push) Successful in 1m2s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 16m15s
A round took 22-29 min an hour: every match went through the insert's
conflict check (60-150 s a rule), scraped and locked each scanned all of
releases. The spot inspector's releases now add their poster to posters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:48:54 +02:00
Daan MeijerandClaude Opus 5.5 aa179f908a release_inspections on the HDD, its indexes on the NVMe
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 2m16s
Test and deploy / deploy (push) Successful in 1m57s
The prober's output, appended and read a release at a time: 40% of the
NVMe's growth while probing. Homelab moved it by hand; a no-op there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 19:53:13 +02:00
Daan MeijerandClaude Opus 5.5 d405b658c3 Prober: a flush with a NUL in a name, or one article twice, no longer fails
Test and deploy / test (push) Successful in 57s
Test and deploy / edge (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 1m51s
jsonb can't hold \u0000, and ON CONFLICT can't take a row twice in one
statement: either failed the whole flush, its outcomes lost and its
candidates taken until a restart. NULs are dropped from strings; the
probe_cache insert takes each message-id once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 17:45:35 +02:00
Daan MeijerandClaude Opus 5.5 9fb36d217c Prober: no claims under PROBE_MIN_FREE_GB free on the NVMe
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m5s
Test and deploy / deploy (push) Successful in 1m1s
At ~200 articles/s what the prober stores (inspections, par2 file lists,
the WAL) took ~0.8-1.1 GB an hour of homelab's root, and nothing stopped it
at the 10 GB floor. Below 12 GB free it claims nothing (resumed 1 GB above):
the lanes go dry and its share shrinks to its minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 17:38:37 +02:00
Daan MeijerandClaude Opus 5.5 de7c758206 probe_cache's primary key on the catalog's tablespace
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 57s
Each claim looks its candidates up in probe_cache by message_id; with the
index on the HDD that took 1.27 s a claim, 48 s of each minute. Moved by
hand on homelab; a no-op there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 12:50:35 +02:00
Daan MeijerandClaude Opus 5.5 38fd19b42e REBUILD.md: decision g, connections shared by priority
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 57s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:20:49 +02:00
Daan MeijerandClaude Opus 5.5 39480c486f NNTP shares: the account's connections shared by priority
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 53s
The coordinator (38) and the prober (8) had a fixed split; with history
paused the coordinator's stood idle while the prober's were its limit.
nntp_shares: each process syncs every 15 s (priority, minimum, what it could
use, what it has open) and gets what it may open: minimums first, then by
priority what each wants, growing only into connections the others closed
or gave up. The pool's size and the prober's lanes follow the grant; the
coordinator doubles its want while work waits for a connection, else asks
for its peak in use and a margin; the prober wants its maximum unless the
queue is dry. Claims scale with the lanes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:20:09 +02:00
Daan MeijerandClaude Opus 5.5 7eba53043a Prober: claim range per value of unreadable, from the index ends
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m59s
Over unreadable >= false Postgres walked every waiting entry of the kind:
2.5 minutes a claim at 28M archive candidates, the prober idle every other
minute. Per value each min and max is one end of the ready index (56 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:00:11 +02:00
Daan MeijerandClaude Opus 5.5 a5e0399ee6 The par2 prune's cursor is (created_at to the microsecond, set_id): 20k sets of one flush share a created_at
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:10:19 +02:00
Daan MeijerandClaude Opus 5.5 f2c5607ec3 Fold only sidecar-only releases (sidecar_only at insert); no queue-wide scan for gone releases
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m56s
Test and deploy / deploy (push) Successful in 54s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:57:59 +02:00
Daan MeijerandClaude Opus 5.5 aa20d6b79f The par2 prune gets 2 min per round (it held the merges back for hours); abandoned ranges past their end are finished
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m42s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:42:14 +02:00
Daan MeijerandClaude Opus 5.5 c55918350e Poster search per poster found (LATERAL on releases_poster_newest_idx), at most 200 posters
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 2m12s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:58:26 +02:00
Daan MeijerandClaude Opus 5.5 c0e00f1707 Poster search through a posters table and releases_poster_newest_idx; complete releases are final; the tail before everything (-3)
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 5m20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:25:30 +02:00
Daan MeijerandClaude Opus 5.5 c9a9c0155e Postgres memory for the freed RAM: shared_buffers 8GB, effective_cache_size 32GB, random_page_cost 1.1, work_mem 16MB
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m6s
Test and deploy / deploy (push) Successful in 2m13s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:57:12 +02:00
Daan MeijerandClaude Opus 5.5 aa3bd82445 At most two leases per group: a group's batches wait for each other, more workers only queue
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 3m19s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:28:42 +02:00
Daan MeijerandClaude Opus 5.5 b73e91219e The judge's unusable waits for final: next to an unread range a release's par2 or files may still come
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m13s
Test and deploy / deploy (push) Successful in 2m20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:06:13 +02:00
Daan MeijerandClaude Opus 5.5 ec2e5a60be Settle every 5 min in rounds of 200k, not ~3M at once hourly; range blocks in a loop of their own
Test and deploy / test (push) Successful in 1m0s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 3m48s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:56:27 +02:00
Daan MeijerandClaude Opus 5.5 ce43099188 Final completeness by time window: ranges keep their posting span, release_final() over blocks of done ranges
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m4s
Test and deploy / deploy (push) Successful in 8m5s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:19:03 +02:00
Daan MeijerandClaude Opus 5.5 3d24db5565 Rules drop posters at the header pass: the CelebMovie family (rule set version 3)
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m47s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:36:46 +02:00
Daan MeijerandClaude Opus 5.5 093d0a71ad The judge hides scraped celebrity pictures; no ? in its patterns (a raw query takes it for a binding)
Test and deploy / test (push) Successful in 52s
Test and deploy / edge (push) Successful in 2m0s
Test and deploy / deploy (push) Successful in 2m11s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:01:07 +02:00
Daan MeijerandClaude Opus 5.5 7d9a81784e A folder's images posted with their sizes are one release: no image set under the subject
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m28s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:13:20 +02:00
Daan MeijerandClaude Opus 5.5 e154b2896f par2_files back on the NVMe: the merger reads it per set, a random read per row on the HDD
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 1m49s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:22:37 +02:00
Daan MeijerandClaude Opus 5.5 1a20fcd2c3 Prune par2_files of complete and judged sets (hourly, from the merger)
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 3m35s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 07:59:40 +02:00
Daan MeijerandClaude Opus 5.5 dd35b49a3f Nightly pause (23:45) and resume (07:45); a deploy while paused leaves the workers stopped
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m12s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 19:13:21 +02:00
Daan MeijerandClaude Opus 5.5 59c93071c7 The sidecar fold every 5 min inside a long par2 pass too
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m50s
Test and deploy / deploy (push) Successful in 2m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:48:30 +02:00
Daan MeijerandClaude Opus 5.5 2e25b7c56e The sidecar fold first in a merger round: a queue, it needn't wait for the par2 pass
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 2m18s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:37:53 +02:00
Daan MeijerandClaude Opus 5.5 e627862a8d The merger keeps its place over a restart (worker_cursors), not from 1970 each deploy
Test and deploy / test (push) Successful in 49s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m4s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:45:25 +02:00
Daan MeijerandClaude Opus 5.5 46525f4595 fragments back on the NVMe: the fragment merger reads every row, not by key
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m14s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:46:14 +02:00
Daan MeijerandClaude Opus 5.5 329c7a1373 Fold sidecars (.nfo, .sfv, .nzb) apart from their upload into it
Test and deploy / test (push) Successful in 44s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m14s
"Murdoch Mysteries S8D1 [00/80] - \"….nzb\"" beside its "[01/94]" upload
was a release of its own (23k sidecar-only releases on the corpus, 15k with
the poster's bigger release within 10 minutes). The settler queues small
settled releases (fold_candidates); the merger's third pass folds one whose
files are all sidecars into the poster's bigger release in the same group
within 10 minutes: the one sharing a 4+ letter word with its subject, else
the one under the same counter total, else none (via 'sidecar').

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:10:48 +02:00
Daan MeijerandClaude Opus 5.5 5897d595cf The junk judge: releases judged by what their probes showed; verdicts hide them
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Successful in 3m29s
Voyager 1's second filter (D42) for voyager 2. Hourly, in order (the first
rule wins): dead (every probed first article gone), fake (a film, episode or
music name whose content is a program: archive entries all executables, or
the file itself by name or sniffed exe), repeated (the same first 16 KiB and
size in 20+ releases), locked (passworded, no password known), unusable
(random names, no inner name, no merge, no spot, a day after settling).
Verdicts in release_verdicts (rule, version) hide releases from search
(judged shows them); a verdict whose rule no longer matches is taken back.
Nothing dropped yet. Service judge; releases:judge --once --dry counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:45:08 +02:00
Daan MeijerandClaude Opus 5.5 99b72ddd8f Sniff Windows executables (MZ with its PE header)
For the junk judge: a "movie" whose first bytes are a program is fake.
Two bytes alone match random data too often, so the PE signature where the
DOS header points must be there too. No PROBE_VERSION bump: new probes only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:42:29 +02:00
Daan MeijerandClaude Opus 5.5 c65282ec5d par2_files, fragments and probe_cache with the chunks (CHUNK_TABLESPACE)
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 2m49s
~20 GB of the NVMe, read by key only; the catalog was at its cap and history
paused. Moved on homelab by hand (3 min, writers stopped; the cap raised to
100); the migration does the same on a new deploy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:11:36 +02:00
Daan MeijerandClaude Opus 5.5 377decaf3d release_chunks' indexes with its chunks (CHUNK_TABLESPACE)
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m8s
Test and deploy / deploy (push) Successful in 2m7s
The October partition's three indexes took 2.9 GB of the NVMe, its pkey
never scanned; chunks are read from the HDD anyway. Moved on homelab by hand
(12 s); the migration does the same on a new deploy, parent indexes too so new
partitions follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:27:58 +02:00
Daan MeijerandClaude Opus 5.5 256fb24c46 A cap on the catalog's NVMe size: above it only the tail and the stored ranges
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s
After reset 24 the catalog grew ~3.7 GB an hour (77 GB, root 31 GB free, 600k
history ranges to go). CATALOG_MAX_NVME_GB (as voyager 1's
BACKFILL_MAX_NVME_GB): once the default tablespace is over it, leases and the
prefetcher take priority < 0 only (the tail, the stored ranges); measured once
a minute. Off when unset; 90 on homelab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:10:11 +02:00
Daan MeijerandClaude Opus 5.5 eddbe2863f Take the padded volume back out: it merged a dump's versions
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 2m34s
208ca1d's volume of 3-5 digits after a digit (YME's "CHT PCK 2 9.26.001")
also took versions for volumes: "K-ML 3.8.233", "3.8.239", "3.8.241" and
Kaizen's "v2.0.1021" ... "v2.0.1057" became one release (505 names on the
corpus). The random name's volumes with their own extensions stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:11:03 +02:00
Daan MeijerandClaude Opus 5.5 208ca1d535 A random name's volumes with extensions of their own; a padded volume after a date
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m31s
- "df2BkeRxIvHcWrsWwwz.part01.cer" … ".part69.bdmv", "jA0eSSUMat8tQtIKUyPMA.part01":
  obfuscators give each volume its own extension (or none); for a random name
  (no dots or spaces) .partNN and that extension go.
- "(????) [0001/1009] - "CHT PCK 2 9.26.001"": a volume of 3-5 digits after a
  dot counts with a digit before it too (5,419 single files of YME).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:10:16 +02:00
Daan MeijerandClaude Opus 5.5 c624a624df Take the image set head back out of the subject key
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m13s
f1c5af5's set head (an image's name up to its first number) split sets whose
images aren't named alike: "a1.jpg" beside "a1 (2).jpg", and "468_1000.jpg"
whose first number is the image's (1 release became 211 and 161 on the
corpus). The random volume stem stays. Tests keep both sets whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:14:17 +02:00
Daan MeijerandClaude Opus 5.5 f1c5af5f36 Under a constant subject: an upload's random volume stem, and an image's set head
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 2m27s
- "[ TOWN ] [ www town ag ] [ TV ] … "Z3mMTMPsoT4ikE_Di3fKnNF21.part01.rar"",
  "….vol4+4.par2": the subject and the total were the key, so every upload of
  a total was one release (2,338 files for 20; also partner-of-ssl-news,
  brothers-of-usenet). A random stem (no dots or spaces) on a volume file
  joins the subject key; a scene name doesn't, so its nfo stays.
- "Molly Set [i/50] "mo103x003.jpg"": every set of 50 was one. An image's
  name up to its first number, when another follows, joins the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 09:12:17 +02:00
Daan MeijerandClaude Opus 5.5 9f255e6ac0 Images and tracks named by a prefix alone are bounded by the counter's total
Test and deploy / test (push) Successful in 1m3s
Test and deploy / edge (push) Successful in 2m10s
Test and deploy / deploy (push) Successful in 3m17s
"[…/559] - "lop1001.jpg"", "pic004.jpg", "DSC_0002[1].jpg", and tracks
"r86_022 - Gwen Guthrie - ….mp3" ("r"): the stem route keyed them by the
letters before the number alone, so every post of a poster was one release
(5,731 images for 559; 6,433 tracks for 457). Under a counter its total is
part of the key; a cover named as its archive's stem keeps the stem's key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:54:02 +02:00
Daan MeijerandClaude Opus 5.5 3f6bb8b162 Spots: rematch the never-tried and newest spots first
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m27s
Oldest inspection first went through the 3-month backfill before today's
spots, whose releases the tail has: 3 h after reset 24, 2 of 16.7k matched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:50:40 +02:00
Daan MeijerandClaude Opus 5.5 c174b3691e PROBE_CONNECTIONS 8: 2 connections to spare
Test and deploy / test (push) Successful in 48s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 52s
At 50 exactly the coordinator still got 502s now and then: a hedged request
closes its connection and opens another while the old one still counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:51:17 +02:00
Daan MeijerandClaude Opus 5.5 9104e9af18 PROBE_CONNECTIONS 9: with 10 the account was at 51
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m54s
Test and deploy / deploy (push) Successful in 1m22s
Coordinator 38 + its plain connection + prober 10 + spotter + spot inspector:
the spot inspector got 502 Too many connections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:50:35 +02:00
Daan MeijerandClaude Opus 5.5 0c5946eb06 Spots: 5,000 unmatched spots matched again an hour
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m21s
Test and deploy / deploy (push) Successful in 1m8s
After a reset every spot is unmatched: 16.5k at 500 an hour took 33 h.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:55:17 +02:00
Daan MeijerandClaude Opus 5.5 1b64fed427 Spots: a matched spot's NZB isn't kept
Test and deploy / test (push) Successful in 1m6s
Test and deploy / edge (push) Successful in 2m47s
Test and deploy / deploy (push) Successful in 1m26s
Only a spot that becomes a release of its own needs its NZB (the header
pass's releases have their chunks); spots took 2.7 GB on the NVMe at 9k.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:42:26 +02:00
Daan MeijerandClaude Opus 5.5 96553a5504 Spaces at a dash are no part of a stem key
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m9s
Test and deploy / deploy (push) Successful in 1m38s
"www.realmom.info - XVID- Farbiges Krabbeltier.r00" beside "… XVID-Farbiges
Krabbeltier.vol03+04.par2": 52 JBinUp uploads were two releases each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:51:47 +02:00
Daan MeijerandClaude Opus 5.5 550f63cd84 Spots: unmatched ones matched again in turns; a standalone release gives way
Test and deploy / test (push) Successful in 58s
Test and deploy / edge (push) Successful in 2m18s
Test and deploy / deploy (push) Successful in 1m52s
A spot older than what the catalog has indexed (the spotter's first backfill
reaches back ~3 months) matched nothing and would have become a release of
its own, a duplicate once the history pass builds the upload. Unmatched
spots are matched again in turns (rematched_at) for 30 days; the 24 h before
a standalone release count from the inspection; a later match deletes the
spot's own release and links the header pass's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:19:24 +02:00
Daan MeijerandClaude Opus 5.5 7df5a638ca Spots: the spotter and the spot inspector, passwords kept apart
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m28s
Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
  seen, each spot's full header verified (lib/spotnet from voyager 1, as
  Spotweb), its description, website and NZB articles; the password its
  title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
  its files listed (spot_files), each matched to the header pass's release
  by the same release identity in one of its groups within 48 h
  (release_spots); unmatched spots matched again hourly, after 24 h a
  release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
  spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:10:10 +02:00
Daan MeijerandClaude Opus 5.5 3fd08b598a An image's post token needs a word in it, not only a size
Test and deploy / test (push) Successful in 46s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m23s
"01of37 ## Alexi uses strange toys … ## alexis001.jpg (1/1) 57295b": with
the whole text as the name, the remainder is the file's size "57295b", which
229189f took for a post token: 16 sets of 36-37 images became 268 releases.
"MDG005b" has letters in a row; "57295b" doesn't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:33:52 +02:00
Daan MeijerandClaude Opus 5.5 652f278474 Postgres WAL on the NVMe; wal_buffers 64 MB
Test and deploy / test (push) Successful in 57s
Test and deploy / edge (push) Successful in 1m58s
Test and deploy / deploy (push) Successful in 1m6s
Every commit waited on the WAL in ./pg on the HDD (WALWrite; the HDD 100%
busy, 56–70 ms a write): prober flushes and claims took most of each minute.
A PG_WAL_DIR volume (default /var/lib/usenet-rebuild-wal) at
/var/lib/postgresql/wal, with data/pg_wal a symlink to it (set up by hand
once, with Postgres stopped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:17:24 +02:00
Daan MeijerandClaude Opus 5.5 8e4ed774c0 release_inspections in the catalog's tablespace (the NVMe)
Test and deploy / test (push) Successful in 53s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 2m33s
The prober writes it every flush and the merger looks inner stems up in it,
so it no longer belongs with the write-once payloads on the HDD. Moved on
homelab by hand (prober and merger stopped); the migration is a no-op there
and moves it on a new deploy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:09:20 +02:00
Daan MeijerandClaude Opus 5.5 229189f35c A date alone isn't an unquoted name; an image under a post token keys by the post
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m12s
Test and deploy / deploy (push) Successful in 3m28s
- "Ballers.2014.S01E01.HDTV.x264-KILLERS[ettv] 15_07_2015.rar", "Computer Bild
  - Ausgabe 152015 - 15_07_2015.rar": the cut after "] " or " - " left
  "15_07_2015.rar" for every item of a Razor1911 dump (1,335 files for 12);
  a date alone after the cut means the whole text is the name.
- "MDG005b - (01/90) "IMG_2307.JPG"": a counter in parentheses makes no post
  key, so every "IMG" set of the poster was one (609 for 90); an image under
  a post token is keyed by it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 09:03:23 +02:00
Daan MeijerandClaude Opus 5.5 590436d02a SCENE_RELEASE only for a track post's lower-case name
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m7s
Test and deploy / deploy (push) Successful in 3m11s
7ad4b70 also keyed rar uploads by artist and total: "VA - Best Of Shamania
Pro 2K16-(SHB 006)-WEB-2016-ZzZz" became 140 files for 20, Ace_Of_Base 72
for 12. Those group by their full name; only a name without capitals or
spaces (Meow's "blaise_mc-in_hellstill_3-ep-web-2022-known.par2") joins
the tracks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 08:05:02 +02:00
Daan MeijerandClaude Opus 5.5 7ad4b70775 A scene music post's par2 under the release name; .par/.part before .par2
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Successful in 1m31s
"[07/14] - "blaise_mc-in_hellstill_3-ep-web-2022-known.par2"": the tracks and the
00- files are keyed by artist and total, the par2 by its stem, so ~1,900 Meow
and Cat posts were two releases. Under a counter a stem that reads as a scene
music release (artist-…-web|cd|ep|single|vinyl|…-…YYYY(-group)) takes the
artist key. And "goh1Phoz5ahXu.par.vol006+08.par2", "X.part.par2" beside
the volumes: .par/.part before a par2 suffix goes too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 06:56:56 +02:00
Daan MeijerandClaude Opus 5.5 2bbabc08de corpus_sweep capture: budget for the marked ranges still filling
Test and deploy / test (push) Successful in 55s
Test and deploy / edge (push) Successful in 2m12s
Test and deploy / deploy (push) Successful in 2m18s
Only the bytes on disk were checked: at 296.6 GB with 2,300 ranges complete it
marked 100 more (~310 GB). The marked ranges not complete count at the stored
ranges' mean size, and no more are marked than fit under 300 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 05:55:26 +02:00
Daan MeijerandClaude Opus 5.5 6f7a1692d6 runOn: a space after the extension, not a bare dash
Test and deploy / test (push) Successful in 54s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 4m21s
0c5ede9 cut "Angry.Birds.Rio.v1.2.2.cracked.READ.NFO-THETA.rar" at .NFO: the
release lost its group, and two groups' releases of one game would merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 05:06:02 +02:00
Daan MeijerandClaude Opus 5.5 0c5ede99dd A quoted name that runs on into a title; .xNNN/.zNNN volumes
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m2s
Test and deploy / deploy (push) Successful in 1m32s
"lois 1258 Billy Joel - Extremes.part01.rar lois 1258 Billy Joel - Extremes ",
"GrooveCoverage.part26.rarGrooveCoveragebestof", "Goga Sekulic.part134.rar -
Goga Sekulic - …": the stem kept the title, and each part was a release of its
own (lois, CPP, DieViper, Maveri, tigra: ~2,400 files). runOn cuts the name
after its archive extension when a space, a dash or the name's own start
follows. And three-digit .x/.z volumes ("Reply 987986177631730.x004"),
not .x264/.x265.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 03:59:24 +02:00
Daan MeijerandClaude Opus 5.5 1c8da1dd12 A long number is a per-post id only under a counter
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m32s
"<kere.ws/illuminatenboard.org> - ID - 1855891249 - Along.The.Ridge.2006.Limited.PAL.DVDr-TWCiSO.rar":
no counter, the board's ID kept isPosterTag from taking the text for a tag,
and a dump of thousands of releases was one of 9,444 files. FTD's ids come
with a counter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:41:40 +02:00
Daan MeijerandClaude Opus 5.5 8d5385fdec Split zip volumes (.zNN) and .xNN as volumes, but not .x32/.x64/.x86
Test and deploy / test (push) Successful in 56s
Test and deploy / edge (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 1m23s
87e15d8's .[s-v]NN let "Licence 700790900887990.z18" and "….x31" uploads
fall apart again (+1,740 releases on the corpus). .zNN is a split zip volume;
.xNN is a volume unless it's a build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 01:47:21 +02:00
Daan MeijerandClaude Opus 5.5 87e15d810d Old rar volumes: .s–.v only, not .x64/.x86
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 1m55s
Test and deploy / deploy (push) Successful in 1m36s
d4e53ca's .[s-z]NN took "Electric.Rain.Swift.3D.PS.v1.0.142.x64" for a
volume, so x86 and x64 builds would be one release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 00:53:38 +02:00