The pending ladder in the prober: unnamed releases probed cheapest first, signatures learned, the named moved into the catalog
Test and deploy / test (push) Failing after 48s
Test and deploy / edge (push) Successful in 1m57s
Test and deploy / deploy (push) Skipped

pending_ladder.ts claims the oldest waiting pending releases' next target
(par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each
claim; a usable name (the largest named file of a par2 set or a listing, or a
lone file's yEnc name, neither packaging nor random) hands the release to the
namer, no name the next target or the end of the climb; climbs count on
pending_signatures per (tool, group, poster) and (tool, group), and a signature
with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are
discarded without a probe, the rules' sample excepted. pending_namer.ts reads
a named release's articles again from the corpus or by XOVER and writes it to
the catalog under the pending key (source weak-stem, the name as inner name)
with its chunk and candidates. Migration 6400; config PENDING_*.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-26 13:41:30 +01:00
1 parent 9cf85e09d8
commit e135076fcd
14 files changed
+1045 -50

No files matched your search

+37 -17
View File
@@ -550,13 +550,17 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
`release_texts`. The category trigger reads the groups from
`group_sets`; the judge's dirty trigger watches `poster_id` instead of
`poster`.
- `pending_releases` (migration 6300, the HDD): the unnamed releases
outside the catalog (decision f): `release_key` (primary), `identity_key`,
`group_id`, `name` (the random stem), `poster` text, the posting span,
`first_article`, `last_article`, counts, `partial`, `finished`, `tool`,
`candidates` jsonb (the probe targets by effort), `status` (0 waiting, 1
named, 2 discarded), `tries`, `created_at`, `probed_at`; indexed by
(group, tool, status, created_at) for the ladder.
- `pending_releases` (migrations 6300 and 6400, the HDD): the unnamed
releases outside the catalog (decision f): `release_key` (primary),
`identity_key`, `group_id`, `name` (the random stem), `poster` text, the
posting span, `first_article`, `last_article`, counts, `file_total`,
`partial`, `finished`, `tool`, `candidates` jsonb (the probe targets),
`inspections` jsonb (what each probe showed), `named`, `status` (0
waiting, 1 named and in the catalog, 2 discarded, 3 exhausted, 4 named
awaiting the move, 9 taken), `tries`, `created_at`, `probed_at`; partial
indexes on the waiting (by `created_at`) and the named (by `probed_at`).
- `pending_signatures` (6400): per (tool, group, poster; poster '' for the
tool and group together) `probed`, `named`, `dropped`, `updated_at`.
- `release_texts(release_id, kind smallint, text varchar(255))` (6200),
primary key on all three, one trigram index (fastupdate): every string
search matches against, appended by whoever learns one and never an
@@ -689,12 +693,9 @@ until the window is done (LOG.md 2026-10-26 (111) has the first local
numbers, (112) the homelab run).
Next: the replay on homelab over the 2016 window (the NVMe bytes per
header for real), the pending ladder and the signature learner (decision
f, amended 2026-10-26: the prober working `pending_releases` cheapest
first, verdicts per tool and group, a named release moved into the
catalog with its segments read again), the ordered range leases with the
carried window in the edge-node and its snapshot (design 1 and 5), the
prober writing `release_texts` kind 2. Gone from the list: block
header for real), the ordered range leases with the carried window in
the edge-node and its snapshot (design 1 and 5), the prober writing
`release_texts` kind 2, the spot/NZB match on pending keys (decision f). Gone from the list: block
leases, the planner and the block merger (design 1, amended).
## Decisions
@@ -729,10 +730,29 @@ f. **Nothing enters the catalog without a usable name** (amended
11% of the segments and nothing anyone can find; the full history has
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
edge sends an unnamed release with an empty payload and its tool, the
batch store writes `pending_releases` (LOG.md (114)). Not built: the
ladder itself (the prober working `pending_releases` cheapest first,
the signature verdicts, the move into the catalog with the segments
read again), and the spot/NZB match on pending keys.
batch store writes `pending_releases` (LOG.md (114)); the ladder in the
prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
the oldest waiting releases' next target (par2, archive, lone, 7z-tail
in that order), at least PENDING_SHARE (25%) of each claim next to the
catalog's own candidates, answered from `probe_cache` when the article
was read before; `usableName` (lib/probe/inspect.ts) takes the largest
named file of a par2 set or a listing, or a lone file's yEnc name, that
is neither packaging nor random; a name sets status 4, no name the next
target or status 3 (exhausted); each finished climb counts on
`pending_signatures` for (tool, group, poster) and (tool, group), and a
signature with PENDING_MIN_PROBES (100) climbs and at most
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
waiting releases of dropped signatures are discarded (status 2), the
rules' `samplePercent` of them left to climb. The namer
(`app/services/pending_namer.ts`, in the prober's process): a named
release's article span read again from the corpus (the stored range's
chunks) or by XOVER on a connection of its own, through the TypeScript
indexer, written as a release under the pending key with `source`
weak-stem, the name as `inner_name`, its chunk and its candidates (the
prober answers those from `probe_cache`: inspections, par2 sets, hashes,
the password flag). Not built: the spot/NZB match on pending keys (step
0 of the ladder beyond signatures), and a signature by poster alone
across groups.
g. The account's NNTP connections shared by priority (`nntp_shares`): the
coordinator (headers) before the prober, each between its minimum and
maximum, 5 kept for the spotter, the spot inspector, the plain