The pending ladder in the prober: unnamed releases probed cheapest first, signatures learned, the named moved into the catalog
pending_ladder.ts claims the oldest waiting pending releases' next target (par2, archive, lone, 7z-tail in that order) as at least PENDING_SHARE of each claim; a usable name (the largest named file of a par2 set or a listing, or a lone file's yEnc name, neither packaging nor random) hands the release to the namer, no name the next target or the end of the climb; climbs count on pending_signatures per (tool, group, poster) and (tool, group), and a signature with 100 climbs and 0.5% named or fewer is dropped: its waiting releases are discarded without a probe, the rules' sample excepted. pending_namer.ts reads a named release's articles again from the corpus or by XOVER and writes it to the catalog under the pending key (source weak-stem, the name as inner name) with its chunk and candidates. Migration 6400; config PENDING_*. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
9cf85e09d8
commit
e135076fcd
14 files changed
+1045
-50
No files matched your search
+37
-17
@@ -550,13 +550,17 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
||||
`release_texts`. The category trigger reads the groups from
|
||||
`group_sets`; the judge's dirty trigger watches `poster_id` instead of
|
||||
`poster`.
|
||||
- `pending_releases` (migration 6300, the HDD): the unnamed releases
|
||||
outside the catalog (decision f): `release_key` (primary), `identity_key`,
|
||||
`group_id`, `name` (the random stem), `poster` text, the posting span,
|
||||
`first_article`, `last_article`, counts, `partial`, `finished`, `tool`,
|
||||
`candidates` jsonb (the probe targets by effort), `status` (0 waiting, 1
|
||||
named, 2 discarded), `tries`, `created_at`, `probed_at`; indexed by
|
||||
(group, tool, status, created_at) for the ladder.
|
||||
- `pending_releases` (migrations 6300 and 6400, the HDD): the unnamed
|
||||
releases outside the catalog (decision f): `release_key` (primary),
|
||||
`identity_key`, `group_id`, `name` (the random stem), `poster` text, the
|
||||
posting span, `first_article`, `last_article`, counts, `file_total`,
|
||||
`partial`, `finished`, `tool`, `candidates` jsonb (the probe targets),
|
||||
`inspections` jsonb (what each probe showed), `named`, `status` (0
|
||||
waiting, 1 named and in the catalog, 2 discarded, 3 exhausted, 4 named
|
||||
awaiting the move, 9 taken), `tries`, `created_at`, `probed_at`; partial
|
||||
indexes on the waiting (by `created_at`) and the named (by `probed_at`).
|
||||
- `pending_signatures` (6400): per (tool, group, poster; poster '' for the
|
||||
tool and group together) `probed`, `named`, `dropped`, `updated_at`.
|
||||
- `release_texts(release_id, kind smallint, text varchar(255))` (6200),
|
||||
primary key on all three, one trigram index (fastupdate): every string
|
||||
search matches against, appended by whoever learns one and never an
|
||||
@@ -689,12 +693,9 @@ until the window is done (LOG.md 2026-10-26 (111) has the first local
|
||||
numbers, (112) the homelab run).
|
||||
|
||||
Next: the replay on homelab over the 2016 window (the NVMe bytes per
|
||||
header for real), the pending ladder and the signature learner (decision
|
||||
f, amended 2026-10-26: the prober working `pending_releases` cheapest
|
||||
first, verdicts per tool and group, a named release moved into the
|
||||
catalog with its segments read again), the ordered range leases with the
|
||||
carried window in the edge-node and its snapshot (design 1 and 5), the
|
||||
prober writing `release_texts` kind 2. Gone from the list: block
|
||||
header for real), the ordered range leases with the carried window in
|
||||
the edge-node and its snapshot (design 1 and 5), the prober writing
|
||||
`release_texts` kind 2, the spot/NZB match on pending keys (decision f). Gone from the list: block
|
||||
leases, the planner and the block merger (design 1, amended).
|
||||
|
||||
## Decisions
|
||||
@@ -729,10 +730,29 @@ f. **Nothing enters the catalog without a usable name** (amended
|
||||
11% of the segments and nothing anyone can find; the full history has
|
||||
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
|
||||
edge sends an unnamed release with an empty payload and its tool, the
|
||||
batch store writes `pending_releases` (LOG.md (114)). Not built: the
|
||||
ladder itself (the prober working `pending_releases` cheapest first,
|
||||
the signature verdicts, the move into the catalog with the segments
|
||||
read again), and the spot/NZB match on pending keys.
|
||||
batch store writes `pending_releases` (LOG.md (114)); the ladder in the
|
||||
prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
|
||||
the oldest waiting releases' next target (par2, archive, lone, 7z-tail
|
||||
in that order), at least PENDING_SHARE (25%) of each claim next to the
|
||||
catalog's own candidates, answered from `probe_cache` when the article
|
||||
was read before; `usableName` (lib/probe/inspect.ts) takes the largest
|
||||
named file of a par2 set or a listing, or a lone file's yEnc name, that
|
||||
is neither packaging nor random; a name sets status 4, no name the next
|
||||
target or status 3 (exhausted); each finished climb counts on
|
||||
`pending_signatures` for (tool, group, poster) and (tool, group), and a
|
||||
signature with PENDING_MIN_PROBES (100) climbs and at most
|
||||
PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
|
||||
waiting releases of dropped signatures are discarded (status 2), the
|
||||
rules' `samplePercent` of them left to climb. The namer
|
||||
(`app/services/pending_namer.ts`, in the prober's process): a named
|
||||
release's article span read again from the corpus (the stored range's
|
||||
chunks) or by XOVER on a connection of its own, through the TypeScript
|
||||
indexer, written as a release under the pending key with `source`
|
||||
weak-stem, the name as `inner_name`, its chunk and its candidates (the
|
||||
prober answers those from `probe_cache`: inspections, par2 sets, hashes,
|
||||
the password flag). Not built: the spot/NZB match on pending keys (step
|
||||
0 of the ladder beyond signatures), and a signature by poster alone
|
||||
across groups.
|
||||
g. The account's NNTP connections shared by priority (`nntp_shares`): the
|
||||
coordinator (headers) before the prober, each between its minimum and
|
||||
maximum, 5 kept for the spotter, the spot inspector, the plain
|
||||
|
||||
Reference in new issue
Block a user