Nothing enters the catalog without a usable name: the pending store (decision f amended)
An unnamed release (no readable file name) is written to pending_releases (migration 6300, the HDD) with its keys, counts, article span, upload tool and probe targets, instead of releases, release_texts, posters and a chunk. The edge sends it with an empty payload and its tool; the batch store splits a batch by readable. The pending ladder (probes cheapest first, signature verdicts, the move into the catalog) comes next. The replay report shows the pending releases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
6860e6ccaf
commit
9cf85e09d8
8 files changed
+286
-15
No files matched your search
+40
-9
@@ -550,6 +550,13 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
||||
`release_texts`. The category trigger reads the groups from
|
||||
`group_sets`; the judge's dirty trigger watches `poster_id` instead of
|
||||
`poster`.
|
||||
- `pending_releases` (migration 6300, the HDD): the unnamed releases
|
||||
outside the catalog (decision f): `release_key` (primary), `identity_key`,
|
||||
`group_id`, `name` (the random stem), `poster` text, the posting span,
|
||||
`first_article`, `last_article`, counts, `partial`, `finished`, `tool`,
|
||||
`candidates` jsonb (the probe targets by effort), `status` (0 waiting, 1
|
||||
named, 2 discarded), `tries`, `created_at`, `probed_at`; indexed by
|
||||
(group, tool, status, created_at) for the ladder.
|
||||
- `release_texts(release_id, kind smallint, text varchar(255))` (6200),
|
||||
primary key on all three, one trigram index (fastupdate): every string
|
||||
search matches against, appended by whoever learns one and never an
|
||||
@@ -682,10 +689,12 @@ until the window is done (LOG.md 2026-10-26 (111) has the first local
|
||||
numbers, (112) the homelab run).
|
||||
|
||||
Next: the replay on homelab over the 2016 window (the NVMe bytes per
|
||||
header for real), the ordered range leases with the carried window in
|
||||
the edge-node and its snapshot (design 1 and 5), the prober writing
|
||||
`release_texts` kind 2, random-named singles without a chunk (decision f:
|
||||
2.3 M of the first replay's 4.0 M singles are "Bob"'s 32-hex posts). Gone from the list: block
|
||||
header for real), the pending ladder and the signature learner (decision
|
||||
f, amended 2026-10-26: the prober working `pending_releases` cheapest
|
||||
first, verdicts per tool and group, a named release moved into the
|
||||
catalog with its segments read again), the ordered range leases with the
|
||||
carried window in the edge-node and its snapshot (design 1 and 5), the
|
||||
prober writing `release_texts` kind 2. Gone from the list: block
|
||||
leases, the planner and the block merger (design 1, amended).
|
||||
|
||||
## Decisions
|
||||
@@ -697,11 +706,33 @@ c. Group list: editable, owner by priority number (above). **Accepted.**
|
||||
d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
|
||||
e. Headers through the coordinator's relay, with prefetch (part 1).
|
||||
**Accepted.**
|
||||
f. Random-named releases (no file with a readable name) keep no chunk in
|
||||
the header pass: their file keys and probe candidates only, until
|
||||
phase 2 names them (bodies) and fetches their segments again. The full
|
||||
history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
|
||||
kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
|
||||
f. **Nothing enters the catalog without a usable name** (amended
|
||||
2026-10-26, the user's direction; before: random-named releases keep
|
||||
no chunk but their rows). A release the window closed on whose files
|
||||
all have random names is no release yet: it goes to `pending_releases`
|
||||
(migration 6300, the HDD) with its keys, counts, article span, upload
|
||||
tool and probe targets, no row in `releases`, `release_texts` or
|
||||
`posters`, no chunk. The pending ladder works them **from least to most
|
||||
effort**: (0) no fetch: a signature verdict (tool and group, poster)
|
||||
learned from earlier probes, a spot or NZB that names the message-ids;
|
||||
(1) a par2 main file's first article (it names the files); (2) the first
|
||||
article of the first file (the yEnc name, an archive's listing in its
|
||||
first 16 KiB); (3) a 7z's tail; then nothing more. A probe that yields
|
||||
a usable name moves the release into the catalog, its segments read
|
||||
again from the corpus (the range stored) or the server; one that
|
||||
yields none counts against its signature, and a signature with
|
||||
≥100 probes and ≤0.5% named (voyager 2's garbage learner, LOG.md
|
||||
2026-10-08 (5)) is **discarded** from then on, a sample of it still
|
||||
probed to notice a change. The reason, in the 2016 replay: one poster's
|
||||
hash-named pieces ("Bob", 11 aliases, 3.2 M releases in five small
|
||||
groups) were 76% of the rows and three quarters of the NVMe bytes for
|
||||
11% of the segments and nothing anyone can find; the full history has
|
||||
725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
|
||||
edge sends an unnamed release with an empty payload and its tool, the
|
||||
batch store writes `pending_releases` (LOG.md (114)). Not built: the
|
||||
ladder itself (the prober working `pending_releases` cheapest first,
|
||||
the signature verdicts, the move into the catalog with the segments
|
||||
read again), and the spot/NZB match on pending keys.
|
||||
g. The account's NNTP connections shared by priority (`nntp_shares`): the
|
||||
coordinator (headers) before the prober, each between its minimum and
|
||||
maximum, 5 kept for the spotter, the spot inspector, the plain
|
||||
|
||||
Reference in new issue
Block a user