Voyager 3 schema: the slim catalog (migration 6000, the reset) and blocks (6100)
releases rebuilt without its text copies: poster_id, group_set_id, bytea keys, source smallint, edge, settled_at as the finish. posters by id, group_sets. blocks as the lease unit, ranges.block_id. fragments, range_blocks, release_final, the settler and the fragment merger gone; release_chunks, rule_counts and rule_samples keyed by block. Cold tables on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses without VOYAGER3_RESET=1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
1a448f37f9
commit
346493b6c3
17 files changed
+527
-545
No files matched your search
+49
@@ -472,6 +472,55 @@ releases, a third of them one poster's image dump (27.7 M rows, LOG.md
|
||||
verdicts, file hashes. Resets are paused (2026-10-12); this one waits
|
||||
for the user's go-ahead, and for the new NVMe.
|
||||
|
||||
### Schema (built 2026-10-25; migrations 6000 and 6100)
|
||||
|
||||
Migration 6000 *is* the reset: it truncates every table the indexing
|
||||
recomputes and refuses to run on a catalog with releases unless
|
||||
`VOYAGER3_RESET=1`, so merging `voyager-3` into `rebuild` is the go-ahead.
|
||||
Survives: `groups`, `ranges` (planned again from their start, with their
|
||||
priorities, `capture`, `est_date`, `stored_at`, `plain_chunks` and measured
|
||||
`posted_low/high`), the corpus, `probe_cache`, `spots` (unlinked from
|
||||
releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
||||
`users`, the rule sets.
|
||||
|
||||
- `releases`: created anew, 8-byte columns first: `id`, `posted_at`,
|
||||
`last_posted_at`, `first_article`, `last_article`, `segments`, `bytes`,
|
||||
`settled_at` (the finish; null while a tail release is provisional),
|
||||
`inspected_at`, `first_seen_at`; `group_id`, `group_set_id`, `poster_id`,
|
||||
`file_total`, `files`, `chunks`, `category`; `source smallint`
|
||||
(`lib/release/source.ts`: 1 subject, 2 post, 3 stem, 4 spot, 5 nzb, 6
|
||||
album); `readable`, `has_par2`, `complete`, `passworded`, `sidecar_only`,
|
||||
`edge` (open at its block's end); `release_key` and `identity_key` as
|
||||
20-byte `bytea`; `name`, `inner_name`. Gone: `poster`, `groups`,
|
||||
`last_seen_at`. Indexes: pkey, release_key unique, (identity_key,
|
||||
group_id), name and inner-name trigram, newest, (poster_id, newest),
|
||||
passworded, `(group_id, identity_key) WHERE edge` for the block merger,
|
||||
`id WHERE settled_at IS NULL` for the provisional ones. The category
|
||||
trigger reads the groups from `group_sets`; the judge's dirty trigger
|
||||
watches `poster_id` instead of `poster`.
|
||||
- `posters(id serial, poster unique)` with its trigram index;
|
||||
`group_sets(id serial, groups jsonb unique)`, a sorted array of names.
|
||||
- `blocks` (6100): `group_id`, `start_id`, `end_id`, `next_id` (the resume
|
||||
point), `posted_low/high`, `status`, `priority`, `node_id`, `attempts`,
|
||||
`rule_version`, `headers`, `releases`, `leased_at`, `heartbeat_at`,
|
||||
`done_at`, `error`; indexes todo (priority, id) and leased.
|
||||
`ranges.block_id` points at the block; the ranges' own lease columns stay
|
||||
until the leases move (then a migration drops them).
|
||||
- `release_chunks`, `rule_counts`, `rule_samples`: `range_id` → `block_id`.
|
||||
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
|
||||
`settled_at IS NOT NULL`), the settler, the fragment merger,
|
||||
`releases_category_newest_idx`.
|
||||
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
|
||||
HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
|
||||
rule read all of `release_inspections` every round, 17 min on the HDD;
|
||||
the judge must read it by release or keep what it needs on the NVMe.
|
||||
|
||||
Next: the batch store (write-once releases, posters and group sets by id,
|
||||
bytea keys, blocks), then the window and albums in the Rust edge, block
|
||||
leases and the block merger, the corpus replay harness. Until the batch
|
||||
store is done the 38 functional tests that write the old columns fail on
|
||||
`voyager-3`.
|
||||
|
||||
## Decisions
|
||||
|
||||
a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
|
||||
|
||||
Reference in new issue
Block a user