voyager 3: the batch store writes releases once, by id references and bytea keys
Each release chunk is a release inserted once: release_key the SHA-1 of the identity, the owner group and the first article (deterministic; no numbering under a group lock), identity_key the identity's SHA-1 bytes, poster_id and group_set_id through catalog_refs (cached per process), source as a code (weak-stem added), the name into release_texts. A finished release's candidates are claimable at once; a provisional tail release is finished by a later chunk of its key, a finished one sent again is left alone. Gone: joins into earlier releases and across groups, key numbering, the group lock, the fragment merger. Interim: chunks, counts and samples keyed by ranges.block_id, a range without a block gets one. Readers follow: search (posters and group_sets joined, words through release_texts, posters by id, source by code), the par2 merger, the sidecar folder, the judge's scraped rule, the spot inspector, the spotter, the NZB service, releases:regroup, edge:bench. Tests insert fixtures through tests/functional/helpers/releases.ts; the suites pass again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc4e17e86f
commit
2580f20a43
32 files changed
+962
-497
No files matched your search
+45
-6
@@ -531,12 +531,51 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
|
||||
rule read all of `release_inspections` every round, 17 min on the HDD;
|
||||
the judge must read it by release or keep what it needs on the NVMe.
|
||||
|
||||
Next: the batch store (write-once releases, posters and group sets by id,
|
||||
bytea keys, blocks, the name into `release_texts`), then the window and
|
||||
albums in the Rust edge, block
|
||||
leases and the block merger, the corpus replay harness. Until the batch
|
||||
store is done the 38 functional tests that write the old columns fail on
|
||||
`voyager-3`.
|
||||
### Batch store (built 2026-10-25)
|
||||
|
||||
`app/services/batch_store.ts`: every release chunk of a batch is one
|
||||
release, inserted once. Its keys (`lib/release/keys.ts`): `identity_key`
|
||||
the identity's SHA-1 as 20 bytes; `release_key` the SHA-1 of the identity,
|
||||
the owner group and the release's first article, so it is deterministic
|
||||
(no numbering under a group lock, no catalog lookups before writing) and a
|
||||
repost, starting at another article, is a release of its own. Its poster
|
||||
and group set by id (`app/services/catalog_refs.ts`: insert what's new,
|
||||
read the ids, cache them per process; the tests forget the cache per
|
||||
test), its source as a code, its name into `release_texts` (kind 1). The
|
||||
chunk's own counts go in as the release's; `complete` is decided then from
|
||||
the counter. A finished release's probe candidates are claimable at once
|
||||
(`ready`, `unreadable`); a provisional tail release (`finished: false` on
|
||||
the chunk, `settled_at` null) is finished by a later chunk of the same
|
||||
release key, which adds the files not sent before (sums, one more chunk),
|
||||
sets `settled_at` and makes its candidates claimable; a finished release
|
||||
sent again is left as it is (`ON CONFLICT … WHERE settled_at IS NULL`).
|
||||
Gone: the join of a chunk into an earlier release of its key (the window
|
||||
keeps an upload together; the block merger joins what a block's end cut),
|
||||
the join across groups (FTD's spread uploads: a gap, to be handled by the
|
||||
block merger or the par2 merger across groups), the numbering of release
|
||||
keys, the group lock, `fragments` (ignored when sent; the Rust edge still
|
||||
sends them). The chunk type carries `finished` and `edge` (both optional;
|
||||
the Rust edge doesn't send them yet, so its releases are finished, not
|
||||
edges). Interim until blocks are leased: chunks, rule counts and samples
|
||||
are keyed by `ranges.block_id`, and a range without a block gets a block
|
||||
of its own in the store (`blockOf`).
|
||||
|
||||
Readers moved with it: search joins `posters` and `group_sets`, matches
|
||||
words in `release_texts` (`r.id IN (SELECT release_id … ILIKE)`), searches
|
||||
posters by id, filters `source` by code and returns the source name; the
|
||||
par2 merger, the sidecar folder and the judge's scraped rule use
|
||||
`poster_id` (`release_merges.from_key` holds the key as hex); the spot
|
||||
inspector matches by bytea identity keys and inserts its spot releases by
|
||||
id references with their title in `release_texts`; the spotter and the NZB
|
||||
service compare the source code. Tests insert fixtures through
|
||||
`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
|
||||
and groups by name, source by name). The unit and functional suites pass.
|
||||
|
||||
Next: the window and albums in the Rust edge (`finished`, `edge`, the
|
||||
provisional tail and its finish), block leases and the planner (then the
|
||||
store keys by the leased block and the lease columns leave `ranges`), the
|
||||
block merger, the prober writing `release_texts` kind 2, the corpus replay
|
||||
harness.
|
||||
|
||||
## Decisions
|
||||
|
||||
|
||||
Reference in new issue
Block a user