Rebuild: the batch store (catalog upserts, write-once chunks, candidates, rule counts, fragments, resume point)
A batch is stored in one transaction and only at its range's resume point, so a retried batch can't count twice. Garbage is judged before per-article: a dropped tool isn't worth reading bodies for either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0714b2c81a
commit
20910647fa
6 files changed
+242
-7
No files matched your search
+5
-2
@@ -65,8 +65,11 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
|
||||
|
||||
3. **Chunk store** (bulk, append-only). One record per release per batch:
|
||||
its files' names, sizes and segment lists, compactly encoded (template
|
||||
codec for message-ids, zstd). Written once, never updated; a re-run of a
|
||||
range replaces that range's chunks (no per-file dedup). ~3 TB.
|
||||
codec for message-ids, zstd). Written once, never updated. A batch is
|
||||
stored in one transaction and only when it starts at its range's resume
|
||||
point, so a batch sent twice or late is refused and the catalog's sums
|
||||
stay right (no per-file dedup); a recovery run for a retracted rule only
|
||||
adds what that rule had dropped. ~3 TB.
|
||||
|
||||
4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
|
||||
name, poster, groups, posted_at, file count, size, segments, expected
|
||||
|
||||
Reference in new issue
Block a user