Voyager 3 schema: the slim catalog (migration 6000, the reset) and blocks (6100)
Test and deploy / test (push) Failing after 50s
Test and deploy / edge (push) Successful in 1m53s
Test and deploy / deploy (push) Skipped

releases rebuilt without its text copies: poster_id, group_set_id,
bytea keys, source smallint, edge, settled_at as the finish. posters by
id, group_sets. blocks as the lease unit, ranges.block_id. fragments,
range_blocks, release_final, the settler and the fragment merger gone;
release_chunks, rule_counts and rule_samples keyed by block. Cold tables
on CHUNK_TABLESPACE. Migration 6000 truncates the catalog and refuses
without VOYAGER3_RESET=1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-25 13:44:54 +01:00
1 parent 1a448f37f9
commit 346493b6c3
17 files changed
+527 -545

No files matched your search

-49
View File
@@ -1,49 +0,0 @@
import db from '@adonisjs/lucid/services/db'
/** Releases unchanged for this long are settled (ranges are read in order; the merger adds within minutes). */
const QUIET_HOURS = 6
/**
* History: a release whose last post is QUIET_HOURS old gets nothing new from the tail, only from the
* neighbouring ranges' fragments, which the merger adds within minutes: settled once unchanged this long.
* Waiting QUIET_HOURS after it was read held every history release back from phase 2 for six hours after
* each reset (the hourly sweeps' resets kept the prober idle; LOG.md 2026-10-12 (13)).
*/
const QUIET_HISTORY_MINUTES = 30
const BATCH = 50_000
/**
* Settles releases (REBUILD.md, part 8): once nothing has been added for
* QUIET_HOURS (QUIET_HISTORY_MINUTES once its last post is that old), its completeness is decided: complete when every file the
* [i/N] counter announced was found, unknown without a counter (phase 2
* decides those by their par2 sets: app/services/par2_completeness.ts).
* Phase 2 picks its work from settled releases.
*/
class Settler {
/** Up to `batches` batches of BATCH (all there are without). */
async run(batches = Infinity): Promise<number> {
let settled = 0
for (let batch = 0; batch < batches; batch++) {
// Their probe candidates become claimable (Prober.claim), the unreadable ones first.
const { rows } = await db.rawQuery(
`WITH settled AS (
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
RETURNING id, readable, files, bytes, poster, sidecar_only),
marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
-- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
-- from it: the merger's sidecar fold looks at them.
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
ON CONFLICT DO NOTHING)
SELECT count(*)::int AS n FROM settled`,
[QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
)
const n = rows[0].n
settled += n
if (n < BATCH) return settled
}
return settled
}
}
export default new Settler()