Pending ladder by class, twins by content, dropped classes at the edge
Test and deploy / test (push) Successful in 3m9s
Test and deploy / deploy (push) Successful in 40m8s

The prober names 6% of the unnamed releases it finishes and the pending
queue (44M) grows eleven times faster than it probes. A sample of 44,530
finished pending rows shows the headers predict a name: size bucket, name
shape, completeness, tool and group separate 0% classes from 95% ones,
while the learner keyed on tool and group alone (LOG.md (130)).

- lib/release/classes.ts, names::pending_class, SQL pending_class():
  a release's class (size × shape × partial, 1..70), the three in step.
- Migration 6800: pending_releases.klass and target_bytes (the first
  target's article; candidates carry bytes now), pending_classes with
  each class's climbs, names, waiting and verdict, the claim's index.
- The ladder claims the classes with the best record first (a prior for
  the unexplored, at most a quarter of a claim from one), the cheapest
  target first within one, then the oldest of the rest; records classes
  like signatures and drops them at the same thresholds; discards their
  waiting releases. Claimed rows sorted in code: an UPDATE returns none.
- The rule set's droppedClasses: the edge keeps unnamed releases of a
  dropped class out of the pending store (counted as garbage), the
  explored sample excepted.
- hash_twins.ts: a probed file whose first 16 KiB and size match a named
  catalog release's names a pending release the probe didn't, and sets
  the inner name of a catalog release without one.
- pending:classes classes the rows from before and seeds the record from
  the finished ones; rules:classes publishes the dropped classes;
  releases:readable also marks the rows' candidates unreadable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-11-02 12:22:32 +01:00
1 parent 81a5dce45a
commit 44c0c73b05
20 files changed
+927 -55

No files matched your search

+12 -1
View File
@@ -9,6 +9,7 @@ import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/i
import { NntpShare } from '#services/nntp_shares'
import { DiskSpace } from '#services/disk_space'
import { decidePar2Completeness } from '#services/par2_completeness'
import { twinKey, twinNames } from '#services/hash_twins'
import PendingLadder, { type PendingProbe } from '#services/pending_ladder'
import { PendingNamer } from '#services/pending_namer'
@@ -392,10 +393,20 @@ export class Prober {
entry.all.push({ kind: r.kind, result: r.result })
byRelease.set(id, entry)
}
// A twin by content names a release its own inspections didn't (hash_twins.ts; LOG.md (130)).
const twins = await twinNames(
done.filter((o) => o.inspection!.hash16k && o.inspection!.size).map((o) => ({ hash16k: o.inspection!.hash16k!, size: o.inspection!.size })),
trx
)
const twinOf = new Map<number, string>()
for (const o of done) {
const twin = o.inspection!.hash16k ? twins.get(twinKey(o.inspection!.hash16k, o.inspection!.size)) : undefined
if (twin && twin !== byRelease.get(o.candidate.releaseId)?.name) twinOf.set(o.candidate.releaseId, twin)
}
const updates = [...byRelease].map(([id, { name, all }]) => ({
id,
passworded: all.some((i) => i.result.passworded === true),
inner_name: innerName(all, name),
inner_name: innerName(all, name) ?? twinOf.get(id) ?? null,
}))
await trx.rawQuery(
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()