Pending ladder by class, twins by content, dropped classes at the edge
The prober names 6% of the unnamed releases it finishes and the pending queue (44M) grows eleven times faster than it probes. A sample of 44,530 finished pending rows shows the headers predict a name: size bucket, name shape, completeness, tool and group separate 0% classes from 95% ones, while the learner keyed on tool and group alone (LOG.md (130)). - lib/release/classes.ts, names::pending_class, SQL pending_class(): a release's class (size × shape × partial, 1..70), the three in step. - Migration 6800: pending_releases.klass and target_bytes (the first target's article; candidates carry bytes now), pending_classes with each class's climbs, names, waiting and verdict, the claim's index. - The ladder claims the classes with the best record first (a prior for the unexplored, at most a quarter of a claim from one), the cheapest target first within one, then the oldest of the rest; records classes like signatures and drops them at the same thresholds; discards their waiting releases. Claimed rows sorted in code: an UPDATE returns none. - The rule set's droppedClasses: the edge keeps unnamed releases of a dropped class out of the pending store (counted as garbage), the explored sample excepted. - hash_twins.ts: a probed file whose first 16 KiB and size match a named catalog release's names a pending release the probe didn't, and sets the inner name of a catalog release without one. - pending:classes classes the rows from before and seeds the record from the finished ones; rules:classes publishes the dropped classes; releases:readable also marks the rows' candidates unreadable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
81a5dce45a
commit
44c0c73b05
20 files changed
+927
-55
No files matched your search
@@ -9,7 +9,8 @@ const CURSOR = 'readable'
|
||||
/**
|
||||
* The readable flag of the catalog's releases by today's classifier (isUnreadableName, LOG.md (129)): rows flagged
|
||||
* readable whose name no one can read ("Zx5AjNaK-mh7CFg6Hf8uu", "Description - Lt5nHjlo3RiuBb3C") get readable =
|
||||
* false, so the readable filter hides them (an inner name from the prober still shows them). By the release name,
|
||||
* false, so the readable filter hides them (an inner name from the prober still shows them), and their probe
|
||||
* candidates marked unreadable, the prober's first tiers. By the release name,
|
||||
* which is the shared file stem for most rows; not by the chunks' file names, which the edge judges from now on.
|
||||
* Batches by id, a dry run by default; --apply writes, and goes on from its cursor when run again.
|
||||
*/
|
||||
@@ -59,10 +60,16 @@ export default class ReleasesReadable extends BaseCommand {
|
||||
}
|
||||
changed += ids.length
|
||||
if (this.apply) {
|
||||
if (ids.length > 0)
|
||||
if (ids.length > 0) {
|
||||
await db.rawQuery(`UPDATE releases SET readable = false WHERE id = ANY(?::bigint[])`, [
|
||||
ids,
|
||||
])
|
||||
// Their probe candidates to the unreadable tiers, which the prober takes first (LOG.md (130)).
|
||||
await db.rawQuery(
|
||||
`UPDATE probe_candidates SET unreadable = true WHERE release_id = ANY(?::bigint[]) AND NOT unreadable`,
|
||||
[ids]
|
||||
)
|
||||
}
|
||||
await db.rawQuery(
|
||||
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
|
||||
[CURSOR, String(to)]
|
||||
|
||||
Reference in new issue
Block a user