Daan MeijerandClaude Opus 5.5
be8a3d4580
Spotnet (CONCEPT D29): - Index free.pt fully and import spots with their descriptions, categories and NZBs; a spot renames its release, since Spotnet names are more trustworthy than names derived from subjects. - Find passwords in spot descriptions too, and tell "needs a password we have" from "needs a password we don't have". Content inspection: - List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads and other archive-like files; mark yEncBin .bin containers as encrypted. - Take real names from yEnc headers for randomly named files, and merge uploads that were split across posts. Storage (D31, D32): - Store segment lists zstd-compressed behind the FileStore accessor. - group:backfill indexes older headers at low priority and pauses when the NVMe or HDD part of the database gets too large. Grouping (D35, D36): - Lone header segments wait in staged_segments until a second segment of their file arrives, which drops the per-article junk binaries. - Counter grouping joins random-subject uploads by poster, [i/M] counter, part count and day. Search and maintenance (D33, D34): - A trigram-indexed search_text column, smaller indexes, estimated stats. - Passes that clean up nonsense names; weekly index maintenance; GC budgets per kind. - Web UI: grey out stale results while a search runs and cancel superseded requests. Count only up to 1,000 matches for the page, which made filter clicks take seconds (Newznab keeps the 10,000 cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
48 lines
2.4 KiB
TypeScript
48 lines
2.4 KiB
TypeScript
import { createHash } from 'node:crypto'
|
|
|
|
/**
|
|
* Identity of one uploaded file: every segment of it has the same poster, the
|
|
* same subject apart from the segment counter, and the same segment count.
|
|
* The group is deliberately left out so crossposts resolve to one file.
|
|
* See CONCEPT.md (D12).
|
|
*/
|
|
export function fileKey(poster: string, base: string, totalSegments: number): string {
|
|
return createHash('sha1').update(`${poster}\n${base}\n${totalSegments}`).digest('hex')
|
|
}
|
|
|
|
/** True when a yEnc name looks like a real filename (it has an extension), not a random token. */
|
|
export function looksLikeFilename(name: string): boolean {
|
|
return /\.[A-Za-z0-9]{1,5}$/.test(name) && !/^[0-9a-f]{32,}$/i.test(name)
|
|
}
|
|
|
|
/**
|
|
* Identity of a file known only from its yEnc header (obfuscated posts).
|
|
* The poster is left out: obfuscating posters randomise it per article.
|
|
* - A real filename is stable across the articles of a file, and needed to
|
|
* tell apart the equal-sized volumes of one archive.
|
|
* - A random, extensionless name can change per article too; the exact
|
|
* byte size and part count are what stays the same.
|
|
* Collisions are caught by the conflict check in IngestService.
|
|
*/
|
|
export function weakFileKey(name: string, size: number, totalParts: number): string {
|
|
const identity = looksLikeFilename(name) ? `name\n${name}` : 'shape'
|
|
return createHash('sha1').update(`weak\n${identity}\n${size}\n${totalParts}`).digest('hex')
|
|
}
|
|
|
|
/**
|
|
* Identity of a file by its counters, for uploads that give every article
|
|
* its own random subject (CONCEPT.md D36): poster, file counter `[i/M]`,
|
|
* part count and posting day stay the same across the articles of one file.
|
|
* The day keeps different uploads with the same shape apart; collisions that
|
|
* remain are caught by the conflict check (these files are weak identities).
|
|
* Null when the subject has no file counter.
|
|
*/
|
|
export function counterFileKey(segment: { poster: string; subject: string; totalParts: number; postedAt?: Date }): string | null {
|
|
const counter = /\[(\d+)\/(\d+)\]/.exec(segment.subject)
|
|
if (!counter || segment.totalParts < 2) return null
|
|
const day = segment.postedAt && !Number.isNaN(segment.postedAt.getTime()) ? segment.postedAt.toISOString().slice(0, 10) : ''
|
|
return createHash('sha1')
|
|
.update(`counter\n${segment.poster}\n${Number(counter[1])}/${Number(counter[2])}\n${segment.totalParts}\n${day}`)
|
|
.digest('hex')
|
|
}
|