Daan MeijerandClaude Opus 5.5
be8a3d4580
Spotnet (CONCEPT D29): - Index free.pt fully and import spots with their descriptions, categories and NZBs; a spot renames its release, since Spotnet names are more trustworthy than names derived from subjects. - Find passwords in spot descriptions too, and tell "needs a password we have" from "needs a password we don't have". Content inspection: - List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads and other archive-like files; mark yEncBin .bin containers as encrypted. - Take real names from yEnc headers for randomly named files, and merge uploads that were split across posts. Storage (D31, D32): - Store segment lists zstd-compressed behind the FileStore accessor. - group:backfill indexes older headers at low priority and pauses when the NVMe or HDD part of the database gets too large. Grouping (D35, D36): - Lone header segments wait in staged_segments until a second segment of their file arrives, which drops the per-article junk binaries. - Counter grouping joins random-subject uploads by poster, [i/M] counter, part count and day. Search and maintenance (D33, D34): - A trigram-indexed search_text column, smaller indexes, estimated stats. - Passes that clean up nonsense names; weekly index maintenance; GC budgets per kind. - Web UI: grey out stale results while a search runs and cancel superseded requests. Count only up to 1,000 matches for the page, which made filter clicks take seconds (Newznab keeps the 10,000 cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
43 lines
2.0 KiB
TypeScript
43 lines
2.0 KiB
TypeScript
import db from '@adonisjs/lucid/services/db'
|
|
import logger from '@adonisjs/core/services/logger'
|
|
|
|
/**
|
|
* Uploads split per poster (CONCEPT.md D35). Some uploaders post every
|
|
* volume of a RAR set under a different poster; the post-id grouping then
|
|
* makes one release per volume, all with the same name
|
|
* (`Masters.of.the.Universe.1987…`: 191 one-file releases of one 199-volume
|
|
* set). Such releases are one upload when all their files share one stem and
|
|
* no file name repeats (a repost would repeat them, another upload would
|
|
* bring another stem); their files move into the oldest of them, and GC
|
|
* deletes the emptied ones.
|
|
*/
|
|
class ReleaseMergeService {
|
|
async mergeSplitPosts({ dryRun = false } = {}): Promise<{ names: number; releases: number }> {
|
|
const { rows } = await db.rawQuery(`
|
|
WITH dup AS (
|
|
SELECT name FROM releases
|
|
WHERE file_count > 0 AND source = 'post' AND is_readable_name(name)
|
|
GROUP BY name HAVING count(*) > 1
|
|
)
|
|
SELECT r.name, array_agg(DISTINCT r.id ORDER BY r.id) AS ids
|
|
FROM dup JOIN releases r ON r.name = dup.name AND r.file_count > 0 AND r.source = 'post'
|
|
JOIN files f ON f.release_id = r.id
|
|
GROUP BY r.name
|
|
HAVING count(DISTINCT name_stem(coalesce(f.real_name, f.filename))) = 1
|
|
AND count(f.id) = count(DISTINCT coalesce(f.real_name, f.filename))
|
|
AND max(r.posted_at) - min(r.posted_at) < interval '30 days'`)
|
|
let releases = 0
|
|
for (const { name, ids } of rows as { name: string; ids: (string | number)[] }[]) {
|
|
const [target, ...others] = ids.map(Number)
|
|
releases += others.length
|
|
if (dryRun) continue
|
|
await db.from('files').whereIn('release_id', others).update({ release_id: target })
|
|
await db.from('binaries').whereIn('release_id', others).update({ release_id: target })
|
|
logger.info(`Merged ${others.length + 1} releases of one upload split per poster into ${target} "${name}".`)
|
|
}
|
|
return { names: rows.length, releases }
|
|
}
|
|
}
|
|
|
|
export default new ReleaseMergeService()
|