Daan MeijerandClaude Opus 5.5
be8a3d4580
Spotnet (CONCEPT D29): - Index free.pt fully and import spots with their descriptions, categories and NZBs; a spot renames its release, since Spotnet names are more trustworthy than names derived from subjects. - Find passwords in spot descriptions too, and tell "needs a password we have" from "needs a password we don't have". Content inspection: - List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads and other archive-like files; mark yEncBin .bin containers as encrypted. - Take real names from yEnc headers for randomly named files, and merge uploads that were split across posts. Storage (D31, D32): - Store segment lists zstd-compressed behind the FileStore accessor. - group:backfill indexes older headers at low priority and pauses when the NVMe or HDD part of the database gets too large. Grouping (D35, D36): - Lone header segments wait in staged_segments until a second segment of their file arrives, which drops the per-article junk binaries. - Counter grouping joins random-subject uploads by poster, [i/M] counter, part count and day. Search and maintenance (D33, D34): - A trigram-indexed search_text column, smaller indexes, estimated stats. - Passes that clean up nonsense names; weekly index maintenance; GC budgets per kind. - Web UI: grey out stale results while a search runs and cancel superseded requests. Count only up to 1,000 matches for the page, which made filter clicks take seconds (Newznab keeps the 10,000 cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
39 lines
1.8 KiB
TypeScript
39 lines
1.8 KiB
TypeScript
import { test } from '@japa/runner'
|
|
import db from '@adonisjs/lucid/services/db'
|
|
import testUtils from '@adonisjs/core/services/test_utils'
|
|
import indexerConfig from '#config/indexer'
|
|
import BackfillService from '#services/BackfillService'
|
|
import FetchRangeService from '#services/FetchRangeService'
|
|
|
|
test.group('BackfillService', (group) => {
|
|
group.each.setup(() => testUtils.db().withGlobalTransaction())
|
|
|
|
test('queues header-only ranges downwards, capped, without holding back the live watermark', async ({ assert }) => {
|
|
const now = new Date()
|
|
const batch = Number(indexerConfig.batchSize)
|
|
const [g] = await db
|
|
.table('groups')
|
|
.insert({
|
|
name: 'alt.binaries.backfilltest', active: true, fetch_bodies: true, last_queued_id: 5_000_000, last_indexed_id: 5_000_000,
|
|
backfill_until_id: 5_000_000 - 100 * batch, backfill_next_id: 4_900_000, created_at: now, updated_at: now,
|
|
})
|
|
.returning('*')
|
|
|
|
const queued = await BackfillService.tick()
|
|
assert.equal(queued, indexerConfig.backfill.maxQueued)
|
|
const ranges = await db.from('fetch_ranges').where('group_id', g.id).orderBy('end_id', 'desc')
|
|
assert.isTrue(ranges.every((r) => r.backfill && r.status === 'queued'))
|
|
assert.equal(Number(ranges[0].end_id), 4_900_000)
|
|
assert.equal(Number(ranges[0].start_id), 4_900_000 - batch + 1)
|
|
const after = await db.from('groups').where('id', g.id).first()
|
|
assert.equal(Number(after.backfill_next_id), 4_900_000 - queued * batch)
|
|
|
|
// The cap holds while those wait.
|
|
assert.equal(await BackfillService.tick(), 0)
|
|
|
|
// Backfill ranges don't hold back the live watermark.
|
|
await db.from('groups').where('id', g.id).update({ last_queued_id: 5_100_000 })
|
|
assert.equal(await FetchRangeService.advanceWatermark({ id: g.id, name: g.name }), 5_100_000n)
|
|
})
|
|
})
|