Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 1322e42fb7 Accounts with limits: a Newznab key and a tier per user, daily API and NZB counts, invites
Test and deploy / test (push) Successful in 47s
Test and deploy / edge (push) Successful in 1m44s
Test and deploy / deploy (push) Skipped
ACCESS=accounts (config/access.ts; open stays the default and behaves as
before): Newznab and the catalog's releases routes name a user (apikey or
the web interface's token), whose tier sets searches and NZBs per UTC day
(Newznab 500/501, the catalog 429) and how far back they see (102/403).
Counted in one upsert per use, so the last one can't be taken twice;
caps, stats and refused requests aren't counted. Signup takes an invite
(SIGNUP), which may give a tier; ace access:invite, access:tier (extends a
running tier), access:suspend. The web interface signs in and shows the
day's usage and the Newznab key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:16:45 +02:00
65 changed files with 1283 additions and 1765 deletions

No files matched your search

+5
View File
@@ -19,6 +19,11 @@ DB_USER=usenet_indexer
DB_PASSWORD=
DB_DATABASE=usenet_rebuild
# Access (config/access.ts): open (NEWZNAB_API_KEY or none, no limits) or accounts
# (a Newznab key per user, tiers with daily limits; ace access:invite, access:tier, access:suspend)
# ACCESS=open
# SIGNUP=open|invite|closed (with accounts: invite unless set)
#--------------------------------------------------------------------
# CORS (configure allowed origins for API access)
#--------------------------------------------------------------------
+4 -29
View File
@@ -238,14 +238,6 @@ files the set describes, into the one with the most files (chunks,
candidates, inspections, counts; the par2's shared name when none was
readable); `release_merges` keeps what went where. Obfuscated uploads whose
files were each a release of their own (`lone` candidates) become one.
It reads what's new an hour (`created_at`) at a time, its place kept after
each, and merges only the sets held or matched by two releases or more; a
file whose first 16 KiB and size more than 20 files share is a common one and
matches nothing (LOG.md 2026-10-20 (85)).
A set's file list is dropped once it can't merge anything more: complete (all
its files in its holder), judged (every holder junk), or final (every holder
inside a run of done ranges, those ranges' candidates probed, its hashes seen
by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).
**Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
@@ -260,7 +252,7 @@ with 68,032 releases with a password):
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed; one release keeps only the tails of its highest volume (the live tail's pieces each queued one; `seven_zip_tails`) |
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
| Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
| Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |
@@ -268,11 +260,9 @@ Live indexing (the tail) runs since 2026-10-09.
**Junk** (voyager 1's second filter, D42). After the header pass's rules
(garbage signatures, executables, per-article and obfuscated subjects), the
judge (`releases:judge`, service `judge`) judges releases by what the
prober, the merges and the spots found: every 2 minutes the releases whose
facts changed (noted by triggers in `judge_dirty`), daily every release (for
what time changes: a day after settling, ranges read beside it). In this
order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
judge (`releases:judge`, service `judge`) judges releases hourly by what the
prober, the merges and the spots found, in this order (the first rule
wins): *dead* (every probed first article gone), *fake* (a film, episode
or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
@@ -336,18 +326,3 @@ f. Random-named releases (no file with a readable name) keep no chunk in
phase 2 names them (bodies) and fetches their segments again. The full
history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
g. The account's NNTP connections shared by priority (`nntp_shares`): the
coordinator (headers) before the prober, each between its minimum and
maximum, 5 kept for the spotter, the spot inspector, the plain
connection and reconnects. A process grows only into connections the
others closed, so the account never goes over its limit. Replaces the
fixed split (38 + 8), under which the prober stayed at 8 while history
was paused and the coordinator's connections idle (LOG.md 2026-10-18
(75)). **Accepted** 2026-10-18.
h. Idle NNTP connections closed: one with nothing to do for
`NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
(the coordinator's pool on its next sync, the plain one too; a dry
prober lane; the spotter between rounds; the spot inspector with no
spots). Eweka dropped the quiet ones and the next command failed
("Spot inspection failed: NNTP connection closed"), and idle ones took
the account's connections from downloads. **Accepted** 2026-10-22.
+26 -3
View File
@@ -1,13 +1,36 @@
import User from '#models/user'
import db from '@adonisjs/lucid/services/db'
import { signupValidator } from '#validators/user'
import type { HttpContext } from '@adonisjs/core/http'
import UserTransformer from '#transformers/user_transformer'
import AccessService from '#services/access_service'
class InviteGone extends Error {}
export default class NewAccountController {
async store({ request, serialize }: HttpContext) {
const { fullName, email, password } = await request.validateUsing(signupValidator)
async store({ request, response, serialize }: HttpContext) {
const { fullName, email, password, invite } = await request.validateUsing(signupValidator)
if (AccessService.signup === 'closed') return response.forbidden({ error: 'Signup is closed' })
if (AccessService.signup === 'invite' && !invite) return response.forbidden({ error: 'An invite code is needed to sign up' })
const user = await User.create({ fullName, email, password })
// The user and their invite together: an invite that's gone rolls the signup back.
const user = await db
.transaction(async (trx) => {
const created = await User.create({ fullName, email, password }, { client: trx })
if (AccessService.signup !== 'invite') return created
const gives = await AccessService.redeem(invite!, created.id, trx)
if (!gives) throw new InviteGone()
if (gives.tier) {
AccessService.grantTier(created, gives.tier, gives.tier_days)
await created.useTransaction(trx).save()
}
return created
})
.catch((error) => {
if (error instanceof InviteGone) return null
throw error
})
if (!user) return response.forbidden({ error: 'Unknown or used invite code' })
const token = await User.accessTokens.create(user)
return serialize({
+38 -13
View File
@@ -2,6 +2,7 @@ import type { HttpContext } from '@adonisjs/core/http'
import env from '#start/env'
import NzbService from '#services/nzb_service'
import SearchService from '#services/search_service'
import AccessService, { AccessDenied } from '#services/access_service'
import { capsXml, errorXml, expandCategories, resultsXml, type NewznabServer } from '#lib/newznab/index'
const LIMITS = { max: 100, default: 50 }
@@ -35,22 +36,37 @@ function int(value: unknown): number | undefined {
* Releases with readable names (or an inner name from the prober, which then is the title) that aren't known to miss files.
*/
export default class NewznabController {
async handle({ request, response }: HttpContext) {
async handle(ctx: HttpContext) {
const xml = (body: string, type = 'application/xml') => {
ctx.response.header('Content-Type', `${type}; charset=utf-8`)
return ctx.response.send(body)
}
try {
return await this.answer(ctx, xml)
} catch (error) {
if (error instanceof AccessDenied) return xml(errorXml(error.newznabCode, error.message))
throw error
}
}
private async answer(ctx: HttpContext, xml: (body: string, type?: string) => void) {
const { request, response } = ctx
const qs = request.qs()
const t = String(qs.t ?? '')
const server = this.server(request)
const xml = (body: string, type = 'application/xml') => {
response.header('Content-Type', `${type}; charset=utf-8`)
return response.send(body)
}
const apiKey = configuredApiKey()
if (apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
if (t === 'caps') return xml(capsXml(server, LIMITS))
// With accounts the key is a user's (AccessService); else the one configured, if any.
const grant = await AccessService.authorize(ctx, qs.apikey)
const apiKey = grant ? grant.user.apiKey : configuredApiKey()
if (!grant && apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
const server = this.server(request, apiKey)
if (t === 'caps') return xml(capsXml(server, LIMITS, { available: AccessService.accounts && AccessService.signup !== 'closed', open: AccessService.accounts && AccessService.signup === 'open' }))
if (t === 'get') {
const id = int(qs.id)
if (id === undefined) return xml(errorXml(200, 'Missing parameter (id)'))
// Unknown ids and releases past the tier's retention aren't counted.
if (!(await AccessService.checkRelease(grant, id))) return xml(errorXml(300, 'No such item'))
await AccessService.charge(grant, 'grab')
const nzb = await NzbService.build(id)
if (!nzb) return xml(errorXml(300, 'No such item'))
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
@@ -59,6 +75,7 @@ export default class NewznabController {
}
if (!(t in SEARCH_FUNCTIONS)) return xml(errorXml(202, `No such function (${t || 'none'})`))
await AccessService.charge(grant, 'api')
const requested = String(qs.cat ?? '')
.split(',')
@@ -79,7 +96,7 @@ export default class NewznabController {
categories,
readable: true,
notIncomplete: true,
maxAgeDays: int(qs.maxage),
maxAgeDays: AccessService.maxAgeDays(grant, int(qs.maxage)),
season: int(qs.season),
episode: int(qs.season) !== undefined ? int(qs.ep) : undefined,
sort: 'newest',
@@ -105,12 +122,20 @@ export default class NewznabController {
episode: tv ? `E${tv[2].padStart(2, '0')}` : undefined,
}
})
return xml(resultsXml(server, items, offset, result?.total ?? 0), 'application/rss+xml')
// What's left of the day, for clients that show it (newznab:apilimits).
const limits = grant && {
apiCurrent: grant.usage.apiHits,
apiMax: grant.limits.apiHits,
grabCurrent: grant.usage.grabs,
grabMax: grant.limits.grabs,
}
return xml(resultsXml(server, items, offset, result?.total ?? 0, limits || undefined), 'application/rss+xml')
}
private server(request: HttpContext['request']): NewznabServer {
/** Download links carry the caller's key. */
private server(request: HttpContext['request'], apiKey: string | undefined): NewznabServer {
const proto = request.header('x-forwarded-proto') ?? 'http'
const host = request.header('x-forwarded-host') ?? request.header('host') ?? 'localhost'
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey: configuredApiKey() }
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey }
}
}
+12 -1
View File
@@ -1,8 +1,19 @@
import UserTransformer from '#transformers/user_transformer'
import type { HttpContext } from '@adonisjs/core/http'
import AccessService from '#services/access_service'
export default class ProfileController {
/** The user, their tier, Newznab key and what they used today. */
async show({ auth, serialize }: HttpContext) {
return serialize(UserTransformer.transform(auth.use('api').getUserOrFail()))
const user = auth.use('api').getUserOrFail()
return serialize({ user: UserTransformer.transform(user), access: await AccessService.summary(user) })
}
/** POST /api/v1/account/api-key: a new Newznab key (the old one stops working). */
async resetApiKey({ auth }: HttpContext) {
const user = auth.use('api').getUserOrFail()
user.apiKey = AccessService.newKey()
await user.save()
return { apiKey: user.apiKey }
}
}
+24 -8
View File
@@ -1,5 +1,6 @@
import type { HttpContext } from '@adonisjs/core/http'
import NzbService from '#services/nzb_service'
import AccessService from '#services/access_service'
import SearchService, { type SortOrder } from '#services/search_service'
import { expandCategories } from '#lib/newznab/index'
@@ -14,13 +15,16 @@ function megabytes(value: unknown): number | undefined {
/** The catalog for the web interface (REBUILD.md, part 9). */
export default class ReleasesController {
/** GET /api/v1/releases?q=&cat=5000,2040&readable=1&complete=1&unlocked=1&min_size=&max_size=(MB)&source=&poster=&sort=&page=&per_page= */
async index({ request }: HttpContext) {
const qs = request.qs()
async index(ctx: HttpContext) {
const qs = ctx.request.qs()
const grant = await AccessService.authorize(ctx, qs.apikey)
await AccessService.charge(grant, 'api')
const categories = String(qs.cat ?? '')
.split(',')
.map((c) => Number(c.trim()))
.filter((c) => Number.isInteger(c) && c > 0)
return SearchService.search({
maxAgeDays: AccessService.maxAgeDays(grant),
q: typeof qs.q === 'string' ? qs.q : undefined,
categories: categories.length ? expandCategories(categories) : undefined,
readable: flag(qs.readable),
@@ -39,13 +43,20 @@ export default class ReleasesController {
}
/** GET /api/v1/releases/:id, with its files */
async show({ params, response }: HttpContext) {
const release = await SearchService.release(Number(params.id))
return release ?? response.notFound({ error: 'Release not found' })
async show(ctx: HttpContext) {
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
if (!(await AccessService.checkRelease(grant, Number(ctx.params.id)))) return ctx.response.notFound({ error: 'Release not found' })
await AccessService.charge(grant, 'api')
const release = await SearchService.release(Number(ctx.params.id))
return release ?? ctx.response.notFound({ error: 'Release not found' })
}
/** GET /api/v1/releases/:id/nzb */
async nzb({ params, response }: HttpContext) {
/** GET /api/v1/releases/:id/nzb (?apikey= for a plain link, as the web interface's) */
async nzb(ctx: HttpContext) {
const { params, response } = ctx
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
if (!(await AccessService.checkRelease(grant, Number(params.id)))) return response.notFound({ error: 'Release not found' })
await AccessService.charge(grant, 'grab')
const nzb = await NzbService.build(Number(params.id))
if (!nzb) return response.notFound({ error: 'Release not found' })
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
@@ -54,7 +65,12 @@ export default class ReleasesController {
return response.send(nzb.xml)
}
/** GET /api/v1/stats */
/** GET /api/v1/access: whether the web interface needs an account, and how one is made. */
async access() {
return { mode: AccessService.mode, signup: AccessService.signup }
}
/** GET /api/v1/stats (open: what the index holds, not what's in it) */
async stats() {
return SearchService.stats()
}
+3
View File
@@ -8,6 +8,9 @@ export default class HttpExceptionHandler extends ExceptionHandler {
*/
protected debug = !app.inProduction
/** A refused request (AccessService: no key, a limit reached) is an answer, not a fault. */
protected ignoreCodes = ['E_ACCESS_DENIED']
/**
* The method is used for handling errors and returning
* response to the client
+8
View File
@@ -1,4 +1,6 @@
import { randomBytes } from 'node:crypto'
import { UserSchema } from '#database/schema'
import { beforeCreate } from '@adonisjs/lucid/orm'
import hash from '@adonisjs/core/services/hash'
import { compose } from '@adonisjs/core/helpers'
import { withAuthFinder } from '@adonisjs/auth/mixins/lucid'
@@ -8,6 +10,12 @@ export default class User extends compose(UserSchema, withAuthFinder(hash)) {
static accessTokens = DbAccessTokensProvider.forModel(User)
declare currentAccessToken?: AccessToken
/** The Newznab key (AccessService), known on the model right away rather than from the column's default. */
@beforeCreate()
static assignApiKey(user: User) {
user.apiKey ??= randomBytes(16).toString('hex')
}
get initials() {
const [first, last] = this.fullName ? this.fullName.split(' ') : this.email.split('@')
if (first && last) {
+171
View File
@@ -0,0 +1,171 @@
import { randomBytes } from 'node:crypto'
import { DateTime } from 'luxon'
import db from '@adonisjs/lucid/services/db'
import { Exception } from '@adonisjs/core/exceptions'
import type { HttpContext } from '@adonisjs/core/http'
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
import User from '#models/user'
import { ACCESS, SIGNUP, TIERS, type TierLimits } from '#config/access'
export type Use = 'api' | 'grab'
/** Newznab's error codes for access (the catalog answers with the HTTP status). */
const STATUS = { 100: 401, 101: 403, 102: 403, 500: 429, 501: 429 } as const
/** Why a request is refused; the catalog's answer (Newznab catches it for its own). */
export class AccessDenied extends Exception {
constructor(
readonly newznabCode: keyof typeof STATUS,
message: string
) {
super(message, { status: STATUS[newznabCode], code: 'E_ACCESS_DENIED' })
}
async handle(error: this, ctx: HttpContext) {
ctx.response.status(error.status).send({ error: error.message, code: error.newznabCode })
}
}
export interface Usage {
apiHits: number
grabs: number
}
/** A caller let in: who, their tier as it is now, and what they used today. */
export interface Grant {
user: User
tier: string
limits: TierLimits
usage: Usage
}
const COLUMNS: Record<Use, string> = { api: 'api_hits', grab: 'grabs' }
const TODAY = `(now() AT TIME ZONE 'UTC')::date`
const DAY_MS = 86_400_000
/** Accounts, tiers and their daily limits (config/access.ts). */
export default class AccessService {
/** From config/access.ts; tests switch it. */
static mode: 'open' | 'accounts' = ACCESS
static signup: 'open' | 'invite' | 'closed' = SIGNUP
static get accounts() {
return this.mode === 'accounts'
}
static newKey() {
return randomBytes(16).toString('hex')
}
/** The user's tier as it is now: one that ran out (or isn't configured) is free. */
static tierOf(user: User): { tier: string; limits: TierLimits } {
const expired = user.tierUntil !== null && user.tierUntil.toMillis() <= Date.now()
const tier = !expired && TIERS[user.tier] ? user.tier : 'free'
return { tier, limits: TIERS[tier] }
}
/** The user a request names: the web interface's token, else an apikey (Newznab, the catalog's NZB links). */
static async caller(ctx: HttpContext, apikey?: unknown): Promise<User | null> {
const guard = ctx.auth.use('api')
if (guard.isAuthenticated || (await guard.check())) return guard.user!
if (typeof apikey === 'string' && /^[0-9a-f]{32}$/.test(apikey)) return User.findBy('apiKey', apikey)
return null
}
/** Lets a request in (counting nothing yet); null when access is open. */
static async authorize(ctx: HttpContext, apikey?: unknown): Promise<Grant | null> {
if (!this.accounts) return null
const user = await this.caller(ctx, apikey)
if (!user) throw new AccessDenied(100, 'Incorrect user credentials')
if (user.suspended) throw new AccessDenied(101, 'Account suspended')
return { user, ...this.tierOf(user), usage: await this.usage(user.id) }
}
/** Counts one use against the grant's tier, or refuses it once the day's limit is reached. */
static async charge(grant: Grant | null, use: Use) {
if (!grant) return
const usage = await this.count(grant.user.id, use, use === 'api' ? grant.limits.apiHits : grant.limits.grabs)
if (!usage) throw use === 'api' ? new AccessDenied(500, 'Request limit reached') : new AccessDenied(501, 'Download limit reached')
grant.usage = usage
}
/**
* Adds one use to today's row unless the limit is reached (null then): one
* statement, so concurrent requests can't both take the last one.
*/
static async count(userId: number, use: Use, limit: number | null): Promise<Usage | null> {
if (limit !== null && limit <= 0) return null
const column = COLUMNS[use]
const { rows } = await db.rawQuery(
`INSERT INTO usage (user_id, day, ${column}) VALUES (?, ${TODAY}, 1)
ON CONFLICT (user_id, day) DO UPDATE SET ${column} = usage.${column} + 1
${limit === null ? '' : `WHERE usage.${column} < ?`}
RETURNING api_hits, grabs`,
limit === null ? [userId] : [userId, limit]
)
return rows[0] ? { apiHits: rows[0].api_hits, grabs: rows[0].grabs } : null
}
static async usage(userId: number): Promise<Usage> {
const { rows } = await db.rawQuery(`SELECT api_hits, grabs FROM usage WHERE user_id = ? AND day = ${TODAY}`, [userId])
return { apiHits: rows[0]?.api_hits ?? 0, grabs: rows[0]?.grabs ?? 0 }
}
/** A search's age bound: the asked one within the tier's retention. */
static maxAgeDays(grant: Grant | null, requested?: number): number | undefined {
const retention = grant?.limits.retentionDays
if (!retention) return requested
return requested ? Math.min(requested, retention) : retention
}
/** Refuses a release posted before the tier's retention (one without a date is let through). */
static checkPosted(grant: Grant | null, postedAt: Date | string | null) {
const retention = grant?.limits.retentionDays
if (!retention || !postedAt) return
if (new Date(postedAt).getTime() < Date.now() - retention * DAY_MS) {
throw new AccessDenied(102, `Older than your tier's retention (${retention} days)`)
}
}
/** Whether a release exists, refusing it when it's older than the tier sees; asked before an NZB is counted. */
static async checkRelease(grant: Grant | null, id: number): Promise<boolean> {
const release = await db.from('releases').where('id', id).select('posted_at').first()
if (!release) return false
this.checkPosted(grant, release.posted_at)
return true
}
/** Takes an unused invite for a new user: what it gives, or null when it's unknown or used. */
static async redeem(code: string, userId: number, trx: TransactionClientContract) {
const { rows } = await trx.rawQuery(
`UPDATE invites SET used_by = ?, used_at = now() WHERE code = ? AND used_by IS NULL AND used_at IS NULL RETURNING tier, tier_days`,
[userId, code]
)
return (rows[0] as { tier: string | null; tier_days: number | null } | undefined) ?? null
}
/**
* Gives a user a tier for `days` (null: without end). The same tier still
* running is extended from where it ends, so a renewal paid early loses nothing.
*/
static grantTier(user: User, tier: string, days: number | null) {
if (!TIERS[tier]) throw new Error(`No such tier: ${tier} (config/access.ts has ${Object.keys(TIERS).join(', ')})`)
const now = DateTime.utc()
const running = user.tier === tier && user.tierUntil && user.tierUntil > now ? user.tierUntil : now
user.tier = tier
user.tierUntil = days === null ? null : running.plus({ days })
}
/** What a profile shows of the account's access. */
static async summary(user: User) {
const { tier, limits } = this.tierOf(user)
return {
mode: this.mode,
tier,
tierUntil: tier === user.tier ? user.tierUntil : null,
apiKey: user.apiKey,
limits,
usage: await this.usage(user.id),
}
}
}
+6 -38
View File
@@ -1,20 +1,8 @@
import db from '@adonisjs/lucid/services/db'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
import ChunkPartitions from '#services/chunk_partitions'
import { decodeChunk } from '#lib/chunks/index'
import { SIDECAR } from '#services/sidecar_folder'
/**
* A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
* fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
*/
function sidecarOnly(r: ReleaseChunk): boolean | null {
if (r.files > 3 || r.bytes >= 1_000_000) return null
const files = decodeChunk(r.payload)
return files.length > 0 && files.every((f) => SIDECAR.test(f.name))
}
export class StaleBatch extends Error {}
@@ -247,21 +235,13 @@ class BatchStore {
}
const fresh = part.filter((_, idx) => !joined.has(idx))
if (fresh.length > 0) {
// The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
// delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
// taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
const { rows: counts } = await trx.rawQuery(
`SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
THEN substr(release_key, length(identity_key) + 2)::int END) AS n
FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
[[...new Set(fresh.map((r) => r.key))]]
)
// The first upload of a key keeps the key; later ones are key:2, key:3, …
const { rows: counts } = await trx.rawQuery(`SELECT identity_key, count(*) AS n FROM releases WHERE identity_key = ANY(?) GROUP BY 1`, [[...new Set(fresh.map((r) => r.key))]])
const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
const numbered = (r: ReleaseChunk) => {
const n = (taken.get(r.key) ?? 0) + 1
taken.set(r.key, n)
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r) }
}
// Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
@@ -272,10 +252,10 @@ class BatchStore {
for (let attempt = 0; inserts.length > 0; attempt++) {
if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
const { rows } = await trx.rawQuery(
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete)
SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)
ON CONFLICT (release_key) DO NOTHING
RETURNING id, release_key`,
[groupId, JSON.stringify(inserts)]
@@ -325,18 +305,6 @@ class BatchStore {
.ignore()
.returning('id')
out.candidates += added.length
// A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
// Files added to a settled release (a later range of its upload): the settler made its candidates claimable
// already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
if (added.length > 0) {
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
[added.map((a: { id: string | number }) => String(a.id))]
)
}
}
return out
-40
View File
@@ -1,40 +0,0 @@
import { statfs } from 'node:fs/promises'
import logger from '@adonisjs/core/services/logger'
/** How often free space is measured. */
const CHECK_MS = 60_000
/** Resumed only this much above the floor: not paused and resumed at every flush. */
const MARGIN_GB = 1
/**
* Free space on the filesystem holding `path`, against a floor: low below it, again fine MARGIN_GB above it. In a
* container `/` is an overlay on the host's Docker root, which on homelab is the NVMe the catalog and the WAL are on.
* The prober filled it ~0.8–1.1 GB an hour at ~200 articles/s, and nothing stopped it at the 10 GB floor (LOG.md
* 2026-10-18 (77)). Off when the floor is 0.
*/
export class DiskSpace {
private checkedAt = 0
private low = false
constructor(
private what: string,
private path: string,
private floorGb: number,
private free: (path: string) => Promise<number> = async (p) => {
const s = await statfs(p)
return s.bavail * s.bsize
},
private checkMs = CHECK_MS
) {}
async short(): Promise<boolean> {
if (!this.floorGb) return false
if (Date.now() - this.checkedAt < this.checkMs) return this.low
this.checkedAt = Date.now()
const gb = (await this.free(this.path)) / 1e9
const was = this.low
this.low = gb < this.floorGb + (was ? MARGIN_GB : 0)
if (this.low !== was) logger.info(`${gb.toFixed(1)} GB free (floor ${this.floorGb} GB): ${this.what} ${this.low ? 'paused' : 'resumed'}.`)
return this.low
}
}
-16
View File
@@ -52,22 +52,6 @@ class Leases {
return { range: { id: Number(row.id), group: group.name, start: Number(row.start_id), end: Number(row.end_id), next: Number(row.next_id) }, rules }
}
/**
* Ranges read to their end whose worker went quiet before finishing them: done. The lease query skips a range past
* its end, so one whose finish never came stayed leased for good (a gap range since 2026-10-16 14:03, during the lock
* pile-up; LOG.md 2026-10-17 (72)). Its batches are all stored; its spooled chunks go.
*/
async finishAbandoned(): Promise<number> {
const { rows } = await db.rawQuery(
`UPDATE ranges SET status = 'done', done_at = now()
WHERE status = 'leased' AND next_id > end_id AND heartbeat_at < now() - make_interval(secs => ?)
RETURNING id`,
[coordinatorConfig.leaseTtlSeconds]
)
for (const { id } of rows) await Spool.remove(Number(id))
return rows.length
}
async leased(rangeId: number, nodeId: number) {
const range = await db.from('ranges').where('id', rangeId).first()
if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId) throw new LeaseLost(`Range ${rangeId} is not leased to this node`)
+13 -93
View File
@@ -2,18 +2,12 @@ import env from '#start/env'
import logger from '@adonisjs/core/services/logger'
import coordinatorConfig from '#config/coordinator'
import { XoverClient } from '#lib/nntp/index'
import type { NntpShare } from '#services/nntp_shares'
/** Connections wanted beyond the most in use at once since the last sync (the planner's, a hedge's). */
const SHARE_MARGIN = 2
/**
* The coordinator's header connections (REBUILD.md, decision e: workers
* have no NNTP account). At most NNTP_CONNECTIONS, reused; a connection
* that failed, or is older than NNTP_CONNECTION_MAX_MINUTES, is closed and
* replaced on the next use. Sharing the account (share()), at most its
* share: over it, connections close as they come free. One left free for
* NNTP_IDLE_CLOSE_SECONDS is closed on the next sync.
* replaced on the next use.
*/
class NntpPool {
private idle: XoverClient[] = []
@@ -23,43 +17,9 @@ class NntpPool {
private openedAt = new Map<XoverClient, number>()
/** When a connection gets closed once free: its max age ±25%, so they don't all reconnect at once. */
private retireAt = new Map<XoverClient, number>()
/** When an idle connection came free. */
private idleSince = new Map<XoverClient, number>()
private limit = coordinatorConfig.nntpConnections
/** Since the last sync: the most connections in use at once, and whether work waited for one (starved()). */
private peak = 0
private short = false
get size() {
return this.limit
}
/** Work that could use another connection didn't get one (the prefetcher with all its loops running). */
starved() {
this.short = true
}
/**
* Tells the share what the pool could use and has open, and takes the share's grant as its size. After work waited:
* twice its size (history going on gets its maximum in a few syncs; a tail round, done in seconds, doesn't take
* the prober's connections for one); else the most in use at once since the last sync and a margin.
*/
async share(share: NntpShare) {
const want = this.short || this.waiting.length > 0 ? Math.max(this.limit * 2, this.limit + SHARE_MARGIN) : this.peak + SHARE_MARGIN
this.short = false
this.peak = this.busy.size
const granted = await share.sync(want, this.open)
const was = this.limit
this.limit = granted
// Over the share: the idle ones close now, the busy ones once free (release()).
while (this.open > this.limit && this.idle.length > 0) this.retire(this.idle.pop()!)
this.closeIdle()
// Under it with work waiting: connections for it.
while (this.waiting.length > 0 && this.open < this.limit) {
const next = this.waiting.shift()!
this.acquire().then(next, () => this.waiting.unshift(next))
}
return { was, granted }
return coordinatorConfig.nntpConnections
}
async use<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
@@ -84,7 +44,6 @@ class NntpPool {
private async run<T>(client: XoverClient, work: (client: XoverClient) => Promise<T>): Promise<T> {
this.busy.add(client)
this.peak = Math.max(this.peak, this.busy.size)
let broken = false
try {
return await work(client)
@@ -97,28 +56,9 @@ class NntpPool {
}
}
/** Closes the connections free for nntpIdleCloseMs, the plain one too. */
private closeIdle() {
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs <= 0) return
const before = Date.now() - idleMs
for (const client of this.idle.filter((c) => (this.idleSince.get(c) ?? Infinity) <= before)) {
this.idle.splice(this.idle.indexOf(client), 1)
this.retire(client)
}
if (this.plain && this.plainPending === 0 && this.plainUsedAt <= before) {
void this.plain.then((c) => c.close()).catch(() => {})
this.plain = null
}
}
private async acquire(): Promise<XoverClient> {
// The one used last: the others stay free long enough to close.
const ready = this.idle.pop()
if (ready) {
this.idleSince.delete(ready)
return ready
}
if (ready) return ready
if (this.open < this.size) {
this.open++
try {
@@ -144,12 +84,15 @@ class NntpPool {
}
private release(client: XoverClient, broken: boolean) {
if (this.open > this.limit) {
this.retire(client)
return
}
if (broken || Date.now() >= (this.retireAt.get(client) ?? Infinity)) {
this.retire(client)
this.open--
this.openedAt.delete(client)
this.retireAt.delete(client)
try {
client.close()
} catch (error: any) {
logger.debug(`Closing a failed NNTP connection: ${error.message}`)
}
// Someone waiting gets a fresh connection.
const next = this.waiting.shift()
if (next) this.acquire().then(next, () => this.waiting.unshift(next))
@@ -157,30 +100,12 @@ class NntpPool {
}
const next = this.waiting.shift()
if (next) next(client)
else {
this.idle.push(client)
this.idleSince.set(client, Date.now())
}
}
private retire(client: XoverClient) {
this.open--
this.openedAt.delete(client)
this.retireAt.delete(client)
this.idleSince.delete(client)
try {
client.close()
} catch (error: any) {
logger.debug(`Closing an NNTP connection: ${error.message}`)
}
else this.idle.push(client)
}
/** Closes every connection, also those in use: their commands fail. */
private plain: Promise<XoverClient> | null = null
private plainQueue: Promise<unknown> = Promise.resolve()
/** Commands queued or running on the plain connection, and when the last one ended. */
private plainPending = 0
private plainUsedAt = 0
/**
* One connection without XFEATURE COMPRESS GZIP, opened when first needed,
@@ -188,7 +113,6 @@ class NntpPool {
* cuts short (Leases.corrupt). Its replies are deflated here.
*/
async usePlain<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
this.plainPending++
const run = this.plainQueue.then(async () => {
this.plain ??= XoverClient.connect({
host: env.get('NNTP_HOST')!,
@@ -208,10 +132,7 @@ class NntpPool {
throw error
}
})
this.plainQueue = run.catch(() => {}).finally(() => {
this.plainPending--
this.plainUsedAt = Date.now()
})
this.plainQueue = run.catch(() => {})
return run
}
@@ -222,7 +143,6 @@ class NntpPool {
this.busy.clear()
this.openedAt.clear()
this.retireAt.clear()
this.idleSince.clear()
this.open = 0
}
}
-90
View File
@@ -1,90 +0,0 @@
import db from '@adonisjs/lucid/services/db'
import coordinatorConfig from '#config/coordinator'
/** A share whose process hasn't synced for this long is gone: its connections closed with it. */
const STALE_SECONDS = 60
/** Syncs one at a time (pg_advisory_xact_lock). */
const LOCK = 2_026_101_875
export interface Share {
name: string
priority: number
/** Kept whatever the others want (as far as the total goes, by priority). */
min: number
/** What it could use now. */
want: number
/** Connections it has open (or may have: a lane without one yet counts). */
open: number
}
/** What each share should have: its minimum first, then by priority what it wants, up to the total. */
export function targets(total: number, shares: Share[]): Map<string, number> {
const ordered = [...shares].sort((a, b) => b.priority - a.priority || a.name.localeCompare(b.name))
const target = new Map<string, number>()
let left = total
for (const s of ordered) {
const t = Math.min(s.min, left)
target.set(s.name, t)
left -= t
}
for (const s of ordered) {
const more = Math.min(Math.max(s.want - target.get(s.name)!, 0), left)
target.set(s.name, target.get(s.name)! + more)
left -= more
}
return target
}
/**
* The connections `self` may have: its target, but no more than the others leave over. Another share holds what it
* has open or was granted, whichever is more (it may open up to its grant until its next sync tells it otherwise),
* so the account never goes over the total: a share grows only into what the others closed or gave up.
*/
export function grant(total: number, self: Share, others: (Share & { granted: number })[]): number {
const target = targets(total, [self, ...others]).get(self.name)!
const held = others.reduce((sum, o) => sum + Math.max(o.open, o.granted), 0)
return Math.max(0, Math.min(target, total - held))
}
/**
* A process's share of the NNTP account's connections (LOG.md 2026-10-18 (75)): the coordinator and the prober had a
* fixed split, and with history paused the coordinator's 38 stood idle while the prober's 8 were its limit. Each
* syncs every nntpShares.everyMs (what it wants and has open) and gets what it may open; one wanting less closes
* connections as they come free, and the other grows as those show up closed.
*/
export class NntpShare {
granted = 0
constructor(
readonly name: string,
private priority: number,
readonly min: number,
readonly max: number
) {}
async sync(want: number, open: number): Promise<number> {
const self: Share = { name: this.name, priority: this.priority, min: this.min, want: Math.min(Math.max(want, this.min), this.max), open }
const total = coordinatorConfig.nntpShares.total
this.granted = await db.transaction(async (trx) => {
await trx.rawQuery(`SELECT pg_advisory_xact_lock(?)`, [LOCK])
const { rows } = await trx.rawQuery(
`SELECT name, priority, min, want, open, granted FROM nntp_shares WHERE name <> ? AND heartbeat_at > now() - make_interval(secs => ?)`,
[this.name, STALE_SECONDS]
)
const granted = grant(total, self, rows)
await trx.rawQuery(
`INSERT INTO nntp_shares (name, priority, min, want, open, granted, heartbeat_at) VALUES (?, ?, ?, ?, ?, ?, now())
ON CONFLICT (name) DO UPDATE SET priority = excluded.priority, min = excluded.min, want = excluded.want,
open = excluded.open, granted = excluded.granted, heartbeat_at = now()`,
[this.name, this.priority, this.min, self.want, open, granted]
)
return granted
})
return this.granted
}
/** Stopping: its connections are closed, the others needn't wait for the row to go stale. */
async leave() {
await db.from('nntp_shares').where('name', this.name).delete()
}
}
+48 -124
View File
@@ -4,7 +4,6 @@ import { par2ReleaseName } from '#lib/release/index'
import { looksRandom } from '#lib/release/random'
import { SidecarFolder } from '#services/sidecar_folder'
import { Par2Pruner } from '#services/par2_pruner'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
/** Releases of a set further apart than this (posted) aren't one upload: a repost is merged separately (decision d). */
const MAX_SPREAD_MS = 48 * 3600 * 1000
@@ -16,28 +15,15 @@ const MAX_HOLDERS = 4
/** Members with more files than this many times the set's, plus SLACK_FILES (its par2 volumes), are more than one upload. */
const MAX_FILES_PER_SET_FILE = 3
const SLACK_FILES = 10
/**
* A par2 file held by more file rows than this (the same first 16 KiB and size) is a common file, not one upload's:
* it matches nothing. One such file was in 190,380 uploads, and a set naming it loaded every one of them; the merger
* ran out of heap every few minutes (LOG.md 2026-10-20 (85)). The judge calls 20 releases repeated.
*/
const COMMON_FILE = 20
/** Between rounds. */
const ROUND_MS = 60_000
/** A window looks back this far past the previous one's end (rows committed after it was read). */
/** A round looks back this far past the previous one's start (rows committed while it ran). */
const OVERLAP_MS = 60_000
/**
* A round goes through what's new in windows of this much (created_at), its place kept after each: a round behind by
* two days of the probe backlog (3.5M sets) never ended, and every restart started it over (LOG.md 2026-10-20 (85)).
*/
const WINDOW_MS = 60 * 60_000
/** The sidecar fold also runs this often inside a long par2 pass: its queue grows by the settler's pace. */
const FOLD_EVERY_MS = 5 * 60_000
/** Between par2_files prunes (complete and judged sets' file lists). */
const PRUNE_EVERY_MS = 60 * 60_000
/** A prune's share of the merger's time: the rest goes on in the next one. */
const PRUNE_BUDGET_MS = 2 * 60_000
/** The merger's row in worker_cursors: the end of its last whole window, kept over a restart. */
/** The merger's row in worker_cursors: the start of its last whole round, kept over a restart. */
const CURSOR = 'merger'
export interface MergeRound {
@@ -58,12 +44,6 @@ export interface MergeRound {
*/
const INNER_STEM = `substring(i.result->>'yencName' from '^([A-Za-z0-9]{16,64})\\.')`
/** The releases holding a par2 file f (its first 16 KiB and size), none when it's a common file (COMMON_FILE). */
const HOLDERS = `SELECT c.release_id FROM (
SELECT l.release_id, count(*) OVER () AS holders
FROM (SELECT h.release_id FROM file_hashes h WHERE h.hash16k = f.hash16k AND h.size = f.length LIMIT ${COMMON_FILE + 1}) l
) c WHERE c.holders <= ${COMMON_FILE}`
interface Member {
id: number
release_key: string
@@ -115,8 +95,8 @@ export class Par2Merger {
// First, at start too: a prune needn't wait for a long round.
if (!this.stopped && Date.now() - prunedAt >= PRUNE_EVERY_MS) {
prunedAt = Date.now()
const pruned = await new Par2Pruner().round(PRUNE_BUDGET_MS)
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged, ${pruned.final} final.`)
const pruned = await new Par2Pruner().round()
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged.`)
}
const round = await this.round()
if (round.merges > 0) logger.info(`Merged ${round.releases} releases into ${round.merges} by par2 set (${round.sets} sets checked).`)
@@ -129,18 +109,14 @@ export class Par2Merger {
}
}
/**
* The sets and inner stems with something new since the last round (a par2 found, a hash of one of their files, a
* lone inspection), a window at a time.
*/
/** The sets with something new since the last round: a par2 found, or a hash of one of their files. */
async round(): Promise<MergeRound> {
// A Date keeps milliseconds, now() microseconds: rounded up, a row made at now() is before the end.
const { rows: now } = await db.rawQuery(`SELECT now() + interval '1 millisecond' AS at`)
const end = new Date(now[0].at)
const { rows: now } = await db.rawQuery(`SELECT now() AS at`)
if (!this.since) {
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
this.since = saved ? new Date(saved.at) : new Date(0)
}
const since = new Date(this.since.getTime() - OVERLAP_MS)
// First: sidecars (.nfo, .sfv, .nzb) apart from their upload join it. A queue, not since: it needn't wait for the
// par2 pass (hours when that starts over).
const folded = await this.folder.round()
@@ -154,96 +130,52 @@ export class Par2Merger {
foldedAt = Date.now()
if (more.folded) logger.info(`Folded ${more.folded} sidecars into their uploads (${more.sidecars} sidecar-only of ${more.looked} small releases).`)
}
const round: MergeRound = { sets: 0, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
while (this.since < end) {
const { rows } = await db.rawQuery(
`SELECT set_id FROM par2_sets WHERE created_at >= ?
UNION
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size WHERE h.created_at >= ?`,
[since, since]
)
const round: MergeRound = { sets: rows.length, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
for (const { set_id: setId } of rows) {
if (this.stopped) return round
const from = await this.firstNew(new Date(this.since.getTime() - OVERLAP_MS), end)
const until = from ? new Date(Math.min(from.getTime() + WINDOW_MS, end.getTime())) : end
if (from) {
const { touched, sets } = await this.setsToMerge(from, until)
round.sets += touched
for (const setId of sets) {
if (this.stopped) return round
await foldAgain()
// One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
const merged = await this.mergeSet(setId.trim()).catch((error) => {
logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.merges++
round.releases += merged + 1
}
}
const { rows: stems } = await db.rawQuery(
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i
WHERE i.kind = 'lone' AND i.created_at >= ? AND i.created_at <= ? AND ${INNER_STEM} IS NOT NULL`,
[from, until]
)
for (const { stem } of stems) {
if (this.stopped) return round
if (!looksRandom(stem)) continue
round.stems++
const merged = await this.mergeStem(stem).catch((error) => {
logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.stemMerges++
round.stemReleases += merged + 1
}
}
await foldAgain()
// One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
const merged = await this.mergeSet(setId.trim()).catch((error) => {
logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.merges++
round.releases += merged + 1
}
// Only after a whole window: a restart halfway goes on from the window before.
this.since = until
await db.rawQuery(
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, this.since]
)
}
const { rows: stems } = await db.rawQuery(
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i WHERE i.kind = 'lone' AND i.created_at >= ? AND ${INNER_STEM} IS NOT NULL`,
[since]
)
for (const { stem } of stems) {
if (this.stopped) return round
if (!looksRandom(stem)) continue
round.stems++
const merged = await this.mergeStem(stem).catch((error) => {
logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.stemMerges++
round.stemReleases += merged + 1
}
}
// Only after a whole round: a restart halfway goes on from the round before.
this.since = now[0].at
await db.rawQuery(
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, this.since]
)
return round
}
/** When the first row a window reads was made, from `from` up to `end`; null when none (a gap, the nightly pause). */
private async firstNew(from: Date, end: Date): Promise<Date | null> {
const { rows } = await db.rawQuery(
`SELECT least(
(SELECT min(created_at) FROM par2_sets WHERE created_at >= ?),
(SELECT min(created_at) FROM file_hashes WHERE created_at >= ?),
(SELECT min(created_at) FROM release_inspections WHERE created_at >= ?)) AS at`,
[from, from, from]
)
const at = rows[0].at ? new Date(rows[0].at) : null
return at && at <= end ? at : null
}
/**
* The sets a window touched (a par2 found, a hash of one of their files), and those among them a merge could
* change: held or matched by two releases or more. Most have every file in one release already (96% of a backlog
* hour's 147k): mergeSet needn't look at them.
*/
private async setsToMerge(from: Date, until: Date): Promise<{ touched: number; sets: string[] }> {
const { rows } = await db.rawQuery(
`WITH touched AS MATERIALIZED (
SELECT set_id FROM par2_sets WHERE created_at >= ? AND created_at <= ?
UNION
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size
WHERE h.created_at >= ? AND h.created_at <= ?
)
SELECT (SELECT count(*) FROM touched) AS touched, array(
SELECT t.set_id FROM touched t
WHERE EXISTS (
SELECT 1 FROM (
SELECT s.release_id FROM par2_sets s WHERE s.set_id = t.set_id
UNION ALL
SELECT m.release_id FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) m WHERE f.set_id = t.set_id
) u HAVING count(DISTINCT u.release_id) >= 2)
) AS sets`,
[from, until, from, until]
)
return { touched: Number(rows[0].touched), sets: rows[0].sets }
}
/** Merges the releases whose lone files' inner names share a random stem; how many went into another. */
async mergeStem(stem: string): Promise<number> {
return db.transaction(async (trx) => {
@@ -288,7 +220,7 @@ export class Par2Merger {
const near = (r: Member) => r.group_id === anchor.group_id && (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
const { rows: matches } = await trx.rawQuery(
`SELECT f.file_id, r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable
FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) h JOIN releases r ON r.id = h.release_id
FROM par2_files f JOIN file_hashes h ON h.hash16k = f.hash16k AND h.size = f.length JOIN releases r ON r.id = h.release_id
WHERE f.set_id = ?`,
[setId]
)
@@ -336,14 +268,6 @@ export class Par2Merger {
for (const table of ['release_chunks', 'probe_candidates', 'release_inspections', 'file_hashes']) {
await trx.rawQuery(`UPDATE ${table} SET release_id = ? WHERE release_id = ANY(?)`, [target.id, others])
}
// An unsettled member's candidates into a settled target: the settler won't come by again to make them claimable.
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE r.id = ? AND r.settled_at IS NOT NULL AND pc.release_id = r.id AND pc.status = 'waiting' AND NOT pc.ready`,
[target.id]
)
// The members' 7z tails together: those below the highest volume go.
await pruneSevenZipTails(trx, [target.id])
// A set held by the target already, or by two of the others (a par2 and its repost), isn't held twice (every
// round failed on the primary key from 20:28 UTC on 2026-10-12; LOG.md 2026-10-12 (30)).
await trx.rawQuery(
+43 -69
View File
@@ -4,98 +4,72 @@ import db from '@adonisjs/lucid/services/db'
const SETTLED_MS = 2 * 86400_000
/** Sets per statement. */
const BATCH = 20_000
/** The sweep's row in worker_cursors: the last set_id it looked at (key); past the last set it starts over. */
/** The complete pass's row in worker_cursors: the newest holder's created_at of the last set it looked at. */
const CURSOR = 'par2-prune'
/** The merger's row in worker_cursors: the end of its last whole window. */
const MERGER = 'merger'
/** The merger's windows look back this far (Par2Merger's OVERLAP_MS). */
const MERGER_OVERLAP_MS = 60_000
/** Verdicts whose releases needn't merge any more; locked isn't one (a password found takes it back). */
const JUNK = ['dead', 'fake', 'repeated']
/** Probe candidates whose results a merge reads: a file's hash (lone), a set's file list (par2). */
const MERGE_KINDS = ['lone', 'par2']
export interface PruneRound {
complete: number
judged: number
final: number
rows: number
}
/**
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60), 2026-10-20 (87)). A set's file list
* matches file hashes to releases and names an unreadable upload once; it's done when
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60)): the table only grew, 35M rows and
* 13.6 GB at 1.8M sets. A set's file list matches file hashes to releases and names an unreadable upload once; it's
* done when
* - complete: SETTLED_MS after its last holder's par2, every file it describes has its hash in a release holding the
* set (all of it merged into one);
* - judged: every release holding it has a junk verdict (dead, fake, repeated);
* - final: SETTLED_MS after its last holder's par2, every holder is final (release_final: a run of done ranges holds
* it with hours to spare, so no more of its upload will be read), the candidates of the ranges around it are probed,
* and the merger has seen every hash of its files. A file still missing then is one no probe will hash: most sets
* describe a readable upload's files, which aren't lone candidates (9 in 10 sets were kept for that, 19 GB).
* - judged: every release holding it has a junk verdict (dead, fake, repeated).
* par2_sets stays (the judge's unusable rule, the merge's holders). A set comes up again only with a new holder: its
* par2 reposted, whose probe writes the rows again; without rows its merge finds no files and the count check holds.
* The sweep goes through the sets with rows by set_id and starts over at the end: a set kept now (a holder at the edge
* of a run, its ranges' probes not done yet) is looked at again on the next pass.
*/
export class Par2Pruner {
/** Sets per batch (BATCH; smaller in the tests). */
constructor(private batch = BATCH) {}
/**
* Batches until a pass is done or `budgetMs` is spent (the cursor is saved per batch, the next round goes on from
* it): run from the merger's loop, a first round over two days of sets held the merges back for hours (LOG.md
* 2026-10-17 (72)).
*/
async round(budgetMs = Infinity): Promise<PruneRound> {
const deadline = Date.now() + budgetMs
const round: PruneRound = { complete: 0, judged: 0, final: 0, rows: 0 }
const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
let after: string = cursors[0]?.key ?? ''
// Ranges done since the oldest merge-kind candidate still unprobed: their releases' hashes and file lists may be
// still to come. Hashes since the merger's last window: it hasn't matched them yet.
const { rows: marks } = await db.rawQuery(
`SELECT coalesce((SELECT min(created_at) FROM probe_candidates WHERE status IN ('waiting', 'taken') AND kind = ANY(?)), now()) AS probed,
coalesce((SELECT at FROM worker_cursors WHERE name = ?), to_timestamp(0)) - make_interval(secs => ?) AS merged,
now() - make_interval(secs => ?) AS settled`,
[MERGE_KINDS, MERGER, MERGER_OVERLAP_MS / 1000, SETTLED_MS / 1000]
)
const { probed, merged, settled } = marks[0]
async round(): Promise<PruneRound> {
const round: PruneRound = { complete: 0, judged: 0, rows: 0 }
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
let since: Date = saved ? new Date(saved.at) : new Date(0)
const until = new Date(Date.now() - SETTLED_MS)
for (;;) {
const { rows: sets } = await db.rawQuery(
`WITH batch AS MATERIALIZED (SELECT DISTINCT set_id FROM par2_files WHERE set_id > ? ORDER BY set_id LIMIT ?),
recent AS MATERIALIZED (SELECT group_id, posted_low, posted_high FROM ranges WHERE status = 'done' AND done_at >= ?),
unseen AS MATERIALIZED (SELECT DISTINCT hash16k, size FROM file_hashes WHERE created_at >= ?)
SELECT b.set_id, CASE
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
AND NOT EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = b.set_id AND NOT EXISTS (
SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
WHERE h.hash16k = f.hash16k AND h.size = f.length)) THEN 'complete'
WHEN NOT EXISTS (SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts v ON v.release_id = o.release_id AND v.rule = ANY(?)
WHERE o.set_id = b.set_id AND v.release_id IS NULL) THEN 'judged'
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
AND NOT EXISTS (SELECT 1 FROM par2_sets s JOIN releases r ON r.id = s.release_id WHERE s.set_id = b.set_id AND (
release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) IS NOT TRUE
OR EXISTS (SELECT 1 FROM recent x WHERE x.group_id = r.group_id
AND x.posted_low <= coalesce(r.last_posted_at, r.posted_at) + interval '6 hours' AND x.posted_high >= r.posted_at - interval '6 hours')))
AND NOT EXISTS (SELECT 1 FROM par2_files f JOIN unseen u ON u.hash16k = f.hash16k AND u.size = f.length WHERE f.set_id = b.set_id)
THEN 'final' END AS reason
FROM batch b ORDER BY b.set_id`,
[after, this.batch, probed, merged, settled, JUNK, settled]
`SELECT set_id, max(created_at) AS at FROM par2_sets GROUP BY set_id
HAVING max(created_at) > ? AND max(created_at) <= ? ORDER BY 2 LIMIT ?`,
[since, until, BATCH]
)
const done = (reason: keyof PruneRound) => sets.filter((s: { reason: string }) => s.reason === reason).map((s: { set_id: string }) => s.set_id)
for (const reason of ['complete', 'judged', 'final'] as const) {
const ids = done(reason)
round[reason] += ids.length
round.rows += await this.drop(ids)
}
// Past the last set: the next batch starts the next pass.
after = sets.length < this.batch ? '' : sets.at(-1).set_id
if (sets.length === 0) break
const ids = sets.map((s: { set_id: string }) => s.set_id)
const { rows: done } = await db.rawQuery(
`SELECT c.set_id FROM unnest(?::char(32)[]) AS c(set_id)
WHERE EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id)
AND NOT EXISTS (
SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id AND NOT EXISTS (
SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
WHERE h.hash16k = f.hash16k AND h.size = f.length))`,
[ids]
)
round.complete += done.length
round.rows += await this.drop(done.map((s: { set_id: string }) => s.set_id))
since = sets.at(-1).at
await db.rawQuery(
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
[CURSOR, after]
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, since]
)
if (after === '' || Date.now() > deadline) break
if (sets.length < BATCH) break
}
// Judged: no cursor (verdicts come and go); only sets with rows left.
const { rows: judged } = await db.rawQuery(
`SELECT DISTINCT s.set_id FROM release_verdicts v JOIN par2_sets s ON s.release_id = v.release_id
WHERE v.rule = ANY(?)
AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
AND NOT EXISTS (
SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts ov ON ov.release_id = o.release_id AND ov.rule = ANY(?)
WHERE o.set_id = s.set_id AND ov.release_id IS NULL)`,
[JUNK, JUNK]
)
round.judged = judged.length
const judgedIds = judged.map((s: { set_id: string }) => s.set_id)
for (let i = 0; i < judgedIds.length; i += BATCH) round.rows += await this.drop(judgedIds.slice(i, i + BATCH))
return round
}
-5
View File
@@ -131,8 +131,6 @@ export class Prefetcher {
.catch((error) => logger.warn(`Prefetching: ${error.message}`))
.finally(() => this.running--)
}
// Work left with every loop running: the pool's share could be bigger.
if (!this.stopped && this.running >= this.loops && this.peek()) NntpPool.starved()
}
stop() {
@@ -147,9 +145,6 @@ export class Prefetcher {
/** One connection's loop: chunks until there is nothing left to fetch. */
private async work() {
while (!this.stopped) {
// The pool's share got smaller: fewer loops (waiting on the pool, they'd count as work short of a connection).
// Two leaving at once is one too many for a tick (TICK_MS), whose loop comes back.
if (this.running > this.loops) return
const next = this.take()
if (!next) return
const { target, span } = next
+64 -136
View File
@@ -6,8 +6,6 @@ import coordinatorConfig from '#config/coordinator'
import { XoverClient } from '#lib/nntp/index'
import { YencError } from '#lib/yenc/index'
import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/index'
import { NntpShare } from '#services/nntp_shares'
import { DiskSpace } from '#services/disk_space'
interface Candidate {
id: number
@@ -30,15 +28,6 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
{ kind: 'archive', unreadable: false },
]
const FLUSH_MS = 2000
/**
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
* and the whole flush failed with it (LOG.md 2026-10-18 (78)).
*/
function jsonb(value: unknown): string {
return JSON.stringify(value, (_key, v) => (typeof v === 'string' ? v.replaceAll('\0', '') : v))
}
/** Slices a claim takes from random points of the queue. */
const SLICES = 20
const REPORT_MS = 60_000
@@ -57,14 +46,6 @@ export class Prober {
private refilling: Promise<void> | null = null
private stopped = false
private stats = Prober.freshStats()
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
private lanes = new Set<Promise<void>>()
private running = 0
/** The last lane to look found nothing to claim: the prober wants its minimum only. */
private dry = false
/** What it stores (inspections, par2 file lists, the WAL) fills the NVMe: no claims below PROBE_MIN_FREE_GB. */
private space = new DiskSpace('probing', '/', coordinatorConfig.probe.minFreeGb)
private share = new NntpShare('prober', coordinatorConfig.nntpShares.proberPriority, coordinatorConfig.probe.minConnections, coordinatorConfig.probe.connections)
/** Per minute: what was read, and where the lanes' time went (ms summed over lanes; flushes ms on their own). */
private static freshStats() {
@@ -75,7 +56,8 @@ export class Prober {
// A previous run's claims: one prober at a time.
const { rowCount } = await db.rawQuery(`UPDATE probe_candidates SET status = 'waiting' WHERE status = 'taken'`)
if (rowCount) logger.info(`Probing: ${rowCount} candidates from a previous run back in the queue.`)
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
const connections = coordinatorConfig.probe.connections
logger.info(`Probing with ${connections} connections.`)
const flusher = (async () => {
while (!this.stopped || this.outcomes.length > 0) {
await sleep(FLUSH_MS)
@@ -85,26 +67,7 @@ export class Prober {
this.report()
}
})()
// Lanes as the share allows: the coordinator's history comes first (app/services/nntp_shares.ts).
while (!this.stopped) {
try {
const was = this.share.granted
const granted = await this.share.sync(this.dry ? 0 : this.share.max, this.running)
if (granted !== was) logger.info(`Probing: ${granted} connections (was ${was}).`)
while (this.running < granted) {
this.running++
const lane: Promise<void> = this.lane()
.catch((error) => logger.error(`Probing: a lane failed: ${error.message}`))
.finally(() => this.lanes.delete(lane))
this.lanes.add(lane)
}
} catch (error: any) {
logger.error(`Probing: NNTP share failed: ${error.message}`)
}
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !this.stopped; waited += 1000) await sleep(1000)
}
await Promise.all(this.lanes)
await this.share.leave().catch(() => {})
await Promise.all(Array.from({ length: connections }, () => this.lane()))
// What was claimed and not probed goes back.
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
this.stopped = true
@@ -115,90 +78,68 @@ export class Prober {
this.stopped = true
}
/**
* One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure; done when
* there are more lanes than the share (its connection closed, the coordinator takes it on its next sync).
*/
/** One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure. */
private async lane() {
let client: XoverClient | null = null
let usedAt = Date.now()
try {
while (!this.stopped) {
if (this.running > this.share.granted) break
const batch: Candidate[] = []
const waited = Date.now()
while (batch.length < coordinatorConfig.probe.pipeline) {
const candidate = await this.next()
if (!candidate) break
batch.push(candidate)
}
this.stats.waitMs += Date.now() - waited
this.dry = batch.length === 0
if (batch.length === 0) {
// Nothing to probe for a while: the connection closes (nntpIdleCloseMs), and opens again with work.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (client && idleMs > 0 && Date.now() - usedAt >= idleMs) {
client.close()
client = null
while (!this.stopped) {
const batch: Candidate[] = []
const waited = Date.now()
while (batch.length < coordinatorConfig.probe.pipeline) {
const candidate = await this.next()
if (!candidate) break
batch.push(candidate)
}
this.stats.waitMs += Date.now() - waited
if (batch.length === 0) {
this.stats.idleMs += 30_000
await sleep(30_000)
continue
}
try {
client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
port: env.get('NNTP_PORT', 563),
user: env.get('NNTP_USER'),
password: env.get('NNTP_PASSWORD'),
secure: env.get('NNTP_SECURE', true),
compress: false,
timeoutMs: coordinatorConfig.nntpTimeoutMs,
})
const fetched = Date.now()
const bodies = await client.bodies(batch.map((c) => c.messageId))
this.stats.fetchMs += Date.now() - fetched
const inspected = Date.now()
for (const [i, candidate] of batch.entries()) {
const body = bodies[i]
this.stats.articles++
if (body === null) {
// Taken down or expired: final.
this.outcomes.push({ candidate, status: 'missing' })
continue
}
this.stats.idleMs += 30_000
await sleep(30_000)
continue
}
try {
client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
port: env.get('NNTP_PORT', 563),
user: env.get('NNTP_USER'),
password: env.get('NNTP_PASSWORD'),
secure: env.get('NNTP_SECURE', true),
compress: false,
timeoutMs: coordinatorConfig.nntpTimeoutMs,
})
const fetched = Date.now()
const bodies = await client.bodies(batch.map((c) => c.messageId))
this.stats.fetchMs += Date.now() - fetched
const inspected = Date.now()
for (const [i, candidate] of batch.entries()) {
const body = bodies[i]
this.stats.articles++
if (body === null) {
// Taken down or expired: final.
this.outcomes.push({ candidate, status: 'missing' })
continue
}
this.stats.bytes += body.length
try {
this.outcomes.push({ candidate, status: 'done', inspection: inspect(body) })
} catch (error: any) {
// Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
this.outcomes.push({ candidate, status: 'error' })
}
this.stats.bytes += body.length
try {
this.outcomes.push({ candidate, status: 'done', inspection: inspect(body) })
} catch (error: any) {
// Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
this.outcomes.push({ candidate, status: 'error' })
}
this.stats.inspectMs += Date.now() - inspected
usedAt = Date.now()
} catch (error: any) {
logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
client?.close()
client = null
await sleep(5000)
}
this.stats.inspectMs += Date.now() - inspected
} catch (error: any) {
logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
client?.close()
client = null
await sleep(5000)
}
} finally {
client?.close()
this.running--
}
}
/** Candidates claimed at a time: a few rounds of every lane's pipeline (a claim takes seconds; 400 lasted ~2 s at 37 lanes). */
private get batch() {
return Math.max(coordinatorConfig.probe.batch, this.running * coordinatorConfig.probe.pipeline * 4)
client?.close()
}
private async next(): Promise<Candidate | undefined> {
if (this.queue.length < this.batch / 2) {
if (this.queue.length < coordinatorConfig.probe.batch / 2) {
this.refilling ??= this.claim().finally(() => (this.refilling = null))
if (this.queue.length === 0) await this.refilling
}
@@ -221,24 +162,13 @@ export class Prober {
}
private async claimTiers() {
// The queue empties, the lanes go dry and the share shrinks to its minimum until there's room again.
if (await this.space.short()) return
for (const { kind, unreadable } of TIERS) {
const want = this.batch - this.queue.length
const want = coordinatorConfig.probe.batch - this.queue.length
if (want <= 0) return
// The starting points chosen here: in the query Postgres drew random() once for all slices.
// Over the claimable ones of this tier only: the rest of the queue isn't settled yet.
// Per value of unreadable, each an end of the index: over `unreadable >= false` Postgres walked every waiting
// entry of the kind, 2.5 minutes a claim at 28M archive candidates (LOG.md 2026-10-18 (74)).
const values = unreadable ? [true] : [false, true]
const ends = values
.map(() => `SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable = ?`)
.join(' UNION ALL ')
const [{ lo, hi }] = (
await db.rawQuery(
`SELECT min(lo) AS lo, max(hi) AS hi FROM (${ends}) e`,
values.flatMap((v) => [kind, v])
)
await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable >= ?`, [kind, unreadable])
).rows
if (lo === null) continue
const starts = Array.from({ length: SLICES }, () => Number(lo) + Math.floor(Math.random() * (Number(hi) - Number(lo) + 1)))
@@ -279,18 +209,16 @@ export class Prober {
await db.transaction(async (trx) => {
await trx.rawQuery(
`UPDATE probe_candidates pc SET status = v.status FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, status text) WHERE pc.id = v.id`,
[jsonb(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
[JSON.stringify(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
)
// Final answers read from the news server, kept by message-id for the next catalog (a reset).
// One per message-id: the same article is a candidate of more than one release, and ON CONFLICT can't take a
// row twice in one statement.
const fresh = [...new Map(outcomes.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
const fresh = outcomes.filter((o) => !o.cached && o.status !== 'waiting')
if (fresh.length > 0) {
await trx.rawQuery(
`INSERT INTO probe_cache (message_id, status, result, version)
SELECT v.message_id, v.status, v.result, ? FROM jsonb_to_recordset(?::jsonb) AS v(message_id text, status text, result jsonb)
ON CONFLICT (message_id) DO UPDATE SET status = excluded.status, result = excluded.result, version = excluded.version, created_at = now()`,
[PROBE_VERSION, jsonb(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
[PROBE_VERSION, JSON.stringify(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
)
}
const done = outcomes.filter((o) => o.inspection)
@@ -300,7 +228,7 @@ export class Prober {
SELECT v.release_id, v.candidate_id, v.kind, v.result FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, candidate_id bigint, kind text, result jsonb)
ON CONFLICT (release_id, candidate_id) DO UPDATE SET result = excluded.result, created_at = now()`,
// A par2 set's file list goes to par2_files, not into every inspection.
[jsonb(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
[JSON.stringify(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
)
await this.storeForMerge(trx, done)
// Each touched release from all its inspections (earlier rounds may have stored others).
@@ -324,7 +252,7 @@ export class Prober {
await trx.rawQuery(
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, passworded boolean, inner_name text) WHERE r.id = v.id`,
[jsonb(updates)]
[JSON.stringify(updates)]
)
})
}
@@ -337,7 +265,7 @@ export class Prober {
`INSERT INTO file_hashes (candidate_id, release_id, hash16k, size)
SELECT v.candidate_id, v.release_id, v.hash16k, v.size FROM jsonb_to_recordset(?::jsonb) AS v(candidate_id bigint, release_id bigint, hash16k text, size bigint)
ON CONFLICT (candidate_id) DO NOTHING`,
[jsonb(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
[JSON.stringify(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
)
}
for (const o of done) {
@@ -347,7 +275,7 @@ export class Prober {
`INSERT INTO par2_files (set_id, file_id, hash16k, length, name)
SELECT ?, v.file_id, v.hash16k, v.length, v.name FROM jsonb_to_recordset(?::jsonb) AS v(file_id text, hash16k text, length bigint, name text)
ON CONFLICT DO NOTHING`,
[set.setId, jsonb(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
[set.setId, JSON.stringify(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
)
await trx.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files) VALUES (?, ?, ?) ON CONFLICT DO NOTHING`, [set.setId, o.candidate.releaseId, set.files.length])
}
+50 -180
View File
@@ -1,18 +1,10 @@
import db from '@adonisjs/lucid/services/db'
import logger from '@adonisjs/core/services/logger'
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
/** Raised when a rule changes: verdicts of an older version are judged again. */
export const JUDGE_VERSION = 1
/** Between rounds of the changed releases (judge_dirty, noted by triggers; migration 4900). */
const ROUND_MS = 2 * 60 * 1000
/**
* A full round (every release, each rule a scan of its table: 35–38 min) this often: for what no trigger notes. Time
* (unusable a day after settling, a release final once the ranges beside it are read) and fewer releases with a file
* after a merge (repeated).
*/
const FULL_MS = 24 * 60 * 60 * 1000
/** Changed releases judged at a time. */
const CHUNK = 50_000
/** Between rounds: each rule is a scan of its table. */
const ROUND_MS = 60 * 60 * 1000
/** A file's first 16 KiB and size in this many releases is one file posted again and again (test files, spam). */
const REPEATED_RELEASES = 20
@@ -32,15 +24,10 @@ export interface Rule {
name: string
/** Why, in a few words (the log, the web interface). */
says: string
/** SELECT of the release ids the rule judges; scoped, only those in judge_scope. */
sql: (scoped: boolean) => string
/** SELECT of the release ids the rule judges. */
sql: string
}
/** A join to the releases in scope (an incremental round), on the given release id column. */
const only = (scoped: boolean, column: string) => (scoped ? `JOIN judge_scope s ON s.release_id = ${column}` : '')
/**
* The junk judge's rules, in order: a release takes the first that matches (REBUILD.md, junk). Each runs on what
* the header pass, the prober, the merges and the spots know by now.
@@ -49,33 +36,26 @@ export const RULES: Rule[] = [
{
name: 'dead',
says: 'every probed first article is gone from the server (taken down or expired)',
sql: (scoped) =>
`SELECT c.release_id FROM probe_candidates c ${only(scoped, 'c.release_id')}
WHERE c.status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(c.status = 'missing')`,
sql: `SELECT release_id FROM probe_candidates WHERE status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(status = 'missing')`,
},
{
name: 'fake',
says: 'named as a film, episode or music, but the content is a program',
// An archive whose entries are all executables, under a media name (its own or its entries'); or a file that is
// an executable by name or by its first bytes, under a media name.
sql: (scoped) => `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
sql: `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
WHERE i.kind = 'archive' AND jsonb_array_length(i.result->'names') > 0
AND NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(i.result->'names') n WHERE n !~* '${EXECUTABLE}')
AND (r.name ~* '${MEDIA}' OR (i.result->>'names') ~* '${MEDIA}')
UNION
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
WHERE i.kind = 'lone' AND ((i.result->>'yencName') ~* '${EXECUTABLE}' OR i.result->>'type' = 'exe')
AND (r.name ~* '${MEDIA}' OR (i.result->>'yencName') ~* '${MEDIA}')`,
},
{
name: 'repeated',
says: `the same file content in ${REPEATED_RELEASES}+ releases (test files, spam)`,
// Scoped: by the files of the releases in scope that are in REPEATED_RELEASES+ releases, counted once a chunk
// (judge_tipped: ReleaseJudge.changed, which first adds the other releases of a file that just tipped over).
sql: (scoped) =>
scoped
? `SELECT DISTINCT h.release_id FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id JOIN judge_tipped USING (hash16k, size)`
: `SELECT DISTINCT h.release_id FROM file_hashes h
sql: `SELECT DISTINCT h.release_id FROM file_hashes h
JOIN (SELECT hash16k, size FROM file_hashes GROUP BY 1, 2 HAVING count(DISTINCT release_id) >= ${REPEATED_RELEASES}) x USING (hash16k, size)`,
},
{
@@ -84,26 +64,19 @@ export const RULES: Rule[] = [
// "@_CelebMovie_@", "NudeCelebs@CelebMovie.com", "@_CelebWow_@", "@_CelebFX_@", "@_iLuvCelebs_@", "@nude-celebs", …:
// a celebrity's scraped pictures as a folder under a typed subject, the same files again and again under other
// titles (LOG.md 2026-10-16 (64)). Not Celeborn, CelebTA or other posters with "celeb" in their name.
// By the posters table's trigram index, each poster's releases then by releases_poster_newest_idx: 7 s, where the
// regex on all of releases was 45 s, every round (LOG.md 2026-10-19 (80)).
sql: (scoped) =>
scoped
? `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`
: `SELECT r.id FROM posters p JOIN releases r ON left(r.poster, 200) = p.poster
WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
sql: `SELECT r.id FROM releases r
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
},
{
name: 'locked',
says: 'passworded, and no spot, NZB or title gives the password',
// releases_passworded_idx: a scan of all of releases was 30 s a round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
sql: `SELECT r.id FROM releases r WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
},
{
name: 'unusable',
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
// Final (release_final): next to an unread range its par2 or the rest of its files may still come (LOG.md 2026-10-16 (67)).
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
sql: `SELECT r.id FROM releases r
WHERE r.settled_at < now() - interval '1 day' AND NOT r.readable AND r.inner_name IS NULL
AND release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete)
AND NOT EXISTS (SELECT 1 FROM release_merges m WHERE m.into_id = r.id)
@@ -113,18 +86,15 @@ export const RULES: Rule[] = [
]
export interface JudgeRound {
/** Releases judged: every one (a full round), or the changed ones and those a hash of theirs tipped over. */
releases?: number
/** Per rule: releases it judges now (in scope), verdicts it gained, verdicts it lost. */
/** Per rule: releases it judges now, verdicts added, verdicts taken back. */
rules: Record<string, { matched: number; added: number; removed: number }>
}
/**
* The junk judge (REBUILD.md, junk; voyager 1's second filter, D42): after the header pass's rules (garbage
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Every few
* minutes the releases whose facts changed (judge_dirty) are judged by every rule; a full round judges every release
* daily. A release gets the verdict of the first rule in order that matches it; one that no longer matches loses its
* verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Each
* round recomputes every rule: a release a rule matches gets a verdict (the first rule in order, over a later one's), one it no longer
* matches loses its verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
*/
export class ReleaseJudge {
private stopped = false
@@ -137,13 +107,9 @@ export class ReleaseJudge {
while (!this.stopped) {
const started = Date.now()
try {
if (await this.fullDue()) {
const round = await this.full()
if (!this.stopped) logger.info(`Judged all releases: ${describe(round)}.`)
} else {
const round = await this.changed()
if (round.releases! > 0) logger.info(`Judged ${round.releases!.toLocaleString('en')} changed releases: ${describe(round)}.`)
}
const round = await this.round()
const parts = Object.entries(round.rules).map(([n, r]) => `${n} ${r.matched} (+${r.added} −${r.removed})`)
logger.info(`Judged: ${parts.join(', ')}.`)
} catch (error: any) {
logger.error(`Judge round failed: ${error.message}`)
}
@@ -151,139 +117,43 @@ export class ReleaseJudge {
}
}
/** A full round is due: a day after the last, or the last was by another rule set. */
async fullDue(): Promise<boolean> {
const state = await db.from('judge_state').first()
return !state || state.version !== JUDGE_VERSION || Date.now() - new Date(state.full_at).getTime() >= FULL_MS
}
/**
* Every release by every rule, the matches gathered in judge_full (each rule its own statement: no transaction of
* half an hour), then applied at once. With dry, only counts. Stopped halfway, nothing is applied.
*/
async full(dry = false): Promise<JudgeRound> {
const started = new Date()
/** One round over every rule; with dry, only counts. */
async round(dry = false): Promise<JudgeRound> {
const round: JudgeRound = { rules: {} }
// Verdicts of an older rule set are judged again from scratch.
if (!dry) await db.from('release_verdicts').where('rule_version', '<>', JUDGE_VERSION).delete()
await db.rawQuery(`TRUNCATE judge_full`)
const judged = new Set<string>()
for (const rule of RULES) {
if (this.stopped) return round
const { rowCount } = await db.rawQuery(
`INSERT INTO judge_full (release_id, rule) SELECT id, ? FROM (${rule.sql(false)}) AS x(id) ON CONFLICT DO NOTHING`,
[rule.name]
)
round.rules[rule.name] = { matched: rowCount ?? 0, added: 0, removed: 0 }
}
if (dry || this.stopped) return round
await db.rawQuery(`ANALYZE judge_full`)
await db.transaction((trx) => this.apply(trx, 'judge_full', false, round))
await db.rawQuery(
`INSERT INTO judge_state (id, full_at, version) VALUES (1, ?, ?) ON CONFLICT (id) DO UPDATE SET full_at = excluded.full_at, version = excluded.version`,
[started, JUDGE_VERSION]
)
return round
}
/**
* The releases noted in judge_dirty since, CHUNK at a time, each chunk in a transaction (a failed one stays noted).
* A changed release can bring a file to REPEATED_RELEASES releases: the others with it are judged too (all but those
* already repeated, or dead or fake, which come first). Only when it's the chunk's releases that bring it there: a
* file in that many without them had its releases judged before.
*/
async changed(): Promise<JudgeRound> {
const round: JudgeRound = { releases: 0, rules: {} }
for (const rule of RULES) round.rules[rule.name] = { matched: 0, added: 0, removed: 0 }
let more = true
while (more && !this.stopped) {
await db.transaction(async (trx) => {
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
await trx.rawQuery(`DROP TABLE IF EXISTS judge_scope`)
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
await trx.rawQuery(`CREATE TEMP TABLE judge_scope (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
const { rows } = await trx.rawQuery(
`WITH taken AS (DELETE FROM judge_dirty WHERE ctid = ANY(ARRAY(SELECT ctid FROM judge_dirty LIMIT ${CHUNK})) RETURNING release_id),
scoped AS (INSERT INTO judge_scope SELECT DISTINCT release_id FROM taken ON CONFLICT DO NOTHING RETURNING 1)
SELECT (SELECT count(*) FROM taken) AS taken`
)
const taken = Number(rows[0].taken)
more = taken === CHUNK
if (taken === 0) return
// The chunk's files in REPEATED_RELEASES+ releases, from file_hashes_hash_size_release_idx alone (migration 5000).
await trx.rawQuery(`DROP TABLE IF EXISTS judge_tipped`)
await trx.rawQuery(
`CREATE TEMP TABLE judge_tipped (hash16k char(32), size bigint, just boolean NOT NULL, PRIMARY KEY (hash16k, size)) ON COMMIT DROP`
)
await trx.rawQuery(
`INSERT INTO judge_tipped
SELECT x.hash16k, x.size, c.outside < ${REPEATED_RELEASES}
FROM (SELECT DISTINCT h.hash16k, h.size FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id) x
CROSS JOIN LATERAL (
SELECT count(DISTINCT f.release_id) AS total,
count(DISTINCT f.release_id) FILTER (WHERE NOT EXISTS (SELECT 1 FROM judge_scope s WHERE s.release_id = f.release_id)) AS outside
FROM file_hashes f WHERE f.hash16k = x.hash16k AND f.size = x.size) c
WHERE c.total >= ${REPEATED_RELEASES}`
)
await trx.rawQuery(
`INSERT INTO judge_scope SELECT DISTINCT f.release_id FROM file_hashes f JOIN judge_tipped t USING (hash16k, size)
WHERE t.just AND NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = f.release_id AND v.rule IN ('dead', 'fake', 'repeated'))
ON CONFLICT DO NOTHING`
)
await trx.rawQuery(`ANALYZE judge_scope`)
const { rows: scope } = await trx.rawQuery(`SELECT count(*) AS n FROM judge_scope`)
round.releases! += Number(scope[0].n)
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL) ON COMMIT DROP`)
for (const rule of RULES) {
const { rowCount } = await trx.rawQuery(
`INSERT INTO judged (release_id, rule) SELECT id, ? FROM (${rule.sql(true)}) AS x(id) ON CONFLICT DO NOTHING`,
if (this.stopped) break
const { rows } = await db.rawQuery(rule.sql)
const ids = rows.map((r: Record<string, unknown>) => String(Object.values(r)[0])).filter((id: string) => !judged.has(id))
for (const id of ids) judged.add(id)
const stats = { matched: ids.length, added: 0, removed: 0 }
if (!dry) {
await db.transaction(async (trx) => {
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
for (let i = 0; i < ids.length; i += 50_000) {
await trx.rawQuery(`INSERT INTO judged SELECT unnest(?::bigint[]) ON CONFLICT DO NOTHING`, [ids.slice(i, i + 50_000)])
}
const removed = await trx.rawQuery(
`DELETE FROM release_verdicts v WHERE v.rule = ? AND NOT EXISTS (SELECT 1 FROM judged j WHERE j.release_id = v.release_id)`,
[rule.name]
)
round.rules[rule.name].matched += rowCount ?? 0
}
await trx.rawQuery(`ANALYZE judged`)
await this.apply(trx, 'judged', true, round)
})
const added = await trx.rawQuery(
`INSERT INTO release_verdicts (release_id, rule, rule_version)
SELECT j.release_id, ?, ? FROM judged j JOIN releases r ON r.id = j.release_id
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
WHERE release_verdicts.rule <> excluded.rule`,
[rule.name, JUDGE_VERSION]
)
stats.removed = removed.rowCount ?? 0
stats.added = added.rowCount ?? 0
})
}
round.rules[rule.name] = stats
}
return round
}
/**
* The matches (judged: release_id, rule; the first rule's each) into release_verdicts: verdicts no rule gives any
* more removed (scoped: of the releases in judge_scope only), changed ones written. Only the changes go through the
* insert: each match through its conflict check took 60–150 s a rule (LOG.md 2026-10-19 (80)).
*/
private async apply(trx: any, judged: string, scoped: boolean, round: JudgeRound) {
const tally = (rows: { rule: string; n: string }[], field: 'added' | 'removed') => {
for (const { rule, n } of rows) {
round.rules[rule] ??= { matched: 0, added: 0, removed: 0 }
round.rules[rule][field] += Number(n)
}
}
const { rows: removed } = await trx.rawQuery(
`WITH gone AS (DELETE FROM release_verdicts v
WHERE ${scoped ? 'v.release_id IN (SELECT release_id FROM judge_scope) AND' : ''}
NOT EXISTS (SELECT 1 FROM ${judged} j WHERE j.release_id = v.release_id)
RETURNING v.rule)
SELECT rule, count(*) AS n FROM gone GROUP BY 1`
)
tally(removed, 'removed')
const { rows: added } = await trx.rawQuery(
`WITH written AS (INSERT INTO release_verdicts (release_id, rule, rule_version)
SELECT j.release_id, j.rule, ? FROM ${judged} j LEFT JOIN release_verdicts v ON v.release_id = j.release_id
WHERE v.rule IS DISTINCT FROM j.rule AND EXISTS (SELECT 1 FROM releases r WHERE r.id = j.release_id)
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
WHERE release_verdicts.rule <> excluded.rule
RETURNING rule)
SELECT rule, count(*) AS n FROM written GROUP BY 1`,
[JUDGE_VERSION]
)
tally(added, 'added')
}
}
/** "dead 1,157,018 (+1,136 −712), …" for the log. */
function describe(round: JudgeRound): string {
return Object.entries(round.rules)
.map(([name, r]) => `${name} ${r.matched.toLocaleString('en')} (+${r.added.toLocaleString('en')} −${r.removed.toLocaleString('en')})`)
.join(', ')
}
+3 -4
View File
@@ -28,11 +28,10 @@ class Settler {
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
RETURNING id, readable, files, bytes, poster, sidecar_only),
RETURNING id, readable, files, bytes, poster),
marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
-- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
-- from it: the merger's sidecar fold looks at them.
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
-- Small ones may be an upload's .nfo or .nzb that came apart from it: the merger's sidecar fold looks at them.
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
ON CONFLICT DO NOTHING)
SELECT count(*)::int AS n FROM settled`,
[QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
-22
View File
@@ -1,22 +0,0 @@
/**
* 7z keeps its listing at the end of the last volume, so a split 7z gets a `7z-tail` probe candidate: the last
* segment of its highest volume (lib/edge/range_indexer.ts). RAR lists its files in every volume's header, and
* its first volume's start (one `archive` candidate) is enough. The live tail reads an upload in pieces while
* it's posted, and each piece queues a tail for the last volume it saw so far (TsIgNBewuFbuwWy: 7, for
* .002–.008; LOG.md 2026-10-23 (94)); merges bring their members' tails together. Of the given releases'
* waiting tails, those below the release's highest volume go. Those of the highest volume stay, all of them:
* a piece that saw only part of it queued a segment that isn't its last.
*/
export async function pruneSevenZipTails(trx: any, releaseIds: (number | string)[]): Promise<number> {
if (releaseIds.length === 0) return 0
const result = await trx.rawQuery(
`DELETE FROM probe_candidates pc USING (
SELECT id, status, volume, max(volume) OVER (PARTITION BY release_id) AS highest
FROM (SELECT id, release_id, status, substring(file from '\\.7z\\.(\\d{1,4})$')::int AS volume
FROM probe_candidates WHERE release_id = ANY(?) AND kind = '7z-tail') t
) s
WHERE pc.id = s.id AND s.status = 'waiting' AND s.volume < s.highest`,
[releaseIds.map(String)]
)
return result.rowCount ?? 0
}
+9 -7
View File
@@ -6,7 +6,7 @@ const BATCH = 2000
/** An upload's sidecar is posted with it: its release is looked for this far from it. */
const WINDOW_MINUTES = 10
/** The files an upload carries beside its content: an index, a checksum list, a scene release's own files. */
export const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
/** The words of a name worth comparing: 4+ letters or digits. */
function words(text: string): Set<string> {
@@ -53,17 +53,19 @@ export class SidecarFolder {
async round(): Promise<FoldRound> {
const round: FoldRound = { looked: 0, sidecars: 0, folded: 0 }
// A candidate whose release went (merged, deleted) leaves with its batch: a separate scan for those over the whole
// queue took 5+ min a round at 13M (LOG.md 2026-10-17 (72)).
const { rows: queued } = await db.rawQuery(
`SELECT f.release_id AS queued_id, r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
`SELECT r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
FROM fold_candidates f JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
[BATCH]
)
const done: number[] = queued.map((r: { queued_id: string }) => Number(r.queued_id))
const { rows: stale } = await db.rawQuery(
`SELECT f.release_id FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id WHERE r.id IS NULL LIMIT ?`,
[BATCH]
)
const done: number[] = stale.map((r: { release_id: string }) => Number(r.release_id))
for (const c of queued) {
if (c.id === null) continue
round.looked++
done.push(Number(c.id))
const { rows: chunks } = await db.rawQuery(`SELECT payload FROM release_chunks WHERE release_id = ?`, [c.id])
const files: ChunkFile[] = chunks.flatMap((x: { payload: Buffer }) => decodeChunk(x.payload))
if (files.length === 0 || !files.every((f) => SIDECAR.test(f.name))) continue
-8
View File
@@ -102,12 +102,6 @@ export class SpotInspector {
this.client = undefined
}
if (!round || round.inspected === 0) {
// Nothing to inspect for a while: closed (nntpIdleCloseMs), opened again with work.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs > 0 && IDLE_MS >= idleMs) {
this.client?.close()
this.client = undefined
}
for (let waited = 0; waited < IDLE_MS && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
}
}
@@ -355,8 +349,6 @@ export class SpotInspector {
const releaseId = Number(row.id)
await db.transaction(async (trx) => {
await trx.table('release_spots').insert({ release_id: releaseId, spot_id: spotId, files: files.length }).onConflict(['release_id', 'spot_id']).ignore()
// Its poster for poster search, as the batch store adds its releases' (migration 4000).
if (spot.poster) await trx.rawQuery(`INSERT INTO posters (poster) VALUES (left(?, 200)) ON CONFLICT DO NOTHING`, [spot.poster])
if (spot.password) {
await trx.table('release_passwords').insert({ release_id: releaseId, password: spot.password, source: spot.password_source ?? 'spot-text', spot_id: spotId }).onConflict(['release_id', 'password']).ignore()
}
-6
View File
@@ -60,12 +60,6 @@ export class Spotter {
this.client = undefined
}
const wait = coordinatorConfig.spots.everySeconds * 1000
// Quiet until the next round: closed (nntpIdleCloseMs), opened again then.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs > 0 && wait - (Date.now() - started) >= idleMs) {
this.client?.close()
this.client = undefined
}
for (let waited = Date.now() - started; waited < wait && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
}
}
+2
View File
@@ -14,6 +14,8 @@ export const signupValidator = vine.create({
email: email().unique({ table: 'users', column: 'email' }),
password: password(),
passwordConfirmation: password().sameAs('password'),
// Needed when SIGNUP is `invite` (config/access.ts).
invite: vine.string().trim().maxLength(32).optional(),
})
/**
+37
View File
@@ -0,0 +1,37 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import AccessService from '#services/access_service'
import { TIERS } from '#config/access'
/** Makes invite codes (SIGNUP=invite), optionally with a tier for the new account, e.g. a sold VIP year. */
export default class AccessInvite extends BaseCommand {
static commandName = 'access:invite'
static description = 'Makes invite codes, optionally giving a tier (e.g. --tier=vip --days=365)'
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'How many codes', default: 1 })
declare count: number
@flags.string({ description: `Tier the new account gets (${Object.keys(TIERS).join(', ')})` })
declare tier?: string
@flags.number({ description: 'For how many days the tier holds (without: no end)' })
declare days?: number
@flags.string({ description: 'A note, e.g. who it is for or the payment' })
declare note?: string
async run() {
if (this.tier && !TIERS[this.tier]) {
this.logger.error(`No such tier: ${this.tier}`)
this.exitCode = 1
return
}
const codes = Array.from({ length: this.count }, () => AccessService.newKey())
await db.table('invites').multiInsert(
codes.map((code) => ({ code, tier: this.tier ?? null, tier_days: this.tier ? (this.days ?? null) : null, note: this.note ?? null }))
)
for (const code of codes) this.logger.log(code)
}
}
+28
View File
@@ -0,0 +1,28 @@
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import User from '#models/user'
/** Suspends a user (Newznab error 101, the catalog 403), e.g. for a shared key; --undo lifts it. */
export default class AccessSuspend extends BaseCommand {
static commandName = 'access:suspend'
static description = 'Suspends a user (or lifts it with --undo)'
static options: CommandOptions = { startApp: true }
@args.string({ description: 'The user’s email' })
declare email: string
@flags.boolean({ description: 'Lift the suspension' })
declare undo: boolean
async run() {
const user = await User.findBy('email', this.email)
if (!user) {
this.logger.error(`No user ${this.email}`)
this.exitCode = 1
return
}
user.suspended = !this.undo
await user.save()
this.logger.info(`${user.email}: ${user.suspended ? 'suspended' : 'active'}`)
}
}
+32
View File
@@ -0,0 +1,32 @@
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import User from '#models/user'
import AccessService from '#services/access_service'
/** Gives a user a tier (a payment, by hand until one is taken on the site); a running one is extended. */
export default class AccessTier extends BaseCommand {
static commandName = 'access:tier'
static description = "Sets a user's tier, e.g. access:tier someone@example.com vip --days=365"
static options: CommandOptions = { startApp: true }
@args.string({ description: 'The user’s email' })
declare email: string
@args.string({ description: 'Tier (config/access.ts)' })
declare tier: string
@flags.number({ description: 'For how many days (without: no end); the same tier still running is extended' })
declare days?: number
async run() {
const user = await User.findBy('email', this.email)
if (!user) {
this.logger.error(`No user ${this.email}`)
this.exitCode = 1
return
}
AccessService.grantTier(user, this.tier, this.days ?? null)
await user.save()
this.logger.info(`${user.email}: ${user.tier} ${user.tierUntil ? `until ${user.tierUntil.toISO()}` : 'without end'}`)
}
}
+1 -22
View File
@@ -6,9 +6,7 @@ import Planner from '#services/planner'
import FragmentMerger from '#services/fragment_merger'
import Settler from '#services/settler'
import RangeBlocks from '#services/range_blocks'
import Leases from '#services/leases'
import NntpPool from '#services/nntp_pool'
import { NntpShare } from '#services/nntp_shares'
import coordinatorConfig from '#config/coordinator'
const TICK_MS = 2000
@@ -44,7 +42,6 @@ export default class CoordinatorRun extends BaseCommand {
const [index, count] = (this.shard ?? '1/1').split('/').map(Number)
if (!(count >= 1 && index >= 1 && index <= count)) throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
const prefetcher = new Prefetcher({ index: index - 1, count })
const share = new NntpShare(count > 1 ? `coordinator ${index}/${count}` : 'coordinator', coordinatorConfig.nntpShares.coordinatorPriority, coordinatorConfig.nntpMinConnections, coordinatorConfig.nntpConnections)
const main = index === 1
let stopping = false
let stopped: () => void
@@ -57,21 +54,6 @@ export default class CoordinatorRun extends BaseCommand {
await done
})
this.logger.info(count > 1 ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).` : 'Coordinator started.')
// The pool's size: its share of the account's connections, the prober the rest (app/services/nntp_shares.ts).
// Before the first connection: until then it's NNTP_CONNECTIONS.
const first = await NntpPool.share(share)
this.logger.info(`NNTP share: ${first.granted} connections.`)
const sharing = (async () => {
while (!stopping) {
try {
const { was, granted } = await NntpPool.share(share)
if (granted !== was) this.logger.info(`NNTP share: ${granted} connections (was ${was}).`)
} catch (error: any) {
this.logger.error(`NNTP share failed: ${error.message}`)
}
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !stopping; waited += TICK_MS) await sleep(TICK_MS)
}
})()
// Two loops: the prefetcher's ticks don't wait for merging or planning (a merge round
// of up to MERGE_BUDGET_MS left the connections idle once their targets were fetched).
const prefetching = (async () => {
@@ -88,8 +70,6 @@ export default class CoordinatorRun extends BaseCommand {
const blocks = (async () => {
while (main && !stopping) {
try {
const abandoned = await Leases.finishAbandoned()
if (abandoned > 0) this.logger.info(`Finished ${abandoned} ranges read to their end whose worker went quiet.`)
await RangeBlocks.rebuild()
} catch (error: any) {
this.logger.error(`Range blocks failed: ${error.message}`)
@@ -143,13 +123,12 @@ export default class CoordinatorRun extends BaseCommand {
}
await sleep(TICK_MS)
}
await Promise.all([prefetching, blocks, sharing])
await Promise.all([prefetching, blocks])
// The chunks in flight, for at most STOP_WAIT_MS (a connection the server holds back can take minutes); then closed.
const waitUntil = Date.now() + STOP_WAIT_MS
while (prefetcher.active > 0 && Date.now() < waitUntil) await sleep(200)
if (prefetcher.active > 0) this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
await NntpPool.close()
await share.leave().catch(() => {})
this.logger.info('Coordinator stopped.')
stopped!()
await this.terminate()
-17
View File
@@ -1,17 +0,0 @@
import { BaseCommand } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { Par2Pruner } from '#services/par2_pruner'
/** The par2 prune (Par2Pruner) in one go: a whole pass from the first set, instead of the merger's 2 minutes an hour. */
export default class Par2Prune extends BaseCommand {
static commandName = 'par2:prune'
static description = 'Drops the par2 file lists the merge can no longer use: one whole pass over every set, then exits'
static options: CommandOptions = { startApp: true }
async run() {
await db.from('worker_cursors').where('name', 'par2-prune').update({ key: '' })
const round = await new Par2Pruner().round()
this.logger.info(`Pruned ${round.rows} par2 file rows: ${round.complete} complete sets, ${round.judged} judged, ${round.final} final.`)
}
}
+3 -3
View File
@@ -5,10 +5,10 @@ import { ReleaseJudge, RULES } from '#services/release_judge'
/** The junk judge (REBUILD.md, junk): releases judged after their probes; verdicts hide them from search. */
export default class ReleasesJudge extends BaseCommand {
static commandName = 'releases:judge'
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable): the changed ones every few minutes, all daily'
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable), hourly'
static options: CommandOptions = { startApp: true, staysAlive: true }
@flags.boolean({ description: 'One full round, then exit' })
@flags.boolean({ description: 'One round, then exit' })
declare once: boolean
@flags.boolean({ description: 'With --once: count only, write no verdicts' })
@@ -17,7 +17,7 @@ export default class ReleasesJudge extends BaseCommand {
async run() {
const judge = new ReleaseJudge()
if (this.once) {
const round = await judge.full(this.dry)
const round = await judge.round(this.dry)
for (const rule of RULES) {
const r = round.rules[rule.name]
if (r) this.logger.info(`${rule.name}: ${r.matched} (+${r.added} −${r.removed}) — ${rule.says}`)
+31
View File
@@ -0,0 +1,31 @@
import env from '#start/env'
/**
* Who may use the catalog and the Newznab API.
*
* `open` (the default): as before, one key for Newznab (NEWZNAB_API_KEY, or
* none) and an open web catalog, no limits. `accounts`: every request names a
* user (a Newznab `apikey`, or the web interface's token), whose tier sets what
* they may do per UTC day and how far back they see.
*/
export const ACCESS = env.get('ACCESS', 'open')
/** Who may sign up: anyone, with an unused invite code, or no one. With accounts, invites unless set. */
export const SIGNUP = env.get('SIGNUP', ACCESS === 'accounts' ? 'invite' : 'open')
export interface TierLimits {
/** Searches and release details per day (Newznab t=search & co., the catalog); null: no limit. */
apiHits: number | null
/** NZBs per day; null: no limit. */
grabs: number | null
/** How many days back releases are found and fetched; null: everything. */
retentionDays: number | null
}
/** A tier that runs out (users.tier_until) falls back to `free`. */
export const TIERS: Record<string, TierLimits> = {
free: { apiHits: 100, grabs: 10, retentionDays: 365 },
vip: { apiHits: 10_000, grabs: 1_000, retentionDays: null },
// Staff and friends.
unlimited: { apiHits: null, grabs: null, retentionDays: null },
}
+2 -26
View File
@@ -31,21 +31,8 @@ export default {
},
/** A leased range without a heartbeat for this long goes back to the queue. */
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
/** NNTP connections the coordinator fetches headers with. */
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
/**
* The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
* all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
* and 2 for a hedged request's or a reconnect's new connection while the old one closes).
*/
nntpShares: {
total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
everyMs: 15_000,
/** Headers before probes: history takes what it can use, the prober the rest. */
coordinatorPriority: 2,
proberPriority: 1,
},
/**
* A reply with no data for this long fails, and its connection is replaced.
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
@@ -81,24 +68,13 @@ export default {
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
*/
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
/**
* A connection with nothing to do for this long is closed, and opened again
* when there's work. 0: kept. Eweka drops connections that stay quiet, and
* the next command on one failed ("Spot inspection failed: NNTP connection
* closed"); idle ones also take the account's connections from downloads.
*/
nntpIdleCloseMs: env.get('NNTP_IDLE_CLOSE_SECONDS', 60) * 1000,
probe: {
/**
* At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
* NNTP connections the prober reads articles with, besides the
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
*/
connections: env.get('PROBE_CONNECTIONS', 6),
/** Of them, always (its share's minimum; nntpShares). */
minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
/** No claims while the filesystem under `/` (the NVMe, in the container) has less free (app/services/disk_space.ts). 0: off. */
minFreeGb: env.get('PROBE_MIN_FREE_GB', 0),
/** Candidates claimed at a time. */
batch: 400,
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
@@ -0,0 +1,49 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Accounts with limits (config/access.ts): a Newznab key and a tier per user,
* invites, and what each user used per day.
*/
export default class extends BaseSchema {
async up() {
this.schema.alterTable('users', (table) => {
// Existing users get a key of their own (a volatile default is evaluated per row).
table.string('api_key', 32).notNullable().unique().defaultTo(this.raw(`md5(random()::text || clock_timestamp()::text)`))
table.string('tier', 16).notNullable().defaultTo('free')
// Null: the tier doesn't run out.
table.timestamp('tier_until', { useTz: true }).nullable()
table.boolean('suspended').notNullable().defaultTo(false)
})
this.schema.createTable('invites', (table) => {
table.string('code', 32).primary()
// What the invite gives (e.g. a sold VIP invite); null: the free tier.
table.string('tier', 16).nullable()
table.integer('tier_days').nullable()
table.string('note').nullable()
table.integer('used_by').nullable().references('id').inTable('users').onDelete('SET NULL')
table.timestamp('used_at', { useTz: true }).nullable()
table.timestamp('created_at', { useTz: true }).notNullable().defaultTo(this.now())
})
this.schema.createTable('usage', (table) => {
table.integer('user_id').notNullable().references('id').inTable('users').onDelete('CASCADE')
// A UTC day.
table.date('day').notNullable()
table.integer('api_hits').notNullable().defaultTo(0)
table.integer('grabs').notNullable().defaultTo(0)
table.primary(['user_id', 'day'])
})
}
async down() {
this.schema.dropTable('usage')
this.schema.dropTable('invites')
this.schema.alterTable('users', (table) => {
table.dropColumn('api_key')
table.dropColumn('tier')
table.dropColumn('tier_until')
table.dropColumn('suspended')
})
}
}
@@ -1,16 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* releases.sidecar_only: set by the batch store when it inserts a small release (at most 3 files, under 1 MB) whose
* files are all sidecars (.nfo, .sfv, .nzb, …). The settler queues only those for the sidecar fold: it queued every
* small release, 13.5M (single images mostly), and the fold's queue scans held the merger back (LOG.md 2026-10-17 (72)).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS sidecar_only boolean`)
}
async down() {
this.schema.raw(`ALTER TABLE releases DROP COLUMN IF EXISTS sidecar_only`)
}
}
@@ -1,16 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* worker_cursors.key: a tie-breaker after `at`. The par2 pruner's cursor was a timestamp alone, read back to the
* millisecond: 20,000 sets of one prober flush share created_at 2026-10-15 14:38:39.883984, and every round took the
* same batch again (LOG.md 2026-10-17 (73)). Now (at to the microsecond, set_id).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`ALTER TABLE worker_cursors ADD COLUMN IF NOT EXISTS key text`)
}
async down() {
this.schema.raw(`ALTER TABLE worker_cursors DROP COLUMN IF EXISTS key`)
}
}
@@ -1,26 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* nntp_shares: the account's connections shared by the processes that read articles (app/services/nntp_shares.ts).
* Each process's row says what it could use and has open, and what it may open; a row without a heartbeat for a
* minute is a process gone (its connections closed with it). The split was fixed per process: with history paused
* the coordinator's 38 stood idle while the prober's 8 were the limit (LOG.md 2026-10-18 (75)).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`
CREATE TABLE IF NOT EXISTS nntp_shares (
name text PRIMARY KEY,
priority integer NOT NULL,
min integer NOT NULL,
want integer NOT NULL,
open integer NOT NULL,
granted integer NOT NULL,
heartbeat_at timestamptz NOT NULL DEFAULT now()
)`)
}
async down() {
this.schema.raw(`DROP TABLE IF EXISTS nntp_shares`)
}
}
@@ -1,26 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* probe_cache's primary key in the catalog's tablespace (default_tablespace: nvme on homelab); the table stays on the
* HDD. Each claim looks its candidates up by message_id (`message_id = ANY(…)`): with the index on the HDD that took
* 1.27 s a claim, 48 s of each minute, the prober's lanes waiting on claims (LOG.md 2026-10-18 (76)). ~97% aren't in it,
* so only the index is read. Homelab moved it by hand (prober stopped); a no-op where it's there already, or without a
* default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE c.relname = 'probe_cache_pkey' AND coalesce(t.spcname, 'pg_default') <> ?`,
[target]
)
if (rows.length > 0) await db.rawQuery(`ALTER INDEX probe_cache_pkey SET TABLESPACE ${target}`)
})
}
async down() {}
}
@@ -1,12 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Was: release_inspections' rows on the HDD (pg_default), for the NVMe's room during the probe backlog (LOG.md
* 2026-10-18 (79)). The judge's fake rule reads all of it every round: on the HDD that took 17–18 min, was ~3
* (LOG.md 2026-10-19 (81)). Migration 4800 moves it back; nothing here now.
*/
export default class extends BaseSchema {
async up() {}
async down() {}
}
@@ -1,22 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The junk judge's locked rule by an index of the passworded releases (1.2M of 74M), not a scan of all of releases
* every round (30 s; LOG.md 2026-10-19 (80)). And the posters of the spot inspector's own releases (source `spot`),
* which it didn't add to posters: poster search, and the judge's scraped rule by posters, missed them.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS releases_passworded_idx ON releases (id) WHERE passworded`)
this.schema.raw(
`INSERT INTO posters (poster) SELECT DISTINCT left(s.poster, 200) FROM spots s
JOIN releases r ON r.id = s.release_id AND r.source = 'spot' WHERE s.poster IS NOT NULL ON CONFLICT DO NOTHING`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS releases_passworded_idx`)
}
}
@@ -1,25 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* release_inspections back in the catalog's tablespace (default_tablespace: nvme on homelab), where migration 4600 had
* moved it to the HDD: the judge's fake rule reads all of it every round, 17–18 min on the HDD, ~3 on the NVMe (LOG.md
* 2026-10-19 (81)). Homelab moved it by hand (prober, merger and judge stopped); a no-op where it's there already, or
* without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE c.relname = 'release_inspections' AND c.relkind = 'r' AND coalesce(t.spcname, 'pg_default') <> ?`,
[target]
)
if (rows.length > 0) await db.rawQuery(`ALTER TABLE release_inspections SET TABLESPACE ${target}`)
})
}
async down() {}
}
@@ -1,119 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The junk judge by what changed (LOG.md 2026-10-19 (83)): a full round over every release took 35–38 min of each
* hour, longer with every inspection the prober added. Triggers note each release whose facts a rule reads changed
* (judge_dirty: its probes' statuses, inspections, file hashes, passwords, spots, par2 sets, merges, its own judged
* columns); the judge judges those every few minutes, and every release in a daily full round (judge_state: when the
* last one was, by which rule set), its matches gathered in judge_full. judge_dirty is unlogged: what a crash loses
* the next full round judges.
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_dirty (release_id bigint NOT NULL)`)
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_full (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL)`)
this.schema.raw(
`CREATE TABLE IF NOT EXISTS judge_state (id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1), full_at timestamptz NOT NULL, version integer NOT NULL)`
)
// The verdicts are an hourly full round's: the next full one a day from now.
this.schema.raw(`INSERT INTO judge_state (full_at, version) VALUES (now(), 1) ON CONFLICT DO NOTHING`)
// Rows of a table with release_id: the releases of the rows inserted, deleted, or updated (both, when moved).
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_by_release_id() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF TG_OP IN ('INSERT', 'UPDATE') THEN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM new_rows WHERE release_id IS NOT NULL;
END IF;
IF TG_OP IN ('UPDATE', 'DELETE') THEN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM old_rows WHERE release_id IS NOT NULL;
END IF;
RETURN NULL;
END $$`)
// probe_candidates: only a status that counts for the dead rule, or a candidate moved by a merge (not the claims'
// 'taken', the settler's ready).
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_probes() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
INSERT INTO judge_dirty (release_id)
SELECT DISTINCT unnest(ARRAY[n.release_id, o.release_id]) FROM new_rows n JOIN old_rows o ON o.id = n.id
WHERE n.release_id IS DISTINCT FROM o.release_id
OR (n.status IS DISTINCT FROM o.status AND (n.status IN ('done', 'missing', 'error') OR o.status IN ('done', 'missing', 'error')));
RETURN NULL;
END $$`)
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_merges() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT into_id FROM new_rows;
RETURN NULL;
END $$`)
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_releases() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO judge_dirty (release_id) SELECT id FROM new_rows;
ELSIF TG_OP = 'DELETE' THEN
INSERT INTO judge_dirty (release_id) SELECT id FROM old_rows;
ELSE
INSERT INTO judge_dirty (release_id) VALUES (NEW.id);
END IF;
RETURN NULL;
END $$`)
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets']) {
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON ${table}`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON ${table}`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON ${table}`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON ${table} REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON ${table} REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON ${table} REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
}
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON probe_candidates`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON probe_candidates`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON probe_candidates REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_probes()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON probe_candidates REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON release_merges`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON release_merges REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_merges()`
)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON releases`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON releases`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON releases`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON releases REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON releases REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
)
// Per row, and only when a column a rule reads changed: the tail and the settler update releases all the time.
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON releases FOR EACH ROW
WHEN (OLD.name IS DISTINCT FROM NEW.name OR OLD.poster IS DISTINCT FROM NEW.poster OR OLD.source IS DISTINCT FROM NEW.source
OR OLD.file_total IS DISTINCT FROM NEW.file_total OR OLD.passworded IS DISTINCT FROM NEW.passworded
OR OLD.settled_at IS DISTINCT FROM NEW.settled_at OR OLD.readable IS DISTINCT FROM NEW.readable
OR OLD.inner_name IS DISTINCT FROM NEW.inner_name OR OLD.group_id IS DISTINCT FROM NEW.group_id
OR OLD.first_article IS DISTINCT FROM NEW.first_article OR OLD.last_article IS DISTINCT FROM NEW.last_article
OR OLD.posted_at IS DISTINCT FROM NEW.posted_at OR OLD.last_posted_at IS DISTINCT FROM NEW.last_posted_at
OR OLD.complete IS DISTINCT FROM NEW.complete)
EXECUTE FUNCTION judge_dirty_releases()`
)
}
async down() {
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets', 'probe_candidates', 'release_merges', 'releases']) {
for (const event of ['insert', 'update', 'delete']) this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_${event} ON ${table}`)
}
this.schema.raw(`DROP FUNCTION IF EXISTS judge_dirty_by_release_id(), judge_dirty_probes(), judge_dirty_merges(), judge_dirty_releases()`)
this.schema.raw(`DROP TABLE IF EXISTS judge_dirty, judge_full, judge_state`)
}
}
@@ -1,21 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* file_hashes by (hash16k, size, release_id), in place of (hash16k): how many releases have a file, from the index
* alone. The judge counts that for every file of each changed release (repeated); with the row fetched for size and
* release_id per entry, a file posted in 100k releases cost 100k heap reads every round (24–32 s a chunk; LOG.md
* 2026-10-19 (84)). The merger and pruner join on (hash16k, size) too.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_size_release_idx ON file_hashes (hash16k, size, release_id)`)
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_idx`)
}
async down() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_idx ON file_hashes (hash16k)`)
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_size_release_idx`)
}
}
@@ -1,17 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Probe candidates added to a release after it settled (a later range of its upload, a merge into it) stayed not
* ready: the settler marks a release's candidates only when it settles it (LOG.md 2026-10-20 (87)). 93,642 waited
* since 2026-10-13 (17k lone, 31k par2, 45k archive). The batch store and the merge mark them now; these are made
* claimable once.
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`
UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.status = 'waiting' AND NOT pc.ready AND r.id = pc.release_id AND r.settled_at IS NOT NULL`)
}
async down() {}
}
+41 -1
View File
@@ -87,6 +87,25 @@ export class GroupSchema extends BaseModel {
declare tail: boolean
}
export class InviteSchema extends BaseModel {
static $columns = ['code', 'createdAt', 'note', 'tier', 'tierDays', 'usedAt', 'usedBy'] as const
$columns = InviteSchema.$columns
@column({ isPrimary: true })
declare code: string
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()
declare note: string | null
@column()
declare tier: string | null
@column()
declare tierDays: number | null
@column.dateTime()
declare usedAt: DateTime | null
@column()
declare usedBy: number | null
}
export class NodeAccessTokenSchema extends BaseModel {
static $columns = ['abilities', 'createdAt', 'expiresAt', 'hash', 'id', 'lastUsedAt', 'name', 'tokenableId', 'type', 'updatedAt'] as const
$columns = NodeAccessTokenSchema.$columns
@@ -399,9 +418,24 @@ export class RuleSetSchema extends BaseModel {
declare version: number
}
export class UsageSchema extends BaseModel {
static $columns = ['apiHits', 'day', 'grabs', 'userId'] as const
$columns = UsageSchema.$columns
@column()
declare apiHits: number
@column.date()
declare day: DateTime
@column()
declare grabs: number
@column({ isPrimary: true })
declare userId: number
}
export class UserSchema extends BaseModel {
static $columns = ['createdAt', 'email', 'fullName', 'id', 'password', 'updatedAt'] as const
static $columns = ['apiKey', 'createdAt', 'email', 'fullName', 'id', 'password', 'suspended', 'tier', 'tierUntil', 'updatedAt'] as const
$columns = UserSchema.$columns
@column()
declare apiKey: string
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()
@@ -412,6 +446,12 @@ export class UserSchema extends BaseModel {
declare id: number
@column({ serializeAs: null })
declare password: string
@column()
declare suspended: boolean
@column()
declare tier: string
@column.dateTime()
declare tierUntil: DateTime | null
@column.dateTime({ autoCreate: true, autoUpdate: true })
declare updatedAt: DateTime | null
}
+7 -13
View File
@@ -115,14 +115,11 @@ services:
<<: *app-env
NNTP_CONNECTIONS: ${NNTP_CONNECTIONS:-38}
# Phase 2's prober: reads the probe queue's articles over connections of its
# own. The account has 50 (NNTP_ACCOUNT_CONNECTIONS): 5 reserved (the
# coordinator's plain one, the spotter's and the spot inspector's 1 each, 2
# to spare: a hedged request closes its connection and opens another; with
# 10 the spot inspector got 502 Too many connections), the other 45 shared
# by the coordinator (NNTP_MIN_CONNECTIONS–NNTP_CONNECTIONS) and the prober
# (PROBE_MIN_CONNECTIONS–PROBE_CONNECTIONS), headers first
# (app/services/nntp_shares.ts).
# Phase 2's prober: reads the probe queue's articles over PROBE_CONNECTIONS
# connections of its own. The account has 50: the coordinator's 38 (+ its
# plain one), the prober's 8, the spotter's and the spot inspector's 1 each,
# 2 to spare (a hedged request closes its connection and opens another)
# (with 10 the spot inspector got 502 Too many connections).
prober:
<<: *app
command: ['node', 'ace', 'probe:run']
@@ -130,9 +127,6 @@ services:
environment:
<<: *app-env
PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-8}
# No claims under this much free on the NVMe (its overlay root): 2 GB above the 10 GB floor, for what the
# tail, the merger and the judge write meanwhile.
PROBE_MIN_FREE_GB: ${PROBE_MIN_FREE_GB:-12}
# The par2 merge: releases whose files one par2 set describes (what the prober read) become one. No NNTP.
merger:
@@ -140,8 +134,8 @@ services:
command: ['node', 'ace', 'releases:merge']
volumes: []
# The junk judge: releases judged by what their probes showed (dead, fake, repeated, locked, unusable), the changed
# ones every 2 minutes, all daily; verdicts hide them from search. No NNTP.
# The junk judge: releases judged hourly by what their probes showed (dead, fake, repeated, locked, unusable);
# verdicts hide them from search. No NNTP.
judge:
<<: *app
command: ['node', 'ace', 'releases:judge']
+1 -1
View File
@@ -2,4 +2,4 @@
* Newznab API formatting, independent of AdonisJS. From voyager 1 (its CONCEPT.md D23).
*/
export { CATEGORIES, categoryLabel, expandCategories, parentCategory } from './categories.js'
export { capsXml, errorXml, resultsXml, type NewznabItem, type NewznabServer } from './xml.js'
export { capsXml, errorXml, resultsXml, type ApiLimits, type NewznabItem, type NewznabServer } from './xml.js'
+24 -4
View File
@@ -27,7 +27,7 @@ export interface NewznabServer {
apiKey?: string
}
/** Newznab error codes: 100 credentials, 200/201 parameters, 202 function, 300 item. */
/** Newznab error codes: 100 credentials, 101 suspended, 102 privileges, 200/201 parameters, 202 function, 300 item, 500/501 request/download limit. */
/** Spotnet UBB markup (or the HTML of old spots) as plain text: [br] as line breaks, other tags dropped. */
export function ubbToText(ubb: string): string {
return ubb
@@ -44,11 +44,23 @@ export function errorXml(code: number, description: string): string {
.end({ prettyPrint: true })
}
export function capsXml(server: NewznabServer, limits: { max: number; default: number }): string {
/** A caller's day so far and its limits (null: none), as nZEDb-style indexers report them. */
export interface ApiLimits {
apiCurrent: number
apiMax: number | null
grabCurrent: number
grabMax: number | null
}
export function capsXml(
server: NewznabServer,
limits: { max: number; default: number },
registration = { available: false, open: false }
): string {
const caps = create({ version: '1.0', encoding: 'UTF-8' }).ele('caps')
caps.ele('server', { version: '1.0', title: server.title, strapline: 'Usenet indexer', url: `${server.baseUrl}/` })
caps.ele('limits', { max: String(limits.max), default: String(limits.default) })
caps.ele('registration', { available: 'no', open: 'no' })
caps.ele('registration', { available: registration.available ? 'yes' : 'no', open: registration.open ? 'yes' : 'no' })
const searching = caps.ele('searching')
searching.ele('search', { available: 'yes', supportedParams: 'q' })
searching.ele('tv-search', { available: 'yes', supportedParams: 'q,season,ep' })
@@ -63,7 +75,7 @@ export function capsXml(server: NewznabServer, limits: { max: number; default: n
return caps.end({ prettyPrint: true })
}
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number): string {
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number, limits?: ApiLimits): string {
const rss = create({ version: '1.0', encoding: 'UTF-8' }).ele('rss', {
'version': '2.0',
'xmlns:atom': 'http://www.w3.org/2005/Atom',
@@ -74,6 +86,14 @@ export function resultsXml(server: NewznabServer, items: NewznabItem[], offset:
channel.ele('description').txt(`${server.title} API results`)
channel.ele('link').txt(`${server.baseUrl}/`)
channel.ele('newznab:response', { offset: String(offset), total: String(total) })
if (limits) {
channel.ele('newznab:apilimits', {
apicurrent: String(limits.apiCurrent),
...(limits.apiMax !== null && { apimax: String(limits.apiMax) }),
grabcurrent: String(limits.grabCurrent),
...(limits.grabMax !== null && { grabmax: String(limits.grabMax) }),
})
}
const key = server.apiKey ? `&apikey=${encodeURIComponent(server.apiKey)}` : ''
for (const item of items) {
+126
View File
@@ -891,3 +891,129 @@ kbd {
margin-top: 4px;
color: var(--faint);
}
/* --- Account (ACCESS=accounts) ------------------------------------------ */
.account-button {
height: 32px;
padding: 0 12px;
font-size: 13px;
}
.account-button .tier {
color: var(--accent);
text-transform: uppercase;
font-size: 11px;
letter-spacing: 0.04em;
}
.dialog {
width: min(420px, calc(100vw - 32px));
padding: 0;
border: 1px solid var(--border);
border-radius: var(--radius);
background: var(--surface);
color: var(--text);
box-shadow: var(--shadow);
}
.dialog::backdrop {
background: rgba(10, 14, 20, 0.35);
}
.dialog-head {
display: flex;
align-items: center;
justify-content: space-between;
padding: 14px 18px;
border-bottom: 1px solid var(--border);
}
.dialog-head h2 {
margin: 0;
font-size: 16px;
}
.dialog-body {
padding: 18px;
}
.form {
display: grid;
gap: 12px;
}
.form label {
display: grid;
gap: 4px;
font-size: 13px;
color: var(--muted);
}
.form input {
height: 38px;
padding: 0 10px;
border: 1px solid var(--border);
border-radius: 8px;
background: var(--bg);
color: var(--text);
font-size: 14px;
}
.form input:focus {
outline: 2px solid var(--accent-soft);
border-color: var(--accent);
}
.form .button {
justify-content: center;
}
.form-switch {
margin: 4px 0 0;
font-size: 13px;
color: var(--muted);
text-align: center;
}
.form-error {
margin: 0;
color: var(--bad);
font-size: 13px;
}
.api-key {
display: flex;
gap: 8px;
align-items: center;
margin-bottom: 18px;
}
.api-key code {
flex: 1;
min-width: 0;
padding: 8px 10px;
border-radius: 8px;
background: var(--bg);
font-family: var(--mono);
font-size: 12.5px;
overflow-wrap: anywhere;
}
.meter {
height: 6px;
margin-top: 4px;
border-radius: 3px;
background: var(--border);
overflow: hidden;
}
.meter span {
display: block;
height: 100%;
background: var(--accent);
}
.meter-full span {
background: var(--bad);
}
+189 -7
View File
@@ -51,6 +51,26 @@ const state = {
let requestId = 0
let searchAbort = null
/** Signing in (GET /api/v1/access): `open` needs none; with `accounts` the catalog takes a token. */
const account = {
mode: 'open',
signup: 'open',
token: storage('get', 'token'),
/** The profile's access: tier, limits, today's usage, the Newznab key. */
access: null,
}
/** localStorage, which a private window may refuse. */
function storage(action, key, value) {
try {
if (action === 'get') return localStorage.getItem(key)
if (action === 'set') localStorage.setItem(key, value)
else localStorage.removeItem(key)
} catch {
return null
}
}
// --- Formatting ------------------------------------------------------------
function formatSize(bytes) {
@@ -147,12 +167,26 @@ function syncControls() {
// --- API -------------------------------------------------------------------
async function api(path, { signal } = {}) {
const response = await fetch(path, { headers: { accept: 'application/json' }, signal })
if (!response.ok) throw new Error(`${response.status} ${response.statusText}`)
async function api(path, { signal, method = 'GET', body } = {}) {
const headers = { accept: 'application/json' }
if (account.token) headers.authorization = `Bearer ${account.token}`
if (body) headers['content-type'] = 'application/json'
const response = await fetch(path, { method, headers, signal, body: body && JSON.stringify(body) })
if (!response.ok) {
const error = await response.json().catch(() => ({}))
// A token that no longer works: sign in again.
if (response.status === 401 && account.mode === 'accounts' && !path.startsWith('/api/v1/auth/')) signOut()
throw new Error(error.error ?? error.errors?.[0]?.message ?? `${response.status} ${response.statusText}`)
}
return response.json()
}
/** A release's NZB as a plain link: with accounts it carries the key (a link can't send the token). */
function nzbHref(id) {
const key = account.mode === 'accounts' && account.access?.apiKey
return `/api/v1/releases/${id}/nzb${key ? `?apikey=${encodeURIComponent(key)}` : ''}`
}
function searchParams(page) {
const params = new URLSearchParams({ sort: state.sort, page: String(page), per_page: '50' })
if (state.q) params.set('q', state.q)
@@ -229,7 +263,7 @@ function releaseRow(release) {
row.querySelector('.release-size').textContent = formatSize(release.size)
const nzb = row.querySelector('.nzb')
nzb.href = `/api/v1/releases/${release.id}/nzb`
nzb.href = nzbHref(release.id)
row.querySelector('.release-main').addEventListener('click', () => openRelease(release.id, true))
return row
}
@@ -264,6 +298,8 @@ async function loadResults({ append = false } = {}) {
(shown < result.total ? ` · showing ${number.format(shown)}` : '')
$('#empty').hidden = result.total !== 0
$('#more').hidden = shown >= result.total && !result.totalCapped
// The day's count in the account button (the profile isn't counted).
if (account.access) loadProfile().catch(() => {})
} catch (error) {
if (id !== requestId) return
$('#list').classList.remove('stale')
@@ -299,7 +335,7 @@ function renderDetail(release) {
const actions = el(
'div',
{ class: 'actions' },
el('a', { class: 'button button-primary', href: `/api/v1/releases/${release.id}/nzb` },
el('a', { class: 'button button-primary', href: nzbHref(release.id) },
el('span', { text: 'Download NZB' })),
el('button', {
class: 'button',
@@ -418,8 +454,151 @@ function closeRelease(push = false) {
document.querySelector(`.release[data-id="${id}"] .release-main`)?.focus()
}
// --- Account (ACCESS=accounts) ---------------------------------------------
function renderAccountButton() {
const button = $('#account')
button.hidden = account.mode !== 'accounts'
if (!account.access) return button.replaceChildren('Sign in')
const { tier, usage, limits } = account.access
const of = (used, max) => (max === null ? number.format(used) : `${number.format(used)}/${number.format(max)}`)
button.title = 'Searches and NZBs today'
button.replaceChildren(el('span', { class: 'tier', text: tier }), ` ${of(usage.apiHits, limits.apiHits)} · ${of(usage.grabs, limits.grabs)} NZB`)
}
async function loadProfile() {
const { data } = await api('/api/v1/account/profile')
account.access = data.access
renderAccountButton()
}
/** Signed in with a new token: the profile, then the search that was waiting. */
async function signedIn(token) {
account.token = token
storage('set', 'token', token)
await loadProfile()
$('#account-dialog').close()
loadResults()
if (state.release) openRelease(state.release)
}
function signOut() {
if (account.token) api('/api/v1/account/logout', { method: 'POST' }).catch(() => {})
account.token = null
account.access = null
storage('remove', 'token')
renderAccountButton()
closeRelease()
$('#list').replaceChildren()
$('#more').hidden = true
$('#empty').hidden = true
$('#summary').textContent = 'Sign in to search the index.'
openAccount('signin')
}
function field(label, props) {
return el('label', {}, label, el('input', props))
}
function accountForm(view) {
const signup = view === 'signup'
const error = el('p', { class: 'form-error', role: 'alert', hidden: true })
const form = el(
'form',
{ class: 'form' },
field('Email', { name: 'email', type: 'email', autocomplete: 'email', required: true }),
field('Password', { name: 'password', type: 'password', autocomplete: signup ? 'new-password' : 'current-password', required: true, minlength: signup ? '8' : null }),
signup ? field('Password again', { name: 'passwordConfirmation', type: 'password', autocomplete: 'new-password', required: true }) : null,
signup && account.signup === 'invite' ? field('Invite code', { name: 'invite', autocomplete: 'off', spellcheck: 'false', required: true }) : null,
error,
el('button', { class: 'button button-primary', type: 'submit', text: signup ? 'Create account' : 'Sign in' })
)
form.addEventListener('submit', async (event) => {
event.preventDefault()
const values = Object.fromEntries(new FormData(form))
error.hidden = true
try {
const path = signup ? '/api/v1/auth/signup' : '/api/v1/auth/login'
const { data } = await api(path, { method: 'POST', body: signup ? { fullName: null, ...values } : values })
await signedIn(data.token)
} catch (failure) {
error.textContent = failure.message
error.hidden = false
}
})
const other = signup
? el('p', { class: 'form-switch' }, 'Have an account? ', el('button', { class: 'link-button', type: 'button', text: 'Sign in', onclick: () => openAccount('signin') }))
: account.signup !== 'closed'
? el('p', { class: 'form-switch' }, 'New here? ', el('button', { class: 'link-button', type: 'button', text: 'Create an account', onclick: () => openAccount('signup') }))
: null
return [form, other]
}
function meter(used, max) {
if (max === null) return null
const share = Math.min(1, used / Math.max(1, max))
return el('div', { class: `meter${used >= max ? ' meter-full' : ''}` }, el('span', { style: `width: ${Math.round(share * 100)}%` }))
}
function profileView() {
const { tier, tierUntil, limits, usage, apiKey } = account.access
const of = (used, max) => (max === null ? `${number.format(used)} (no limit)` : `${number.format(used)} of ${number.format(max)}`)
const key = el('code', { text: apiKey })
return [
el('dl', { class: 'facts' },
fact('Tier', tierUntil ? `${tier} until ${new Date(tierUntil).toLocaleDateString()}` : tier),
fact('Searches today', el('div', {}, of(usage.apiHits, limits.apiHits), meter(usage.apiHits, limits.apiHits))),
fact('NZBs today', el('div', {}, of(usage.grabs, limits.grabs), meter(usage.grabs, limits.grabs))),
fact('Retention', limits.retentionDays ? `${number.format(limits.retentionDays)} days` : 'Everything')),
...section('Newznab API key', 'for Prowlarr, Sonarr, Radarr', el('div', { class: 'api-key' },
key,
el('button', { class: 'button', type: 'button', text: 'Copy', onclick: async (event) => {
await navigator.clipboard?.writeText(account.access.apiKey)
event.target.textContent = 'Copied'
setTimeout(() => (event.target.textContent = 'Copy'), 1200)
} }),
el('button', { class: 'button', type: 'button', text: 'New key', title: 'The old key stops working', onclick: async () => {
if (!confirm('Make a new key? Apps using the old one stop working until they get the new one.')) return
const { apiKey: fresh } = await api('/api/v1/account/api-key', { method: 'POST' })
account.access.apiKey = fresh
key.textContent = fresh
} }))),
el('p', { class: 'form-switch' }, `URL for your apps: ${location.origin}`),
el('button', { class: 'button', type: 'button', text: 'Sign out', onclick: () => signOut() }),
]
}
async function openAccount(view) {
const dialog = $('#account-dialog')
if (view === 'profile') await loadProfile().catch(() => {})
if (view === 'profile' && !account.access) return
$('#account-title').textContent = { signin: 'Sign in', signup: 'Create an account', profile: 'Your account' }[view]
$('#account-body').replaceChildren(...(view === 'profile' ? profileView() : accountForm(view)))
if (!dialog.open) dialog.showModal()
dialog.querySelector('input')?.focus()
}
/** With accounts, nothing is searched before signing in. */
async function startAccess() {
try {
Object.assign(account, await api('/api/v1/access'))
} catch {
// An older server: open.
}
if (account.mode !== 'accounts') return true
if (account.token) await loadProfile().catch(() => {})
renderAccountButton()
if (account.access) return true
$('#summary').textContent = 'Sign in to search the index.'
if (!$('#account-dialog').open) openAccount('signin')
return false
}
// --- Wiring ----------------------------------------------------------------
$('#account').addEventListener('click', () => openAccount(account.access ? 'profile' : 'signin'))
$('#account-close').addEventListener('click', () => $('#account-dialog').close())
$('#q').addEventListener('input', (event) => {
state.q = event.target.value.trim()
scheduleSearch()
@@ -495,5 +674,8 @@ window.addEventListener('popstate', () => {
readUrl()
syncControls()
loadStats()
loadResults()
if (state.release) openRelease(state.release)
startAccess().then((ready) => {
if (!ready) return
loadResults()
if (state.release) openRelease(state.release)
})
+11
View File
@@ -14,8 +14,19 @@
<span>Voyager 2</span>
</a>
<dl class="stats" id="stats" aria-label="Index statistics"></dl>
<button type="button" class="button account-button" id="account" hidden></button>
</header>
<dialog class="dialog" id="account-dialog" aria-labelledby="account-title">
<div class="dialog-head">
<h2 id="account-title"></h2>
<button type="button" class="icon-button" id="account-close" aria-label="Close">
<svg aria-hidden="true" viewBox="0 0 24 24"><path d="M6 6l12 12M18 6 6 18" /></svg>
</button>
</div>
<div class="dialog-body" id="account-body"></div>
</dialog>
<main class="layout">
<section class="search" aria-label="Search">
<div class="search-row">
+4 -6
View File
@@ -43,19 +43,19 @@ export default await Env.create(new URL('../', import.meta.url), {
NNTP_PASSWORD: Env.schema.string.optional(),
NNTP_SECURE: Env.schema.boolean.optional(),
NNTP_CONNECTIONS: Env.schema.number.optional(),
NNTP_MIN_CONNECTIONS: Env.schema.number.optional(),
NNTP_ACCOUNT_CONNECTIONS: Env.schema.number.optional(),
NNTP_RESERVED_CONNECTIONS: Env.schema.number.optional(),
NNTP_TIMEOUT_MS: Env.schema.number.optional(),
NNTP_HEDGE_AFTER_SECONDS: Env.schema.number.optional(),
NNTP_HEDGE_CONNECTIONS: Env.schema.number.optional(),
NNTP_CONNECTION_MAX_MINUTES: Env.schema.number.optional(),
NNTP_IDLE_CLOSE_SECONDS: Env.schema.number.optional(),
NNTP_STALL_KB_PER_MINUTE: Env.schema.number.optional(),
// Newznab API key (empty: no key needed)
NEWZNAB_API_KEY: Env.schema.secret.optional(),
// Access (config/access.ts): open (one key, no limits) or accounts (a key per user, tiers with daily limits)
ACCESS: Env.schema.enum.optional(['open', 'accounts'] as const),
SIGNUP: Env.schema.enum.optional(['open', 'invite', 'closed'] as const),
// Coordinator
RANGE_ARTICLES: Env.schema.number.optional(),
TAIL_EVERY_SECONDS: Env.schema.number.optional(),
@@ -78,8 +78,6 @@ export default await Env.create(new URL('../', import.meta.url), {
SPOOL_MAX_GB: Env.schema.number.optional(),
PREFETCH_PER_RANGE: Env.schema.number.optional(),
PROBE_CONNECTIONS: Env.schema.number.optional(),
PROBE_MIN_CONNECTIONS: Env.schema.number.optional(),
PROBE_MIN_FREE_GB: Env.schema.number.optional(),
HTTP_ROUTES: Env.schema.enum.optional(['all', 'node', 'public'] as const),
PREFETCH_PER_GROUP: Env.schema.number.optional(),
})
+6 -1
View File
@@ -45,13 +45,18 @@ router
router
.group(() => {
router.get('profile', [controllers.Profile, 'show'])
router.post('api-key', [controllers.Profile, 'resetApiKey'])
router.post('logout', [controllers.AccessTokens, 'destroy'])
})
.prefix('account')
.as('profile')
.use(middleware.auth())
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it); Newznab has its key.
// How the web interface signs in: open, or accounts with limits (config/access.ts).
router.get('access', [controllers.Releases, 'access'])
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it), unless ACCESS
// is `accounts`: then a user's token or apikey, counted as Newznab's are (AccessService).
router.get('releases', [controllers.Releases, 'index'])
router.get('releases/:id', [controllers.Releases, 'show']).where('id', router.matchers.number())
router.get('releases/:id/nzb', [controllers.Releases, 'nzb']).where('id', router.matchers.number())
+155
View File
@@ -0,0 +1,155 @@
import { test } from '@japa/runner'
import { DateTime } from 'luxon'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import User from '#models/user'
import BatchStore from '#services/batch_store'
import AccessService from '#services/access_service'
import { TIERS } from '#config/access'
import { RangeIndexer } from '#lib/edge/index'
import { DEFAULT_RULES } from '#lib/rules/index'
import type { Overview } from '#lib/nntp/index'
const GROUP = 'alt.binaries.test'
function file(name: string, counter: string, parts: number, at: number): Overview[] {
return Array.from({ length: parts }, (_, i) => [at + i, {
'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
'message-id': `<${name}-${i + 1}@test>`, ':bytes': String(700000 + i), 'xref': `news.example ${GROUP}:${at + i}`,
}] as Overview)
}
/** Two releases: a show (recent) and a movie posted two years ago. */
async function catalog() {
const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
const [n] = await db.table('nodes').insert({ name: 'access-node' }).returning('id')
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: n.id }).returning('id')
const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
indexer.add([...file('Some.Show.S01E01.1080p.WEB.mkv', 'Some.Show.S01E01.1080p.WEB [1/1]', 2, 100), ...file('Old.Movie.2024.1080p.BluRay.mkv', 'Old.Movie.2024.1080p.BluRay [1/1]', 1, 300)], 1001)
indexer.finish()
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
const show = await db.from('releases').where('name', 'ilike', '%show%').firstOrFail()
const movie = await db.from('releases').where('name', 'ilike', '%movie%').firstOrFail()
await db.from('releases').where('id', movie.id).update({ posted_at: DateTime.now().minus({ years: 2 }).toJSDate() })
return { show: Number(show.id), movie: Number(movie.id) }
}
const user = (email = 'someone@example.com') => User.create({ email, password: 'secret-password' })
test.group('Access with accounts', (group) => {
const free = { ...TIERS.free }
group.each.setup(() => testUtils.db().withGlobalTransaction())
group.each.setup(() => {
AccessService.mode = 'accounts'
AccessService.signup = 'invite'
Object.assign(TIERS.free, { apiHits: 3, grabs: 1, retentionDays: 365 })
return () => {
AccessService.mode = 'open'
AccessService.signup = 'open'
Object.assign(TIERS.free, free)
}
})
test("Newznab takes a user's key, counts searches and grabs per day and keeps to the tier's retention", async ({ client, assert }) => {
const { show, movie } = await catalog()
const someone = await user()
assert.include((await client.get('/api?t=search&q=')).text(), 'code="100"')
assert.include((await client.get('/api?t=search&q=&apikey=0123456789abcdef0123456789abcdef')).text(), 'code="100"')
// caps isn't counted.
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), '<registration available="yes" open="no"/>')
const feed = (await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text()
// The free tier sees a year back: the movie isn't found.
assert.include(feed, 'Some Show S01E01')
assert.notInclude(feed, 'Old Movie')
assert.include(feed, `t=get&amp;id=${show}&amp;apikey=${someone.apiKey}`)
assert.include(feed, '<newznab:apilimits apicurrent="1" apimax="3" grabcurrent="0" grabmax="1"/>')
// Past the retention: refused, and not counted as a grab.
assert.include((await client.get(`/api?t=get&id=${movie}&apikey=${someone.apiKey}`)).text(), 'code="102"')
assert.include((await client.get(`/api?t=get&id=999999&apikey=${someone.apiKey}`)).text(), 'code="300"')
const nzb = await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)
assert.include(nzb.header('content-type'), 'application/x-nzb')
assert.include((await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)).text(), 'code="501"')
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
assert.include((await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text(), 'code="500"')
assert.deepEqual(await AccessService.usage(someone.id), { apiHits: 3, grabs: 1 })
// VIP sees everything; a tier that ran out is free again.
AccessService.grantTier(someone, 'vip', 30)
await someone.save()
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'Old Movie 2024')
someone.tierUntil = DateTime.now().minus({ days: 1 })
await someone.save()
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'code="500"')
someone.suspended = true
await someone.save()
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="101"')
})
test('the catalog needs a token or a key, and counts as Newznab does', async ({ client, assert }) => {
const { show, movie } = await catalog()
const someone = await user()
;(await client.get('/api/v1/releases')).assertStatus(401)
;(await client.get(`/api/v1/releases/${show}/nzb`)).assertStatus(401)
// Stats and how to sign in stay open.
;(await client.get('/api/v1/stats')).assertStatus(200)
assert.deepEqual((await client.get('/api/v1/access')).body(), { mode: 'accounts', signup: 'invite' })
const found = await client.get('/api/v1/releases').loginAs(someone)
found.assertStatus(200)
assert.deepEqual((found.body() as { releases: { id: number }[] }).releases.map((r) => r.id), [show])
;(await client.get(`/api/v1/releases/${movie}`).loginAs(someone)).assertStatus(403)
;(await client.get(`/api/v1/releases/${show}`).loginAs(someone)).assertStatus(200)
// A plain link (the web interface's download button) carries the key.
const nzb = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
nzb.assertStatus(200)
const over = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
over.assertStatus(429)
assert.deepEqual(over.body(), { error: 'Download limit reached', code: 501 })
// The refused movie wasn't counted: a third search is left.
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(200)
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(429)
const profile = ((await client.get('/api/v1/account/profile').loginAs(someone)).body() as any).data
assert.equal(profile.access.tier, 'free')
assert.equal(profile.access.apiKey, someone.apiKey)
assert.deepEqual(profile.access.usage, { apiHits: 3, grabs: 1 })
const reset = (await client.post('/api/v1/account/api-key').loginAs(someone)).body() as { apiKey: string }
assert.notEqual(reset.apiKey, someone.apiKey)
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="100"')
})
test('signup takes an unused invite, which may give a tier', async ({ client, assert }) => {
const form = (email: string, invite?: string) => ({ email, password: 'secret-password', passwordConfirmation: 'secret-password', fullName: null, invite })
await db.table('invites').insert([{ code: 'plain-invite' }, { code: 'vip-invite', tier: 'vip', tier_days: 365 }])
;(await client.post('/api/v1/auth/signup').json(form('a@example.com'))).assertStatus(403)
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'no-such-code'))).assertStatus(403)
// A refused invite leaves no account behind.
assert.isNull(await User.findBy('email', 'a@example.com'))
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'plain-invite'))).assertStatus(200)
;(await client.post('/api/v1/auth/signup').json(form('b@example.com', 'plain-invite'))).assertStatus(403)
const vip = await client.post('/api/v1/auth/signup').json(form('c@example.com', 'vip-invite'))
vip.assertStatus(200)
const c = await User.findByOrFail('email', 'c@example.com')
assert.equal(c.tier, 'vip')
assert.closeTo(c.tierUntil!.diffNow('days').days, 365, 1)
assert.match(c.apiKey, /^[0-9a-f]{32}$/)
// Renewed early, the year is added to what's left.
AccessService.grantTier(c, 'vip', 365)
assert.closeTo(c.tierUntil!.diffNow('days').days, 730, 1)
AccessService.signup = 'closed'
;(await client.post('/api/v1/auth/signup').json(form('d@example.com', 'vip-invite'))).assertStatus(403)
})
})
-59
View File
@@ -120,40 +120,6 @@ test.group('BatchStore', (group) => {
assert.equal(Number(chunks!.n), 4)
})
test('candidates added to a settled release (a later range of its upload) are claimable at once', async ({ assert }) => {
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E02.1080p.mkv', 2, 'Some.Show.S01E02.1080p [1/2]', '2026-10-05T10:00:00Z'))
// Settled before the next range came (the settler marks only what it settles).
await db.from('releases').update({ settled_at: new Date() })
await db.from('probe_candidates').update({ ready: true })
await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E02.1080p.par2', 1, 'Some.Show.S01E02.1080p [2/2]', '2026-10-05T10:20:00Z'))
const candidates = await db.from('probe_candidates').orderBy('id').select('kind', 'ready')
assert.deepEqual(candidates.map((c) => c.kind), ['par2'])
assert.isTrue(candidates.every((c) => c.ready))
assert.lengthOf(await releasesOf(), 1)
})
test('a split 7z read in pieces keeps the tail of its highest volume only; RAR gets none', async ({ assert }) => {
// As the live tail reads it while posted: each piece queues a tail for the last volume it saw so far.
await storeIn(GROUP, 1, [
...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
])
await storeIn(GROUP, 100_001, upload(100_010, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'))
// A lower volume posted late: its tail goes at once.
await storeIn(GROUP, 200_001, upload(200_010, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'))
await storeIn(GROUP, 300_001, upload(300_010, 'Some.Show.S01E03.part01.rar', 2, 'Some.Show.S01E03 [1/2]', '2026-10-23T09:00:00Z'))
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E03.part02.rar', 2, 'Some.Show.S01E03 [2/2]', '2026-10-23T09:02:00Z'))
const candidates = await db.from('probe_candidates').orderBy('file').select('kind', 'file')
assert.deepEqual(
candidates.map((c) => [c.kind, c.file]),
[
['archive', 'Some.Show.S01E03.part01.rar'],
['archive', 'TsIgNBewuFbuwWy.7z.001'],
['7z-tail', 'TsIgNBewuFbuwWy.7z.004'],
]
)
})
test('the same upload in another group (multi-posted) is a release of its own', async ({ assert }) => {
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
@@ -180,29 +146,4 @@ test.group('BatchStore', (group) => {
const keys = (await releasesOf()).filter((r) => r.identity_key === first.identity_key).map((r) => r.release_key)
assert.deepEqual(keys, [first.identity_key, `${first.identity_key}:3`])
})
test('a re-upload is numbered after the highest number, not into the holes merges left', async ({ assert }) => {
for (let day = 0; day < 4; day++) {
await storeIn(GROUP, 1 + day * 100_000, upload(10 + day * 100_000, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', `2026-10-0${day * 2 + 1}T10:00:00Z`))
}
const [first] = await releasesOf()
// Merged away: key:2 and key:3 gone, key and key:4 left.
await db.from('releases').whereIn('release_key', [`${first.identity_key}:2`, `${first.identity_key}:3`]).delete()
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-09T10:00:00Z'))
const keys = (await releasesOf()).map((r) => r.release_key.split(':')[1] ?? '1')
assert.deepEqual(keys, ['1', '4', '5'])
})
test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
await storeIn(GROUP, 1_000_000, [
...upload(1_000_000, 'Murdoch Mysteries S8D1.nzb', 1, '[00/80]', '2026-10-05T10:00:00Z'),
...upload(1_000_010, 'Some.Tool.v1.zip', 1, '[01/01]', '2026-10-05T10:00:00Z'),
...upload(1_000_020, 'Some.Show.S01E01.part01.rar', 2, '[02/30]', '2026-10-05T10:00:00Z'),
])
const flags = new Map((await db.from('releases').select('name', 'sidecar_only')).map((r) => [r.name, r.sidecar_only]))
assert.isTrue(flags.get('Murdoch Mysteries S8D1'))
assert.isFalse(flags.get('Some.Tool.v1'))
// Over 1 MB: not looked at.
assert.isNull(flags.get('Some.Show.S01E01'))
})
})
-7
View File
@@ -31,12 +31,5 @@ test.group('Leases', (group) => {
// a1's worker went quiet: its lease is taken over (a2 alone counts).
await db.from('ranges').where('id', a1).update({ heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
assert.equal(await leased(5), a1)
// Read to its end, its worker gone before finishing: done. One still reading, or quiet but not at its end, stays.
await db.from('ranges').where('id', a2).update({ next_id: 201, heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
await db.from('ranges').where('id', b1).update({ next_id: 101 })
assert.equal(await Leases.finishAbandoned(), 1)
assert.equal((await db.from('ranges').where('id', a2).firstOrFail()).status, 'done')
assert.equal((await db.from('ranges').where('id', b1).firstOrFail()).status, 'leased')
})
})
-53
View File
@@ -1,53 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import coordinatorConfig from '#config/coordinator'
import { grant, targets, NntpShare } from '#services/nntp_shares'
test.group('NNTP shares', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('targets: minimums first, then by priority what each wants', ({ assert }) => {
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 0 }
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 0 }
// History running: the coordinator all it wants, the prober the rest.
assert.deepEqual(Object.fromEntries(targets(45, [prober, coordinator])), { coordinator: 38, prober: 7 })
// History paused: the coordinator its minimum (or what it uses), the prober the rest.
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 8 }])), { coordinator: 8, prober: 37 })
// Wanting more than its minimum doesn't take another's minimum.
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 60 }])), { coordinator: 43, prober: 2 })
})
test('grant: growing only into connections the others closed or gave up', ({ assert }) => {
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 37, granted: 37 }
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 8 }
// History came back: the prober still holds 37, the coordinator keeps its 8 for now.
assert.equal(grant(45, coordinator, [prober]), 8)
// The prober syncs: its target is 7, its grant goes down at once.
assert.equal(grant(45, { ...prober, open: 37 }, [{ ...coordinator, granted: 8 }]), 7)
// Its connections not closed yet: still held.
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 30 }]), 15)
// Closed: the coordinator gets all it wants.
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 7 }]), 38)
})
test('sync: each process its row; a process gone (no heartbeat) holds nothing', async ({ assert }) => {
const total = coordinatorConfig.nntpShares.total
const coordinator = new NntpShare('coordinator', 2, 8, 38)
const prober = new NntpShare('prober', 1, 2, total)
assert.equal(await coordinator.sync(8, 0), 8)
assert.equal(await prober.sync(total, 0), total - 8)
// The coordinator wants all: the prober holds its grant until it has synced and closed them.
assert.equal(await coordinator.sync(38, 8), 8)
assert.equal(await prober.sync(total, total - 8), total - 38)
assert.equal(await coordinator.sync(38, 8), 8)
assert.equal(await prober.sync(total, total - 38), total - 38)
assert.equal(await coordinator.sync(38, 8), 38)
// The prober's process died: after a minute without a sync its share is free.
await db.from('nntp_shares').where('name', 'prober').update({ open: total, granted: total, heartbeat_at: new Date(Date.now() - 120_000) })
assert.equal(await coordinator.sync(38, 38), 38)
await coordinator.leave()
assert.equal(await prober.sync(total, 0), total)
})
})
-40
View File
@@ -78,44 +78,6 @@ test.group('Par2Merger', (group) => {
assert.equal((await new Par2Merger().round()).sets, 1)
})
test('a round behind by hours goes through them a window at a time, past the gaps', async ({ assert }) => {
const ago = (hours: number) => new Date(Date.now() - hours * 3600_000)
await db.table('par2_sets').multiInsert([
{ set_id: 'a1'.repeat(16), release_id: 1, files: 1, created_at: ago(30) },
{ set_id: 'a2'.repeat(16), release_id: 2, files: 1, created_at: ago(2.5) },
{ set_id: 'a3'.repeat(16), release_id: 3, files: 1, created_at: ago(0.5) },
])
await db.table('worker_cursors').insert({ name: 'merger', at: ago(48) })
assert.equal((await new Par2Merger().round()).sets, 3)
const saved = await db.from('worker_cursors').where('name', 'merger').firstOrFail()
assert.isAbove(new Date(saved.at).getTime(), ago(0.5).getTime())
})
test('a par2 file held by more than 20 files is a common one: it merges nothing', async ({ assert }) => {
const at = new Date('2022-07-08T03:46:00Z')
const ids: number[] = []
for (const key of ['dd000001', 'dd000002']) {
const [row] = await db
.table('releases')
.insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
.returning('id')
ids.push(Number(row.id))
}
// The par2 is held by the first; the second holds the file, and so do 20 releases elsewhere and years apart.
const setId = hash('1')
await db.table('par2_sets').insert({ set_id: setId, release_id: ids[0], files: 1 })
await db.table('par2_files').insert({ set_id: setId, file_id: hash('0'), hash16k: hash('9'), length: 30214, name: 'sample.nfo' })
await db.table('file_hashes').insert({ candidate_id: 4000, release_id: ids[1], hash16k: hash('9'), size: 30214 })
for (let i = 1; i <= 20; i++) await db.table('file_hashes').insert({ candidate_id: 4000 + i, release_id: 900000 + i, hash16k: hash('9'), size: 30214 })
const round = await new Par2Merger().round()
assert.deepEqual([round.sets, round.merges], [1, 0])
assert.equal(await new Par2Merger().mergeSet(setId), 0)
assert.equal((await db.from('releases').whereIn('id', ids).select('id')).length, 2)
// One holder fewer: the file is one upload's again.
await db.from('file_hashes').where('candidate_id', 4020).delete()
assert.equal(await new Par2Merger().mergeSet(setId), 1)
})
test('a set held by two of the releases merged is held once by the target', async ({ assert }) => {
const at = new Date('2022-07-08T03:46:00Z')
const ids: number[] = []
@@ -258,8 +220,6 @@ test.group('Par2Merger', (group) => {
// A picture isn't a sidecar; an nfo an hour later belongs to nothing here.
const picture = await release('cover', 1, null, at, ['cover.jpg'])
const late = await release('Murdoch Mysteries S8D1', 1, null, new Date(at.getTime() + 3600_000), ['Murdoch Mysteries S8D1.nfo'])
// Queued, its release gone meanwhile (merged): leaves the queue with its batch.
await db.table('fold_candidates').insert({ release_id: 987654321 })
const round = await new Par2Merger().round()
assert.deepEqual(round.folded, { looked: 5, sidecars: 2, folded: 1 })
+5 -61
View File
@@ -53,72 +53,16 @@ test.group('Par2Pruner', (group) => {
const locked = await set('e', e, 2, null)
await db.table('release_verdicts').insert({ release_id: e, rule: 'locked', rule_version: 1 })
// Final: its holder held by a run of done ranges with hours to spare, nothing hashed (a readable upload's files).
const f = await release()
const final = await set('f', f, 2, null)
const posted = new Date('2024-03-01T12:00:00Z')
const [group] = await db.table('groups').insert({ name: 'alt.binaries.pruned', priority: 1 }).returning('id')
const groupId = Number(group.id)
await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100 })
const [range] = await db
.table('ranges')
.insert({ group_id: groupId, start_id: 1, end_id: 10000, next_id: 10001, status: 'done', done_at: old, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z') })
.returning('id')
await db.table('range_blocks').insert({ group_id: groupId, start_id: 1, end_id: 10000, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z'), at_group_start: false })
// A lone candidate unprobed since before the range was done: its hash may still match. Kept.
await db.table('probe_candidates').insert({ id: 9001, release_id: b, kind: 'lone', message_id: '<9001@x>', file: 'x', status: 'waiting', created_at: new Date(old.getTime() - 3600_000) })
// Out of budget: one batch of two sets (a, b), the next round goes on.
assert.deepEqual(await new Par2Pruner(2).round(0), { complete: 1, judged: 0, final: 0, rows: 2 })
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 1, final: 0, rows: 2 })
const round = await new Par2Pruner().round()
assert.deepEqual(round, { complete: 1, judged: 1, rows: 4 })
assert.equal(await count(complete), 0)
assert.equal(await count(apart), 2)
assert.equal(await count(recent), 2)
assert.equal(await count(dead), 0)
assert.equal(await count(locked), 2)
assert.equal(await count(final), 2)
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '5')
// Probed: final. The next pass starts over from the first set.
await db.from('probe_candidates').where('id', 9001).update({ status: 'done' })
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 0, final: 1, rows: 2 })
assert.equal(await count(final), 0)
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '6')
// A hash of its files since the merger's last window: the merger hasn't matched it yet. Kept.
const g = await release()
const unseen = await set('g', g, 1, null)
await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100 })
await db.table('worker_cursors').insert({ name: 'merger', at: new Date(Date.now() - 3600_000) })
await db.table('probe_candidates').insert({ id: 9002, release_id: await release(), kind: 'lone', message_id: '<9002@x>', file: 'x', status: 'done' })
await db.table('file_hashes').insert({ candidate_id: 9002, release_id: 1, hash16k: 'g0'.padEnd(32, '0'), size: 1000 })
assert.equal((await new Par2Pruner().round()).final, 0)
// Past the hash by more than the merger's overlap.
await db.from('worker_cursors').where('name', 'merger').update({ at: new Date(Date.now() + 120_000) })
assert.equal((await new Par2Pruner().round()).final, 1)
assert.equal(await count(unseen), 0)
assert.isNotNull(range)
})
test('sets of one created_at, a batch each: every one looked at', async ({ assert }) => {
// One prober flush gives thousands of sets the same created_at (to the microsecond).
const flush = new Date('2026-10-15T14:38:39.883Z')
let candidate = 7000
for (const letter of ['p', 'q', 'r']) {
const key = `pr${letter}`.padEnd(40, '0')
const [row] = await db.table('releases').insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }).returning('id')
const id = Number(row.id)
const setId = letter.repeat(32)
const h = `${letter}0`.padEnd(32, '0')
await db.table('par2_files').insert({ set_id: setId, file_id: h, hash16k: h, length: 1000, name: 'f.rar' })
candidate++
await db.table('probe_candidates').insert({ id: candidate, release_id: id, kind: 'lone', message_id: `<${candidate}@x>`, file: 'x', status: 'done' })
await db.table('file_hashes').insert({ candidate_id: candidate, release_id: id, hash16k: h, size: 1000 })
await db.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files, created_at) VALUES (?, ?, 1, ?::timestamptz + interval '984 microseconds')`, [setId, id, flush])
}
const round = await new Par2Pruner(1).round()
assert.equal(round.complete, 3)
assert.equal((await db.from('par2_files').whereIn('set_id', ['p'.repeat(32), 'q'.repeat(32), 'r'.repeat(32)]).count('* as n'))[0].n, '0')
// The pass ended: the next one starts from the first set.
assert.equal((await db.from('worker_cursors').where('name', 'par2-prune').firstOrFail()).key, '')
// Again: the cursor is past them, nothing more.
assert.deepEqual(await new Par2Pruner().round(), { complete: 0, judged: 0, rows: 0 })
})
})
-45
View File
@@ -1,45 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import { Prober } from '#services/prober'
test.group('Prober', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('a claim takes each tier: the readable tiers also take unreadable candidates, a kind with only one of them too', async ({ assert }) => {
const [row] = await db
.table('releases')
.insert({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 })
.returning('id')
const candidate = (kind: string, unreadable: boolean, ready = true) => ({
release_id: row.id, kind, message_id: `<${kind}-${unreadable}-${ready}@x>`, file: kind, ready, unreadable,
})
// One per kind, so each tier's id range is a single id (the slices' starts aren't random then).
await db.table('probe_candidates').multiInsert([candidate('lone', true), candidate('par2', true), candidate('archive', false), candidate('7z-tail', false, false)])
const prober = new Prober() as any
await prober.claimTiers()
assert.sameMembers(prober.queue.map((c: { kind: string }) => c.kind), ['lone', 'par2', 'archive'])
const status = Object.fromEntries((await db.from('probe_candidates').select('kind', 'status')).map((c) => [c.kind, c.status]))
assert.deepEqual(status, { 'lone': 'taken', 'par2': 'taken', 'archive': 'taken', '7z-tail': 'waiting' })
})
test('a flush stores a name with a NUL in it, and one article that two releases have as a candidate', async ({ assert }) => {
const release = async (key: string) =>
Number((await db.table('releases').insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 }).returning('id'))[0].id)
const a = await release('fa')
const b = await release('fb')
const candidate = async (releaseId: number) =>
Number((await db.table('probe_candidates').insert({ release_id: releaseId, kind: 'archive', message_id: '<same@x>', file: 'x.rar', status: 'taken' }).returning('id'))[0].id)
const inspection = { yencName: 'x.rar', size: 1000, type: null, names: ['Movie\u0000.mkv'], entries: 1, passworded: false }
const prober = new Prober() as any
for (const releaseId of [a, b]) {
prober.outcomes.push({ candidate: { id: await candidate(releaseId), releaseId, kind: 'archive', messageId: '<same@x>' }, status: 'done', inspection })
}
await prober.flush()
assert.deepEqual((await db.from('probe_candidates').select('status')).map((c) => c.status), ['done', 'done'])
const cached = await db.from('probe_cache').where('message_id', '<same@x>').firstOrFail()
assert.deepEqual(cached.result.names, ['Movie.mkv'])
assert.equal((await db.from('release_inspections').count('* as n'))[0].n, '2')
})
})
+2 -84
View File
@@ -59,8 +59,6 @@ test.group('ReleaseJudge', (group) => {
const scraped = await release('Jamie Lee Curtis Nude Celebrity Pictures', { poster: '@_CelebMovie_@', source: 'subject' })
const celeborn = await release('Some.Album.2010', { poster: 'celeborn@not.here (Celeborn)', source: 'subject' })
// Scraped posters are found through posters (the batch store adds each release's).
await db.table('posters').multiInsert([{ poster: '@_CelebMovie_@' }, { poster: 'celeborn@not.here (Celeborn)' }])
const locked = await release('Locked.Thing', { passworded: true })
const unlocked = await release('Known.Password', { passworded: true })
@@ -80,7 +78,7 @@ test.group('ReleaseJudge', (group) => {
const spotted = await release('yy12CD34ef56GH78ij90', { readable: false, ...placed })
await db.table('release_spots').insert({ release_id: spotted, spot_id: 's@x', files: 1 })
const round = await new ReleaseJudge().full()
const round = await new ReleaseJudge().round()
const verdicts = new Map((await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [Number(v.release_id), v.rule]))
assert.equal(verdicts.get(dead), 'dead')
assert.isUndefined(verdicts.get(halfDead))
@@ -107,88 +105,8 @@ test.group('ReleaseJudge', (group) => {
// A password found: the next round takes the verdict back.
await db.table('release_passwords').insert({ release_id: locked, password: 'found', source: 'nzb' })
const again = await new ReleaseJudge().full()
const again = await new ReleaseJudge().round()
assert.equal(again.rules.locked.removed, 1)
// Verdicts that stay are not written again.
assert.equal(again.rules.repeated.added, 0)
assert.equal(again.rules.fake.added, 0)
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
})
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({ assert }) => {
let n = 0
const settled = new Date(Date.now() - 3 * 86400_000)
const release = async (name: string, extra: Record<string, unknown> = {}) => {
n++
const key = `jc${String(n).padStart(6, '0')}`.padEnd(40, '0')
const [row] = await db
.table('releases')
.insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', files: 3, posted_at: settled, readable: true, settled_at: settled, ...extra })
.returning('id')
return Number(row.id)
}
const judge = new ReleaseJudge()
const verdict = async (id: number) => (await db.from('release_verdicts').where('release_id', id).first())?.rule
const dirty = async () => Number((await db.from('judge_dirty').count('* as n'))[0].n)
// New releases are noted; a passworded one is locked, until a password turns up.
const locked = await release('Locked.Thing', { passworded: true })
assert.isAbove(await dirty(), 0)
let round = await judge.changed()
assert.equal(await verdict(locked), 'locked')
assert.equal(round.rules.locked.added, 1)
assert.equal(await dirty(), 0)
await db.table('release_passwords').insert({ release_id: locked, password: 'pw', source: 'nzb' })
round = await judge.changed()
assert.isUndefined(await verdict(locked))
assert.equal(round.rules.locked.removed, 1)
// Probes: a release whose probed articles are all missing is dead once the prober's statuses come in.
const dead = await release('Gone.Movie.2020.1080p')
const ids = await db.table('probe_candidates').multiInsert([
{ release_id: dead, kind: 'lone', message_id: '<jc1@x>', file: 'a', status: 'waiting' },
{ release_id: dead, kind: 'lone', message_id: '<jc2@x>', file: 'b', status: 'waiting' },
]).returning('id')
await judge.changed()
assert.isUndefined(await verdict(dead))
await db.from('probe_candidates').whereIn('id', ids.map((r: { id: string }) => r.id)).update({ status: 'missing' })
await judge.changed()
assert.equal(await verdict(dead), 'dead')
// One file in 19 releases, one of them locked: the 20th makes them all repeated, the locked one too (repeated
// comes first), though only the 20th changed.
const hash = 'b'.repeat(32)
const holders: number[] = []
for (let i = 0; i < 19; i++) holders.push(await release(`holder${i}`, i === 0 ? { passworded: true } : {}))
for (const [i, id] of holders.entries()) await db.table('file_hashes').insert({ candidate_id: 30000 + i, release_id: id, hash16k: hash, size: 5000 })
await judge.changed()
assert.equal(await verdict(holders[0]), 'locked')
assert.isUndefined(await verdict(holders[1]))
const twentieth = await release('holder19')
await db.table('file_hashes').insert({ candidate_id: 30019, release_id: twentieth, hash16k: hash, size: 5000 })
round = await judge.changed()
assert.equal(round.releases, 20)
const rules = new Set()
for (const id of [...holders, twentieth]) rules.add(await verdict(id))
assert.deepEqual([...rules], ['repeated'])
// A 21st: the file was in 20 releases without it, so only it is judged.
const twentyFirst = await release('holder20')
await db.table('file_hashes').insert({ candidate_id: 30020, release_id: twentyFirst, hash16k: hash, size: 5000 })
round = await judge.changed()
assert.equal(round.releases, 1)
assert.equal(await verdict(twentyFirst), 'repeated')
// A verdict nothing noted stays until the full round; it's due a day after the last one.
const stale = await release('Fine.Release')
await judge.changed()
await db.table('release_verdicts').insert({ release_id: stale, rule: 'fake', rule_version: 1 })
await judge.changed()
assert.equal(await verdict(stale), 'fake')
await db.from('judge_state').update({ full_at: new Date(Date.now() - 2 * 86400_000) })
assert.isTrue(await judge.fullDue())
round = await judge.full()
assert.isUndefined(await verdict(stale))
assert.equal(await verdict(dead), 'dead')
assert.isFalse(await judge.fullDue())
})
})
-30
View File
@@ -1,30 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
test.group('7z tails', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test("of a release's waiting tails, those below its highest volume go; the highest volume's all stay", async ({ assert }) => {
const row = (key: string) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', files: 4 })
const [merged, other] = (await db.table('releases').multiInsert([row('a'), row('b')]).returning('id')).map((r) => r.id)
const tail = (releaseId: number, volume: string, mid: string, status = 'waiting') => ({ release_id: releaseId, kind: '7z-tail', message_id: mid, file: `x.7z.${volume}`, status })
await db.table('probe_candidates').multiInsert([
// Two merged members' tails, one of the highest volume probed already.
tail(merged, '002', '<1@x>'),
tail(merged, '005', '<2@x>'),
tail(merged, '003', '<3@x>', 'done'),
tail(merged, '009', '<4@x>', 'taken'),
// Two pieces that each saw part of volume 009.
tail(merged, '009', '<5@x>'),
{ release_id: merged, kind: 'archive', message_id: '<6@x>', file: 'x.7z.001' },
// Another release: not asked for.
tail(other, '002', '<7@x>'),
tail(other, '004', '<8@x>'),
])
assert.equal(await pruneSevenZipTails(db, [merged]), 2)
const left = await db.from('probe_candidates').orderBy('message_id').select('message_id')
assert.deepEqual(left.map((c) => c.message_id), ['<3@x>', '<4@x>', '<5@x>', '<6@x>', '<7@x>', '<8@x>'])
})
})
-2
View File
@@ -103,8 +103,6 @@ test.group('Spots', (group) => {
const release = await db.from('releases').where('source', 'spot').firstOrFail()
assert.equal(release.name, 'Lonely File')
assert.equal(release.files, 1)
assert.isNotNull(release.poster)
assert.isNotNull(await db.from('posters').where('poster', release.poster).first())
const { default: NzbService } = await import('#services/nzb_service')
assert.include((await NzbService.build(Number(release.id)))!.xml, 'lonely.file.mkv')
// Not again while it has one; once the header pass builds the file's release, the spot moves there.
-22
View File
@@ -1,22 +0,0 @@
import { test } from '@japa/runner'
import { DiskSpace } from '#services/disk_space'
test.group('DiskSpace', () => {
test('short below the floor, fine again a margin above it; off without a floor', async ({ assert }) => {
let gb = 15
const space = new DiskSpace('probing', '/', 12, async () => gb * 1e9, 0)
assert.isFalse(await space.short())
gb = 11.5
assert.isTrue(await space.short())
// Back over the floor, not yet over the margin: still short.
gb = 12.5
assert.isTrue(await space.short())
gb = 13.1
assert.isFalse(await space.short())
assert.isFalse(await new DiskSpace('probing', '/', 0, async () => 0, 0).short())
})
test('measures a real filesystem', async ({ assert }) => {
assert.isFalse(await new DiskSpace('probing', '/', 0.001).short())
})
})
-69
View File
@@ -1,69 +0,0 @@
import { test } from '@japa/runner'
import net from 'node:net'
import env from '#start/env'
import coordinatorConfig from '#config/coordinator'
import pool from '#services/nntp_pool'
import type { NntpShare } from '#services/nntp_shares'
/** A tiny NNTP server that only logs in, counting the connections open. */
function server(): Promise<{ port: number; open: () => number; close: () => void }> {
return new Promise((resolve) => {
let open = 0
const srv = net.createServer((sock) => {
open++
sock.on('close', () => open--)
sock.write('200 welcome\r\n')
sock.on('data', (chunk) => {
for (const line of chunk.toString().split('\r\n')) {
if (line.startsWith('AUTHINFO USER')) sock.write('381 more\r\n')
else if (line.startsWith('AUTHINFO PASS')) sock.write('281 ok\r\n')
else if (line === 'XFEATURE COMPRESS GZIP') sock.write('290 Feature Enabled\r\n')
else if (line === 'QUIT') sock.end('205 bye\r\n')
}
})
})
srv.listen(0, '127.0.0.1', () =>
resolve({
port: (srv.address() as net.AddressInfo).port,
open: () => open,
close: () => srv.close(),
})
)
})
}
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
const share = { sync: async () => 4 } as unknown as NntpShare
test.group('nntp / pool', () => {
test('a connection free for nntpIdleCloseMs closes on the next sync; one used since stays', async ({
assert,
cleanup,
}) => {
const { port, open, close } = await server()
const idleCloseMs = coordinatorConfig.nntpIdleCloseMs
env.set('NNTP_HOST', '127.0.0.1')
env.set('NNTP_PORT', port)
env.set('NNTP_SECURE', false)
coordinatorConfig.nntpIdleCloseMs = 100
cleanup(async () => {
coordinatorConfig.nntpIdleCloseMs = idleCloseMs
await pool.close()
close()
})
await pool.share(share)
// Two at once: two connections.
await Promise.all([pool.use(() => sleep(20)), pool.use(() => sleep(20))])
assert.equal(open(), 2)
await sleep(150)
// One used again: the other has been free for longer than 100 ms.
await pool.use(async () => {})
await pool.share(share)
await sleep(50)
assert.equal(open(), 1)
await sleep(150)
await pool.share(share)
await sleep(50)
assert.equal(open(), 0)
})
})