Compare commits
20
Commits
No files matched your search
@@ -19,11 +19,6 @@ DB_USER=usenet_indexer
|
||||
DB_PASSWORD=
|
||||
DB_DATABASE=usenet_rebuild
|
||||
|
||||
# Access (config/access.ts): open (NEWZNAB_API_KEY or none, no limits) or accounts
|
||||
# (a Newznab key per user, tiers with daily limits; ace access:invite, access:tier, access:suspend)
|
||||
# ACCESS=open
|
||||
# SIGNUP=open|invite|closed (with accounts: invite unless set)
|
||||
|
||||
#--------------------------------------------------------------------
|
||||
# CORS (configure allowed origins for API access)
|
||||
#--------------------------------------------------------------------
|
||||
|
||||
+29
-4
@@ -238,6 +238,14 @@ files the set describes, into the one with the most files (chunks,
|
||||
candidates, inspections, counts; the par2's shared name when none was
|
||||
readable); `release_merges` keeps what went where. Obfuscated uploads whose
|
||||
files were each a release of their own (`lone` candidates) become one.
|
||||
It reads what's new an hour (`created_at`) at a time, its place kept after
|
||||
each, and merges only the sets held or matched by two releases or more; a
|
||||
file whose first 16 KiB and size more than 20 files share is a common one and
|
||||
matches nothing (LOG.md 2026-10-20 (85)).
|
||||
A set's file list is dropped once it can't merge anything more: complete (all
|
||||
its files in its holder), judged (every holder junk), or final (every holder
|
||||
inside a run of done ranges, those ranges' candidates probed, its hashes seen
|
||||
by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).
|
||||
|
||||
**Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
|
||||
voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
|
||||
@@ -252,7 +260,7 @@ with 68,032 releases with a password):
|
||||
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
|
||||
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
|
||||
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
|
||||
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
|
||||
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed; one release keeps only the tails of its highest volume (the live tail's pieces each queued one; `seven_zip_tails`) |
|
||||
| Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
|
||||
| Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |
|
||||
|
||||
@@ -260,9 +268,11 @@ Live indexing (the tail) runs since 2026-10-09.
|
||||
|
||||
**Junk** (voyager 1's second filter, D42). After the header pass's rules
|
||||
(garbage signatures, executables, per-article and obfuscated subjects), the
|
||||
judge (`releases:judge`, service `judge`) judges releases hourly by what the
|
||||
prober, the merges and the spots found, in this order (the first rule
|
||||
wins): *dead* (every probed first article gone), *fake* (a film, episode
|
||||
judge (`releases:judge`, service `judge`) judges releases by what the
|
||||
prober, the merges and the spots found: every 2 minutes the releases whose
|
||||
facts changed (noted by triggers in `judge_dirty`), daily every release (for
|
||||
what time changes: a day after settling, ranges read beside it). In this
|
||||
order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
|
||||
or music name whose content is a program: archive entries all
|
||||
executables, or the file itself one by name or by its first bytes, sniffed
|
||||
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
|
||||
@@ -326,3 +336,18 @@ f. Random-named releases (no file with a readable name) keep no chunk in
|
||||
phase 2 names them (bodies) and fetches their segments again. The full
|
||||
history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
|
||||
kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
|
||||
g. The account's NNTP connections shared by priority (`nntp_shares`): the
|
||||
coordinator (headers) before the prober, each between its minimum and
|
||||
maximum, 5 kept for the spotter, the spot inspector, the plain
|
||||
connection and reconnects. A process grows only into connections the
|
||||
others closed, so the account never goes over its limit. Replaces the
|
||||
fixed split (38 + 8), under which the prober stayed at 8 while history
|
||||
was paused and the coordinator's connections idle (LOG.md 2026-10-18
|
||||
(75)). **Accepted** 2026-10-18.
|
||||
h. Idle NNTP connections closed: one with nothing to do for
|
||||
`NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
|
||||
(the coordinator's pool on its next sync, the plain one too; a dry
|
||||
prober lane; the spotter between rounds; the spot inspector with no
|
||||
spots). Eweka dropped the quiet ones and the next command failed
|
||||
("Spot inspection failed: NNTP connection closed"), and idle ones took
|
||||
the account's connections from downloads. **Accepted** 2026-10-22.
|
||||
@@ -1,36 +1,13 @@
|
||||
import User from '#models/user'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import { signupValidator } from '#validators/user'
|
||||
import type { HttpContext } from '@adonisjs/core/http'
|
||||
import UserTransformer from '#transformers/user_transformer'
|
||||
import AccessService from '#services/access_service'
|
||||
|
||||
class InviteGone extends Error {}
|
||||
|
||||
export default class NewAccountController {
|
||||
async store({ request, response, serialize }: HttpContext) {
|
||||
const { fullName, email, password, invite } = await request.validateUsing(signupValidator)
|
||||
if (AccessService.signup === 'closed') return response.forbidden({ error: 'Signup is closed' })
|
||||
if (AccessService.signup === 'invite' && !invite) return response.forbidden({ error: 'An invite code is needed to sign up' })
|
||||
async store({ request, serialize }: HttpContext) {
|
||||
const { fullName, email, password } = await request.validateUsing(signupValidator)
|
||||
|
||||
// The user and their invite together: an invite that's gone rolls the signup back.
|
||||
const user = await db
|
||||
.transaction(async (trx) => {
|
||||
const created = await User.create({ fullName, email, password }, { client: trx })
|
||||
if (AccessService.signup !== 'invite') return created
|
||||
const gives = await AccessService.redeem(invite!, created.id, trx)
|
||||
if (!gives) throw new InviteGone()
|
||||
if (gives.tier) {
|
||||
AccessService.grantTier(created, gives.tier, gives.tier_days)
|
||||
await created.useTransaction(trx).save()
|
||||
}
|
||||
return created
|
||||
})
|
||||
.catch((error) => {
|
||||
if (error instanceof InviteGone) return null
|
||||
throw error
|
||||
})
|
||||
if (!user) return response.forbidden({ error: 'Unknown or used invite code' })
|
||||
const user = await User.create({ fullName, email, password })
|
||||
const token = await User.accessTokens.create(user)
|
||||
|
||||
return serialize({
|
||||
|
||||
@@ -2,7 +2,6 @@ import type { HttpContext } from '@adonisjs/core/http'
|
||||
import env from '#start/env'
|
||||
import NzbService from '#services/nzb_service'
|
||||
import SearchService from '#services/search_service'
|
||||
import AccessService, { AccessDenied } from '#services/access_service'
|
||||
import { capsXml, errorXml, expandCategories, resultsXml, type NewznabServer } from '#lib/newznab/index'
|
||||
|
||||
const LIMITS = { max: 100, default: 50 }
|
||||
@@ -36,37 +35,22 @@ function int(value: unknown): number | undefined {
|
||||
* Releases with readable names (or an inner name from the prober, which then is the title) that aren't known to miss files.
|
||||
*/
|
||||
export default class NewznabController {
|
||||
async handle(ctx: HttpContext) {
|
||||
const xml = (body: string, type = 'application/xml') => {
|
||||
ctx.response.header('Content-Type', `${type}; charset=utf-8`)
|
||||
return ctx.response.send(body)
|
||||
}
|
||||
try {
|
||||
return await this.answer(ctx, xml)
|
||||
} catch (error) {
|
||||
if (error instanceof AccessDenied) return xml(errorXml(error.newznabCode, error.message))
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
private async answer(ctx: HttpContext, xml: (body: string, type?: string) => void) {
|
||||
const { request, response } = ctx
|
||||
async handle({ request, response }: HttpContext) {
|
||||
const qs = request.qs()
|
||||
const t = String(qs.t ?? '')
|
||||
const server = this.server(request)
|
||||
const xml = (body: string, type = 'application/xml') => {
|
||||
response.header('Content-Type', `${type}; charset=utf-8`)
|
||||
return response.send(body)
|
||||
}
|
||||
|
||||
// With accounts the key is a user's (AccessService); else the one configured, if any.
|
||||
const grant = await AccessService.authorize(ctx, qs.apikey)
|
||||
const apiKey = grant ? grant.user.apiKey : configuredApiKey()
|
||||
if (!grant && apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
|
||||
const server = this.server(request, apiKey)
|
||||
if (t === 'caps') return xml(capsXml(server, LIMITS, { available: AccessService.accounts && AccessService.signup !== 'closed', open: AccessService.accounts && AccessService.signup === 'open' }))
|
||||
const apiKey = configuredApiKey()
|
||||
if (apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
|
||||
if (t === 'caps') return xml(capsXml(server, LIMITS))
|
||||
|
||||
if (t === 'get') {
|
||||
const id = int(qs.id)
|
||||
if (id === undefined) return xml(errorXml(200, 'Missing parameter (id)'))
|
||||
// Unknown ids and releases past the tier's retention aren't counted.
|
||||
if (!(await AccessService.checkRelease(grant, id))) return xml(errorXml(300, 'No such item'))
|
||||
await AccessService.charge(grant, 'grab')
|
||||
const nzb = await NzbService.build(id)
|
||||
if (!nzb) return xml(errorXml(300, 'No such item'))
|
||||
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
|
||||
@@ -75,7 +59,6 @@ export default class NewznabController {
|
||||
}
|
||||
|
||||
if (!(t in SEARCH_FUNCTIONS)) return xml(errorXml(202, `No such function (${t || 'none'})`))
|
||||
await AccessService.charge(grant, 'api')
|
||||
|
||||
const requested = String(qs.cat ?? '')
|
||||
.split(',')
|
||||
@@ -96,7 +79,7 @@ export default class NewznabController {
|
||||
categories,
|
||||
readable: true,
|
||||
notIncomplete: true,
|
||||
maxAgeDays: AccessService.maxAgeDays(grant, int(qs.maxage)),
|
||||
maxAgeDays: int(qs.maxage),
|
||||
season: int(qs.season),
|
||||
episode: int(qs.season) !== undefined ? int(qs.ep) : undefined,
|
||||
sort: 'newest',
|
||||
@@ -122,20 +105,12 @@ export default class NewznabController {
|
||||
episode: tv ? `E${tv[2].padStart(2, '0')}` : undefined,
|
||||
}
|
||||
})
|
||||
// What's left of the day, for clients that show it (newznab:apilimits).
|
||||
const limits = grant && {
|
||||
apiCurrent: grant.usage.apiHits,
|
||||
apiMax: grant.limits.apiHits,
|
||||
grabCurrent: grant.usage.grabs,
|
||||
grabMax: grant.limits.grabs,
|
||||
}
|
||||
return xml(resultsXml(server, items, offset, result?.total ?? 0, limits || undefined), 'application/rss+xml')
|
||||
return xml(resultsXml(server, items, offset, result?.total ?? 0), 'application/rss+xml')
|
||||
}
|
||||
|
||||
/** Download links carry the caller's key. */
|
||||
private server(request: HttpContext['request'], apiKey: string | undefined): NewznabServer {
|
||||
private server(request: HttpContext['request']): NewznabServer {
|
||||
const proto = request.header('x-forwarded-proto') ?? 'http'
|
||||
const host = request.header('x-forwarded-host') ?? request.header('host') ?? 'localhost'
|
||||
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey }
|
||||
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey: configuredApiKey() }
|
||||
}
|
||||
}
|
||||
@@ -1,19 +1,8 @@
|
||||
import UserTransformer from '#transformers/user_transformer'
|
||||
import type { HttpContext } from '@adonisjs/core/http'
|
||||
import AccessService from '#services/access_service'
|
||||
|
||||
export default class ProfileController {
|
||||
/** The user, their tier, Newznab key and what they used today. */
|
||||
async show({ auth, serialize }: HttpContext) {
|
||||
const user = auth.use('api').getUserOrFail()
|
||||
return serialize({ user: UserTransformer.transform(user), access: await AccessService.summary(user) })
|
||||
}
|
||||
|
||||
/** POST /api/v1/account/api-key: a new Newznab key (the old one stops working). */
|
||||
async resetApiKey({ auth }: HttpContext) {
|
||||
const user = auth.use('api').getUserOrFail()
|
||||
user.apiKey = AccessService.newKey()
|
||||
await user.save()
|
||||
return { apiKey: user.apiKey }
|
||||
return serialize(UserTransformer.transform(auth.use('api').getUserOrFail()))
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,5 @@
|
||||
import type { HttpContext } from '@adonisjs/core/http'
|
||||
import NzbService from '#services/nzb_service'
|
||||
import AccessService from '#services/access_service'
|
||||
import SearchService, { type SortOrder } from '#services/search_service'
|
||||
import { expandCategories } from '#lib/newznab/index'
|
||||
|
||||
@@ -15,16 +14,13 @@ function megabytes(value: unknown): number | undefined {
|
||||
/** The catalog for the web interface (REBUILD.md, part 9). */
|
||||
export default class ReleasesController {
|
||||
/** GET /api/v1/releases?q=&cat=5000,2040&readable=1&complete=1&unlocked=1&min_size=&max_size=(MB)&source=&poster=&sort=&page=&per_page= */
|
||||
async index(ctx: HttpContext) {
|
||||
const qs = ctx.request.qs()
|
||||
const grant = await AccessService.authorize(ctx, qs.apikey)
|
||||
await AccessService.charge(grant, 'api')
|
||||
async index({ request }: HttpContext) {
|
||||
const qs = request.qs()
|
||||
const categories = String(qs.cat ?? '')
|
||||
.split(',')
|
||||
.map((c) => Number(c.trim()))
|
||||
.filter((c) => Number.isInteger(c) && c > 0)
|
||||
return SearchService.search({
|
||||
maxAgeDays: AccessService.maxAgeDays(grant),
|
||||
q: typeof qs.q === 'string' ? qs.q : undefined,
|
||||
categories: categories.length ? expandCategories(categories) : undefined,
|
||||
readable: flag(qs.readable),
|
||||
@@ -43,20 +39,13 @@ export default class ReleasesController {
|
||||
}
|
||||
|
||||
/** GET /api/v1/releases/:id, with its files */
|
||||
async show(ctx: HttpContext) {
|
||||
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
|
||||
if (!(await AccessService.checkRelease(grant, Number(ctx.params.id)))) return ctx.response.notFound({ error: 'Release not found' })
|
||||
await AccessService.charge(grant, 'api')
|
||||
const release = await SearchService.release(Number(ctx.params.id))
|
||||
return release ?? ctx.response.notFound({ error: 'Release not found' })
|
||||
async show({ params, response }: HttpContext) {
|
||||
const release = await SearchService.release(Number(params.id))
|
||||
return release ?? response.notFound({ error: 'Release not found' })
|
||||
}
|
||||
|
||||
/** GET /api/v1/releases/:id/nzb (?apikey= for a plain link, as the web interface's) */
|
||||
async nzb(ctx: HttpContext) {
|
||||
const { params, response } = ctx
|
||||
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
|
||||
if (!(await AccessService.checkRelease(grant, Number(params.id)))) return response.notFound({ error: 'Release not found' })
|
||||
await AccessService.charge(grant, 'grab')
|
||||
/** GET /api/v1/releases/:id/nzb */
|
||||
async nzb({ params, response }: HttpContext) {
|
||||
const nzb = await NzbService.build(Number(params.id))
|
||||
if (!nzb) return response.notFound({ error: 'Release not found' })
|
||||
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
|
||||
@@ -65,12 +54,7 @@ export default class ReleasesController {
|
||||
return response.send(nzb.xml)
|
||||
}
|
||||
|
||||
/** GET /api/v1/access: whether the web interface needs an account, and how one is made. */
|
||||
async access() {
|
||||
return { mode: AccessService.mode, signup: AccessService.signup }
|
||||
}
|
||||
|
||||
/** GET /api/v1/stats (open: what the index holds, not what's in it) */
|
||||
/** GET /api/v1/stats */
|
||||
async stats() {
|
||||
return SearchService.stats()
|
||||
}
|
||||
|
||||
@@ -8,9 +8,6 @@ export default class HttpExceptionHandler extends ExceptionHandler {
|
||||
*/
|
||||
protected debug = !app.inProduction
|
||||
|
||||
/** A refused request (AccessService: no key, a limit reached) is an answer, not a fault. */
|
||||
protected ignoreCodes = ['E_ACCESS_DENIED']
|
||||
|
||||
/**
|
||||
* The method is used for handling errors and returning
|
||||
* response to the client
|
||||
|
||||
@@ -1,6 +1,4 @@
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import { UserSchema } from '#database/schema'
|
||||
import { beforeCreate } from '@adonisjs/lucid/orm'
|
||||
import hash from '@adonisjs/core/services/hash'
|
||||
import { compose } from '@adonisjs/core/helpers'
|
||||
import { withAuthFinder } from '@adonisjs/auth/mixins/lucid'
|
||||
@@ -10,12 +8,6 @@ export default class User extends compose(UserSchema, withAuthFinder(hash)) {
|
||||
static accessTokens = DbAccessTokensProvider.forModel(User)
|
||||
declare currentAccessToken?: AccessToken
|
||||
|
||||
/** The Newznab key (AccessService), known on the model right away rather than from the column's default. */
|
||||
@beforeCreate()
|
||||
static assignApiKey(user: User) {
|
||||
user.apiKey ??= randomBytes(16).toString('hex')
|
||||
}
|
||||
|
||||
get initials() {
|
||||
const [first, last] = this.fullName ? this.fullName.split(' ') : this.email.split('@')
|
||||
if (first && last) {
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import { DateTime } from 'luxon'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import { Exception } from '@adonisjs/core/exceptions'
|
||||
import type { HttpContext } from '@adonisjs/core/http'
|
||||
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
|
||||
import User from '#models/user'
|
||||
import { ACCESS, SIGNUP, TIERS, type TierLimits } from '#config/access'
|
||||
|
||||
export type Use = 'api' | 'grab'
|
||||
|
||||
/** Newznab's error codes for access (the catalog answers with the HTTP status). */
|
||||
const STATUS = { 100: 401, 101: 403, 102: 403, 500: 429, 501: 429 } as const
|
||||
|
||||
/** Why a request is refused; the catalog's answer (Newznab catches it for its own). */
|
||||
export class AccessDenied extends Exception {
|
||||
constructor(
|
||||
readonly newznabCode: keyof typeof STATUS,
|
||||
message: string
|
||||
) {
|
||||
super(message, { status: STATUS[newznabCode], code: 'E_ACCESS_DENIED' })
|
||||
}
|
||||
|
||||
async handle(error: this, ctx: HttpContext) {
|
||||
ctx.response.status(error.status).send({ error: error.message, code: error.newznabCode })
|
||||
}
|
||||
}
|
||||
|
||||
export interface Usage {
|
||||
apiHits: number
|
||||
grabs: number
|
||||
}
|
||||
|
||||
/** A caller let in: who, their tier as it is now, and what they used today. */
|
||||
export interface Grant {
|
||||
user: User
|
||||
tier: string
|
||||
limits: TierLimits
|
||||
usage: Usage
|
||||
}
|
||||
|
||||
const COLUMNS: Record<Use, string> = { api: 'api_hits', grab: 'grabs' }
|
||||
const TODAY = `(now() AT TIME ZONE 'UTC')::date`
|
||||
const DAY_MS = 86_400_000
|
||||
|
||||
/** Accounts, tiers and their daily limits (config/access.ts). */
|
||||
export default class AccessService {
|
||||
/** From config/access.ts; tests switch it. */
|
||||
static mode: 'open' | 'accounts' = ACCESS
|
||||
static signup: 'open' | 'invite' | 'closed' = SIGNUP
|
||||
|
||||
static get accounts() {
|
||||
return this.mode === 'accounts'
|
||||
}
|
||||
|
||||
static newKey() {
|
||||
return randomBytes(16).toString('hex')
|
||||
}
|
||||
|
||||
/** The user's tier as it is now: one that ran out (or isn't configured) is free. */
|
||||
static tierOf(user: User): { tier: string; limits: TierLimits } {
|
||||
const expired = user.tierUntil !== null && user.tierUntil.toMillis() <= Date.now()
|
||||
const tier = !expired && TIERS[user.tier] ? user.tier : 'free'
|
||||
return { tier, limits: TIERS[tier] }
|
||||
}
|
||||
|
||||
/** The user a request names: the web interface's token, else an apikey (Newznab, the catalog's NZB links). */
|
||||
static async caller(ctx: HttpContext, apikey?: unknown): Promise<User | null> {
|
||||
const guard = ctx.auth.use('api')
|
||||
if (guard.isAuthenticated || (await guard.check())) return guard.user!
|
||||
if (typeof apikey === 'string' && /^[0-9a-f]{32}$/.test(apikey)) return User.findBy('apiKey', apikey)
|
||||
return null
|
||||
}
|
||||
|
||||
/** Lets a request in (counting nothing yet); null when access is open. */
|
||||
static async authorize(ctx: HttpContext, apikey?: unknown): Promise<Grant | null> {
|
||||
if (!this.accounts) return null
|
||||
const user = await this.caller(ctx, apikey)
|
||||
if (!user) throw new AccessDenied(100, 'Incorrect user credentials')
|
||||
if (user.suspended) throw new AccessDenied(101, 'Account suspended')
|
||||
return { user, ...this.tierOf(user), usage: await this.usage(user.id) }
|
||||
}
|
||||
|
||||
/** Counts one use against the grant's tier, or refuses it once the day's limit is reached. */
|
||||
static async charge(grant: Grant | null, use: Use) {
|
||||
if (!grant) return
|
||||
const usage = await this.count(grant.user.id, use, use === 'api' ? grant.limits.apiHits : grant.limits.grabs)
|
||||
if (!usage) throw use === 'api' ? new AccessDenied(500, 'Request limit reached') : new AccessDenied(501, 'Download limit reached')
|
||||
grant.usage = usage
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds one use to today's row unless the limit is reached (null then): one
|
||||
* statement, so concurrent requests can't both take the last one.
|
||||
*/
|
||||
static async count(userId: number, use: Use, limit: number | null): Promise<Usage | null> {
|
||||
if (limit !== null && limit <= 0) return null
|
||||
const column = COLUMNS[use]
|
||||
const { rows } = await db.rawQuery(
|
||||
`INSERT INTO usage (user_id, day, ${column}) VALUES (?, ${TODAY}, 1)
|
||||
ON CONFLICT (user_id, day) DO UPDATE SET ${column} = usage.${column} + 1
|
||||
${limit === null ? '' : `WHERE usage.${column} < ?`}
|
||||
RETURNING api_hits, grabs`,
|
||||
limit === null ? [userId] : [userId, limit]
|
||||
)
|
||||
return rows[0] ? { apiHits: rows[0].api_hits, grabs: rows[0].grabs } : null
|
||||
}
|
||||
|
||||
static async usage(userId: number): Promise<Usage> {
|
||||
const { rows } = await db.rawQuery(`SELECT api_hits, grabs FROM usage WHERE user_id = ? AND day = ${TODAY}`, [userId])
|
||||
return { apiHits: rows[0]?.api_hits ?? 0, grabs: rows[0]?.grabs ?? 0 }
|
||||
}
|
||||
|
||||
/** A search's age bound: the asked one within the tier's retention. */
|
||||
static maxAgeDays(grant: Grant | null, requested?: number): number | undefined {
|
||||
const retention = grant?.limits.retentionDays
|
||||
if (!retention) return requested
|
||||
return requested ? Math.min(requested, retention) : retention
|
||||
}
|
||||
|
||||
/** Refuses a release posted before the tier's retention (one without a date is let through). */
|
||||
static checkPosted(grant: Grant | null, postedAt: Date | string | null) {
|
||||
const retention = grant?.limits.retentionDays
|
||||
if (!retention || !postedAt) return
|
||||
if (new Date(postedAt).getTime() < Date.now() - retention * DAY_MS) {
|
||||
throw new AccessDenied(102, `Older than your tier's retention (${retention} days)`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a release exists, refusing it when it's older than the tier sees; asked before an NZB is counted. */
|
||||
static async checkRelease(grant: Grant | null, id: number): Promise<boolean> {
|
||||
const release = await db.from('releases').where('id', id).select('posted_at').first()
|
||||
if (!release) return false
|
||||
this.checkPosted(grant, release.posted_at)
|
||||
return true
|
||||
}
|
||||
|
||||
/** Takes an unused invite for a new user: what it gives, or null when it's unknown or used. */
|
||||
static async redeem(code: string, userId: number, trx: TransactionClientContract) {
|
||||
const { rows } = await trx.rawQuery(
|
||||
`UPDATE invites SET used_by = ?, used_at = now() WHERE code = ? AND used_by IS NULL AND used_at IS NULL RETURNING tier, tier_days`,
|
||||
[userId, code]
|
||||
)
|
||||
return (rows[0] as { tier: string | null; tier_days: number | null } | undefined) ?? null
|
||||
}
|
||||
|
||||
/**
|
||||
* Gives a user a tier for `days` (null: without end). The same tier still
|
||||
* running is extended from where it ends, so a renewal paid early loses nothing.
|
||||
*/
|
||||
static grantTier(user: User, tier: string, days: number | null) {
|
||||
if (!TIERS[tier]) throw new Error(`No such tier: ${tier} (config/access.ts has ${Object.keys(TIERS).join(', ')})`)
|
||||
const now = DateTime.utc()
|
||||
const running = user.tier === tier && user.tierUntil && user.tierUntil > now ? user.tierUntil : now
|
||||
user.tier = tier
|
||||
user.tierUntil = days === null ? null : running.plus({ days })
|
||||
}
|
||||
|
||||
/** What a profile shows of the account's access. */
|
||||
static async summary(user: User) {
|
||||
const { tier, limits } = this.tierOf(user)
|
||||
return {
|
||||
mode: this.mode,
|
||||
tier,
|
||||
tierUntil: tier === user.tier ? user.tierUntil : null,
|
||||
apiKey: user.apiKey,
|
||||
limits,
|
||||
usage: await this.usage(user.id),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,20 @@
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import { pruneSevenZipTails } from '#services/seven_zip_tails'
|
||||
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
|
||||
import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
|
||||
import ChunkPartitions from '#services/chunk_partitions'
|
||||
import { decodeChunk } from '#lib/chunks/index'
|
||||
import { SIDECAR } from '#services/sidecar_folder'
|
||||
|
||||
/**
|
||||
* A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
|
||||
* fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
|
||||
*/
|
||||
function sidecarOnly(r: ReleaseChunk): boolean | null {
|
||||
if (r.files > 3 || r.bytes >= 1_000_000) return null
|
||||
const files = decodeChunk(r.payload)
|
||||
return files.length > 0 && files.every((f) => SIDECAR.test(f.name))
|
||||
}
|
||||
|
||||
export class StaleBatch extends Error {}
|
||||
|
||||
@@ -235,13 +247,21 @@ class BatchStore {
|
||||
}
|
||||
const fresh = part.filter((_, idx) => !joined.has(idx))
|
||||
if (fresh.length > 0) {
|
||||
// The first upload of a key keeps the key; later ones are key:2, key:3, …
|
||||
const { rows: counts } = await trx.rawQuery(`SELECT identity_key, count(*) AS n FROM releases WHERE identity_key = ANY(?) GROUP BY 1`, [[...new Set(fresh.map((r) => r.key))]])
|
||||
// The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
|
||||
// delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
|
||||
// taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
|
||||
const { rows: counts } = await trx.rawQuery(
|
||||
`SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
|
||||
WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
|
||||
THEN substr(release_key, length(identity_key) + 2)::int END) AS n
|
||||
FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
|
||||
[[...new Set(fresh.map((r) => r.key))]]
|
||||
)
|
||||
const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
|
||||
const numbered = (r: ReleaseChunk) => {
|
||||
const n = (taken.get(r.key) ?? 0) + 1
|
||||
taken.set(r.key, n)
|
||||
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r) }
|
||||
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
|
||||
}
|
||||
// Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
|
||||
const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
|
||||
@@ -252,10 +272,10 @@ class BatchStore {
|
||||
for (let attempt = 0; inserts.length > 0; attempt++) {
|
||||
if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
|
||||
const { rows } = await trx.rawQuery(
|
||||
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete)
|
||||
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
|
||||
SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
|
||||
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)
|
||||
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
|
||||
ON CONFLICT (release_key) DO NOTHING
|
||||
RETURNING id, release_key`,
|
||||
[groupId, JSON.stringify(inserts)]
|
||||
@@ -305,6 +325,18 @@ class BatchStore {
|
||||
.ignore()
|
||||
.returning('id')
|
||||
out.candidates += added.length
|
||||
// A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
|
||||
const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
|
||||
if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
|
||||
// Files added to a settled release (a later range of its upload): the settler made its candidates claimable
|
||||
// already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
|
||||
if (added.length > 0) {
|
||||
await trx.rawQuery(
|
||||
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
|
||||
WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
|
||||
[added.map((a: { id: string | number }) => String(a.id))]
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
import { statfs } from 'node:fs/promises'
|
||||
import logger from '@adonisjs/core/services/logger'
|
||||
|
||||
/** How often free space is measured. */
|
||||
const CHECK_MS = 60_000
|
||||
/** Resumed only this much above the floor: not paused and resumed at every flush. */
|
||||
const MARGIN_GB = 1
|
||||
|
||||
/**
|
||||
* Free space on the filesystem holding `path`, against a floor: low below it, again fine MARGIN_GB above it. In a
|
||||
* container `/` is an overlay on the host's Docker root, which on homelab is the NVMe the catalog and the WAL are on.
|
||||
* The prober filled it ~0.8–1.1 GB an hour at ~200 articles/s, and nothing stopped it at the 10 GB floor (LOG.md
|
||||
* 2026-10-18 (77)). Off when the floor is 0.
|
||||
*/
|
||||
export class DiskSpace {
|
||||
private checkedAt = 0
|
||||
private low = false
|
||||
|
||||
constructor(
|
||||
private what: string,
|
||||
private path: string,
|
||||
private floorGb: number,
|
||||
private free: (path: string) => Promise<number> = async (p) => {
|
||||
const s = await statfs(p)
|
||||
return s.bavail * s.bsize
|
||||
},
|
||||
private checkMs = CHECK_MS
|
||||
) {}
|
||||
|
||||
async short(): Promise<boolean> {
|
||||
if (!this.floorGb) return false
|
||||
if (Date.now() - this.checkedAt < this.checkMs) return this.low
|
||||
this.checkedAt = Date.now()
|
||||
const gb = (await this.free(this.path)) / 1e9
|
||||
const was = this.low
|
||||
this.low = gb < this.floorGb + (was ? MARGIN_GB : 0)
|
||||
if (this.low !== was) logger.info(`${gb.toFixed(1)} GB free (floor ${this.floorGb} GB): ${this.what} ${this.low ? 'paused' : 'resumed'}.`)
|
||||
return this.low
|
||||
}
|
||||
}
|
||||
@@ -52,6 +52,22 @@ class Leases {
|
||||
return { range: { id: Number(row.id), group: group.name, start: Number(row.start_id), end: Number(row.end_id), next: Number(row.next_id) }, rules }
|
||||
}
|
||||
|
||||
/**
|
||||
* Ranges read to their end whose worker went quiet before finishing them: done. The lease query skips a range past
|
||||
* its end, so one whose finish never came stayed leased for good (a gap range since 2026-10-16 14:03, during the lock
|
||||
* pile-up; LOG.md 2026-10-17 (72)). Its batches are all stored; its spooled chunks go.
|
||||
*/
|
||||
async finishAbandoned(): Promise<number> {
|
||||
const { rows } = await db.rawQuery(
|
||||
`UPDATE ranges SET status = 'done', done_at = now()
|
||||
WHERE status = 'leased' AND next_id > end_id AND heartbeat_at < now() - make_interval(secs => ?)
|
||||
RETURNING id`,
|
||||
[coordinatorConfig.leaseTtlSeconds]
|
||||
)
|
||||
for (const { id } of rows) await Spool.remove(Number(id))
|
||||
return rows.length
|
||||
}
|
||||
|
||||
async leased(rangeId: number, nodeId: number) {
|
||||
const range = await db.from('ranges').where('id', rangeId).first()
|
||||
if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId) throw new LeaseLost(`Range ${rangeId} is not leased to this node`)
|
||||
|
||||
+93
-13
@@ -2,12 +2,18 @@ import env from '#start/env'
|
||||
import logger from '@adonisjs/core/services/logger'
|
||||
import coordinatorConfig from '#config/coordinator'
|
||||
import { XoverClient } from '#lib/nntp/index'
|
||||
import type { NntpShare } from '#services/nntp_shares'
|
||||
|
||||
/** Connections wanted beyond the most in use at once since the last sync (the planner's, a hedge's). */
|
||||
const SHARE_MARGIN = 2
|
||||
|
||||
/**
|
||||
* The coordinator's header connections (REBUILD.md, decision e: workers
|
||||
* have no NNTP account). At most NNTP_CONNECTIONS, reused; a connection
|
||||
* that failed, or is older than NNTP_CONNECTION_MAX_MINUTES, is closed and
|
||||
* replaced on the next use.
|
||||
* replaced on the next use. Sharing the account (share()), at most its
|
||||
* share: over it, connections close as they come free. One left free for
|
||||
* NNTP_IDLE_CLOSE_SECONDS is closed on the next sync.
|
||||
*/
|
||||
class NntpPool {
|
||||
private idle: XoverClient[] = []
|
||||
@@ -17,9 +23,43 @@ class NntpPool {
|
||||
private openedAt = new Map<XoverClient, number>()
|
||||
/** When a connection gets closed once free: its max age ±25%, so they don't all reconnect at once. */
|
||||
private retireAt = new Map<XoverClient, number>()
|
||||
/** When an idle connection came free. */
|
||||
private idleSince = new Map<XoverClient, number>()
|
||||
private limit = coordinatorConfig.nntpConnections
|
||||
/** Since the last sync: the most connections in use at once, and whether work waited for one (starved()). */
|
||||
private peak = 0
|
||||
private short = false
|
||||
|
||||
get size() {
|
||||
return coordinatorConfig.nntpConnections
|
||||
return this.limit
|
||||
}
|
||||
|
||||
/** Work that could use another connection didn't get one (the prefetcher with all its loops running). */
|
||||
starved() {
|
||||
this.short = true
|
||||
}
|
||||
|
||||
/**
|
||||
* Tells the share what the pool could use and has open, and takes the share's grant as its size. After work waited:
|
||||
* twice its size (history going on gets its maximum in a few syncs; a tail round, done in seconds, doesn't take
|
||||
* the prober's connections for one); else the most in use at once since the last sync and a margin.
|
||||
*/
|
||||
async share(share: NntpShare) {
|
||||
const want = this.short || this.waiting.length > 0 ? Math.max(this.limit * 2, this.limit + SHARE_MARGIN) : this.peak + SHARE_MARGIN
|
||||
this.short = false
|
||||
this.peak = this.busy.size
|
||||
const granted = await share.sync(want, this.open)
|
||||
const was = this.limit
|
||||
this.limit = granted
|
||||
// Over the share: the idle ones close now, the busy ones once free (release()).
|
||||
while (this.open > this.limit && this.idle.length > 0) this.retire(this.idle.pop()!)
|
||||
this.closeIdle()
|
||||
// Under it with work waiting: connections for it.
|
||||
while (this.waiting.length > 0 && this.open < this.limit) {
|
||||
const next = this.waiting.shift()!
|
||||
this.acquire().then(next, () => this.waiting.unshift(next))
|
||||
}
|
||||
return { was, granted }
|
||||
}
|
||||
|
||||
async use<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
|
||||
@@ -44,6 +84,7 @@ class NntpPool {
|
||||
|
||||
private async run<T>(client: XoverClient, work: (client: XoverClient) => Promise<T>): Promise<T> {
|
||||
this.busy.add(client)
|
||||
this.peak = Math.max(this.peak, this.busy.size)
|
||||
let broken = false
|
||||
try {
|
||||
return await work(client)
|
||||
@@ -56,9 +97,28 @@ class NntpPool {
|
||||
}
|
||||
}
|
||||
|
||||
/** Closes the connections free for nntpIdleCloseMs, the plain one too. */
|
||||
private closeIdle() {
|
||||
const idleMs = coordinatorConfig.nntpIdleCloseMs
|
||||
if (idleMs <= 0) return
|
||||
const before = Date.now() - idleMs
|
||||
for (const client of this.idle.filter((c) => (this.idleSince.get(c) ?? Infinity) <= before)) {
|
||||
this.idle.splice(this.idle.indexOf(client), 1)
|
||||
this.retire(client)
|
||||
}
|
||||
if (this.plain && this.plainPending === 0 && this.plainUsedAt <= before) {
|
||||
void this.plain.then((c) => c.close()).catch(() => {})
|
||||
this.plain = null
|
||||
}
|
||||
}
|
||||
|
||||
private async acquire(): Promise<XoverClient> {
|
||||
// The one used last: the others stay free long enough to close.
|
||||
const ready = this.idle.pop()
|
||||
if (ready) return ready
|
||||
if (ready) {
|
||||
this.idleSince.delete(ready)
|
||||
return ready
|
||||
}
|
||||
if (this.open < this.size) {
|
||||
this.open++
|
||||
try {
|
||||
@@ -84,15 +144,12 @@ class NntpPool {
|
||||
}
|
||||
|
||||
private release(client: XoverClient, broken: boolean) {
|
||||
if (broken || Date.now() >= (this.retireAt.get(client) ?? Infinity)) {
|
||||
this.open--
|
||||
this.openedAt.delete(client)
|
||||
this.retireAt.delete(client)
|
||||
try {
|
||||
client.close()
|
||||
} catch (error: any) {
|
||||
logger.debug(`Closing a failed NNTP connection: ${error.message}`)
|
||||
if (this.open > this.limit) {
|
||||
this.retire(client)
|
||||
return
|
||||
}
|
||||
if (broken || Date.now() >= (this.retireAt.get(client) ?? Infinity)) {
|
||||
this.retire(client)
|
||||
// Someone waiting gets a fresh connection.
|
||||
const next = this.waiting.shift()
|
||||
if (next) this.acquire().then(next, () => this.waiting.unshift(next))
|
||||
@@ -100,12 +157,30 @@ class NntpPool {
|
||||
}
|
||||
const next = this.waiting.shift()
|
||||
if (next) next(client)
|
||||
else this.idle.push(client)
|
||||
else {
|
||||
this.idle.push(client)
|
||||
this.idleSince.set(client, Date.now())
|
||||
}
|
||||
}
|
||||
|
||||
private retire(client: XoverClient) {
|
||||
this.open--
|
||||
this.openedAt.delete(client)
|
||||
this.retireAt.delete(client)
|
||||
this.idleSince.delete(client)
|
||||
try {
|
||||
client.close()
|
||||
} catch (error: any) {
|
||||
logger.debug(`Closing an NNTP connection: ${error.message}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Closes every connection, also those in use: their commands fail. */
|
||||
private plain: Promise<XoverClient> | null = null
|
||||
private plainQueue: Promise<unknown> = Promise.resolve()
|
||||
/** Commands queued or running on the plain connection, and when the last one ended. */
|
||||
private plainPending = 0
|
||||
private plainUsedAt = 0
|
||||
|
||||
/**
|
||||
* One connection without XFEATURE COMPRESS GZIP, opened when first needed,
|
||||
@@ -113,6 +188,7 @@ class NntpPool {
|
||||
* cuts short (Leases.corrupt). Its replies are deflated here.
|
||||
*/
|
||||
async usePlain<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
|
||||
this.plainPending++
|
||||
const run = this.plainQueue.then(async () => {
|
||||
this.plain ??= XoverClient.connect({
|
||||
host: env.get('NNTP_HOST')!,
|
||||
@@ -132,7 +208,10 @@ class NntpPool {
|
||||
throw error
|
||||
}
|
||||
})
|
||||
this.plainQueue = run.catch(() => {})
|
||||
this.plainQueue = run.catch(() => {}).finally(() => {
|
||||
this.plainPending--
|
||||
this.plainUsedAt = Date.now()
|
||||
})
|
||||
return run
|
||||
}
|
||||
|
||||
@@ -143,6 +222,7 @@ class NntpPool {
|
||||
this.busy.clear()
|
||||
this.openedAt.clear()
|
||||
this.retireAt.clear()
|
||||
this.idleSince.clear()
|
||||
this.open = 0
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import coordinatorConfig from '#config/coordinator'
|
||||
|
||||
/** A share whose process hasn't synced for this long is gone: its connections closed with it. */
|
||||
const STALE_SECONDS = 60
|
||||
/** Syncs one at a time (pg_advisory_xact_lock). */
|
||||
const LOCK = 2_026_101_875
|
||||
|
||||
export interface Share {
|
||||
name: string
|
||||
priority: number
|
||||
/** Kept whatever the others want (as far as the total goes, by priority). */
|
||||
min: number
|
||||
/** What it could use now. */
|
||||
want: number
|
||||
/** Connections it has open (or may have: a lane without one yet counts). */
|
||||
open: number
|
||||
}
|
||||
|
||||
/** What each share should have: its minimum first, then by priority what it wants, up to the total. */
|
||||
export function targets(total: number, shares: Share[]): Map<string, number> {
|
||||
const ordered = [...shares].sort((a, b) => b.priority - a.priority || a.name.localeCompare(b.name))
|
||||
const target = new Map<string, number>()
|
||||
let left = total
|
||||
for (const s of ordered) {
|
||||
const t = Math.min(s.min, left)
|
||||
target.set(s.name, t)
|
||||
left -= t
|
||||
}
|
||||
for (const s of ordered) {
|
||||
const more = Math.min(Math.max(s.want - target.get(s.name)!, 0), left)
|
||||
target.set(s.name, target.get(s.name)! + more)
|
||||
left -= more
|
||||
}
|
||||
return target
|
||||
}
|
||||
|
||||
/**
|
||||
* The connections `self` may have: its target, but no more than the others leave over. Another share holds what it
|
||||
* has open or was granted, whichever is more (it may open up to its grant until its next sync tells it otherwise),
|
||||
* so the account never goes over the total: a share grows only into what the others closed or gave up.
|
||||
*/
|
||||
export function grant(total: number, self: Share, others: (Share & { granted: number })[]): number {
|
||||
const target = targets(total, [self, ...others]).get(self.name)!
|
||||
const held = others.reduce((sum, o) => sum + Math.max(o.open, o.granted), 0)
|
||||
return Math.max(0, Math.min(target, total - held))
|
||||
}
|
||||
|
||||
/**
|
||||
* A process's share of the NNTP account's connections (LOG.md 2026-10-18 (75)): the coordinator and the prober had a
|
||||
* fixed split, and with history paused the coordinator's 38 stood idle while the prober's 8 were its limit. Each
|
||||
* syncs every nntpShares.everyMs (what it wants and has open) and gets what it may open; one wanting less closes
|
||||
* connections as they come free, and the other grows as those show up closed.
|
||||
*/
|
||||
export class NntpShare {
|
||||
granted = 0
|
||||
|
||||
constructor(
|
||||
readonly name: string,
|
||||
private priority: number,
|
||||
readonly min: number,
|
||||
readonly max: number
|
||||
) {}
|
||||
|
||||
async sync(want: number, open: number): Promise<number> {
|
||||
const self: Share = { name: this.name, priority: this.priority, min: this.min, want: Math.min(Math.max(want, this.min), this.max), open }
|
||||
const total = coordinatorConfig.nntpShares.total
|
||||
this.granted = await db.transaction(async (trx) => {
|
||||
await trx.rawQuery(`SELECT pg_advisory_xact_lock(?)`, [LOCK])
|
||||
const { rows } = await trx.rawQuery(
|
||||
`SELECT name, priority, min, want, open, granted FROM nntp_shares WHERE name <> ? AND heartbeat_at > now() - make_interval(secs => ?)`,
|
||||
[this.name, STALE_SECONDS]
|
||||
)
|
||||
const granted = grant(total, self, rows)
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO nntp_shares (name, priority, min, want, open, granted, heartbeat_at) VALUES (?, ?, ?, ?, ?, ?, now())
|
||||
ON CONFLICT (name) DO UPDATE SET priority = excluded.priority, min = excluded.min, want = excluded.want,
|
||||
open = excluded.open, granted = excluded.granted, heartbeat_at = now()`,
|
||||
[this.name, this.priority, this.min, self.want, open, granted]
|
||||
)
|
||||
return granted
|
||||
})
|
||||
return this.granted
|
||||
}
|
||||
|
||||
/** Stopping: its connections are closed, the others needn't wait for the row to go stale. */
|
||||
async leave() {
|
||||
await db.from('nntp_shares').where('name', this.name).delete()
|
||||
}
|
||||
}
|
||||
+96
-20
@@ -4,6 +4,7 @@ import { par2ReleaseName } from '#lib/release/index'
|
||||
import { looksRandom } from '#lib/release/random'
|
||||
import { SidecarFolder } from '#services/sidecar_folder'
|
||||
import { Par2Pruner } from '#services/par2_pruner'
|
||||
import { pruneSevenZipTails } from '#services/seven_zip_tails'
|
||||
|
||||
/** Releases of a set further apart than this (posted) aren't one upload: a repost is merged separately (decision d). */
|
||||
const MAX_SPREAD_MS = 48 * 3600 * 1000
|
||||
@@ -15,15 +16,28 @@ const MAX_HOLDERS = 4
|
||||
/** Members with more files than this many times the set's, plus SLACK_FILES (its par2 volumes), are more than one upload. */
|
||||
const MAX_FILES_PER_SET_FILE = 3
|
||||
const SLACK_FILES = 10
|
||||
/**
|
||||
* A par2 file held by more file rows than this (the same first 16 KiB and size) is a common file, not one upload's:
|
||||
* it matches nothing. One such file was in 190,380 uploads, and a set naming it loaded every one of them; the merger
|
||||
* ran out of heap every few minutes (LOG.md 2026-10-20 (85)). The judge calls 20 releases repeated.
|
||||
*/
|
||||
const COMMON_FILE = 20
|
||||
/** Between rounds. */
|
||||
const ROUND_MS = 60_000
|
||||
/** A round looks back this far past the previous one's start (rows committed while it ran). */
|
||||
/** A window looks back this far past the previous one's end (rows committed after it was read). */
|
||||
const OVERLAP_MS = 60_000
|
||||
/**
|
||||
* A round goes through what's new in windows of this much (created_at), its place kept after each: a round behind by
|
||||
* two days of the probe backlog (3.5M sets) never ended, and every restart started it over (LOG.md 2026-10-20 (85)).
|
||||
*/
|
||||
const WINDOW_MS = 60 * 60_000
|
||||
/** The sidecar fold also runs this often inside a long par2 pass: its queue grows by the settler's pace. */
|
||||
const FOLD_EVERY_MS = 5 * 60_000
|
||||
/** Between par2_files prunes (complete and judged sets' file lists). */
|
||||
const PRUNE_EVERY_MS = 60 * 60_000
|
||||
/** The merger's row in worker_cursors: the start of its last whole round, kept over a restart. */
|
||||
/** A prune's share of the merger's time: the rest goes on in the next one. */
|
||||
const PRUNE_BUDGET_MS = 2 * 60_000
|
||||
/** The merger's row in worker_cursors: the end of its last whole window, kept over a restart. */
|
||||
const CURSOR = 'merger'
|
||||
|
||||
export interface MergeRound {
|
||||
@@ -44,6 +58,12 @@ export interface MergeRound {
|
||||
*/
|
||||
const INNER_STEM = `substring(i.result->>'yencName' from '^([A-Za-z0-9]{16,64})\\.')`
|
||||
|
||||
/** The releases holding a par2 file f (its first 16 KiB and size), none when it's a common file (COMMON_FILE). */
|
||||
const HOLDERS = `SELECT c.release_id FROM (
|
||||
SELECT l.release_id, count(*) OVER () AS holders
|
||||
FROM (SELECT h.release_id FROM file_hashes h WHERE h.hash16k = f.hash16k AND h.size = f.length LIMIT ${COMMON_FILE + 1}) l
|
||||
) c WHERE c.holders <= ${COMMON_FILE}`
|
||||
|
||||
interface Member {
|
||||
id: number
|
||||
release_key: string
|
||||
@@ -95,8 +115,8 @@ export class Par2Merger {
|
||||
// First, at start too: a prune needn't wait for a long round.
|
||||
if (!this.stopped && Date.now() - prunedAt >= PRUNE_EVERY_MS) {
|
||||
prunedAt = Date.now()
|
||||
const pruned = await new Par2Pruner().round()
|
||||
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged.`)
|
||||
const pruned = await new Par2Pruner().round(PRUNE_BUDGET_MS)
|
||||
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged, ${pruned.final} final.`)
|
||||
}
|
||||
const round = await this.round()
|
||||
if (round.merges > 0) logger.info(`Merged ${round.releases} releases into ${round.merges} by par2 set (${round.sets} sets checked).`)
|
||||
@@ -109,14 +129,18 @@ export class Par2Merger {
|
||||
}
|
||||
}
|
||||
|
||||
/** The sets with something new since the last round: a par2 found, or a hash of one of their files. */
|
||||
/**
|
||||
* The sets and inner stems with something new since the last round (a par2 found, a hash of one of their files, a
|
||||
* lone inspection), a window at a time.
|
||||
*/
|
||||
async round(): Promise<MergeRound> {
|
||||
const { rows: now } = await db.rawQuery(`SELECT now() AS at`)
|
||||
// A Date keeps milliseconds, now() microseconds: rounded up, a row made at now() is before the end.
|
||||
const { rows: now } = await db.rawQuery(`SELECT now() + interval '1 millisecond' AS at`)
|
||||
const end = new Date(now[0].at)
|
||||
if (!this.since) {
|
||||
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
|
||||
this.since = saved ? new Date(saved.at) : new Date(0)
|
||||
}
|
||||
const since = new Date(this.since.getTime() - OVERLAP_MS)
|
||||
// First: sidecars (.nfo, .sfv, .nzb) apart from their upload join it. A queue, not since: it needn't wait for the
|
||||
// par2 pass (hours when that starts over).
|
||||
const folded = await this.folder.round()
|
||||
@@ -130,14 +154,15 @@ export class Par2Merger {
|
||||
foldedAt = Date.now()
|
||||
if (more.folded) logger.info(`Folded ${more.folded} sidecars into their uploads (${more.sidecars} sidecar-only of ${more.looked} small releases).`)
|
||||
}
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT set_id FROM par2_sets WHERE created_at >= ?
|
||||
UNION
|
||||
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size WHERE h.created_at >= ?`,
|
||||
[since, since]
|
||||
)
|
||||
const round: MergeRound = { sets: rows.length, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
|
||||
for (const { set_id: setId } of rows) {
|
||||
const round: MergeRound = { sets: 0, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
|
||||
while (this.since < end) {
|
||||
if (this.stopped) return round
|
||||
const from = await this.firstNew(new Date(this.since.getTime() - OVERLAP_MS), end)
|
||||
const until = from ? new Date(Math.min(from.getTime() + WINDOW_MS, end.getTime())) : end
|
||||
if (from) {
|
||||
const { touched, sets } = await this.setsToMerge(from, until)
|
||||
round.sets += touched
|
||||
for (const setId of sets) {
|
||||
if (this.stopped) return round
|
||||
await foldAgain()
|
||||
// One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
|
||||
@@ -151,8 +176,9 @@ export class Par2Merger {
|
||||
}
|
||||
}
|
||||
const { rows: stems } = await db.rawQuery(
|
||||
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i WHERE i.kind = 'lone' AND i.created_at >= ? AND ${INNER_STEM} IS NOT NULL`,
|
||||
[since]
|
||||
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i
|
||||
WHERE i.kind = 'lone' AND i.created_at >= ? AND i.created_at <= ? AND ${INNER_STEM} IS NOT NULL`,
|
||||
[from, until]
|
||||
)
|
||||
for (const { stem } of stems) {
|
||||
if (this.stopped) return round
|
||||
@@ -167,15 +193,57 @@ export class Par2Merger {
|
||||
round.stemReleases += merged + 1
|
||||
}
|
||||
}
|
||||
// Only after a whole round: a restart halfway goes on from the round before.
|
||||
this.since = now[0].at
|
||||
}
|
||||
// Only after a whole window: a restart halfway goes on from the window before.
|
||||
this.since = until
|
||||
await db.rawQuery(
|
||||
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
|
||||
[CURSOR, this.since]
|
||||
)
|
||||
}
|
||||
return round
|
||||
}
|
||||
|
||||
/** When the first row a window reads was made, from `from` up to `end`; null when none (a gap, the nightly pause). */
|
||||
private async firstNew(from: Date, end: Date): Promise<Date | null> {
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT least(
|
||||
(SELECT min(created_at) FROM par2_sets WHERE created_at >= ?),
|
||||
(SELECT min(created_at) FROM file_hashes WHERE created_at >= ?),
|
||||
(SELECT min(created_at) FROM release_inspections WHERE created_at >= ?)) AS at`,
|
||||
[from, from, from]
|
||||
)
|
||||
const at = rows[0].at ? new Date(rows[0].at) : null
|
||||
return at && at <= end ? at : null
|
||||
}
|
||||
|
||||
/**
|
||||
* The sets a window touched (a par2 found, a hash of one of their files), and those among them a merge could
|
||||
* change: held or matched by two releases or more. Most have every file in one release already (96% of a backlog
|
||||
* hour's 147k): mergeSet needn't look at them.
|
||||
*/
|
||||
private async setsToMerge(from: Date, until: Date): Promise<{ touched: number; sets: string[] }> {
|
||||
const { rows } = await db.rawQuery(
|
||||
`WITH touched AS MATERIALIZED (
|
||||
SELECT set_id FROM par2_sets WHERE created_at >= ? AND created_at <= ?
|
||||
UNION
|
||||
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size
|
||||
WHERE h.created_at >= ? AND h.created_at <= ?
|
||||
)
|
||||
SELECT (SELECT count(*) FROM touched) AS touched, array(
|
||||
SELECT t.set_id FROM touched t
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM (
|
||||
SELECT s.release_id FROM par2_sets s WHERE s.set_id = t.set_id
|
||||
UNION ALL
|
||||
SELECT m.release_id FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) m WHERE f.set_id = t.set_id
|
||||
) u HAVING count(DISTINCT u.release_id) >= 2)
|
||||
) AS sets`,
|
||||
[from, until, from, until]
|
||||
)
|
||||
return { touched: Number(rows[0].touched), sets: rows[0].sets }
|
||||
}
|
||||
|
||||
/** Merges the releases whose lone files' inner names share a random stem; how many went into another. */
|
||||
async mergeStem(stem: string): Promise<number> {
|
||||
return db.transaction(async (trx) => {
|
||||
@@ -220,7 +288,7 @@ export class Par2Merger {
|
||||
const near = (r: Member) => r.group_id === anchor.group_id && (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
|
||||
const { rows: matches } = await trx.rawQuery(
|
||||
`SELECT f.file_id, r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable
|
||||
FROM par2_files f JOIN file_hashes h ON h.hash16k = f.hash16k AND h.size = f.length JOIN releases r ON r.id = h.release_id
|
||||
FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) h JOIN releases r ON r.id = h.release_id
|
||||
WHERE f.set_id = ?`,
|
||||
[setId]
|
||||
)
|
||||
@@ -268,6 +336,14 @@ export class Par2Merger {
|
||||
for (const table of ['release_chunks', 'probe_candidates', 'release_inspections', 'file_hashes']) {
|
||||
await trx.rawQuery(`UPDATE ${table} SET release_id = ? WHERE release_id = ANY(?)`, [target.id, others])
|
||||
}
|
||||
// An unsettled member's candidates into a settled target: the settler won't come by again to make them claimable.
|
||||
await trx.rawQuery(
|
||||
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
|
||||
WHERE r.id = ? AND r.settled_at IS NOT NULL AND pc.release_id = r.id AND pc.status = 'waiting' AND NOT pc.ready`,
|
||||
[target.id]
|
||||
)
|
||||
// The members' 7z tails together: those below the highest volume go.
|
||||
await pruneSevenZipTails(trx, [target.id])
|
||||
// A set held by the target already, or by two of the others (a par2 and its repost), isn't held twice (every
|
||||
// round failed on the primary key from 20:28 UTC on 2026-10-12; LOG.md 2026-10-12 (30)).
|
||||
await trx.rawQuery(
|
||||
|
||||
+69
-43
@@ -4,72 +4,98 @@ import db from '@adonisjs/lucid/services/db'
|
||||
const SETTLED_MS = 2 * 86400_000
|
||||
/** Sets per statement. */
|
||||
const BATCH = 20_000
|
||||
/** The complete pass's row in worker_cursors: the newest holder's created_at of the last set it looked at. */
|
||||
/** The sweep's row in worker_cursors: the last set_id it looked at (key); past the last set it starts over. */
|
||||
const CURSOR = 'par2-prune'
|
||||
/** The merger's row in worker_cursors: the end of its last whole window. */
|
||||
const MERGER = 'merger'
|
||||
/** The merger's windows look back this far (Par2Merger's OVERLAP_MS). */
|
||||
const MERGER_OVERLAP_MS = 60_000
|
||||
/** Verdicts whose releases needn't merge any more; locked isn't one (a password found takes it back). */
|
||||
const JUNK = ['dead', 'fake', 'repeated']
|
||||
/** Probe candidates whose results a merge reads: a file's hash (lone), a set's file list (par2). */
|
||||
const MERGE_KINDS = ['lone', 'par2']
|
||||
|
||||
export interface PruneRound {
|
||||
complete: number
|
||||
judged: number
|
||||
final: number
|
||||
rows: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60)): the table only grew, 35M rows and
|
||||
* 13.6 GB at 1.8M sets. A set's file list matches file hashes to releases and names an unreadable upload once; it's
|
||||
* done when
|
||||
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60), 2026-10-20 (87)). A set's file list
|
||||
* matches file hashes to releases and names an unreadable upload once; it's done when
|
||||
* - complete: SETTLED_MS after its last holder's par2, every file it describes has its hash in a release holding the
|
||||
* set (all of it merged into one);
|
||||
* - judged: every release holding it has a junk verdict (dead, fake, repeated).
|
||||
* - judged: every release holding it has a junk verdict (dead, fake, repeated);
|
||||
* - final: SETTLED_MS after its last holder's par2, every holder is final (release_final: a run of done ranges holds
|
||||
* it with hours to spare, so no more of its upload will be read), the candidates of the ranges around it are probed,
|
||||
* and the merger has seen every hash of its files. A file still missing then is one no probe will hash: most sets
|
||||
* describe a readable upload's files, which aren't lone candidates (9 in 10 sets were kept for that, 19 GB).
|
||||
* par2_sets stays (the judge's unusable rule, the merge's holders). A set comes up again only with a new holder: its
|
||||
* par2 reposted, whose probe writes the rows again; without rows its merge finds no files and the count check holds.
|
||||
* The sweep goes through the sets with rows by set_id and starts over at the end: a set kept now (a holder at the edge
|
||||
* of a run, its ranges' probes not done yet) is looked at again on the next pass.
|
||||
*/
|
||||
export class Par2Pruner {
|
||||
async round(): Promise<PruneRound> {
|
||||
const round: PruneRound = { complete: 0, judged: 0, rows: 0 }
|
||||
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
|
||||
let since: Date = saved ? new Date(saved.at) : new Date(0)
|
||||
const until = new Date(Date.now() - SETTLED_MS)
|
||||
/** Sets per batch (BATCH; smaller in the tests). */
|
||||
constructor(private batch = BATCH) {}
|
||||
|
||||
/**
|
||||
* Batches until a pass is done or `budgetMs` is spent (the cursor is saved per batch, the next round goes on from
|
||||
* it): run from the merger's loop, a first round over two days of sets held the merges back for hours (LOG.md
|
||||
* 2026-10-17 (72)).
|
||||
*/
|
||||
async round(budgetMs = Infinity): Promise<PruneRound> {
|
||||
const deadline = Date.now() + budgetMs
|
||||
const round: PruneRound = { complete: 0, judged: 0, final: 0, rows: 0 }
|
||||
const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
|
||||
let after: string = cursors[0]?.key ?? ''
|
||||
// Ranges done since the oldest merge-kind candidate still unprobed: their releases' hashes and file lists may be
|
||||
// still to come. Hashes since the merger's last window: it hasn't matched them yet.
|
||||
const { rows: marks } = await db.rawQuery(
|
||||
`SELECT coalesce((SELECT min(created_at) FROM probe_candidates WHERE status IN ('waiting', 'taken') AND kind = ANY(?)), now()) AS probed,
|
||||
coalesce((SELECT at FROM worker_cursors WHERE name = ?), to_timestamp(0)) - make_interval(secs => ?) AS merged,
|
||||
now() - make_interval(secs => ?) AS settled`,
|
||||
[MERGE_KINDS, MERGER, MERGER_OVERLAP_MS / 1000, SETTLED_MS / 1000]
|
||||
)
|
||||
const { probed, merged, settled } = marks[0]
|
||||
for (;;) {
|
||||
const { rows: sets } = await db.rawQuery(
|
||||
`SELECT set_id, max(created_at) AS at FROM par2_sets GROUP BY set_id
|
||||
HAVING max(created_at) > ? AND max(created_at) <= ? ORDER BY 2 LIMIT ?`,
|
||||
[since, until, BATCH]
|
||||
)
|
||||
if (sets.length === 0) break
|
||||
const ids = sets.map((s: { set_id: string }) => s.set_id)
|
||||
const { rows: done } = await db.rawQuery(
|
||||
`SELECT c.set_id FROM unnest(?::char(32)[]) AS c(set_id)
|
||||
WHERE EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id AND NOT EXISTS (
|
||||
`WITH batch AS MATERIALIZED (SELECT DISTINCT set_id FROM par2_files WHERE set_id > ? ORDER BY set_id LIMIT ?),
|
||||
recent AS MATERIALIZED (SELECT group_id, posted_low, posted_high FROM ranges WHERE status = 'done' AND done_at >= ?),
|
||||
unseen AS MATERIALIZED (SELECT DISTINCT hash16k, size FROM file_hashes WHERE created_at >= ?)
|
||||
SELECT b.set_id, CASE
|
||||
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
|
||||
AND NOT EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = b.set_id AND NOT EXISTS (
|
||||
SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
|
||||
WHERE h.hash16k = f.hash16k AND h.size = f.length))`,
|
||||
[ids]
|
||||
WHERE h.hash16k = f.hash16k AND h.size = f.length)) THEN 'complete'
|
||||
WHEN NOT EXISTS (SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts v ON v.release_id = o.release_id AND v.rule = ANY(?)
|
||||
WHERE o.set_id = b.set_id AND v.release_id IS NULL) THEN 'judged'
|
||||
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
|
||||
AND NOT EXISTS (SELECT 1 FROM par2_sets s JOIN releases r ON r.id = s.release_id WHERE s.set_id = b.set_id AND (
|
||||
release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) IS NOT TRUE
|
||||
OR EXISTS (SELECT 1 FROM recent x WHERE x.group_id = r.group_id
|
||||
AND x.posted_low <= coalesce(r.last_posted_at, r.posted_at) + interval '6 hours' AND x.posted_high >= r.posted_at - interval '6 hours')))
|
||||
AND NOT EXISTS (SELECT 1 FROM par2_files f JOIN unseen u ON u.hash16k = f.hash16k AND u.size = f.length WHERE f.set_id = b.set_id)
|
||||
THEN 'final' END AS reason
|
||||
FROM batch b ORDER BY b.set_id`,
|
||||
[after, this.batch, probed, merged, settled, JUNK, settled]
|
||||
)
|
||||
round.complete += done.length
|
||||
round.rows += await this.drop(done.map((s: { set_id: string }) => s.set_id))
|
||||
since = sets.at(-1).at
|
||||
await db.rawQuery(
|
||||
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
|
||||
[CURSOR, since]
|
||||
)
|
||||
if (sets.length < BATCH) break
|
||||
const done = (reason: keyof PruneRound) => sets.filter((s: { reason: string }) => s.reason === reason).map((s: { set_id: string }) => s.set_id)
|
||||
for (const reason of ['complete', 'judged', 'final'] as const) {
|
||||
const ids = done(reason)
|
||||
round[reason] += ids.length
|
||||
round.rows += await this.drop(ids)
|
||||
}
|
||||
// Judged: no cursor (verdicts come and go); only sets with rows left.
|
||||
const { rows: judged } = await db.rawQuery(
|
||||
`SELECT DISTINCT s.set_id FROM release_verdicts v JOIN par2_sets s ON s.release_id = v.release_id
|
||||
WHERE v.rule = ANY(?)
|
||||
AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts ov ON ov.release_id = o.release_id AND ov.rule = ANY(?)
|
||||
WHERE o.set_id = s.set_id AND ov.release_id IS NULL)`,
|
||||
[JUNK, JUNK]
|
||||
// Past the last set: the next batch starts the next pass.
|
||||
after = sets.length < this.batch ? '' : sets.at(-1).set_id
|
||||
await db.rawQuery(
|
||||
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
|
||||
[CURSOR, after]
|
||||
)
|
||||
round.judged = judged.length
|
||||
const judgedIds = judged.map((s: { set_id: string }) => s.set_id)
|
||||
for (let i = 0; i < judgedIds.length; i += BATCH) round.rows += await this.drop(judgedIds.slice(i, i + BATCH))
|
||||
if (after === '' || Date.now() > deadline) break
|
||||
}
|
||||
return round
|
||||
}
|
||||
|
||||
|
||||
@@ -131,6 +131,8 @@ export class Prefetcher {
|
||||
.catch((error) => logger.warn(`Prefetching: ${error.message}`))
|
||||
.finally(() => this.running--)
|
||||
}
|
||||
// Work left with every loop running: the pool's share could be bigger.
|
||||
if (!this.stopped && this.running >= this.loops && this.peek()) NntpPool.starved()
|
||||
}
|
||||
|
||||
stop() {
|
||||
@@ -145,6 +147,9 @@ export class Prefetcher {
|
||||
/** One connection's loop: chunks until there is nothing left to fetch. */
|
||||
private async work() {
|
||||
while (!this.stopped) {
|
||||
// The pool's share got smaller: fewer loops (waiting on the pool, they'd count as work short of a connection).
|
||||
// Two leaving at once is one too many for a tick (TICK_MS), whose loop comes back.
|
||||
if (this.running > this.loops) return
|
||||
const next = this.take()
|
||||
if (!next) return
|
||||
const { target, span } = next
|
||||
|
||||
+86
-14
@@ -6,6 +6,8 @@ import coordinatorConfig from '#config/coordinator'
|
||||
import { XoverClient } from '#lib/nntp/index'
|
||||
import { YencError } from '#lib/yenc/index'
|
||||
import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/index'
|
||||
import { NntpShare } from '#services/nntp_shares'
|
||||
import { DiskSpace } from '#services/disk_space'
|
||||
|
||||
interface Candidate {
|
||||
id: number
|
||||
@@ -28,6 +30,15 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
|
||||
{ kind: 'archive', unreadable: false },
|
||||
]
|
||||
const FLUSH_MS = 2000
|
||||
|
||||
/**
|
||||
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
|
||||
* and the whole flush failed with it (LOG.md 2026-10-18 (78)).
|
||||
*/
|
||||
function jsonb(value: unknown): string {
|
||||
return JSON.stringify(value, (_key, v) => (typeof v === 'string' ? v.replaceAll('\0', '') : v))
|
||||
}
|
||||
|
||||
/** Slices a claim takes from random points of the queue. */
|
||||
const SLICES = 20
|
||||
const REPORT_MS = 60_000
|
||||
@@ -46,6 +57,14 @@ export class Prober {
|
||||
private refilling: Promise<void> | null = null
|
||||
private stopped = false
|
||||
private stats = Prober.freshStats()
|
||||
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
|
||||
private lanes = new Set<Promise<void>>()
|
||||
private running = 0
|
||||
/** The last lane to look found nothing to claim: the prober wants its minimum only. */
|
||||
private dry = false
|
||||
/** What it stores (inspections, par2 file lists, the WAL) fills the NVMe: no claims below PROBE_MIN_FREE_GB. */
|
||||
private space = new DiskSpace('probing', '/', coordinatorConfig.probe.minFreeGb)
|
||||
private share = new NntpShare('prober', coordinatorConfig.nntpShares.proberPriority, coordinatorConfig.probe.minConnections, coordinatorConfig.probe.connections)
|
||||
|
||||
/** Per minute: what was read, and where the lanes' time went (ms summed over lanes; flushes ms on their own). */
|
||||
private static freshStats() {
|
||||
@@ -56,8 +75,7 @@ export class Prober {
|
||||
// A previous run's claims: one prober at a time.
|
||||
const { rowCount } = await db.rawQuery(`UPDATE probe_candidates SET status = 'waiting' WHERE status = 'taken'`)
|
||||
if (rowCount) logger.info(`Probing: ${rowCount} candidates from a previous run back in the queue.`)
|
||||
const connections = coordinatorConfig.probe.connections
|
||||
logger.info(`Probing with ${connections} connections.`)
|
||||
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
|
||||
const flusher = (async () => {
|
||||
while (!this.stopped || this.outcomes.length > 0) {
|
||||
await sleep(FLUSH_MS)
|
||||
@@ -67,7 +85,26 @@ export class Prober {
|
||||
this.report()
|
||||
}
|
||||
})()
|
||||
await Promise.all(Array.from({ length: connections }, () => this.lane()))
|
||||
// Lanes as the share allows: the coordinator's history comes first (app/services/nntp_shares.ts).
|
||||
while (!this.stopped) {
|
||||
try {
|
||||
const was = this.share.granted
|
||||
const granted = await this.share.sync(this.dry ? 0 : this.share.max, this.running)
|
||||
if (granted !== was) logger.info(`Probing: ${granted} connections (was ${was}).`)
|
||||
while (this.running < granted) {
|
||||
this.running++
|
||||
const lane: Promise<void> = this.lane()
|
||||
.catch((error) => logger.error(`Probing: a lane failed: ${error.message}`))
|
||||
.finally(() => this.lanes.delete(lane))
|
||||
this.lanes.add(lane)
|
||||
}
|
||||
} catch (error: any) {
|
||||
logger.error(`Probing: NNTP share failed: ${error.message}`)
|
||||
}
|
||||
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !this.stopped; waited += 1000) await sleep(1000)
|
||||
}
|
||||
await Promise.all(this.lanes)
|
||||
await this.share.leave().catch(() => {})
|
||||
// What was claimed and not probed goes back.
|
||||
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
|
||||
this.stopped = true
|
||||
@@ -78,10 +115,16 @@ export class Prober {
|
||||
this.stopped = true
|
||||
}
|
||||
|
||||
/** One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure. */
|
||||
/**
|
||||
* One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure; done when
|
||||
* there are more lanes than the share (its connection closed, the coordinator takes it on its next sync).
|
||||
*/
|
||||
private async lane() {
|
||||
let client: XoverClient | null = null
|
||||
let usedAt = Date.now()
|
||||
try {
|
||||
while (!this.stopped) {
|
||||
if (this.running > this.share.granted) break
|
||||
const batch: Candidate[] = []
|
||||
const waited = Date.now()
|
||||
while (batch.length < coordinatorConfig.probe.pipeline) {
|
||||
@@ -90,7 +133,14 @@ export class Prober {
|
||||
batch.push(candidate)
|
||||
}
|
||||
this.stats.waitMs += Date.now() - waited
|
||||
this.dry = batch.length === 0
|
||||
if (batch.length === 0) {
|
||||
// Nothing to probe for a while: the connection closes (nntpIdleCloseMs), and opens again with work.
|
||||
const idleMs = coordinatorConfig.nntpIdleCloseMs
|
||||
if (client && idleMs > 0 && Date.now() - usedAt >= idleMs) {
|
||||
client.close()
|
||||
client = null
|
||||
}
|
||||
this.stats.idleMs += 30_000
|
||||
await sleep(30_000)
|
||||
continue
|
||||
@@ -127,6 +177,7 @@ export class Prober {
|
||||
}
|
||||
}
|
||||
this.stats.inspectMs += Date.now() - inspected
|
||||
usedAt = Date.now()
|
||||
} catch (error: any) {
|
||||
logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
|
||||
for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
|
||||
@@ -135,11 +186,19 @@ export class Prober {
|
||||
await sleep(5000)
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
client?.close()
|
||||
this.running--
|
||||
}
|
||||
}
|
||||
|
||||
/** Candidates claimed at a time: a few rounds of every lane's pipeline (a claim takes seconds; 400 lasted ~2 s at 37 lanes). */
|
||||
private get batch() {
|
||||
return Math.max(coordinatorConfig.probe.batch, this.running * coordinatorConfig.probe.pipeline * 4)
|
||||
}
|
||||
|
||||
private async next(): Promise<Candidate | undefined> {
|
||||
if (this.queue.length < coordinatorConfig.probe.batch / 2) {
|
||||
if (this.queue.length < this.batch / 2) {
|
||||
this.refilling ??= this.claim().finally(() => (this.refilling = null))
|
||||
if (this.queue.length === 0) await this.refilling
|
||||
}
|
||||
@@ -162,13 +221,24 @@ export class Prober {
|
||||
}
|
||||
|
||||
private async claimTiers() {
|
||||
// The queue empties, the lanes go dry and the share shrinks to its minimum until there's room again.
|
||||
if (await this.space.short()) return
|
||||
for (const { kind, unreadable } of TIERS) {
|
||||
const want = coordinatorConfig.probe.batch - this.queue.length
|
||||
const want = this.batch - this.queue.length
|
||||
if (want <= 0) return
|
||||
// The starting points chosen here: in the query Postgres drew random() once for all slices.
|
||||
// Over the claimable ones of this tier only: the rest of the queue isn't settled yet.
|
||||
// Per value of unreadable, each an end of the index: over `unreadable >= false` Postgres walked every waiting
|
||||
// entry of the kind, 2.5 minutes a claim at 28M archive candidates (LOG.md 2026-10-18 (74)).
|
||||
const values = unreadable ? [true] : [false, true]
|
||||
const ends = values
|
||||
.map(() => `SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable = ?`)
|
||||
.join(' UNION ALL ')
|
||||
const [{ lo, hi }] = (
|
||||
await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable >= ?`, [kind, unreadable])
|
||||
await db.rawQuery(
|
||||
`SELECT min(lo) AS lo, max(hi) AS hi FROM (${ends}) e`,
|
||||
values.flatMap((v) => [kind, v])
|
||||
)
|
||||
).rows
|
||||
if (lo === null) continue
|
||||
const starts = Array.from({ length: SLICES }, () => Number(lo) + Math.floor(Math.random() * (Number(hi) - Number(lo) + 1)))
|
||||
@@ -209,16 +279,18 @@ export class Prober {
|
||||
await db.transaction(async (trx) => {
|
||||
await trx.rawQuery(
|
||||
`UPDATE probe_candidates pc SET status = v.status FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, status text) WHERE pc.id = v.id`,
|
||||
[JSON.stringify(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
|
||||
[jsonb(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
|
||||
)
|
||||
// Final answers read from the news server, kept by message-id for the next catalog (a reset).
|
||||
const fresh = outcomes.filter((o) => !o.cached && o.status !== 'waiting')
|
||||
// One per message-id: the same article is a candidate of more than one release, and ON CONFLICT can't take a
|
||||
// row twice in one statement.
|
||||
const fresh = [...new Map(outcomes.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
|
||||
if (fresh.length > 0) {
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO probe_cache (message_id, status, result, version)
|
||||
SELECT v.message_id, v.status, v.result, ? FROM jsonb_to_recordset(?::jsonb) AS v(message_id text, status text, result jsonb)
|
||||
ON CONFLICT (message_id) DO UPDATE SET status = excluded.status, result = excluded.result, version = excluded.version, created_at = now()`,
|
||||
[PROBE_VERSION, JSON.stringify(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
|
||||
[PROBE_VERSION, jsonb(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
|
||||
)
|
||||
}
|
||||
const done = outcomes.filter((o) => o.inspection)
|
||||
@@ -228,7 +300,7 @@ export class Prober {
|
||||
SELECT v.release_id, v.candidate_id, v.kind, v.result FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, candidate_id bigint, kind text, result jsonb)
|
||||
ON CONFLICT (release_id, candidate_id) DO UPDATE SET result = excluded.result, created_at = now()`,
|
||||
// A par2 set's file list goes to par2_files, not into every inspection.
|
||||
[JSON.stringify(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
|
||||
[jsonb(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
|
||||
)
|
||||
await this.storeForMerge(trx, done)
|
||||
// Each touched release from all its inspections (earlier rounds may have stored others).
|
||||
@@ -252,7 +324,7 @@ export class Prober {
|
||||
await trx.rawQuery(
|
||||
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
|
||||
FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, passworded boolean, inner_name text) WHERE r.id = v.id`,
|
||||
[JSON.stringify(updates)]
|
||||
[jsonb(updates)]
|
||||
)
|
||||
})
|
||||
}
|
||||
@@ -265,7 +337,7 @@ export class Prober {
|
||||
`INSERT INTO file_hashes (candidate_id, release_id, hash16k, size)
|
||||
SELECT v.candidate_id, v.release_id, v.hash16k, v.size FROM jsonb_to_recordset(?::jsonb) AS v(candidate_id bigint, release_id bigint, hash16k text, size bigint)
|
||||
ON CONFLICT (candidate_id) DO NOTHING`,
|
||||
[JSON.stringify(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
|
||||
[jsonb(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
|
||||
)
|
||||
}
|
||||
for (const o of done) {
|
||||
@@ -275,7 +347,7 @@ export class Prober {
|
||||
`INSERT INTO par2_files (set_id, file_id, hash16k, length, name)
|
||||
SELECT ?, v.file_id, v.hash16k, v.length, v.name FROM jsonb_to_recordset(?::jsonb) AS v(file_id text, hash16k text, length bigint, name text)
|
||||
ON CONFLICT DO NOTHING`,
|
||||
[set.setId, JSON.stringify(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
|
||||
[set.setId, jsonb(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
|
||||
)
|
||||
await trx.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files) VALUES (?, ?, ?) ON CONFLICT DO NOTHING`, [set.setId, o.candidate.releaseId, set.files.length])
|
||||
}
|
||||
|
||||
+179
-49
@@ -1,10 +1,18 @@
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import logger from '@adonisjs/core/services/logger'
|
||||
|
||||
/** Raised when a rule changes: verdicts of an older version are judged again. */
|
||||
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
|
||||
export const JUDGE_VERSION = 1
|
||||
/** Between rounds: each rule is a scan of its table. */
|
||||
const ROUND_MS = 60 * 60 * 1000
|
||||
/** Between rounds of the changed releases (judge_dirty, noted by triggers; migration 4900). */
|
||||
const ROUND_MS = 2 * 60 * 1000
|
||||
/**
|
||||
* A full round (every release, each rule a scan of its table: 35–38 min) this often: for what no trigger notes. Time
|
||||
* (unusable a day after settling, a release final once the ranges beside it are read) and fewer releases with a file
|
||||
* after a merge (repeated).
|
||||
*/
|
||||
const FULL_MS = 24 * 60 * 60 * 1000
|
||||
/** Changed releases judged at a time. */
|
||||
const CHUNK = 50_000
|
||||
/** A file's first 16 KiB and size in this many releases is one file posted again and again (test files, spam). */
|
||||
const REPEATED_RELEASES = 20
|
||||
|
||||
@@ -24,10 +32,15 @@ export interface Rule {
|
||||
name: string
|
||||
/** Why, in a few words (the log, the web interface). */
|
||||
says: string
|
||||
/** SELECT of the release ids the rule judges. */
|
||||
sql: string
|
||||
/** SELECT of the release ids the rule judges; scoped, only those in judge_scope. */
|
||||
sql: (scoped: boolean) => string
|
||||
}
|
||||
|
||||
/** A join to the releases in scope (an incremental round), on the given release id column. */
|
||||
const only = (scoped: boolean, column: string) => (scoped ? `JOIN judge_scope s ON s.release_id = ${column}` : '')
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* The junk judge's rules, in order: a release takes the first that matches (REBUILD.md, junk). Each runs on what
|
||||
* the header pass, the prober, the merges and the spots know by now.
|
||||
@@ -36,26 +49,33 @@ export const RULES: Rule[] = [
|
||||
{
|
||||
name: 'dead',
|
||||
says: 'every probed first article is gone from the server (taken down or expired)',
|
||||
sql: `SELECT release_id FROM probe_candidates WHERE status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(status = 'missing')`,
|
||||
sql: (scoped) =>
|
||||
`SELECT c.release_id FROM probe_candidates c ${only(scoped, 'c.release_id')}
|
||||
WHERE c.status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(c.status = 'missing')`,
|
||||
},
|
||||
{
|
||||
name: 'fake',
|
||||
says: 'named as a film, episode or music, but the content is a program',
|
||||
// An archive whose entries are all executables, under a media name (its own or its entries'); or a file that is
|
||||
// an executable by name or by its first bytes, under a media name.
|
||||
sql: `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
|
||||
sql: (scoped) => `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
|
||||
WHERE i.kind = 'archive' AND jsonb_array_length(i.result->'names') > 0
|
||||
AND NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(i.result->'names') n WHERE n !~* '${EXECUTABLE}')
|
||||
AND (r.name ~* '${MEDIA}' OR (i.result->>'names') ~* '${MEDIA}')
|
||||
UNION
|
||||
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
|
||||
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
|
||||
WHERE i.kind = 'lone' AND ((i.result->>'yencName') ~* '${EXECUTABLE}' OR i.result->>'type' = 'exe')
|
||||
AND (r.name ~* '${MEDIA}' OR (i.result->>'yencName') ~* '${MEDIA}')`,
|
||||
},
|
||||
{
|
||||
name: 'repeated',
|
||||
says: `the same file content in ${REPEATED_RELEASES}+ releases (test files, spam)`,
|
||||
sql: `SELECT DISTINCT h.release_id FROM file_hashes h
|
||||
// Scoped: by the files of the releases in scope that are in REPEATED_RELEASES+ releases, counted once a chunk
|
||||
// (judge_tipped: ReleaseJudge.changed, which first adds the other releases of a file that just tipped over).
|
||||
sql: (scoped) =>
|
||||
scoped
|
||||
? `SELECT DISTINCT h.release_id FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id JOIN judge_tipped USING (hash16k, size)`
|
||||
: `SELECT DISTINCT h.release_id FROM file_hashes h
|
||||
JOIN (SELECT hash16k, size FROM file_hashes GROUP BY 1, 2 HAVING count(DISTINCT release_id) >= ${REPEATED_RELEASES}) x USING (hash16k, size)`,
|
||||
},
|
||||
{
|
||||
@@ -64,19 +84,26 @@ export const RULES: Rule[] = [
|
||||
// "@_CelebMovie_@", "NudeCelebs@CelebMovie.com", "@_CelebWow_@", "@_CelebFX_@", "@_iLuvCelebs_@", "@nude-celebs", …:
|
||||
// a celebrity's scraped pictures as a folder under a typed subject, the same files again and again under other
|
||||
// titles (LOG.md 2026-10-16 (64)). Not Celeborn, CelebTA or other posters with "celeb" in their name.
|
||||
sql: `SELECT r.id FROM releases r
|
||||
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
|
||||
// By the posters table's trigram index, each poster's releases then by releases_poster_newest_idx: 7 s, where the
|
||||
// regex on all of releases was 45 s, every round (LOG.md 2026-10-19 (80)).
|
||||
sql: (scoped) =>
|
||||
scoped
|
||||
? `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
|
||||
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`
|
||||
: `SELECT r.id FROM posters p JOIN releases r ON left(r.poster, 200) = p.poster
|
||||
WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
|
||||
},
|
||||
{
|
||||
name: 'locked',
|
||||
says: 'passworded, and no spot, NZB or title gives the password',
|
||||
sql: `SELECT r.id FROM releases r WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
|
||||
// releases_passworded_idx: a scan of all of releases was 30 s a round.
|
||||
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
|
||||
},
|
||||
{
|
||||
name: 'unusable',
|
||||
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
|
||||
// Final (release_final): next to an unread range its par2 or the rest of its files may still come (LOG.md 2026-10-16 (67)).
|
||||
sql: `SELECT r.id FROM releases r
|
||||
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
|
||||
WHERE r.settled_at < now() - interval '1 day' AND NOT r.readable AND r.inner_name IS NULL
|
||||
AND release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete)
|
||||
AND NOT EXISTS (SELECT 1 FROM release_merges m WHERE m.into_id = r.id)
|
||||
@@ -86,15 +113,18 @@ export const RULES: Rule[] = [
|
||||
]
|
||||
|
||||
export interface JudgeRound {
|
||||
/** Per rule: releases it judges now, verdicts added, verdicts taken back. */
|
||||
/** Releases judged: every one (a full round), or the changed ones and those a hash of theirs tipped over. */
|
||||
releases?: number
|
||||
/** Per rule: releases it judges now (in scope), verdicts it gained, verdicts it lost. */
|
||||
rules: Record<string, { matched: number; added: number; removed: number }>
|
||||
}
|
||||
|
||||
/**
|
||||
* The junk judge (REBUILD.md, junk; voyager 1's second filter, D42): after the header pass's rules (garbage
|
||||
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Each
|
||||
* round recomputes every rule: a release a rule matches gets a verdict (the first rule in order, over a later one's), one it no longer
|
||||
* matches loses its verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
|
||||
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Every few
|
||||
* minutes the releases whose facts changed (judge_dirty) are judged by every rule; a full round judges every release
|
||||
* daily. A release gets the verdict of the first rule in order that matches it; one that no longer matches loses its
|
||||
* verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
|
||||
*/
|
||||
export class ReleaseJudge {
|
||||
private stopped = false
|
||||
@@ -107,9 +137,13 @@ export class ReleaseJudge {
|
||||
while (!this.stopped) {
|
||||
const started = Date.now()
|
||||
try {
|
||||
const round = await this.round()
|
||||
const parts = Object.entries(round.rules).map(([n, r]) => `${n} ${r.matched} (+${r.added} −${r.removed})`)
|
||||
logger.info(`Judged: ${parts.join(', ')}.`)
|
||||
if (await this.fullDue()) {
|
||||
const round = await this.full()
|
||||
if (!this.stopped) logger.info(`Judged all releases: ${describe(round)}.`)
|
||||
} else {
|
||||
const round = await this.changed()
|
||||
if (round.releases! > 0) logger.info(`Judged ${round.releases!.toLocaleString('en')} changed releases: ${describe(round)}.`)
|
||||
}
|
||||
} catch (error: any) {
|
||||
logger.error(`Judge round failed: ${error.message}`)
|
||||
}
|
||||
@@ -117,43 +151,139 @@ export class ReleaseJudge {
|
||||
}
|
||||
}
|
||||
|
||||
/** One round over every rule; with dry, only counts. */
|
||||
async round(dry = false): Promise<JudgeRound> {
|
||||
/** A full round is due: a day after the last, or the last was by another rule set. */
|
||||
async fullDue(): Promise<boolean> {
|
||||
const state = await db.from('judge_state').first()
|
||||
return !state || state.version !== JUDGE_VERSION || Date.now() - new Date(state.full_at).getTime() >= FULL_MS
|
||||
}
|
||||
|
||||
/**
|
||||
* Every release by every rule, the matches gathered in judge_full (each rule its own statement: no transaction of
|
||||
* half an hour), then applied at once. With dry, only counts. Stopped halfway, nothing is applied.
|
||||
*/
|
||||
async full(dry = false): Promise<JudgeRound> {
|
||||
const started = new Date()
|
||||
const round: JudgeRound = { rules: {} }
|
||||
// Verdicts of an older rule set are judged again from scratch.
|
||||
if (!dry) await db.from('release_verdicts').where('rule_version', '<>', JUDGE_VERSION).delete()
|
||||
const judged = new Set<string>()
|
||||
await db.rawQuery(`TRUNCATE judge_full`)
|
||||
for (const rule of RULES) {
|
||||
if (this.stopped) break
|
||||
const { rows } = await db.rawQuery(rule.sql)
|
||||
const ids = rows.map((r: Record<string, unknown>) => String(Object.values(r)[0])).filter((id: string) => !judged.has(id))
|
||||
for (const id of ids) judged.add(id)
|
||||
const stats = { matched: ids.length, added: 0, removed: 0 }
|
||||
if (!dry) {
|
||||
await db.transaction(async (trx) => {
|
||||
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
|
||||
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
|
||||
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
|
||||
for (let i = 0; i < ids.length; i += 50_000) {
|
||||
await trx.rawQuery(`INSERT INTO judged SELECT unnest(?::bigint[]) ON CONFLICT DO NOTHING`, [ids.slice(i, i + 50_000)])
|
||||
}
|
||||
const removed = await trx.rawQuery(
|
||||
`DELETE FROM release_verdicts v WHERE v.rule = ? AND NOT EXISTS (SELECT 1 FROM judged j WHERE j.release_id = v.release_id)`,
|
||||
if (this.stopped) return round
|
||||
const { rowCount } = await db.rawQuery(
|
||||
`INSERT INTO judge_full (release_id, rule) SELECT id, ? FROM (${rule.sql(false)}) AS x(id) ON CONFLICT DO NOTHING`,
|
||||
[rule.name]
|
||||
)
|
||||
const added = await trx.rawQuery(
|
||||
`INSERT INTO release_verdicts (release_id, rule, rule_version)
|
||||
SELECT j.release_id, ?, ? FROM judged j JOIN releases r ON r.id = j.release_id
|
||||
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
|
||||
WHERE release_verdicts.rule <> excluded.rule`,
|
||||
[rule.name, JUDGE_VERSION]
|
||||
)
|
||||
stats.removed = removed.rowCount ?? 0
|
||||
stats.added = added.rowCount ?? 0
|
||||
})
|
||||
round.rules[rule.name] = { matched: rowCount ?? 0, added: 0, removed: 0 }
|
||||
}
|
||||
round.rules[rule.name] = stats
|
||||
if (dry || this.stopped) return round
|
||||
await db.rawQuery(`ANALYZE judge_full`)
|
||||
await db.transaction((trx) => this.apply(trx, 'judge_full', false, round))
|
||||
await db.rawQuery(
|
||||
`INSERT INTO judge_state (id, full_at, version) VALUES (1, ?, ?) ON CONFLICT (id) DO UPDATE SET full_at = excluded.full_at, version = excluded.version`,
|
||||
[started, JUDGE_VERSION]
|
||||
)
|
||||
return round
|
||||
}
|
||||
|
||||
/**
|
||||
* The releases noted in judge_dirty since, CHUNK at a time, each chunk in a transaction (a failed one stays noted).
|
||||
* A changed release can bring a file to REPEATED_RELEASES releases: the others with it are judged too (all but those
|
||||
* already repeated, or dead or fake, which come first). Only when it's the chunk's releases that bring it there: a
|
||||
* file in that many without them had its releases judged before.
|
||||
*/
|
||||
async changed(): Promise<JudgeRound> {
|
||||
const round: JudgeRound = { releases: 0, rules: {} }
|
||||
for (const rule of RULES) round.rules[rule.name] = { matched: 0, added: 0, removed: 0 }
|
||||
let more = true
|
||||
while (more && !this.stopped) {
|
||||
await db.transaction(async (trx) => {
|
||||
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
|
||||
await trx.rawQuery(`DROP TABLE IF EXISTS judge_scope`)
|
||||
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
|
||||
await trx.rawQuery(`CREATE TEMP TABLE judge_scope (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
|
||||
const { rows } = await trx.rawQuery(
|
||||
`WITH taken AS (DELETE FROM judge_dirty WHERE ctid = ANY(ARRAY(SELECT ctid FROM judge_dirty LIMIT ${CHUNK})) RETURNING release_id),
|
||||
scoped AS (INSERT INTO judge_scope SELECT DISTINCT release_id FROM taken ON CONFLICT DO NOTHING RETURNING 1)
|
||||
SELECT (SELECT count(*) FROM taken) AS taken`
|
||||
)
|
||||
const taken = Number(rows[0].taken)
|
||||
more = taken === CHUNK
|
||||
if (taken === 0) return
|
||||
// The chunk's files in REPEATED_RELEASES+ releases, from file_hashes_hash_size_release_idx alone (migration 5000).
|
||||
await trx.rawQuery(`DROP TABLE IF EXISTS judge_tipped`)
|
||||
await trx.rawQuery(
|
||||
`CREATE TEMP TABLE judge_tipped (hash16k char(32), size bigint, just boolean NOT NULL, PRIMARY KEY (hash16k, size)) ON COMMIT DROP`
|
||||
)
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO judge_tipped
|
||||
SELECT x.hash16k, x.size, c.outside < ${REPEATED_RELEASES}
|
||||
FROM (SELECT DISTINCT h.hash16k, h.size FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id) x
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT count(DISTINCT f.release_id) AS total,
|
||||
count(DISTINCT f.release_id) FILTER (WHERE NOT EXISTS (SELECT 1 FROM judge_scope s WHERE s.release_id = f.release_id)) AS outside
|
||||
FROM file_hashes f WHERE f.hash16k = x.hash16k AND f.size = x.size) c
|
||||
WHERE c.total >= ${REPEATED_RELEASES}`
|
||||
)
|
||||
await trx.rawQuery(
|
||||
`INSERT INTO judge_scope SELECT DISTINCT f.release_id FROM file_hashes f JOIN judge_tipped t USING (hash16k, size)
|
||||
WHERE t.just AND NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = f.release_id AND v.rule IN ('dead', 'fake', 'repeated'))
|
||||
ON CONFLICT DO NOTHING`
|
||||
)
|
||||
await trx.rawQuery(`ANALYZE judge_scope`)
|
||||
const { rows: scope } = await trx.rawQuery(`SELECT count(*) AS n FROM judge_scope`)
|
||||
round.releases! += Number(scope[0].n)
|
||||
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL) ON COMMIT DROP`)
|
||||
for (const rule of RULES) {
|
||||
const { rowCount } = await trx.rawQuery(
|
||||
`INSERT INTO judged (release_id, rule) SELECT id, ? FROM (${rule.sql(true)}) AS x(id) ON CONFLICT DO NOTHING`,
|
||||
[rule.name]
|
||||
)
|
||||
round.rules[rule.name].matched += rowCount ?? 0
|
||||
}
|
||||
await trx.rawQuery(`ANALYZE judged`)
|
||||
await this.apply(trx, 'judged', true, round)
|
||||
})
|
||||
}
|
||||
return round
|
||||
}
|
||||
|
||||
/**
|
||||
* The matches (judged: release_id, rule; the first rule's each) into release_verdicts: verdicts no rule gives any
|
||||
* more removed (scoped: of the releases in judge_scope only), changed ones written. Only the changes go through the
|
||||
* insert: each match through its conflict check took 60–150 s a rule (LOG.md 2026-10-19 (80)).
|
||||
*/
|
||||
private async apply(trx: any, judged: string, scoped: boolean, round: JudgeRound) {
|
||||
const tally = (rows: { rule: string; n: string }[], field: 'added' | 'removed') => {
|
||||
for (const { rule, n } of rows) {
|
||||
round.rules[rule] ??= { matched: 0, added: 0, removed: 0 }
|
||||
round.rules[rule][field] += Number(n)
|
||||
}
|
||||
}
|
||||
const { rows: removed } = await trx.rawQuery(
|
||||
`WITH gone AS (DELETE FROM release_verdicts v
|
||||
WHERE ${scoped ? 'v.release_id IN (SELECT release_id FROM judge_scope) AND' : ''}
|
||||
NOT EXISTS (SELECT 1 FROM ${judged} j WHERE j.release_id = v.release_id)
|
||||
RETURNING v.rule)
|
||||
SELECT rule, count(*) AS n FROM gone GROUP BY 1`
|
||||
)
|
||||
tally(removed, 'removed')
|
||||
const { rows: added } = await trx.rawQuery(
|
||||
`WITH written AS (INSERT INTO release_verdicts (release_id, rule, rule_version)
|
||||
SELECT j.release_id, j.rule, ? FROM ${judged} j LEFT JOIN release_verdicts v ON v.release_id = j.release_id
|
||||
WHERE v.rule IS DISTINCT FROM j.rule AND EXISTS (SELECT 1 FROM releases r WHERE r.id = j.release_id)
|
||||
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
|
||||
WHERE release_verdicts.rule <> excluded.rule
|
||||
RETURNING rule)
|
||||
SELECT rule, count(*) AS n FROM written GROUP BY 1`,
|
||||
[JUDGE_VERSION]
|
||||
)
|
||||
tally(added, 'added')
|
||||
}
|
||||
}
|
||||
|
||||
/** "dead 1,157,018 (+1,136 −712), …" for the log. */
|
||||
function describe(round: JudgeRound): string {
|
||||
return Object.entries(round.rules)
|
||||
.map(([name, r]) => `${name} ${r.matched.toLocaleString('en')} (+${r.added.toLocaleString('en')} −${r.removed.toLocaleString('en')})`)
|
||||
.join(', ')
|
||||
}
|
||||
@@ -28,10 +28,11 @@ class Settler {
|
||||
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
|
||||
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
|
||||
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
|
||||
RETURNING id, readable, files, bytes, poster),
|
||||
RETURNING id, readable, files, bytes, poster, sidecar_only),
|
||||
marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
|
||||
-- Small ones may be an upload's .nfo or .nzb that came apart from it: the merger's sidecar fold looks at them.
|
||||
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
|
||||
-- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
|
||||
-- from it: the merger's sidecar fold looks at them.
|
||||
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
|
||||
ON CONFLICT DO NOTHING)
|
||||
SELECT count(*)::int AS n FROM settled`,
|
||||
[QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
/**
|
||||
* 7z keeps its listing at the end of the last volume, so a split 7z gets a `7z-tail` probe candidate: the last
|
||||
* segment of its highest volume (lib/edge/range_indexer.ts). RAR lists its files in every volume's header, and
|
||||
* its first volume's start (one `archive` candidate) is enough. The live tail reads an upload in pieces while
|
||||
* it's posted, and each piece queues a tail for the last volume it saw so far (TsIgNBewuFbuwWy: 7, for
|
||||
* .002–.008; LOG.md 2026-10-23 (94)); merges bring their members' tails together. Of the given releases'
|
||||
* waiting tails, those below the release's highest volume go. Those of the highest volume stay, all of them:
|
||||
* a piece that saw only part of it queued a segment that isn't its last.
|
||||
*/
|
||||
export async function pruneSevenZipTails(trx: any, releaseIds: (number | string)[]): Promise<number> {
|
||||
if (releaseIds.length === 0) return 0
|
||||
const result = await trx.rawQuery(
|
||||
`DELETE FROM probe_candidates pc USING (
|
||||
SELECT id, status, volume, max(volume) OVER (PARTITION BY release_id) AS highest
|
||||
FROM (SELECT id, release_id, status, substring(file from '\\.7z\\.(\\d{1,4})$')::int AS volume
|
||||
FROM probe_candidates WHERE release_id = ANY(?) AND kind = '7z-tail') t
|
||||
) s
|
||||
WHERE pc.id = s.id AND s.status = 'waiting' AND s.volume < s.highest`,
|
||||
[releaseIds.map(String)]
|
||||
)
|
||||
return result.rowCount ?? 0
|
||||
}
|
||||
@@ -6,7 +6,7 @@ const BATCH = 2000
|
||||
/** An upload's sidecar is posted with it: its release is looked for this far from it. */
|
||||
const WINDOW_MINUTES = 10
|
||||
/** The files an upload carries beside its content: an index, a checksum list, a scene release's own files. */
|
||||
const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
|
||||
export const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
|
||||
|
||||
/** The words of a name worth comparing: 4+ letters or digits. */
|
||||
function words(text: string): Set<string> {
|
||||
@@ -53,19 +53,17 @@ export class SidecarFolder {
|
||||
|
||||
async round(): Promise<FoldRound> {
|
||||
const round: FoldRound = { looked: 0, sidecars: 0, folded: 0 }
|
||||
// A candidate whose release went (merged, deleted) leaves with its batch: a separate scan for those over the whole
|
||||
// queue took 5+ min a round at 13M (LOG.md 2026-10-17 (72)).
|
||||
const { rows: queued } = await db.rawQuery(
|
||||
`SELECT r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
|
||||
FROM fold_candidates f JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
|
||||
`SELECT f.release_id AS queued_id, r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
|
||||
FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
|
||||
[BATCH]
|
||||
)
|
||||
const { rows: stale } = await db.rawQuery(
|
||||
`SELECT f.release_id FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id WHERE r.id IS NULL LIMIT ?`,
|
||||
[BATCH]
|
||||
)
|
||||
const done: number[] = stale.map((r: { release_id: string }) => Number(r.release_id))
|
||||
const done: number[] = queued.map((r: { queued_id: string }) => Number(r.queued_id))
|
||||
for (const c of queued) {
|
||||
if (c.id === null) continue
|
||||
round.looked++
|
||||
done.push(Number(c.id))
|
||||
const { rows: chunks } = await db.rawQuery(`SELECT payload FROM release_chunks WHERE release_id = ?`, [c.id])
|
||||
const files: ChunkFile[] = chunks.flatMap((x: { payload: Buffer }) => decodeChunk(x.payload))
|
||||
if (files.length === 0 || !files.every((f) => SIDECAR.test(f.name))) continue
|
||||
|
||||
@@ -102,6 +102,12 @@ export class SpotInspector {
|
||||
this.client = undefined
|
||||
}
|
||||
if (!round || round.inspected === 0) {
|
||||
// Nothing to inspect for a while: closed (nntpIdleCloseMs), opened again with work.
|
||||
const idleMs = coordinatorConfig.nntpIdleCloseMs
|
||||
if (idleMs > 0 && IDLE_MS >= idleMs) {
|
||||
this.client?.close()
|
||||
this.client = undefined
|
||||
}
|
||||
for (let waited = 0; waited < IDLE_MS && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
|
||||
}
|
||||
}
|
||||
@@ -349,6 +355,8 @@ export class SpotInspector {
|
||||
const releaseId = Number(row.id)
|
||||
await db.transaction(async (trx) => {
|
||||
await trx.table('release_spots').insert({ release_id: releaseId, spot_id: spotId, files: files.length }).onConflict(['release_id', 'spot_id']).ignore()
|
||||
// Its poster for poster search, as the batch store adds its releases' (migration 4000).
|
||||
if (spot.poster) await trx.rawQuery(`INSERT INTO posters (poster) VALUES (left(?, 200)) ON CONFLICT DO NOTHING`, [spot.poster])
|
||||
if (spot.password) {
|
||||
await trx.table('release_passwords').insert({ release_id: releaseId, password: spot.password, source: spot.password_source ?? 'spot-text', spot_id: spotId }).onConflict(['release_id', 'password']).ignore()
|
||||
}
|
||||
|
||||
@@ -60,6 +60,12 @@ export class Spotter {
|
||||
this.client = undefined
|
||||
}
|
||||
const wait = coordinatorConfig.spots.everySeconds * 1000
|
||||
// Quiet until the next round: closed (nntpIdleCloseMs), opened again then.
|
||||
const idleMs = coordinatorConfig.nntpIdleCloseMs
|
||||
if (idleMs > 0 && wait - (Date.now() - started) >= idleMs) {
|
||||
this.client?.close()
|
||||
this.client = undefined
|
||||
}
|
||||
for (let waited = Date.now() - started; waited < wait && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,8 +14,6 @@ export const signupValidator = vine.create({
|
||||
email: email().unique({ table: 'users', column: 'email' }),
|
||||
password: password(),
|
||||
passwordConfirmation: password().sameAs('password'),
|
||||
// Needed when SIGNUP is `invite` (config/access.ts).
|
||||
invite: vine.string().trim().maxLength(32).optional(),
|
||||
})
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
import { BaseCommand, flags } from '@adonisjs/core/ace'
|
||||
import type { CommandOptions } from '@adonisjs/core/types/ace'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import AccessService from '#services/access_service'
|
||||
import { TIERS } from '#config/access'
|
||||
|
||||
/** Makes invite codes (SIGNUP=invite), optionally with a tier for the new account, e.g. a sold VIP year. */
|
||||
export default class AccessInvite extends BaseCommand {
|
||||
static commandName = 'access:invite'
|
||||
static description = 'Makes invite codes, optionally giving a tier (e.g. --tier=vip --days=365)'
|
||||
static options: CommandOptions = { startApp: true }
|
||||
|
||||
@flags.number({ description: 'How many codes', default: 1 })
|
||||
declare count: number
|
||||
|
||||
@flags.string({ description: `Tier the new account gets (${Object.keys(TIERS).join(', ')})` })
|
||||
declare tier?: string
|
||||
|
||||
@flags.number({ description: 'For how many days the tier holds (without: no end)' })
|
||||
declare days?: number
|
||||
|
||||
@flags.string({ description: 'A note, e.g. who it is for or the payment' })
|
||||
declare note?: string
|
||||
|
||||
async run() {
|
||||
if (this.tier && !TIERS[this.tier]) {
|
||||
this.logger.error(`No such tier: ${this.tier}`)
|
||||
this.exitCode = 1
|
||||
return
|
||||
}
|
||||
const codes = Array.from({ length: this.count }, () => AccessService.newKey())
|
||||
await db.table('invites').multiInsert(
|
||||
codes.map((code) => ({ code, tier: this.tier ?? null, tier_days: this.tier ? (this.days ?? null) : null, note: this.note ?? null }))
|
||||
)
|
||||
for (const code of codes) this.logger.log(code)
|
||||
}
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
|
||||
import type { CommandOptions } from '@adonisjs/core/types/ace'
|
||||
import User from '#models/user'
|
||||
|
||||
/** Suspends a user (Newznab error 101, the catalog 403), e.g. for a shared key; --undo lifts it. */
|
||||
export default class AccessSuspend extends BaseCommand {
|
||||
static commandName = 'access:suspend'
|
||||
static description = 'Suspends a user (or lifts it with --undo)'
|
||||
static options: CommandOptions = { startApp: true }
|
||||
|
||||
@args.string({ description: 'The user’s email' })
|
||||
declare email: string
|
||||
|
||||
@flags.boolean({ description: 'Lift the suspension' })
|
||||
declare undo: boolean
|
||||
|
||||
async run() {
|
||||
const user = await User.findBy('email', this.email)
|
||||
if (!user) {
|
||||
this.logger.error(`No user ${this.email}`)
|
||||
this.exitCode = 1
|
||||
return
|
||||
}
|
||||
user.suspended = !this.undo
|
||||
await user.save()
|
||||
this.logger.info(`${user.email}: ${user.suspended ? 'suspended' : 'active'}`)
|
||||
}
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
|
||||
import type { CommandOptions } from '@adonisjs/core/types/ace'
|
||||
import User from '#models/user'
|
||||
import AccessService from '#services/access_service'
|
||||
|
||||
/** Gives a user a tier (a payment, by hand until one is taken on the site); a running one is extended. */
|
||||
export default class AccessTier extends BaseCommand {
|
||||
static commandName = 'access:tier'
|
||||
static description = "Sets a user's tier, e.g. access:tier someone@example.com vip --days=365"
|
||||
static options: CommandOptions = { startApp: true }
|
||||
|
||||
@args.string({ description: 'The user’s email' })
|
||||
declare email: string
|
||||
|
||||
@args.string({ description: 'Tier (config/access.ts)' })
|
||||
declare tier: string
|
||||
|
||||
@flags.number({ description: 'For how many days (without: no end); the same tier still running is extended' })
|
||||
declare days?: number
|
||||
|
||||
async run() {
|
||||
const user = await User.findBy('email', this.email)
|
||||
if (!user) {
|
||||
this.logger.error(`No user ${this.email}`)
|
||||
this.exitCode = 1
|
||||
return
|
||||
}
|
||||
AccessService.grantTier(user, this.tier, this.days ?? null)
|
||||
await user.save()
|
||||
this.logger.info(`${user.email}: ${user.tier} ${user.tierUntil ? `until ${user.tierUntil.toISO()}` : 'without end'}`)
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,9 @@ import Planner from '#services/planner'
|
||||
import FragmentMerger from '#services/fragment_merger'
|
||||
import Settler from '#services/settler'
|
||||
import RangeBlocks from '#services/range_blocks'
|
||||
import Leases from '#services/leases'
|
||||
import NntpPool from '#services/nntp_pool'
|
||||
import { NntpShare } from '#services/nntp_shares'
|
||||
import coordinatorConfig from '#config/coordinator'
|
||||
|
||||
const TICK_MS = 2000
|
||||
@@ -42,6 +44,7 @@ export default class CoordinatorRun extends BaseCommand {
|
||||
const [index, count] = (this.shard ?? '1/1').split('/').map(Number)
|
||||
if (!(count >= 1 && index >= 1 && index <= count)) throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
|
||||
const prefetcher = new Prefetcher({ index: index - 1, count })
|
||||
const share = new NntpShare(count > 1 ? `coordinator ${index}/${count}` : 'coordinator', coordinatorConfig.nntpShares.coordinatorPriority, coordinatorConfig.nntpMinConnections, coordinatorConfig.nntpConnections)
|
||||
const main = index === 1
|
||||
let stopping = false
|
||||
let stopped: () => void
|
||||
@@ -54,6 +57,21 @@ export default class CoordinatorRun extends BaseCommand {
|
||||
await done
|
||||
})
|
||||
this.logger.info(count > 1 ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).` : 'Coordinator started.')
|
||||
// The pool's size: its share of the account's connections, the prober the rest (app/services/nntp_shares.ts).
|
||||
// Before the first connection: until then it's NNTP_CONNECTIONS.
|
||||
const first = await NntpPool.share(share)
|
||||
this.logger.info(`NNTP share: ${first.granted} connections.`)
|
||||
const sharing = (async () => {
|
||||
while (!stopping) {
|
||||
try {
|
||||
const { was, granted } = await NntpPool.share(share)
|
||||
if (granted !== was) this.logger.info(`NNTP share: ${granted} connections (was ${was}).`)
|
||||
} catch (error: any) {
|
||||
this.logger.error(`NNTP share failed: ${error.message}`)
|
||||
}
|
||||
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !stopping; waited += TICK_MS) await sleep(TICK_MS)
|
||||
}
|
||||
})()
|
||||
// Two loops: the prefetcher's ticks don't wait for merging or planning (a merge round
|
||||
// of up to MERGE_BUDGET_MS left the connections idle once their targets were fetched).
|
||||
const prefetching = (async () => {
|
||||
@@ -70,6 +88,8 @@ export default class CoordinatorRun extends BaseCommand {
|
||||
const blocks = (async () => {
|
||||
while (main && !stopping) {
|
||||
try {
|
||||
const abandoned = await Leases.finishAbandoned()
|
||||
if (abandoned > 0) this.logger.info(`Finished ${abandoned} ranges read to their end whose worker went quiet.`)
|
||||
await RangeBlocks.rebuild()
|
||||
} catch (error: any) {
|
||||
this.logger.error(`Range blocks failed: ${error.message}`)
|
||||
@@ -123,12 +143,13 @@ export default class CoordinatorRun extends BaseCommand {
|
||||
}
|
||||
await sleep(TICK_MS)
|
||||
}
|
||||
await Promise.all([prefetching, blocks])
|
||||
await Promise.all([prefetching, blocks, sharing])
|
||||
// The chunks in flight, for at most STOP_WAIT_MS (a connection the server holds back can take minutes); then closed.
|
||||
const waitUntil = Date.now() + STOP_WAIT_MS
|
||||
while (prefetcher.active > 0 && Date.now() < waitUntil) await sleep(200)
|
||||
if (prefetcher.active > 0) this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
|
||||
await NntpPool.close()
|
||||
await share.leave().catch(() => {})
|
||||
this.logger.info('Coordinator stopped.')
|
||||
stopped!()
|
||||
await this.terminate()
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { BaseCommand } from '@adonisjs/core/ace'
|
||||
import type { CommandOptions } from '@adonisjs/core/types/ace'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import { Par2Pruner } from '#services/par2_pruner'
|
||||
|
||||
/** The par2 prune (Par2Pruner) in one go: a whole pass from the first set, instead of the merger's 2 minutes an hour. */
|
||||
export default class Par2Prune extends BaseCommand {
|
||||
static commandName = 'par2:prune'
|
||||
static description = 'Drops the par2 file lists the merge can no longer use: one whole pass over every set, then exits'
|
||||
static options: CommandOptions = { startApp: true }
|
||||
|
||||
async run() {
|
||||
await db.from('worker_cursors').where('name', 'par2-prune').update({ key: '' })
|
||||
const round = await new Par2Pruner().round()
|
||||
this.logger.info(`Pruned ${round.rows} par2 file rows: ${round.complete} complete sets, ${round.judged} judged, ${round.final} final.`)
|
||||
}
|
||||
}
|
||||
@@ -5,10 +5,10 @@ import { ReleaseJudge, RULES } from '#services/release_judge'
|
||||
/** The junk judge (REBUILD.md, junk): releases judged after their probes; verdicts hide them from search. */
|
||||
export default class ReleasesJudge extends BaseCommand {
|
||||
static commandName = 'releases:judge'
|
||||
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable), hourly'
|
||||
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable): the changed ones every few minutes, all daily'
|
||||
static options: CommandOptions = { startApp: true, staysAlive: true }
|
||||
|
||||
@flags.boolean({ description: 'One round, then exit' })
|
||||
@flags.boolean({ description: 'One full round, then exit' })
|
||||
declare once: boolean
|
||||
|
||||
@flags.boolean({ description: 'With --once: count only, write no verdicts' })
|
||||
@@ -17,7 +17,7 @@ export default class ReleasesJudge extends BaseCommand {
|
||||
async run() {
|
||||
const judge = new ReleaseJudge()
|
||||
if (this.once) {
|
||||
const round = await judge.round(this.dry)
|
||||
const round = await judge.full(this.dry)
|
||||
for (const rule of RULES) {
|
||||
const r = round.rules[rule.name]
|
||||
if (r) this.logger.info(`${rule.name}: ${r.matched} (+${r.added} −${r.removed}) — ${rule.says}`)
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
import env from '#start/env'
|
||||
|
||||
/**
|
||||
* Who may use the catalog and the Newznab API.
|
||||
*
|
||||
* `open` (the default): as before, one key for Newznab (NEWZNAB_API_KEY, or
|
||||
* none) and an open web catalog, no limits. `accounts`: every request names a
|
||||
* user (a Newznab `apikey`, or the web interface's token), whose tier sets what
|
||||
* they may do per UTC day and how far back they see.
|
||||
*/
|
||||
export const ACCESS = env.get('ACCESS', 'open')
|
||||
|
||||
/** Who may sign up: anyone, with an unused invite code, or no one. With accounts, invites unless set. */
|
||||
export const SIGNUP = env.get('SIGNUP', ACCESS === 'accounts' ? 'invite' : 'open')
|
||||
|
||||
export interface TierLimits {
|
||||
/** Searches and release details per day (Newznab t=search & co., the catalog); null: no limit. */
|
||||
apiHits: number | null
|
||||
/** NZBs per day; null: no limit. */
|
||||
grabs: number | null
|
||||
/** How many days back releases are found and fetched; null: everything. */
|
||||
retentionDays: number | null
|
||||
}
|
||||
|
||||
/** A tier that runs out (users.tier_until) falls back to `free`. */
|
||||
export const TIERS: Record<string, TierLimits> = {
|
||||
free: { apiHits: 100, grabs: 10, retentionDays: 365 },
|
||||
vip: { apiHits: 10_000, grabs: 1_000, retentionDays: null },
|
||||
// Staff and friends.
|
||||
unlimited: { apiHits: null, grabs: null, retentionDays: null },
|
||||
}
|
||||
+26
-2
@@ -31,8 +31,21 @@ export default {
|
||||
},
|
||||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||||
/** NNTP connections the coordinator fetches headers with. */
|
||||
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
|
||||
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
|
||||
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
|
||||
/**
|
||||
* The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
|
||||
* all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
|
||||
* and 2 for a hedged request's or a reconnect's new connection while the old one closes).
|
||||
*/
|
||||
nntpShares: {
|
||||
total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
|
||||
everyMs: 15_000,
|
||||
/** Headers before probes: history takes what it can use, the prober the rest. */
|
||||
coordinatorPriority: 2,
|
||||
proberPriority: 1,
|
||||
},
|
||||
/**
|
||||
* A reply with no data for this long fails, and its connection is replaced.
|
||||
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
|
||||
@@ -68,13 +81,24 @@ export default {
|
||||
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
|
||||
*/
|
||||
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
|
||||
/**
|
||||
* A connection with nothing to do for this long is closed, and opened again
|
||||
* when there's work. 0: kept. Eweka drops connections that stay quiet, and
|
||||
* the next command on one failed ("Spot inspection failed: NNTP connection
|
||||
* closed"); idle ones also take the account's connections from downloads.
|
||||
*/
|
||||
nntpIdleCloseMs: env.get('NNTP_IDLE_CLOSE_SECONDS', 60) * 1000,
|
||||
probe: {
|
||||
/**
|
||||
* NNTP connections the prober reads articles with, besides the
|
||||
* At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
|
||||
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
|
||||
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
|
||||
*/
|
||||
connections: env.get('PROBE_CONNECTIONS', 6),
|
||||
/** Of them, always (its share's minimum; nntpShares). */
|
||||
minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
|
||||
/** No claims while the filesystem under `/` (the NVMe, in the container) has less free (app/services/disk_space.ts). 0: off. */
|
||||
minFreeGb: env.get('PROBE_MIN_FREE_GB', 0),
|
||||
/** Candidates claimed at a time. */
|
||||
batch: 400,
|
||||
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Accounts with limits (config/access.ts): a Newznab key and a tier per user,
|
||||
* invites, and what each user used per day.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.alterTable('users', (table) => {
|
||||
// Existing users get a key of their own (a volatile default is evaluated per row).
|
||||
table.string('api_key', 32).notNullable().unique().defaultTo(this.raw(`md5(random()::text || clock_timestamp()::text)`))
|
||||
table.string('tier', 16).notNullable().defaultTo('free')
|
||||
// Null: the tier doesn't run out.
|
||||
table.timestamp('tier_until', { useTz: true }).nullable()
|
||||
table.boolean('suspended').notNullable().defaultTo(false)
|
||||
})
|
||||
|
||||
this.schema.createTable('invites', (table) => {
|
||||
table.string('code', 32).primary()
|
||||
// What the invite gives (e.g. a sold VIP invite); null: the free tier.
|
||||
table.string('tier', 16).nullable()
|
||||
table.integer('tier_days').nullable()
|
||||
table.string('note').nullable()
|
||||
table.integer('used_by').nullable().references('id').inTable('users').onDelete('SET NULL')
|
||||
table.timestamp('used_at', { useTz: true }).nullable()
|
||||
table.timestamp('created_at', { useTz: true }).notNullable().defaultTo(this.now())
|
||||
})
|
||||
|
||||
this.schema.createTable('usage', (table) => {
|
||||
table.integer('user_id').notNullable().references('id').inTable('users').onDelete('CASCADE')
|
||||
// A UTC day.
|
||||
table.date('day').notNullable()
|
||||
table.integer('api_hits').notNullable().defaultTo(0)
|
||||
table.integer('grabs').notNullable().defaultTo(0)
|
||||
table.primary(['user_id', 'day'])
|
||||
})
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.dropTable('usage')
|
||||
this.schema.dropTable('invites')
|
||||
this.schema.alterTable('users', (table) => {
|
||||
table.dropColumn('api_key')
|
||||
table.dropColumn('tier')
|
||||
table.dropColumn('tier_until')
|
||||
table.dropColumn('suspended')
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* releases.sidecar_only: set by the batch store when it inserts a small release (at most 3 files, under 1 MB) whose
|
||||
* files are all sidecars (.nfo, .sfv, .nzb, …). The settler queues only those for the sidecar fold: it queued every
|
||||
* small release, 13.5M (single images mostly), and the fold's queue scans held the merger back (LOG.md 2026-10-17 (72)).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS sidecar_only boolean`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`ALTER TABLE releases DROP COLUMN IF EXISTS sidecar_only`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* worker_cursors.key: a tie-breaker after `at`. The par2 pruner's cursor was a timestamp alone, read back to the
|
||||
* millisecond: 20,000 sets of one prober flush share created_at 2026-10-15 14:38:39.883984, and every round took the
|
||||
* same batch again (LOG.md 2026-10-17 (73)). Now (at to the microsecond, set_id).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`ALTER TABLE worker_cursors ADD COLUMN IF NOT EXISTS key text`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`ALTER TABLE worker_cursors DROP COLUMN IF EXISTS key`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* nntp_shares: the account's connections shared by the processes that read articles (app/services/nntp_shares.ts).
|
||||
* Each process's row says what it could use and has open, and what it may open; a row without a heartbeat for a
|
||||
* minute is a process gone (its connections closed with it). The split was fixed per process: with history paused
|
||||
* the coordinator's 38 stood idle while the prober's 8 were the limit (LOG.md 2026-10-18 (75)).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`
|
||||
CREATE TABLE IF NOT EXISTS nntp_shares (
|
||||
name text PRIMARY KEY,
|
||||
priority integer NOT NULL,
|
||||
min integer NOT NULL,
|
||||
want integer NOT NULL,
|
||||
open integer NOT NULL,
|
||||
granted integer NOT NULL,
|
||||
heartbeat_at timestamptz NOT NULL DEFAULT now()
|
||||
)`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP TABLE IF EXISTS nntp_shares`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* probe_cache's primary key in the catalog's tablespace (default_tablespace: nvme on homelab); the table stays on the
|
||||
* HDD. Each claim looks its candidates up by message_id (`message_id = ANY(…)`): with the index on the HDD that took
|
||||
* 1.27 s a claim, 48 s of each minute, the prober's lanes waiting on claims (LOG.md 2026-10-18 (76)). ~97% aren't in it,
|
||||
* so only the index is read. Homelab moved it by hand (prober stopped); a no-op where it's there already, or without a
|
||||
* default tablespace (tests).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.defer(async (db) => {
|
||||
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
|
||||
const target = setting[0].target
|
||||
if (!target) return
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||
WHERE c.relname = 'probe_cache_pkey' AND coalesce(t.spcname, 'pg_default') <> ?`,
|
||||
[target]
|
||||
)
|
||||
if (rows.length > 0) await db.rawQuery(`ALTER INDEX probe_cache_pkey SET TABLESPACE ${target}`)
|
||||
})
|
||||
}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Was: release_inspections' rows on the HDD (pg_default), for the NVMe's room during the probe backlog (LOG.md
|
||||
* 2026-10-18 (79)). The judge's fake rule reads all of it every round: on the HDD that took 17–18 min, was ~3
|
||||
* (LOG.md 2026-10-19 (81)). Migration 4800 moves it back; nothing here now.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The junk judge's locked rule by an index of the passworded releases (1.2M of 74M), not a scan of all of releases
|
||||
* every round (30 s; LOG.md 2026-10-19 (80)). And the posters of the spot inspector's own releases (source `spot`),
|
||||
* which it didn't add to posters: poster search, and the judge's scraped rule by posters, missed them.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
static disableTransactions = true
|
||||
|
||||
async up() {
|
||||
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS releases_passworded_idx ON releases (id) WHERE passworded`)
|
||||
this.schema.raw(
|
||||
`INSERT INTO posters (poster) SELECT DISTINCT left(s.poster, 200) FROM spots s
|
||||
JOIN releases r ON r.id = s.release_id AND r.source = 'spot' WHERE s.poster IS NOT NULL ON CONFLICT DO NOTHING`
|
||||
)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS releases_passworded_idx`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* release_inspections back in the catalog's tablespace (default_tablespace: nvme on homelab), where migration 4600 had
|
||||
* moved it to the HDD: the judge's fake rule reads all of it every round, 17–18 min on the HDD, ~3 on the NVMe (LOG.md
|
||||
* 2026-10-19 (81)). Homelab moved it by hand (prober, merger and judge stopped); a no-op where it's there already, or
|
||||
* without a default tablespace (tests).
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.defer(async (db) => {
|
||||
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
|
||||
const target = setting[0].target
|
||||
if (!target) return
|
||||
const { rows } = await db.rawQuery(
|
||||
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
|
||||
WHERE c.relname = 'release_inspections' AND c.relkind = 'r' AND coalesce(t.spcname, 'pg_default') <> ?`,
|
||||
[target]
|
||||
)
|
||||
if (rows.length > 0) await db.rawQuery(`ALTER TABLE release_inspections SET TABLESPACE ${target}`)
|
||||
})
|
||||
}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* The junk judge by what changed (LOG.md 2026-10-19 (83)): a full round over every release took 35–38 min of each
|
||||
* hour, longer with every inspection the prober added. Triggers note each release whose facts a rule reads changed
|
||||
* (judge_dirty: its probes' statuses, inspections, file hashes, passwords, spots, par2 sets, merges, its own judged
|
||||
* columns); the judge judges those every few minutes, and every release in a daily full round (judge_state: when the
|
||||
* last one was, by which rule set), its matches gathered in judge_full. judge_dirty is unlogged: what a crash loses
|
||||
* the next full round judges.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_dirty (release_id bigint NOT NULL)`)
|
||||
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_full (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL)`)
|
||||
this.schema.raw(
|
||||
`CREATE TABLE IF NOT EXISTS judge_state (id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1), full_at timestamptz NOT NULL, version integer NOT NULL)`
|
||||
)
|
||||
// The verdicts are an hourly full round's: the next full one a day from now.
|
||||
this.schema.raw(`INSERT INTO judge_state (full_at, version) VALUES (now(), 1) ON CONFLICT DO NOTHING`)
|
||||
|
||||
// Rows of a table with release_id: the releases of the rows inserted, deleted, or updated (both, when moved).
|
||||
this.schema.raw(`
|
||||
CREATE OR REPLACE FUNCTION judge_dirty_by_release_id() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF TG_OP IN ('INSERT', 'UPDATE') THEN
|
||||
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM new_rows WHERE release_id IS NOT NULL;
|
||||
END IF;
|
||||
IF TG_OP IN ('UPDATE', 'DELETE') THEN
|
||||
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM old_rows WHERE release_id IS NOT NULL;
|
||||
END IF;
|
||||
RETURN NULL;
|
||||
END $$`)
|
||||
// probe_candidates: only a status that counts for the dead rule, or a candidate moved by a merge (not the claims'
|
||||
// 'taken', the settler's ready).
|
||||
this.schema.raw(`
|
||||
CREATE OR REPLACE FUNCTION judge_dirty_probes() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
INSERT INTO judge_dirty (release_id)
|
||||
SELECT DISTINCT unnest(ARRAY[n.release_id, o.release_id]) FROM new_rows n JOIN old_rows o ON o.id = n.id
|
||||
WHERE n.release_id IS DISTINCT FROM o.release_id
|
||||
OR (n.status IS DISTINCT FROM o.status AND (n.status IN ('done', 'missing', 'error') OR o.status IN ('done', 'missing', 'error')));
|
||||
RETURN NULL;
|
||||
END $$`)
|
||||
this.schema.raw(`
|
||||
CREATE OR REPLACE FUNCTION judge_dirty_merges() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
INSERT INTO judge_dirty (release_id) SELECT DISTINCT into_id FROM new_rows;
|
||||
RETURN NULL;
|
||||
END $$`)
|
||||
this.schema.raw(`
|
||||
CREATE OR REPLACE FUNCTION judge_dirty_releases() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF TG_OP = 'INSERT' THEN
|
||||
INSERT INTO judge_dirty (release_id) SELECT id FROM new_rows;
|
||||
ELSIF TG_OP = 'DELETE' THEN
|
||||
INSERT INTO judge_dirty (release_id) SELECT id FROM old_rows;
|
||||
ELSE
|
||||
INSERT INTO judge_dirty (release_id) VALUES (NEW.id);
|
||||
END IF;
|
||||
RETURN NULL;
|
||||
END $$`)
|
||||
|
||||
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets']) {
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON ${table}`)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON ${table}`)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON ${table}`)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON ${table} REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
|
||||
)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON ${table} REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
|
||||
)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON ${table} REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
|
||||
)
|
||||
}
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON probe_candidates`)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON probe_candidates`)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON probe_candidates REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_probes()`
|
||||
)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON probe_candidates REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
|
||||
)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON release_merges`)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON release_merges REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_merges()`
|
||||
)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON releases`)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON releases`)
|
||||
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON releases`)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON releases REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
|
||||
)
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON releases REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
|
||||
)
|
||||
// Per row, and only when a column a rule reads changed: the tail and the settler update releases all the time.
|
||||
this.schema.raw(
|
||||
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON releases FOR EACH ROW
|
||||
WHEN (OLD.name IS DISTINCT FROM NEW.name OR OLD.poster IS DISTINCT FROM NEW.poster OR OLD.source IS DISTINCT FROM NEW.source
|
||||
OR OLD.file_total IS DISTINCT FROM NEW.file_total OR OLD.passworded IS DISTINCT FROM NEW.passworded
|
||||
OR OLD.settled_at IS DISTINCT FROM NEW.settled_at OR OLD.readable IS DISTINCT FROM NEW.readable
|
||||
OR OLD.inner_name IS DISTINCT FROM NEW.inner_name OR OLD.group_id IS DISTINCT FROM NEW.group_id
|
||||
OR OLD.first_article IS DISTINCT FROM NEW.first_article OR OLD.last_article IS DISTINCT FROM NEW.last_article
|
||||
OR OLD.posted_at IS DISTINCT FROM NEW.posted_at OR OLD.last_posted_at IS DISTINCT FROM NEW.last_posted_at
|
||||
OR OLD.complete IS DISTINCT FROM NEW.complete)
|
||||
EXECUTE FUNCTION judge_dirty_releases()`
|
||||
)
|
||||
}
|
||||
|
||||
async down() {
|
||||
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets', 'probe_candidates', 'release_merges', 'releases']) {
|
||||
for (const event of ['insert', 'update', 'delete']) this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_${event} ON ${table}`)
|
||||
}
|
||||
this.schema.raw(`DROP FUNCTION IF EXISTS judge_dirty_by_release_id(), judge_dirty_probes(), judge_dirty_merges(), judge_dirty_releases()`)
|
||||
this.schema.raw(`DROP TABLE IF EXISTS judge_dirty, judge_full, judge_state`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* file_hashes by (hash16k, size, release_id), in place of (hash16k): how many releases have a file, from the index
|
||||
* alone. The judge counts that for every file of each changed release (repeated); with the row fetched for size and
|
||||
* release_id per entry, a file posted in 100k releases cost 100k heap reads every round (24–32 s a chunk; LOG.md
|
||||
* 2026-10-19 (84)). The merger and pruner join on (hash16k, size) too.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
static disableTransactions = true
|
||||
|
||||
async up() {
|
||||
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_size_release_idx ON file_hashes (hash16k, size, release_id)`)
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_idx`)
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_idx ON file_hashes (hash16k)`)
|
||||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_size_release_idx`)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
/**
|
||||
* Probe candidates added to a release after it settled (a later range of its upload, a merge into it) stayed not
|
||||
* ready: the settler marks a release's candidates only when it settles it (LOG.md 2026-10-20 (87)). 93,642 waited
|
||||
* since 2026-10-13 (17k lone, 31k par2, 45k archive). The batch store and the merge mark them now; these are made
|
||||
* claimable once.
|
||||
*/
|
||||
export default class extends BaseSchema {
|
||||
async up() {
|
||||
this.schema.raw(`
|
||||
UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
|
||||
WHERE pc.status = 'waiting' AND NOT pc.ready AND r.id = pc.release_id AND r.settled_at IS NOT NULL`)
|
||||
}
|
||||
|
||||
async down() {}
|
||||
}
|
||||
+1
-41
@@ -87,25 +87,6 @@ export class GroupSchema extends BaseModel {
|
||||
declare tail: boolean
|
||||
}
|
||||
|
||||
export class InviteSchema extends BaseModel {
|
||||
static $columns = ['code', 'createdAt', 'note', 'tier', 'tierDays', 'usedAt', 'usedBy'] as const
|
||||
$columns = InviteSchema.$columns
|
||||
@column({ isPrimary: true })
|
||||
declare code: string
|
||||
@column.dateTime({ autoCreate: true })
|
||||
declare createdAt: DateTime
|
||||
@column()
|
||||
declare note: string | null
|
||||
@column()
|
||||
declare tier: string | null
|
||||
@column()
|
||||
declare tierDays: number | null
|
||||
@column.dateTime()
|
||||
declare usedAt: DateTime | null
|
||||
@column()
|
||||
declare usedBy: number | null
|
||||
}
|
||||
|
||||
export class NodeAccessTokenSchema extends BaseModel {
|
||||
static $columns = ['abilities', 'createdAt', 'expiresAt', 'hash', 'id', 'lastUsedAt', 'name', 'tokenableId', 'type', 'updatedAt'] as const
|
||||
$columns = NodeAccessTokenSchema.$columns
|
||||
@@ -418,24 +399,9 @@ export class RuleSetSchema extends BaseModel {
|
||||
declare version: number
|
||||
}
|
||||
|
||||
export class UsageSchema extends BaseModel {
|
||||
static $columns = ['apiHits', 'day', 'grabs', 'userId'] as const
|
||||
$columns = UsageSchema.$columns
|
||||
@column()
|
||||
declare apiHits: number
|
||||
@column.date()
|
||||
declare day: DateTime
|
||||
@column()
|
||||
declare grabs: number
|
||||
@column({ isPrimary: true })
|
||||
declare userId: number
|
||||
}
|
||||
|
||||
export class UserSchema extends BaseModel {
|
||||
static $columns = ['apiKey', 'createdAt', 'email', 'fullName', 'id', 'password', 'suspended', 'tier', 'tierUntil', 'updatedAt'] as const
|
||||
static $columns = ['createdAt', 'email', 'fullName', 'id', 'password', 'updatedAt'] as const
|
||||
$columns = UserSchema.$columns
|
||||
@column()
|
||||
declare apiKey: string
|
||||
@column.dateTime({ autoCreate: true })
|
||||
declare createdAt: DateTime
|
||||
@column()
|
||||
@@ -446,12 +412,6 @@ export class UserSchema extends BaseModel {
|
||||
declare id: number
|
||||
@column({ serializeAs: null })
|
||||
declare password: string
|
||||
@column()
|
||||
declare suspended: boolean
|
||||
@column()
|
||||
declare tier: string
|
||||
@column.dateTime()
|
||||
declare tierUntil: DateTime | null
|
||||
@column.dateTime({ autoCreate: true, autoUpdate: true })
|
||||
declare updatedAt: DateTime | null
|
||||
}
|
||||
+13
-7
@@ -115,11 +115,14 @@ services:
|
||||
<<: *app-env
|
||||
NNTP_CONNECTIONS: ${NNTP_CONNECTIONS:-38}
|
||||
|
||||
# Phase 2's prober: reads the probe queue's articles over PROBE_CONNECTIONS
|
||||
# connections of its own. The account has 50: the coordinator's 38 (+ its
|
||||
# plain one), the prober's 8, the spotter's and the spot inspector's 1 each,
|
||||
# 2 to spare (a hedged request closes its connection and opens another)
|
||||
# (with 10 the spot inspector got 502 Too many connections).
|
||||
# Phase 2's prober: reads the probe queue's articles over connections of its
|
||||
# own. The account has 50 (NNTP_ACCOUNT_CONNECTIONS): 5 reserved (the
|
||||
# coordinator's plain one, the spotter's and the spot inspector's 1 each, 2
|
||||
# to spare: a hedged request closes its connection and opens another; with
|
||||
# 10 the spot inspector got 502 Too many connections), the other 45 shared
|
||||
# by the coordinator (NNTP_MIN_CONNECTIONS–NNTP_CONNECTIONS) and the prober
|
||||
# (PROBE_MIN_CONNECTIONS–PROBE_CONNECTIONS), headers first
|
||||
# (app/services/nntp_shares.ts).
|
||||
prober:
|
||||
<<: *app
|
||||
command: ['node', 'ace', 'probe:run']
|
||||
@@ -127,6 +130,9 @@ services:
|
||||
environment:
|
||||
<<: *app-env
|
||||
PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-8}
|
||||
# No claims under this much free on the NVMe (its overlay root): 2 GB above the 10 GB floor, for what the
|
||||
# tail, the merger and the judge write meanwhile.
|
||||
PROBE_MIN_FREE_GB: ${PROBE_MIN_FREE_GB:-12}
|
||||
|
||||
# The par2 merge: releases whose files one par2 set describes (what the prober read) become one. No NNTP.
|
||||
merger:
|
||||
@@ -134,8 +140,8 @@ services:
|
||||
command: ['node', 'ace', 'releases:merge']
|
||||
volumes: []
|
||||
|
||||
# The junk judge: releases judged hourly by what their probes showed (dead, fake, repeated, locked, unusable);
|
||||
# verdicts hide them from search. No NNTP.
|
||||
# The junk judge: releases judged by what their probes showed (dead, fake, repeated, locked, unusable), the changed
|
||||
# ones every 2 minutes, all daily; verdicts hide them from search. No NNTP.
|
||||
judge:
|
||||
<<: *app
|
||||
command: ['node', 'ace', 'releases:judge']
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
* Newznab API formatting, independent of AdonisJS. From voyager 1 (its CONCEPT.md D23).
|
||||
*/
|
||||
export { CATEGORIES, categoryLabel, expandCategories, parentCategory } from './categories.js'
|
||||
export { capsXml, errorXml, resultsXml, type ApiLimits, type NewznabItem, type NewznabServer } from './xml.js'
|
||||
export { capsXml, errorXml, resultsXml, type NewznabItem, type NewznabServer } from './xml.js'
|
||||
+4
-24
@@ -27,7 +27,7 @@ export interface NewznabServer {
|
||||
apiKey?: string
|
||||
}
|
||||
|
||||
/** Newznab error codes: 100 credentials, 101 suspended, 102 privileges, 200/201 parameters, 202 function, 300 item, 500/501 request/download limit. */
|
||||
/** Newznab error codes: 100 credentials, 200/201 parameters, 202 function, 300 item. */
|
||||
/** Spotnet UBB markup (or the HTML of old spots) as plain text: [br] as line breaks, other tags dropped. */
|
||||
export function ubbToText(ubb: string): string {
|
||||
return ubb
|
||||
@@ -44,23 +44,11 @@ export function errorXml(code: number, description: string): string {
|
||||
.end({ prettyPrint: true })
|
||||
}
|
||||
|
||||
/** A caller's day so far and its limits (null: none), as nZEDb-style indexers report them. */
|
||||
export interface ApiLimits {
|
||||
apiCurrent: number
|
||||
apiMax: number | null
|
||||
grabCurrent: number
|
||||
grabMax: number | null
|
||||
}
|
||||
|
||||
export function capsXml(
|
||||
server: NewznabServer,
|
||||
limits: { max: number; default: number },
|
||||
registration = { available: false, open: false }
|
||||
): string {
|
||||
export function capsXml(server: NewznabServer, limits: { max: number; default: number }): string {
|
||||
const caps = create({ version: '1.0', encoding: 'UTF-8' }).ele('caps')
|
||||
caps.ele('server', { version: '1.0', title: server.title, strapline: 'Usenet indexer', url: `${server.baseUrl}/` })
|
||||
caps.ele('limits', { max: String(limits.max), default: String(limits.default) })
|
||||
caps.ele('registration', { available: registration.available ? 'yes' : 'no', open: registration.open ? 'yes' : 'no' })
|
||||
caps.ele('registration', { available: 'no', open: 'no' })
|
||||
const searching = caps.ele('searching')
|
||||
searching.ele('search', { available: 'yes', supportedParams: 'q' })
|
||||
searching.ele('tv-search', { available: 'yes', supportedParams: 'q,season,ep' })
|
||||
@@ -75,7 +63,7 @@ export function capsXml(
|
||||
return caps.end({ prettyPrint: true })
|
||||
}
|
||||
|
||||
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number, limits?: ApiLimits): string {
|
||||
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number): string {
|
||||
const rss = create({ version: '1.0', encoding: 'UTF-8' }).ele('rss', {
|
||||
'version': '2.0',
|
||||
'xmlns:atom': 'http://www.w3.org/2005/Atom',
|
||||
@@ -86,14 +74,6 @@ export function resultsXml(server: NewznabServer, items: NewznabItem[], offset:
|
||||
channel.ele('description').txt(`${server.title} API results`)
|
||||
channel.ele('link').txt(`${server.baseUrl}/`)
|
||||
channel.ele('newznab:response', { offset: String(offset), total: String(total) })
|
||||
if (limits) {
|
||||
channel.ele('newznab:apilimits', {
|
||||
apicurrent: String(limits.apiCurrent),
|
||||
...(limits.apiMax !== null && { apimax: String(limits.apiMax) }),
|
||||
grabcurrent: String(limits.grabCurrent),
|
||||
...(limits.grabMax !== null && { grabmax: String(limits.grabMax) }),
|
||||
})
|
||||
}
|
||||
|
||||
const key = server.apiKey ? `&apikey=${encodeURIComponent(server.apiKey)}` : ''
|
||||
for (const item of items) {
|
||||
|
||||
-126
@@ -891,129 +891,3 @@ kbd {
|
||||
margin-top: 4px;
|
||||
color: var(--faint);
|
||||
}
|
||||
|
||||
/* --- Account (ACCESS=accounts) ------------------------------------------ */
|
||||
|
||||
.account-button {
|
||||
height: 32px;
|
||||
padding: 0 12px;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.account-button .tier {
|
||||
color: var(--accent);
|
||||
text-transform: uppercase;
|
||||
font-size: 11px;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
|
||||
.dialog {
|
||||
width: min(420px, calc(100vw - 32px));
|
||||
padding: 0;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
background: var(--surface);
|
||||
color: var(--text);
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
|
||||
.dialog::backdrop {
|
||||
background: rgba(10, 14, 20, 0.35);
|
||||
}
|
||||
|
||||
.dialog-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 14px 18px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.dialog-head h2 {
|
||||
margin: 0;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.dialog-body {
|
||||
padding: 18px;
|
||||
}
|
||||
|
||||
.form {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.form label {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
font-size: 13px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.form input {
|
||||
height: 38px;
|
||||
padding: 0 10px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.form input:focus {
|
||||
outline: 2px solid var(--accent-soft);
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.form .button {
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.form-switch {
|
||||
margin: 4px 0 0;
|
||||
font-size: 13px;
|
||||
color: var(--muted);
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.form-error {
|
||||
margin: 0;
|
||||
color: var(--bad);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.api-key {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
margin-bottom: 18px;
|
||||
}
|
||||
|
||||
.api-key code {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
padding: 8px 10px;
|
||||
border-radius: 8px;
|
||||
background: var(--bg);
|
||||
font-family: var(--mono);
|
||||
font-size: 12.5px;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.meter {
|
||||
height: 6px;
|
||||
margin-top: 4px;
|
||||
border-radius: 3px;
|
||||
background: var(--border);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.meter span {
|
||||
display: block;
|
||||
height: 100%;
|
||||
background: var(--accent);
|
||||
}
|
||||
|
||||
.meter-full span {
|
||||
background: var(--bad);
|
||||
}
|
||||
+5
-187
@@ -51,26 +51,6 @@ const state = {
|
||||
let requestId = 0
|
||||
let searchAbort = null
|
||||
|
||||
/** Signing in (GET /api/v1/access): `open` needs none; with `accounts` the catalog takes a token. */
|
||||
const account = {
|
||||
mode: 'open',
|
||||
signup: 'open',
|
||||
token: storage('get', 'token'),
|
||||
/** The profile's access: tier, limits, today's usage, the Newznab key. */
|
||||
access: null,
|
||||
}
|
||||
|
||||
/** localStorage, which a private window may refuse. */
|
||||
function storage(action, key, value) {
|
||||
try {
|
||||
if (action === 'get') return localStorage.getItem(key)
|
||||
if (action === 'set') localStorage.setItem(key, value)
|
||||
else localStorage.removeItem(key)
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
// --- Formatting ------------------------------------------------------------
|
||||
|
||||
function formatSize(bytes) {
|
||||
@@ -167,26 +147,12 @@ function syncControls() {
|
||||
|
||||
// --- API -------------------------------------------------------------------
|
||||
|
||||
async function api(path, { signal, method = 'GET', body } = {}) {
|
||||
const headers = { accept: 'application/json' }
|
||||
if (account.token) headers.authorization = `Bearer ${account.token}`
|
||||
if (body) headers['content-type'] = 'application/json'
|
||||
const response = await fetch(path, { method, headers, signal, body: body && JSON.stringify(body) })
|
||||
if (!response.ok) {
|
||||
const error = await response.json().catch(() => ({}))
|
||||
// A token that no longer works: sign in again.
|
||||
if (response.status === 401 && account.mode === 'accounts' && !path.startsWith('/api/v1/auth/')) signOut()
|
||||
throw new Error(error.error ?? error.errors?.[0]?.message ?? `${response.status} ${response.statusText}`)
|
||||
}
|
||||
async function api(path, { signal } = {}) {
|
||||
const response = await fetch(path, { headers: { accept: 'application/json' }, signal })
|
||||
if (!response.ok) throw new Error(`${response.status} ${response.statusText}`)
|
||||
return response.json()
|
||||
}
|
||||
|
||||
/** A release's NZB as a plain link: with accounts it carries the key (a link can't send the token). */
|
||||
function nzbHref(id) {
|
||||
const key = account.mode === 'accounts' && account.access?.apiKey
|
||||
return `/api/v1/releases/${id}/nzb${key ? `?apikey=${encodeURIComponent(key)}` : ''}`
|
||||
}
|
||||
|
||||
function searchParams(page) {
|
||||
const params = new URLSearchParams({ sort: state.sort, page: String(page), per_page: '50' })
|
||||
if (state.q) params.set('q', state.q)
|
||||
@@ -263,7 +229,7 @@ function releaseRow(release) {
|
||||
|
||||
row.querySelector('.release-size').textContent = formatSize(release.size)
|
||||
const nzb = row.querySelector('.nzb')
|
||||
nzb.href = nzbHref(release.id)
|
||||
nzb.href = `/api/v1/releases/${release.id}/nzb`
|
||||
row.querySelector('.release-main').addEventListener('click', () => openRelease(release.id, true))
|
||||
return row
|
||||
}
|
||||
@@ -298,8 +264,6 @@ async function loadResults({ append = false } = {}) {
|
||||
(shown < result.total ? ` · showing ${number.format(shown)}` : '')
|
||||
$('#empty').hidden = result.total !== 0
|
||||
$('#more').hidden = shown >= result.total && !result.totalCapped
|
||||
// The day's count in the account button (the profile isn't counted).
|
||||
if (account.access) loadProfile().catch(() => {})
|
||||
} catch (error) {
|
||||
if (id !== requestId) return
|
||||
$('#list').classList.remove('stale')
|
||||
@@ -335,7 +299,7 @@ function renderDetail(release) {
|
||||
const actions = el(
|
||||
'div',
|
||||
{ class: 'actions' },
|
||||
el('a', { class: 'button button-primary', href: nzbHref(release.id) },
|
||||
el('a', { class: 'button button-primary', href: `/api/v1/releases/${release.id}/nzb` },
|
||||
el('span', { text: 'Download NZB' })),
|
||||
el('button', {
|
||||
class: 'button',
|
||||
@@ -454,151 +418,8 @@ function closeRelease(push = false) {
|
||||
document.querySelector(`.release[data-id="${id}"] .release-main`)?.focus()
|
||||
}
|
||||
|
||||
// --- Account (ACCESS=accounts) ---------------------------------------------
|
||||
|
||||
function renderAccountButton() {
|
||||
const button = $('#account')
|
||||
button.hidden = account.mode !== 'accounts'
|
||||
if (!account.access) return button.replaceChildren('Sign in')
|
||||
const { tier, usage, limits } = account.access
|
||||
const of = (used, max) => (max === null ? number.format(used) : `${number.format(used)}/${number.format(max)}`)
|
||||
button.title = 'Searches and NZBs today'
|
||||
button.replaceChildren(el('span', { class: 'tier', text: tier }), ` ${of(usage.apiHits, limits.apiHits)} · ${of(usage.grabs, limits.grabs)} NZB`)
|
||||
}
|
||||
|
||||
async function loadProfile() {
|
||||
const { data } = await api('/api/v1/account/profile')
|
||||
account.access = data.access
|
||||
renderAccountButton()
|
||||
}
|
||||
|
||||
/** Signed in with a new token: the profile, then the search that was waiting. */
|
||||
async function signedIn(token) {
|
||||
account.token = token
|
||||
storage('set', 'token', token)
|
||||
await loadProfile()
|
||||
$('#account-dialog').close()
|
||||
loadResults()
|
||||
if (state.release) openRelease(state.release)
|
||||
}
|
||||
|
||||
function signOut() {
|
||||
if (account.token) api('/api/v1/account/logout', { method: 'POST' }).catch(() => {})
|
||||
account.token = null
|
||||
account.access = null
|
||||
storage('remove', 'token')
|
||||
renderAccountButton()
|
||||
closeRelease()
|
||||
$('#list').replaceChildren()
|
||||
$('#more').hidden = true
|
||||
$('#empty').hidden = true
|
||||
$('#summary').textContent = 'Sign in to search the index.'
|
||||
openAccount('signin')
|
||||
}
|
||||
|
||||
function field(label, props) {
|
||||
return el('label', {}, label, el('input', props))
|
||||
}
|
||||
|
||||
function accountForm(view) {
|
||||
const signup = view === 'signup'
|
||||
const error = el('p', { class: 'form-error', role: 'alert', hidden: true })
|
||||
const form = el(
|
||||
'form',
|
||||
{ class: 'form' },
|
||||
field('Email', { name: 'email', type: 'email', autocomplete: 'email', required: true }),
|
||||
field('Password', { name: 'password', type: 'password', autocomplete: signup ? 'new-password' : 'current-password', required: true, minlength: signup ? '8' : null }),
|
||||
signup ? field('Password again', { name: 'passwordConfirmation', type: 'password', autocomplete: 'new-password', required: true }) : null,
|
||||
signup && account.signup === 'invite' ? field('Invite code', { name: 'invite', autocomplete: 'off', spellcheck: 'false', required: true }) : null,
|
||||
error,
|
||||
el('button', { class: 'button button-primary', type: 'submit', text: signup ? 'Create account' : 'Sign in' })
|
||||
)
|
||||
form.addEventListener('submit', async (event) => {
|
||||
event.preventDefault()
|
||||
const values = Object.fromEntries(new FormData(form))
|
||||
error.hidden = true
|
||||
try {
|
||||
const path = signup ? '/api/v1/auth/signup' : '/api/v1/auth/login'
|
||||
const { data } = await api(path, { method: 'POST', body: signup ? { fullName: null, ...values } : values })
|
||||
await signedIn(data.token)
|
||||
} catch (failure) {
|
||||
error.textContent = failure.message
|
||||
error.hidden = false
|
||||
}
|
||||
})
|
||||
const other = signup
|
||||
? el('p', { class: 'form-switch' }, 'Have an account? ', el('button', { class: 'link-button', type: 'button', text: 'Sign in', onclick: () => openAccount('signin') }))
|
||||
: account.signup !== 'closed'
|
||||
? el('p', { class: 'form-switch' }, 'New here? ', el('button', { class: 'link-button', type: 'button', text: 'Create an account', onclick: () => openAccount('signup') }))
|
||||
: null
|
||||
return [form, other]
|
||||
}
|
||||
|
||||
function meter(used, max) {
|
||||
if (max === null) return null
|
||||
const share = Math.min(1, used / Math.max(1, max))
|
||||
return el('div', { class: `meter${used >= max ? ' meter-full' : ''}` }, el('span', { style: `width: ${Math.round(share * 100)}%` }))
|
||||
}
|
||||
|
||||
function profileView() {
|
||||
const { tier, tierUntil, limits, usage, apiKey } = account.access
|
||||
const of = (used, max) => (max === null ? `${number.format(used)} (no limit)` : `${number.format(used)} of ${number.format(max)}`)
|
||||
const key = el('code', { text: apiKey })
|
||||
return [
|
||||
el('dl', { class: 'facts' },
|
||||
fact('Tier', tierUntil ? `${tier} until ${new Date(tierUntil).toLocaleDateString()}` : tier),
|
||||
fact('Searches today', el('div', {}, of(usage.apiHits, limits.apiHits), meter(usage.apiHits, limits.apiHits))),
|
||||
fact('NZBs today', el('div', {}, of(usage.grabs, limits.grabs), meter(usage.grabs, limits.grabs))),
|
||||
fact('Retention', limits.retentionDays ? `${number.format(limits.retentionDays)} days` : 'Everything')),
|
||||
...section('Newznab API key', 'for Prowlarr, Sonarr, Radarr', el('div', { class: 'api-key' },
|
||||
key,
|
||||
el('button', { class: 'button', type: 'button', text: 'Copy', onclick: async (event) => {
|
||||
await navigator.clipboard?.writeText(account.access.apiKey)
|
||||
event.target.textContent = 'Copied'
|
||||
setTimeout(() => (event.target.textContent = 'Copy'), 1200)
|
||||
} }),
|
||||
el('button', { class: 'button', type: 'button', text: 'New key', title: 'The old key stops working', onclick: async () => {
|
||||
if (!confirm('Make a new key? Apps using the old one stop working until they get the new one.')) return
|
||||
const { apiKey: fresh } = await api('/api/v1/account/api-key', { method: 'POST' })
|
||||
account.access.apiKey = fresh
|
||||
key.textContent = fresh
|
||||
} }))),
|
||||
el('p', { class: 'form-switch' }, `URL for your apps: ${location.origin}`),
|
||||
el('button', { class: 'button', type: 'button', text: 'Sign out', onclick: () => signOut() }),
|
||||
]
|
||||
}
|
||||
|
||||
async function openAccount(view) {
|
||||
const dialog = $('#account-dialog')
|
||||
if (view === 'profile') await loadProfile().catch(() => {})
|
||||
if (view === 'profile' && !account.access) return
|
||||
$('#account-title').textContent = { signin: 'Sign in', signup: 'Create an account', profile: 'Your account' }[view]
|
||||
$('#account-body').replaceChildren(...(view === 'profile' ? profileView() : accountForm(view)))
|
||||
if (!dialog.open) dialog.showModal()
|
||||
dialog.querySelector('input')?.focus()
|
||||
}
|
||||
|
||||
/** With accounts, nothing is searched before signing in. */
|
||||
async function startAccess() {
|
||||
try {
|
||||
Object.assign(account, await api('/api/v1/access'))
|
||||
} catch {
|
||||
// An older server: open.
|
||||
}
|
||||
if (account.mode !== 'accounts') return true
|
||||
if (account.token) await loadProfile().catch(() => {})
|
||||
renderAccountButton()
|
||||
if (account.access) return true
|
||||
$('#summary').textContent = 'Sign in to search the index.'
|
||||
if (!$('#account-dialog').open) openAccount('signin')
|
||||
return false
|
||||
}
|
||||
|
||||
// --- Wiring ----------------------------------------------------------------
|
||||
|
||||
$('#account').addEventListener('click', () => openAccount(account.access ? 'profile' : 'signin'))
|
||||
$('#account-close').addEventListener('click', () => $('#account-dialog').close())
|
||||
|
||||
$('#q').addEventListener('input', (event) => {
|
||||
state.q = event.target.value.trim()
|
||||
scheduleSearch()
|
||||
@@ -674,8 +495,5 @@ window.addEventListener('popstate', () => {
|
||||
readUrl()
|
||||
syncControls()
|
||||
loadStats()
|
||||
startAccess().then((ready) => {
|
||||
if (!ready) return
|
||||
loadResults()
|
||||
if (state.release) openRelease(state.release)
|
||||
})
|
||||
@@ -14,19 +14,8 @@
|
||||
<span>Voyager 2</span>
|
||||
</a>
|
||||
<dl class="stats" id="stats" aria-label="Index statistics"></dl>
|
||||
<button type="button" class="button account-button" id="account" hidden></button>
|
||||
</header>
|
||||
|
||||
<dialog class="dialog" id="account-dialog" aria-labelledby="account-title">
|
||||
<div class="dialog-head">
|
||||
<h2 id="account-title"></h2>
|
||||
<button type="button" class="icon-button" id="account-close" aria-label="Close">
|
||||
<svg aria-hidden="true" viewBox="0 0 24 24"><path d="M6 6l12 12M18 6 6 18" /></svg>
|
||||
</button>
|
||||
</div>
|
||||
<div class="dialog-body" id="account-body"></div>
|
||||
</dialog>
|
||||
|
||||
<main class="layout">
|
||||
<section class="search" aria-label="Search">
|
||||
<div class="search-row">
|
||||
|
||||
+6
-4
@@ -43,19 +43,19 @@ export default await Env.create(new URL('../', import.meta.url), {
|
||||
NNTP_PASSWORD: Env.schema.string.optional(),
|
||||
NNTP_SECURE: Env.schema.boolean.optional(),
|
||||
NNTP_CONNECTIONS: Env.schema.number.optional(),
|
||||
NNTP_MIN_CONNECTIONS: Env.schema.number.optional(),
|
||||
NNTP_ACCOUNT_CONNECTIONS: Env.schema.number.optional(),
|
||||
NNTP_RESERVED_CONNECTIONS: Env.schema.number.optional(),
|
||||
NNTP_TIMEOUT_MS: Env.schema.number.optional(),
|
||||
NNTP_HEDGE_AFTER_SECONDS: Env.schema.number.optional(),
|
||||
NNTP_HEDGE_CONNECTIONS: Env.schema.number.optional(),
|
||||
NNTP_CONNECTION_MAX_MINUTES: Env.schema.number.optional(),
|
||||
NNTP_IDLE_CLOSE_SECONDS: Env.schema.number.optional(),
|
||||
NNTP_STALL_KB_PER_MINUTE: Env.schema.number.optional(),
|
||||
|
||||
// Newznab API key (empty: no key needed)
|
||||
NEWZNAB_API_KEY: Env.schema.secret.optional(),
|
||||
|
||||
// Access (config/access.ts): open (one key, no limits) or accounts (a key per user, tiers with daily limits)
|
||||
ACCESS: Env.schema.enum.optional(['open', 'accounts'] as const),
|
||||
SIGNUP: Env.schema.enum.optional(['open', 'invite', 'closed'] as const),
|
||||
|
||||
// Coordinator
|
||||
RANGE_ARTICLES: Env.schema.number.optional(),
|
||||
TAIL_EVERY_SECONDS: Env.schema.number.optional(),
|
||||
@@ -78,6 +78,8 @@ export default await Env.create(new URL('../', import.meta.url), {
|
||||
SPOOL_MAX_GB: Env.schema.number.optional(),
|
||||
PREFETCH_PER_RANGE: Env.schema.number.optional(),
|
||||
PROBE_CONNECTIONS: Env.schema.number.optional(),
|
||||
PROBE_MIN_CONNECTIONS: Env.schema.number.optional(),
|
||||
PROBE_MIN_FREE_GB: Env.schema.number.optional(),
|
||||
HTTP_ROUTES: Env.schema.enum.optional(['all', 'node', 'public'] as const),
|
||||
PREFETCH_PER_GROUP: Env.schema.number.optional(),
|
||||
})
|
||||
+1
-6
@@ -45,18 +45,13 @@ router
|
||||
router
|
||||
.group(() => {
|
||||
router.get('profile', [controllers.Profile, 'show'])
|
||||
router.post('api-key', [controllers.Profile, 'resetApiKey'])
|
||||
router.post('logout', [controllers.AccessTokens, 'destroy'])
|
||||
})
|
||||
.prefix('account')
|
||||
.as('profile')
|
||||
.use(middleware.auth())
|
||||
|
||||
// How the web interface signs in: open, or accounts with limits (config/access.ts).
|
||||
router.get('access', [controllers.Releases, 'access'])
|
||||
|
||||
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it), unless ACCESS
|
||||
// is `accounts`: then a user's token or apikey, counted as Newznab's are (AccessService).
|
||||
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it); Newznab has its key.
|
||||
router.get('releases', [controllers.Releases, 'index'])
|
||||
router.get('releases/:id', [controllers.Releases, 'show']).where('id', router.matchers.number())
|
||||
router.get('releases/:id/nzb', [controllers.Releases, 'nzb']).where('id', router.matchers.number())
|
||||
|
||||
@@ -1,155 +0,0 @@
|
||||
import { test } from '@japa/runner'
|
||||
import { DateTime } from 'luxon'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import testUtils from '@adonisjs/core/services/test_utils'
|
||||
import User from '#models/user'
|
||||
import BatchStore from '#services/batch_store'
|
||||
import AccessService from '#services/access_service'
|
||||
import { TIERS } from '#config/access'
|
||||
import { RangeIndexer } from '#lib/edge/index'
|
||||
import { DEFAULT_RULES } from '#lib/rules/index'
|
||||
import type { Overview } from '#lib/nntp/index'
|
||||
|
||||
const GROUP = 'alt.binaries.test'
|
||||
|
||||
function file(name: string, counter: string, parts: number, at: number): Overview[] {
|
||||
return Array.from({ length: parts }, (_, i) => [at + i, {
|
||||
'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
|
||||
'message-id': `<${name}-${i + 1}@test>`, ':bytes': String(700000 + i), 'xref': `news.example ${GROUP}:${at + i}`,
|
||||
}] as Overview)
|
||||
}
|
||||
|
||||
/** Two releases: a show (recent) and a movie posted two years ago. */
|
||||
async function catalog() {
|
||||
const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
|
||||
const [n] = await db.table('nodes').insert({ name: 'access-node' }).returning('id')
|
||||
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: n.id }).returning('id')
|
||||
const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
|
||||
indexer.add([...file('Some.Show.S01E01.1080p.WEB.mkv', 'Some.Show.S01E01.1080p.WEB [1/1]', 2, 100), ...file('Old.Movie.2024.1080p.BluRay.mkv', 'Old.Movie.2024.1080p.BluRay [1/1]', 1, 300)], 1001)
|
||||
indexer.finish()
|
||||
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
|
||||
const show = await db.from('releases').where('name', 'ilike', '%show%').firstOrFail()
|
||||
const movie = await db.from('releases').where('name', 'ilike', '%movie%').firstOrFail()
|
||||
await db.from('releases').where('id', movie.id).update({ posted_at: DateTime.now().minus({ years: 2 }).toJSDate() })
|
||||
return { show: Number(show.id), movie: Number(movie.id) }
|
||||
}
|
||||
|
||||
const user = (email = 'someone@example.com') => User.create({ email, password: 'secret-password' })
|
||||
|
||||
test.group('Access with accounts', (group) => {
|
||||
const free = { ...TIERS.free }
|
||||
group.each.setup(() => testUtils.db().withGlobalTransaction())
|
||||
group.each.setup(() => {
|
||||
AccessService.mode = 'accounts'
|
||||
AccessService.signup = 'invite'
|
||||
Object.assign(TIERS.free, { apiHits: 3, grabs: 1, retentionDays: 365 })
|
||||
return () => {
|
||||
AccessService.mode = 'open'
|
||||
AccessService.signup = 'open'
|
||||
Object.assign(TIERS.free, free)
|
||||
}
|
||||
})
|
||||
|
||||
test("Newznab takes a user's key, counts searches and grabs per day and keeps to the tier's retention", async ({ client, assert }) => {
|
||||
const { show, movie } = await catalog()
|
||||
const someone = await user()
|
||||
|
||||
assert.include((await client.get('/api?t=search&q=')).text(), 'code="100"')
|
||||
assert.include((await client.get('/api?t=search&q=&apikey=0123456789abcdef0123456789abcdef')).text(), 'code="100"')
|
||||
// caps isn't counted.
|
||||
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), '<registration available="yes" open="no"/>')
|
||||
|
||||
const feed = (await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text()
|
||||
// The free tier sees a year back: the movie isn't found.
|
||||
assert.include(feed, 'Some Show S01E01')
|
||||
assert.notInclude(feed, 'Old Movie')
|
||||
assert.include(feed, `t=get&id=${show}&apikey=${someone.apiKey}`)
|
||||
assert.include(feed, '<newznab:apilimits apicurrent="1" apimax="3" grabcurrent="0" grabmax="1"/>')
|
||||
|
||||
// Past the retention: refused, and not counted as a grab.
|
||||
assert.include((await client.get(`/api?t=get&id=${movie}&apikey=${someone.apiKey}`)).text(), 'code="102"')
|
||||
assert.include((await client.get(`/api?t=get&id=999999&apikey=${someone.apiKey}`)).text(), 'code="300"')
|
||||
const nzb = await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)
|
||||
assert.include(nzb.header('content-type'), 'application/x-nzb')
|
||||
assert.include((await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)).text(), 'code="501"')
|
||||
|
||||
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
|
||||
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
|
||||
assert.include((await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text(), 'code="500"')
|
||||
assert.deepEqual(await AccessService.usage(someone.id), { apiHits: 3, grabs: 1 })
|
||||
|
||||
// VIP sees everything; a tier that ran out is free again.
|
||||
AccessService.grantTier(someone, 'vip', 30)
|
||||
await someone.save()
|
||||
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'Old Movie 2024')
|
||||
someone.tierUntil = DateTime.now().minus({ days: 1 })
|
||||
await someone.save()
|
||||
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'code="500"')
|
||||
|
||||
someone.suspended = true
|
||||
await someone.save()
|
||||
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="101"')
|
||||
})
|
||||
|
||||
test('the catalog needs a token or a key, and counts as Newznab does', async ({ client, assert }) => {
|
||||
const { show, movie } = await catalog()
|
||||
const someone = await user()
|
||||
|
||||
;(await client.get('/api/v1/releases')).assertStatus(401)
|
||||
;(await client.get(`/api/v1/releases/${show}/nzb`)).assertStatus(401)
|
||||
// Stats and how to sign in stay open.
|
||||
;(await client.get('/api/v1/stats')).assertStatus(200)
|
||||
assert.deepEqual((await client.get('/api/v1/access')).body(), { mode: 'accounts', signup: 'invite' })
|
||||
|
||||
const found = await client.get('/api/v1/releases').loginAs(someone)
|
||||
found.assertStatus(200)
|
||||
assert.deepEqual((found.body() as { releases: { id: number }[] }).releases.map((r) => r.id), [show])
|
||||
;(await client.get(`/api/v1/releases/${movie}`).loginAs(someone)).assertStatus(403)
|
||||
;(await client.get(`/api/v1/releases/${show}`).loginAs(someone)).assertStatus(200)
|
||||
|
||||
// A plain link (the web interface's download button) carries the key.
|
||||
const nzb = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
|
||||
nzb.assertStatus(200)
|
||||
const over = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
|
||||
over.assertStatus(429)
|
||||
assert.deepEqual(over.body(), { error: 'Download limit reached', code: 501 })
|
||||
// The refused movie wasn't counted: a third search is left.
|
||||
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(200)
|
||||
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(429)
|
||||
|
||||
const profile = ((await client.get('/api/v1/account/profile').loginAs(someone)).body() as any).data
|
||||
assert.equal(profile.access.tier, 'free')
|
||||
assert.equal(profile.access.apiKey, someone.apiKey)
|
||||
assert.deepEqual(profile.access.usage, { apiHits: 3, grabs: 1 })
|
||||
|
||||
const reset = (await client.post('/api/v1/account/api-key').loginAs(someone)).body() as { apiKey: string }
|
||||
assert.notEqual(reset.apiKey, someone.apiKey)
|
||||
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="100"')
|
||||
})
|
||||
|
||||
test('signup takes an unused invite, which may give a tier', async ({ client, assert }) => {
|
||||
const form = (email: string, invite?: string) => ({ email, password: 'secret-password', passwordConfirmation: 'secret-password', fullName: null, invite })
|
||||
await db.table('invites').insert([{ code: 'plain-invite' }, { code: 'vip-invite', tier: 'vip', tier_days: 365 }])
|
||||
|
||||
;(await client.post('/api/v1/auth/signup').json(form('a@example.com'))).assertStatus(403)
|
||||
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'no-such-code'))).assertStatus(403)
|
||||
// A refused invite leaves no account behind.
|
||||
assert.isNull(await User.findBy('email', 'a@example.com'))
|
||||
|
||||
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'plain-invite'))).assertStatus(200)
|
||||
;(await client.post('/api/v1/auth/signup').json(form('b@example.com', 'plain-invite'))).assertStatus(403)
|
||||
const vip = await client.post('/api/v1/auth/signup').json(form('c@example.com', 'vip-invite'))
|
||||
vip.assertStatus(200)
|
||||
const c = await User.findByOrFail('email', 'c@example.com')
|
||||
assert.equal(c.tier, 'vip')
|
||||
assert.closeTo(c.tierUntil!.diffNow('days').days, 365, 1)
|
||||
assert.match(c.apiKey, /^[0-9a-f]{32}$/)
|
||||
|
||||
// Renewed early, the year is added to what's left.
|
||||
AccessService.grantTier(c, 'vip', 365)
|
||||
assert.closeTo(c.tierUntil!.diffNow('days').days, 730, 1)
|
||||
|
||||
AccessService.signup = 'closed'
|
||||
;(await client.post('/api/v1/auth/signup').json(form('d@example.com', 'vip-invite'))).assertStatus(403)
|
||||
})
|
||||
})
|
||||
@@ -120,6 +120,40 @@ test.group('BatchStore', (group) => {
|
||||
assert.equal(Number(chunks!.n), 4)
|
||||
})
|
||||
|
||||
test('candidates added to a settled release (a later range of its upload) are claimable at once', async ({ assert }) => {
|
||||
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E02.1080p.mkv', 2, 'Some.Show.S01E02.1080p [1/2]', '2026-10-05T10:00:00Z'))
|
||||
// Settled before the next range came (the settler marks only what it settles).
|
||||
await db.from('releases').update({ settled_at: new Date() })
|
||||
await db.from('probe_candidates').update({ ready: true })
|
||||
await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E02.1080p.par2', 1, 'Some.Show.S01E02.1080p [2/2]', '2026-10-05T10:20:00Z'))
|
||||
const candidates = await db.from('probe_candidates').orderBy('id').select('kind', 'ready')
|
||||
assert.deepEqual(candidates.map((c) => c.kind), ['par2'])
|
||||
assert.isTrue(candidates.every((c) => c.ready))
|
||||
assert.lengthOf(await releasesOf(), 1)
|
||||
})
|
||||
|
||||
test('a split 7z read in pieces keeps the tail of its highest volume only; RAR gets none', async ({ assert }) => {
|
||||
// As the live tail reads it while posted: each piece queues a tail for the last volume it saw so far.
|
||||
await storeIn(GROUP, 1, [
|
||||
...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
|
||||
...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
|
||||
])
|
||||
await storeIn(GROUP, 100_001, upload(100_010, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'))
|
||||
// A lower volume posted late: its tail goes at once.
|
||||
await storeIn(GROUP, 200_001, upload(200_010, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'))
|
||||
await storeIn(GROUP, 300_001, upload(300_010, 'Some.Show.S01E03.part01.rar', 2, 'Some.Show.S01E03 [1/2]', '2026-10-23T09:00:00Z'))
|
||||
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E03.part02.rar', 2, 'Some.Show.S01E03 [2/2]', '2026-10-23T09:02:00Z'))
|
||||
const candidates = await db.from('probe_candidates').orderBy('file').select('kind', 'file')
|
||||
assert.deepEqual(
|
||||
candidates.map((c) => [c.kind, c.file]),
|
||||
[
|
||||
['archive', 'Some.Show.S01E03.part01.rar'],
|
||||
['archive', 'TsIgNBewuFbuwWy.7z.001'],
|
||||
['7z-tail', 'TsIgNBewuFbuwWy.7z.004'],
|
||||
]
|
||||
)
|
||||
})
|
||||
|
||||
test('the same upload in another group (multi-posted) is a release of its own', async ({ assert }) => {
|
||||
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
|
||||
await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
|
||||
@@ -146,4 +180,29 @@ test.group('BatchStore', (group) => {
|
||||
const keys = (await releasesOf()).filter((r) => r.identity_key === first.identity_key).map((r) => r.release_key)
|
||||
assert.deepEqual(keys, [first.identity_key, `${first.identity_key}:3`])
|
||||
})
|
||||
|
||||
test('a re-upload is numbered after the highest number, not into the holes merges left', async ({ assert }) => {
|
||||
for (let day = 0; day < 4; day++) {
|
||||
await storeIn(GROUP, 1 + day * 100_000, upload(10 + day * 100_000, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', `2026-10-0${day * 2 + 1}T10:00:00Z`))
|
||||
}
|
||||
const [first] = await releasesOf()
|
||||
// Merged away: key:2 and key:3 gone, key and key:4 left.
|
||||
await db.from('releases').whereIn('release_key', [`${first.identity_key}:2`, `${first.identity_key}:3`]).delete()
|
||||
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-09T10:00:00Z'))
|
||||
const keys = (await releasesOf()).map((r) => r.release_key.split(':')[1] ?? '1')
|
||||
assert.deepEqual(keys, ['1', '4', '5'])
|
||||
})
|
||||
|
||||
test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
|
||||
await storeIn(GROUP, 1_000_000, [
|
||||
...upload(1_000_000, 'Murdoch Mysteries S8D1.nzb', 1, '[00/80]', '2026-10-05T10:00:00Z'),
|
||||
...upload(1_000_010, 'Some.Tool.v1.zip', 1, '[01/01]', '2026-10-05T10:00:00Z'),
|
||||
...upload(1_000_020, 'Some.Show.S01E01.part01.rar', 2, '[02/30]', '2026-10-05T10:00:00Z'),
|
||||
])
|
||||
const flags = new Map((await db.from('releases').select('name', 'sidecar_only')).map((r) => [r.name, r.sidecar_only]))
|
||||
assert.isTrue(flags.get('Murdoch Mysteries S8D1'))
|
||||
assert.isFalse(flags.get('Some.Tool.v1'))
|
||||
// Over 1 MB: not looked at.
|
||||
assert.isNull(flags.get('Some.Show.S01E01'))
|
||||
})
|
||||
})
|
||||
@@ -31,5 +31,12 @@ test.group('Leases', (group) => {
|
||||
// a1's worker went quiet: its lease is taken over (a2 alone counts).
|
||||
await db.from('ranges').where('id', a1).update({ heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
|
||||
assert.equal(await leased(5), a1)
|
||||
|
||||
// Read to its end, its worker gone before finishing: done. One still reading, or quiet but not at its end, stays.
|
||||
await db.from('ranges').where('id', a2).update({ next_id: 201, heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
|
||||
await db.from('ranges').where('id', b1).update({ next_id: 101 })
|
||||
assert.equal(await Leases.finishAbandoned(), 1)
|
||||
assert.equal((await db.from('ranges').where('id', a2).firstOrFail()).status, 'done')
|
||||
assert.equal((await db.from('ranges').where('id', b1).firstOrFail()).status, 'leased')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,53 @@
|
||||
import { test } from '@japa/runner'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import testUtils from '@adonisjs/core/services/test_utils'
|
||||
import coordinatorConfig from '#config/coordinator'
|
||||
import { grant, targets, NntpShare } from '#services/nntp_shares'
|
||||
|
||||
test.group('NNTP shares', (group) => {
|
||||
group.each.setup(() => testUtils.db().withGlobalTransaction())
|
||||
|
||||
test('targets: minimums first, then by priority what each wants', ({ assert }) => {
|
||||
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 0 }
|
||||
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 0 }
|
||||
// History running: the coordinator all it wants, the prober the rest.
|
||||
assert.deepEqual(Object.fromEntries(targets(45, [prober, coordinator])), { coordinator: 38, prober: 7 })
|
||||
// History paused: the coordinator its minimum (or what it uses), the prober the rest.
|
||||
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 8 }])), { coordinator: 8, prober: 37 })
|
||||
// Wanting more than its minimum doesn't take another's minimum.
|
||||
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 60 }])), { coordinator: 43, prober: 2 })
|
||||
})
|
||||
|
||||
test('grant: growing only into connections the others closed or gave up', ({ assert }) => {
|
||||
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 37, granted: 37 }
|
||||
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 8 }
|
||||
// History came back: the prober still holds 37, the coordinator keeps its 8 for now.
|
||||
assert.equal(grant(45, coordinator, [prober]), 8)
|
||||
// The prober syncs: its target is 7, its grant goes down at once.
|
||||
assert.equal(grant(45, { ...prober, open: 37 }, [{ ...coordinator, granted: 8 }]), 7)
|
||||
// Its connections not closed yet: still held.
|
||||
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 30 }]), 15)
|
||||
// Closed: the coordinator gets all it wants.
|
||||
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 7 }]), 38)
|
||||
})
|
||||
|
||||
test('sync: each process its row; a process gone (no heartbeat) holds nothing', async ({ assert }) => {
|
||||
const total = coordinatorConfig.nntpShares.total
|
||||
const coordinator = new NntpShare('coordinator', 2, 8, 38)
|
||||
const prober = new NntpShare('prober', 1, 2, total)
|
||||
assert.equal(await coordinator.sync(8, 0), 8)
|
||||
assert.equal(await prober.sync(total, 0), total - 8)
|
||||
// The coordinator wants all: the prober holds its grant until it has synced and closed them.
|
||||
assert.equal(await coordinator.sync(38, 8), 8)
|
||||
assert.equal(await prober.sync(total, total - 8), total - 38)
|
||||
assert.equal(await coordinator.sync(38, 8), 8)
|
||||
assert.equal(await prober.sync(total, total - 38), total - 38)
|
||||
assert.equal(await coordinator.sync(38, 8), 38)
|
||||
|
||||
// The prober's process died: after a minute without a sync its share is free.
|
||||
await db.from('nntp_shares').where('name', 'prober').update({ open: total, granted: total, heartbeat_at: new Date(Date.now() - 120_000) })
|
||||
assert.equal(await coordinator.sync(38, 38), 38)
|
||||
await coordinator.leave()
|
||||
assert.equal(await prober.sync(total, 0), total)
|
||||
})
|
||||
})
|
||||
@@ -78,6 +78,44 @@ test.group('Par2Merger', (group) => {
|
||||
assert.equal((await new Par2Merger().round()).sets, 1)
|
||||
})
|
||||
|
||||
test('a round behind by hours goes through them a window at a time, past the gaps', async ({ assert }) => {
|
||||
const ago = (hours: number) => new Date(Date.now() - hours * 3600_000)
|
||||
await db.table('par2_sets').multiInsert([
|
||||
{ set_id: 'a1'.repeat(16), release_id: 1, files: 1, created_at: ago(30) },
|
||||
{ set_id: 'a2'.repeat(16), release_id: 2, files: 1, created_at: ago(2.5) },
|
||||
{ set_id: 'a3'.repeat(16), release_id: 3, files: 1, created_at: ago(0.5) },
|
||||
])
|
||||
await db.table('worker_cursors').insert({ name: 'merger', at: ago(48) })
|
||||
assert.equal((await new Par2Merger().round()).sets, 3)
|
||||
const saved = await db.from('worker_cursors').where('name', 'merger').firstOrFail()
|
||||
assert.isAbove(new Date(saved.at).getTime(), ago(0.5).getTime())
|
||||
})
|
||||
|
||||
test('a par2 file held by more than 20 files is a common one: it merges nothing', async ({ assert }) => {
|
||||
const at = new Date('2022-07-08T03:46:00Z')
|
||||
const ids: number[] = []
|
||||
for (const key of ['dd000001', 'dd000002']) {
|
||||
const [row] = await db
|
||||
.table('releases')
|
||||
.insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
|
||||
.returning('id')
|
||||
ids.push(Number(row.id))
|
||||
}
|
||||
// The par2 is held by the first; the second holds the file, and so do 20 releases elsewhere and years apart.
|
||||
const setId = hash('1')
|
||||
await db.table('par2_sets').insert({ set_id: setId, release_id: ids[0], files: 1 })
|
||||
await db.table('par2_files').insert({ set_id: setId, file_id: hash('0'), hash16k: hash('9'), length: 30214, name: 'sample.nfo' })
|
||||
await db.table('file_hashes').insert({ candidate_id: 4000, release_id: ids[1], hash16k: hash('9'), size: 30214 })
|
||||
for (let i = 1; i <= 20; i++) await db.table('file_hashes').insert({ candidate_id: 4000 + i, release_id: 900000 + i, hash16k: hash('9'), size: 30214 })
|
||||
const round = await new Par2Merger().round()
|
||||
assert.deepEqual([round.sets, round.merges], [1, 0])
|
||||
assert.equal(await new Par2Merger().mergeSet(setId), 0)
|
||||
assert.equal((await db.from('releases').whereIn('id', ids).select('id')).length, 2)
|
||||
// One holder fewer: the file is one upload's again.
|
||||
await db.from('file_hashes').where('candidate_id', 4020).delete()
|
||||
assert.equal(await new Par2Merger().mergeSet(setId), 1)
|
||||
})
|
||||
|
||||
test('a set held by two of the releases merged is held once by the target', async ({ assert }) => {
|
||||
const at = new Date('2022-07-08T03:46:00Z')
|
||||
const ids: number[] = []
|
||||
@@ -220,6 +258,8 @@ test.group('Par2Merger', (group) => {
|
||||
// A picture isn't a sidecar; an nfo an hour later belongs to nothing here.
|
||||
const picture = await release('cover', 1, null, at, ['cover.jpg'])
|
||||
const late = await release('Murdoch Mysteries S8D1', 1, null, new Date(at.getTime() + 3600_000), ['Murdoch Mysteries S8D1.nfo'])
|
||||
// Queued, its release gone meanwhile (merged): leaves the queue with its batch.
|
||||
await db.table('fold_candidates').insert({ release_id: 987654321 })
|
||||
|
||||
const round = await new Par2Merger().round()
|
||||
assert.deepEqual(round.folded, { looked: 5, sidecars: 2, folded: 1 })
|
||||
|
||||
@@ -53,16 +53,72 @@ test.group('Par2Pruner', (group) => {
|
||||
const locked = await set('e', e, 2, null)
|
||||
await db.table('release_verdicts').insert({ release_id: e, rule: 'locked', rule_version: 1 })
|
||||
|
||||
const round = await new Par2Pruner().round()
|
||||
assert.deepEqual(round, { complete: 1, judged: 1, rows: 4 })
|
||||
// Final: its holder held by a run of done ranges with hours to spare, nothing hashed (a readable upload's files).
|
||||
const f = await release()
|
||||
const final = await set('f', f, 2, null)
|
||||
const posted = new Date('2024-03-01T12:00:00Z')
|
||||
const [group] = await db.table('groups').insert({ name: 'alt.binaries.pruned', priority: 1 }).returning('id')
|
||||
const groupId = Number(group.id)
|
||||
await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100 })
|
||||
const [range] = await db
|
||||
.table('ranges')
|
||||
.insert({ group_id: groupId, start_id: 1, end_id: 10000, next_id: 10001, status: 'done', done_at: old, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z') })
|
||||
.returning('id')
|
||||
await db.table('range_blocks').insert({ group_id: groupId, start_id: 1, end_id: 10000, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z'), at_group_start: false })
|
||||
// A lone candidate unprobed since before the range was done: its hash may still match. Kept.
|
||||
await db.table('probe_candidates').insert({ id: 9001, release_id: b, kind: 'lone', message_id: '<9001@x>', file: 'x', status: 'waiting', created_at: new Date(old.getTime() - 3600_000) })
|
||||
|
||||
// Out of budget: one batch of two sets (a, b), the next round goes on.
|
||||
assert.deepEqual(await new Par2Pruner(2).round(0), { complete: 1, judged: 0, final: 0, rows: 2 })
|
||||
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 1, final: 0, rows: 2 })
|
||||
assert.equal(await count(complete), 0)
|
||||
assert.equal(await count(apart), 2)
|
||||
assert.equal(await count(recent), 2)
|
||||
assert.equal(await count(dead), 0)
|
||||
assert.equal(await count(locked), 2)
|
||||
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '5')
|
||||
assert.equal(await count(final), 2)
|
||||
|
||||
// Again: the cursor is past them, nothing more.
|
||||
assert.deepEqual(await new Par2Pruner().round(), { complete: 0, judged: 0, rows: 0 })
|
||||
// Probed: final. The next pass starts over from the first set.
|
||||
await db.from('probe_candidates').where('id', 9001).update({ status: 'done' })
|
||||
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 0, final: 1, rows: 2 })
|
||||
assert.equal(await count(final), 0)
|
||||
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '6')
|
||||
|
||||
// A hash of its files since the merger's last window: the merger hasn't matched it yet. Kept.
|
||||
const g = await release()
|
||||
const unseen = await set('g', g, 1, null)
|
||||
await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100 })
|
||||
await db.table('worker_cursors').insert({ name: 'merger', at: new Date(Date.now() - 3600_000) })
|
||||
await db.table('probe_candidates').insert({ id: 9002, release_id: await release(), kind: 'lone', message_id: '<9002@x>', file: 'x', status: 'done' })
|
||||
await db.table('file_hashes').insert({ candidate_id: 9002, release_id: 1, hash16k: 'g0'.padEnd(32, '0'), size: 1000 })
|
||||
assert.equal((await new Par2Pruner().round()).final, 0)
|
||||
// Past the hash by more than the merger's overlap.
|
||||
await db.from('worker_cursors').where('name', 'merger').update({ at: new Date(Date.now() + 120_000) })
|
||||
assert.equal((await new Par2Pruner().round()).final, 1)
|
||||
assert.equal(await count(unseen), 0)
|
||||
assert.isNotNull(range)
|
||||
})
|
||||
|
||||
test('sets of one created_at, a batch each: every one looked at', async ({ assert }) => {
|
||||
// One prober flush gives thousands of sets the same created_at (to the microsecond).
|
||||
const flush = new Date('2026-10-15T14:38:39.883Z')
|
||||
let candidate = 7000
|
||||
for (const letter of ['p', 'q', 'r']) {
|
||||
const key = `pr${letter}`.padEnd(40, '0')
|
||||
const [row] = await db.table('releases').insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }).returning('id')
|
||||
const id = Number(row.id)
|
||||
const setId = letter.repeat(32)
|
||||
const h = `${letter}0`.padEnd(32, '0')
|
||||
await db.table('par2_files').insert({ set_id: setId, file_id: h, hash16k: h, length: 1000, name: 'f.rar' })
|
||||
candidate++
|
||||
await db.table('probe_candidates').insert({ id: candidate, release_id: id, kind: 'lone', message_id: `<${candidate}@x>`, file: 'x', status: 'done' })
|
||||
await db.table('file_hashes').insert({ candidate_id: candidate, release_id: id, hash16k: h, size: 1000 })
|
||||
await db.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files, created_at) VALUES (?, ?, 1, ?::timestamptz + interval '984 microseconds')`, [setId, id, flush])
|
||||
}
|
||||
const round = await new Par2Pruner(1).round()
|
||||
assert.equal(round.complete, 3)
|
||||
assert.equal((await db.from('par2_files').whereIn('set_id', ['p'.repeat(32), 'q'.repeat(32), 'r'.repeat(32)]).count('* as n'))[0].n, '0')
|
||||
// The pass ended: the next one starts from the first set.
|
||||
assert.equal((await db.from('worker_cursors').where('name', 'par2-prune').firstOrFail()).key, '')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,45 @@
|
||||
import { test } from '@japa/runner'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import testUtils from '@adonisjs/core/services/test_utils'
|
||||
import { Prober } from '#services/prober'
|
||||
|
||||
test.group('Prober', (group) => {
|
||||
group.each.setup(() => testUtils.db().withGlobalTransaction())
|
||||
|
||||
test('a claim takes each tier: the readable tiers also take unreadable candidates, a kind with only one of them too', async ({ assert }) => {
|
||||
const [row] = await db
|
||||
.table('releases')
|
||||
.insert({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 })
|
||||
.returning('id')
|
||||
const candidate = (kind: string, unreadable: boolean, ready = true) => ({
|
||||
release_id: row.id, kind, message_id: `<${kind}-${unreadable}-${ready}@x>`, file: kind, ready, unreadable,
|
||||
})
|
||||
// One per kind, so each tier's id range is a single id (the slices' starts aren't random then).
|
||||
await db.table('probe_candidates').multiInsert([candidate('lone', true), candidate('par2', true), candidate('archive', false), candidate('7z-tail', false, false)])
|
||||
|
||||
const prober = new Prober() as any
|
||||
await prober.claimTiers()
|
||||
assert.sameMembers(prober.queue.map((c: { kind: string }) => c.kind), ['lone', 'par2', 'archive'])
|
||||
const status = Object.fromEntries((await db.from('probe_candidates').select('kind', 'status')).map((c) => [c.kind, c.status]))
|
||||
assert.deepEqual(status, { 'lone': 'taken', 'par2': 'taken', 'archive': 'taken', '7z-tail': 'waiting' })
|
||||
})
|
||||
|
||||
test('a flush stores a name with a NUL in it, and one article that two releases have as a candidate', async ({ assert }) => {
|
||||
const release = async (key: string) =>
|
||||
Number((await db.table('releases').insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 }).returning('id'))[0].id)
|
||||
const a = await release('fa')
|
||||
const b = await release('fb')
|
||||
const candidate = async (releaseId: number) =>
|
||||
Number((await db.table('probe_candidates').insert({ release_id: releaseId, kind: 'archive', message_id: '<same@x>', file: 'x.rar', status: 'taken' }).returning('id'))[0].id)
|
||||
const inspection = { yencName: 'x.rar', size: 1000, type: null, names: ['Movie\u0000.mkv'], entries: 1, passworded: false }
|
||||
const prober = new Prober() as any
|
||||
for (const releaseId of [a, b]) {
|
||||
prober.outcomes.push({ candidate: { id: await candidate(releaseId), releaseId, kind: 'archive', messageId: '<same@x>' }, status: 'done', inspection })
|
||||
}
|
||||
await prober.flush()
|
||||
assert.deepEqual((await db.from('probe_candidates').select('status')).map((c) => c.status), ['done', 'done'])
|
||||
const cached = await db.from('probe_cache').where('message_id', '<same@x>').firstOrFail()
|
||||
assert.deepEqual(cached.result.names, ['Movie.mkv'])
|
||||
assert.equal((await db.from('release_inspections').count('* as n'))[0].n, '2')
|
||||
})
|
||||
})
|
||||
@@ -59,6 +59,8 @@ test.group('ReleaseJudge', (group) => {
|
||||
|
||||
const scraped = await release('Jamie Lee Curtis Nude Celebrity Pictures', { poster: '@_CelebMovie_@', source: 'subject' })
|
||||
const celeborn = await release('Some.Album.2010', { poster: 'celeborn@not.here (Celeborn)', source: 'subject' })
|
||||
// Scraped posters are found through posters (the batch store adds each release's).
|
||||
await db.table('posters').multiInsert([{ poster: '@_CelebMovie_@' }, { poster: 'celeborn@not.here (Celeborn)' }])
|
||||
const locked = await release('Locked.Thing', { passworded: true })
|
||||
|
||||
const unlocked = await release('Known.Password', { passworded: true })
|
||||
@@ -78,7 +80,7 @@ test.group('ReleaseJudge', (group) => {
|
||||
const spotted = await release('yy12CD34ef56GH78ij90', { readable: false, ...placed })
|
||||
await db.table('release_spots').insert({ release_id: spotted, spot_id: 's@x', files: 1 })
|
||||
|
||||
const round = await new ReleaseJudge().round()
|
||||
const round = await new ReleaseJudge().full()
|
||||
const verdicts = new Map((await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [Number(v.release_id), v.rule]))
|
||||
assert.equal(verdicts.get(dead), 'dead')
|
||||
assert.isUndefined(verdicts.get(halfDead))
|
||||
@@ -105,8 +107,88 @@ test.group('ReleaseJudge', (group) => {
|
||||
|
||||
// A password found: the next round takes the verdict back.
|
||||
await db.table('release_passwords').insert({ release_id: locked, password: 'found', source: 'nzb' })
|
||||
const again = await new ReleaseJudge().round()
|
||||
const again = await new ReleaseJudge().full()
|
||||
assert.equal(again.rules.locked.removed, 1)
|
||||
// Verdicts that stay are not written again.
|
||||
assert.equal(again.rules.repeated.added, 0)
|
||||
assert.equal(again.rules.fake.added, 0)
|
||||
assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
|
||||
})
|
||||
|
||||
test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({ assert }) => {
|
||||
let n = 0
|
||||
const settled = new Date(Date.now() - 3 * 86400_000)
|
||||
const release = async (name: string, extra: Record<string, unknown> = {}) => {
|
||||
n++
|
||||
const key = `jc${String(n).padStart(6, '0')}`.padEnd(40, '0')
|
||||
const [row] = await db
|
||||
.table('releases')
|
||||
.insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', files: 3, posted_at: settled, readable: true, settled_at: settled, ...extra })
|
||||
.returning('id')
|
||||
return Number(row.id)
|
||||
}
|
||||
const judge = new ReleaseJudge()
|
||||
const verdict = async (id: number) => (await db.from('release_verdicts').where('release_id', id).first())?.rule
|
||||
const dirty = async () => Number((await db.from('judge_dirty').count('* as n'))[0].n)
|
||||
|
||||
// New releases are noted; a passworded one is locked, until a password turns up.
|
||||
const locked = await release('Locked.Thing', { passworded: true })
|
||||
assert.isAbove(await dirty(), 0)
|
||||
let round = await judge.changed()
|
||||
assert.equal(await verdict(locked), 'locked')
|
||||
assert.equal(round.rules.locked.added, 1)
|
||||
assert.equal(await dirty(), 0)
|
||||
await db.table('release_passwords').insert({ release_id: locked, password: 'pw', source: 'nzb' })
|
||||
round = await judge.changed()
|
||||
assert.isUndefined(await verdict(locked))
|
||||
assert.equal(round.rules.locked.removed, 1)
|
||||
|
||||
// Probes: a release whose probed articles are all missing is dead once the prober's statuses come in.
|
||||
const dead = await release('Gone.Movie.2020.1080p')
|
||||
const ids = await db.table('probe_candidates').multiInsert([
|
||||
{ release_id: dead, kind: 'lone', message_id: '<jc1@x>', file: 'a', status: 'waiting' },
|
||||
{ release_id: dead, kind: 'lone', message_id: '<jc2@x>', file: 'b', status: 'waiting' },
|
||||
]).returning('id')
|
||||
await judge.changed()
|
||||
assert.isUndefined(await verdict(dead))
|
||||
await db.from('probe_candidates').whereIn('id', ids.map((r: { id: string }) => r.id)).update({ status: 'missing' })
|
||||
await judge.changed()
|
||||
assert.equal(await verdict(dead), 'dead')
|
||||
|
||||
// One file in 19 releases, one of them locked: the 20th makes them all repeated, the locked one too (repeated
|
||||
// comes first), though only the 20th changed.
|
||||
const hash = 'b'.repeat(32)
|
||||
const holders: number[] = []
|
||||
for (let i = 0; i < 19; i++) holders.push(await release(`holder${i}`, i === 0 ? { passworded: true } : {}))
|
||||
for (const [i, id] of holders.entries()) await db.table('file_hashes').insert({ candidate_id: 30000 + i, release_id: id, hash16k: hash, size: 5000 })
|
||||
await judge.changed()
|
||||
assert.equal(await verdict(holders[0]), 'locked')
|
||||
assert.isUndefined(await verdict(holders[1]))
|
||||
const twentieth = await release('holder19')
|
||||
await db.table('file_hashes').insert({ candidate_id: 30019, release_id: twentieth, hash16k: hash, size: 5000 })
|
||||
round = await judge.changed()
|
||||
assert.equal(round.releases, 20)
|
||||
const rules = new Set()
|
||||
for (const id of [...holders, twentieth]) rules.add(await verdict(id))
|
||||
assert.deepEqual([...rules], ['repeated'])
|
||||
// A 21st: the file was in 20 releases without it, so only it is judged.
|
||||
const twentyFirst = await release('holder20')
|
||||
await db.table('file_hashes').insert({ candidate_id: 30020, release_id: twentyFirst, hash16k: hash, size: 5000 })
|
||||
round = await judge.changed()
|
||||
assert.equal(round.releases, 1)
|
||||
assert.equal(await verdict(twentyFirst), 'repeated')
|
||||
|
||||
// A verdict nothing noted stays until the full round; it's due a day after the last one.
|
||||
const stale = await release('Fine.Release')
|
||||
await judge.changed()
|
||||
await db.table('release_verdicts').insert({ release_id: stale, rule: 'fake', rule_version: 1 })
|
||||
await judge.changed()
|
||||
assert.equal(await verdict(stale), 'fake')
|
||||
await db.from('judge_state').update({ full_at: new Date(Date.now() - 2 * 86400_000) })
|
||||
assert.isTrue(await judge.fullDue())
|
||||
round = await judge.full()
|
||||
assert.isUndefined(await verdict(stale))
|
||||
assert.equal(await verdict(dead), 'dead')
|
||||
assert.isFalse(await judge.fullDue())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,30 @@
|
||||
import { test } from '@japa/runner'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
import testUtils from '@adonisjs/core/services/test_utils'
|
||||
import { pruneSevenZipTails } from '#services/seven_zip_tails'
|
||||
|
||||
test.group('7z tails', (group) => {
|
||||
group.each.setup(() => testUtils.db().withGlobalTransaction())
|
||||
|
||||
test("of a release's waiting tails, those below its highest volume go; the highest volume's all stay", async ({ assert }) => {
|
||||
const row = (key: string) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', files: 4 })
|
||||
const [merged, other] = (await db.table('releases').multiInsert([row('a'), row('b')]).returning('id')).map((r) => r.id)
|
||||
const tail = (releaseId: number, volume: string, mid: string, status = 'waiting') => ({ release_id: releaseId, kind: '7z-tail', message_id: mid, file: `x.7z.${volume}`, status })
|
||||
await db.table('probe_candidates').multiInsert([
|
||||
// Two merged members' tails, one of the highest volume probed already.
|
||||
tail(merged, '002', '<1@x>'),
|
||||
tail(merged, '005', '<2@x>'),
|
||||
tail(merged, '003', '<3@x>', 'done'),
|
||||
tail(merged, '009', '<4@x>', 'taken'),
|
||||
// Two pieces that each saw part of volume 009.
|
||||
tail(merged, '009', '<5@x>'),
|
||||
{ release_id: merged, kind: 'archive', message_id: '<6@x>', file: 'x.7z.001' },
|
||||
// Another release: not asked for.
|
||||
tail(other, '002', '<7@x>'),
|
||||
tail(other, '004', '<8@x>'),
|
||||
])
|
||||
assert.equal(await pruneSevenZipTails(db, [merged]), 2)
|
||||
const left = await db.from('probe_candidates').orderBy('message_id').select('message_id')
|
||||
assert.deepEqual(left.map((c) => c.message_id), ['<3@x>', '<4@x>', '<5@x>', '<6@x>', '<7@x>', '<8@x>'])
|
||||
})
|
||||
})
|
||||
@@ -103,6 +103,8 @@ test.group('Spots', (group) => {
|
||||
const release = await db.from('releases').where('source', 'spot').firstOrFail()
|
||||
assert.equal(release.name, 'Lonely File')
|
||||
assert.equal(release.files, 1)
|
||||
assert.isNotNull(release.poster)
|
||||
assert.isNotNull(await db.from('posters').where('poster', release.poster).first())
|
||||
const { default: NzbService } = await import('#services/nzb_service')
|
||||
assert.include((await NzbService.build(Number(release.id)))!.xml, 'lonely.file.mkv')
|
||||
// Not again while it has one; once the header pass builds the file's release, the spot moves there.
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { test } from '@japa/runner'
|
||||
import { DiskSpace } from '#services/disk_space'
|
||||
|
||||
test.group('DiskSpace', () => {
|
||||
test('short below the floor, fine again a margin above it; off without a floor', async ({ assert }) => {
|
||||
let gb = 15
|
||||
const space = new DiskSpace('probing', '/', 12, async () => gb * 1e9, 0)
|
||||
assert.isFalse(await space.short())
|
||||
gb = 11.5
|
||||
assert.isTrue(await space.short())
|
||||
// Back over the floor, not yet over the margin: still short.
|
||||
gb = 12.5
|
||||
assert.isTrue(await space.short())
|
||||
gb = 13.1
|
||||
assert.isFalse(await space.short())
|
||||
assert.isFalse(await new DiskSpace('probing', '/', 0, async () => 0, 0).short())
|
||||
})
|
||||
|
||||
test('measures a real filesystem', async ({ assert }) => {
|
||||
assert.isFalse(await new DiskSpace('probing', '/', 0.001).short())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,69 @@
|
||||
import { test } from '@japa/runner'
|
||||
import net from 'node:net'
|
||||
import env from '#start/env'
|
||||
import coordinatorConfig from '#config/coordinator'
|
||||
import pool from '#services/nntp_pool'
|
||||
import type { NntpShare } from '#services/nntp_shares'
|
||||
|
||||
/** A tiny NNTP server that only logs in, counting the connections open. */
|
||||
function server(): Promise<{ port: number; open: () => number; close: () => void }> {
|
||||
return new Promise((resolve) => {
|
||||
let open = 0
|
||||
const srv = net.createServer((sock) => {
|
||||
open++
|
||||
sock.on('close', () => open--)
|
||||
sock.write('200 welcome\r\n')
|
||||
sock.on('data', (chunk) => {
|
||||
for (const line of chunk.toString().split('\r\n')) {
|
||||
if (line.startsWith('AUTHINFO USER')) sock.write('381 more\r\n')
|
||||
else if (line.startsWith('AUTHINFO PASS')) sock.write('281 ok\r\n')
|
||||
else if (line === 'XFEATURE COMPRESS GZIP') sock.write('290 Feature Enabled\r\n')
|
||||
else if (line === 'QUIT') sock.end('205 bye\r\n')
|
||||
}
|
||||
})
|
||||
})
|
||||
srv.listen(0, '127.0.0.1', () =>
|
||||
resolve({
|
||||
port: (srv.address() as net.AddressInfo).port,
|
||||
open: () => open,
|
||||
close: () => srv.close(),
|
||||
})
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
|
||||
const share = { sync: async () => 4 } as unknown as NntpShare
|
||||
|
||||
test.group('nntp / pool', () => {
|
||||
test('a connection free for nntpIdleCloseMs closes on the next sync; one used since stays', async ({
|
||||
assert,
|
||||
cleanup,
|
||||
}) => {
|
||||
const { port, open, close } = await server()
|
||||
const idleCloseMs = coordinatorConfig.nntpIdleCloseMs
|
||||
env.set('NNTP_HOST', '127.0.0.1')
|
||||
env.set('NNTP_PORT', port)
|
||||
env.set('NNTP_SECURE', false)
|
||||
coordinatorConfig.nntpIdleCloseMs = 100
|
||||
cleanup(async () => {
|
||||
coordinatorConfig.nntpIdleCloseMs = idleCloseMs
|
||||
await pool.close()
|
||||
close()
|
||||
})
|
||||
await pool.share(share)
|
||||
// Two at once: two connections.
|
||||
await Promise.all([pool.use(() => sleep(20)), pool.use(() => sleep(20))])
|
||||
assert.equal(open(), 2)
|
||||
await sleep(150)
|
||||
// One used again: the other has been free for longer than 100 ms.
|
||||
await pool.use(async () => {})
|
||||
await pool.share(share)
|
||||
await sleep(50)
|
||||
assert.equal(open(), 1)
|
||||
await sleep(150)
|
||||
await pool.share(share)
|
||||
await sleep(50)
|
||||
assert.equal(open(), 0)
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user