daan/usenet-indexer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: daan/usenet-indexer:37750fa4b3c680ba715fef0b62ef3b6d8246ff40
daan/usenet-indexer:master daan/usenet-indexer:rebuild daan/usenet-indexer:voyager-3 daan/usenet-indexer:monetization daan/usenet-indexer:experiment/adonisjs
..
compare: daan/usenet-indexer:monetization
daan/usenet-indexer:rebuild daan/usenet-indexer:voyager-3 daan/usenet-indexer:monetization daan/usenet-indexer:master daan/usenet-indexer:experiment/adonisjs
1 Commits
37750fa4b3 .. monetization
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 1322e42fb7 Accounts with limits: a Newznab key and a tier per user, daily API and NZB counts, invites
Test and deploy / test (push) Successful in 47s
Details
Test and deploy / edge (push) Successful in 1m44s
Details
Test and deploy / deploy (push) Skipped
Details 
ACCESS=accounts (config/access.ts; open stays the default and behaves as
before): Newznab and the catalog's releases routes name a user (apikey or
the web interface's token), whose tier sets searches and NZBs per UTC day
(Newznab 500/501, the catalog 429) and how far back they see (102/403).
Counted in one upsert per use, so the last one can't be taken twice;
caps, stats and refused requests aren't counted. Signup takes an invite
(SIGNUP), which may give a tier; ace access:invite, access:tier (extends a
running tier), access:suspend. The web interface signs in and shows the
day's usage and the Newznab key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 00:16:45 +02:00
115 changed files with 1376 additions and 8681 deletions
Expand all files Collapse all files

No files matched your search

.env.example
+5
View File
@@ -19,6 +19,11 @@ DB_USER=usenet_indexer
DB_PASSWORD=
DB_DATABASE=usenet_rebuild
# Access (config/access.ts): open (NEWZNAB_API_KEY or none, no limits) or accounts
# (a Newznab key per user, tiers with daily limits; ace access:invite, access:tier, access:suspend)
# ACCESS=open
# SIGNUP=open|invite|closed (with accounts: invite unless set)
#--------------------------------------------------------------------
# CORS (configure allowed origins for API access)
#--------------------------------------------------------------------
REBUILD.md
+11 -183
View File
@@ -5,8 +5,7 @@ rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its
decisions D1–D42 are superseded; its domain model still holds, and its
features not yet here are the phase 2 backlog below.
Branch `rebuild`; its successor is designed under *Voyager 3* below (branch
`voyager-3`). Phase 1 finds releases from headers and makes their NZBs.
Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
Everything learned from articles (names from yEnc headers, par2, archive
listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
@@ -190,22 +189,13 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
number; merged when the neighbouring range is done, then written as a
chunk. Never staged per segment.
8. **Settling**. A release is settled when nothing was added to it for 6
hours (30 minutes once its last post is that old: history), and its
completeness decided: complete when every file its [i/N] counter
announced was found. Without a counter, by its par2 sets once their index
is probed (`par2_completeness`): every file a set describes is there by
name or by its first 16 KiB, or, renamed, as many files and bytes; decided
again after a par2 merge and before the pruner drops a set's file list.
Segments aren't compared. Phase 2 picks its work from settled releases.
8. **Settling**. A release is settled when the ranges covering its posting
span are done and no chunk arrived for a day: completeness is computed
then (expected vs found files and segments). Phase 2 picks its work from
settled releases.
9. **NZB and API**. NZB built from a release's chunks, with a known
password in its meta; Newznab search (names, inner names) and get; a web
interface. Categories (Newznab's) by a trigger on releases (migration
5200): from the name, else the inner name the prober found, else a spot's
category (spot_category, set by the spot inspector), else the groups
(erotica, sounds, ebooks, …). `releases:recategorize` recomputes old rows
in batches.
9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
search (names) and get. Web UI later.
## Group list
@@ -231,36 +221,6 @@ range with the deployed build's edge-bench (`replay`, ~seconds a range) and
reports suspects and the change since the previous build (`report`). A rule
change is judged on the whole corpus before a reset.
Since 2026-10-23 also history fetched ahead of the indexing: while the
catalog is over its NVMe cap, ranges marked `capture` (priority >= 0, not
leased) are fetched into the corpus by idle loops, at most perGroup /
perRange of a group at a time, to be indexed from disk once there's room
(LOG.md 2026-10-23 (100)). Stored filtered (`CORPUS_FILTER`, default on):
without the lines the edge worker only counts, decided per line as it does
(a crosspost another indexed group owns, a subject that names no file part;
~48% of headers), the chunk compressed again as zlib; beside it a `.drops`
(gzip JSON: rule version, the owners' priorities, per rule the article
numbers as differences, crossposts by owner) and `filtered` in range.json.
The worker makes the same releases of a filtered chunk; rule_counts lack
those two rules' counts. `corpus:filter` filters chunks stored raw. A
corpus filtered this way no longer serves a sweep of those two rules.
Windows by posting time (2026-10-25; voyager 3's test data, LOG.md (104)):
`ranges:dates` samples when each active group's articles were posted (one
XOVER of 32 articles every 500M articles of the group, 16 … 1000 samples,
the median Date of each slice, kept monotone; `group_dates`) and estimates
every range's posting date from the curve (`ranges.est_date`; the edges of
a window are off by up to one sample interval, three weeks in boneless).
`ranges:capture --from 2025-01-01 --to 2026-01-01` marks every active
group's ranges posted in the window `capture`: planned ones not started at
`--priority` (0), done ones as they are, both fetched into the corpus by the
fill, which takes capture ranges in priority order, done ones too (they're
fetched again, corpus only), and skips those complete (`ranges.stored_at`,
set when it sees the range.json). A year of every group is the data to
replay voyager 3's 30-day blocks on: marked 2026-10-25, 2016 (priority 0,
30B articles, ~0.56 TB) and 2021 (priority 1, 87B, ~1.6 TB); a recent year
is ~200B articles, ~3.6 TB.
## Phase 2
Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It
@@ -278,21 +238,6 @@ files the set describes, into the one with the most files (chunks,
candidates, inspections, counts; the par2's shared name when none was
readable); `release_merges` keeps what went where. Obfuscated uploads whose
files were each a release of their own (`lone` candidates) become one.
Readable files have no hash: a name pass (2026-10-23) joins the releases
of the set's poster in its group posted within 12 hours whose files the
set all names (no case or punctuation; sidecars, par2 and images aside),
as `via = 'name'`: an album's tracks and its index, an index beside its
differently spelled video. `releases:merge-names` for sets from before.
Also merged: lone files whose inner names share a random stem (`inner`),
and small sidecar-only releases into their upload (`sidecar`).
It reads what's new an hour (`created_at`) at a time, its place kept after
each, and merges only the sets held or matched by two releases or more; a
file whose first 16 KiB and size more than 20 files share is a common one and
matches nothing (LOG.md 2026-10-20 (85)).
A set's file list is dropped once it can't merge anything more: complete (all
its files in its holder), judged (every holder junk), or final (every holder
inside a run of done ranges, those ranges' candidates probed, its hashes seen
by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).
**Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
@@ -307,7 +252,7 @@ with 68,032 releases with a password):
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
| 7z listings | D19 | yes (2026-10-23): the prober reads the `7z-tail` (the last segment of the highest volume, or of an unsplit multi-segment .7z) and `lib/sevenzip` lists it from the archive's end header (LZMA-packed, decoded in TypeScript); an encrypted header (`-mhe`) means passworded without names. One release keeps only the tails of its highest volume (`seven_zip_tails`). ZIP: the first segment's local headers (first entries, encryption) |
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
| Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
| Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |
@@ -315,11 +260,9 @@ Live indexing (the tail) runs since 2026-10-09.
**Junk** (voyager 1's second filter, D42). After the header pass's rules
(garbage signatures, executables, per-article and obfuscated subjects), the
judge (`releases:judge`, service `judge`) judges releases by what the
prober, the merges and the spots found: every 2 minutes the releases whose
facts changed (noted by triggers in `judge_dirty`), daily every release (for
what time changes: a day after settling, ranges read beside it). In this
order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
judge (`releases:judge`, service `judge`) judges releases hourly by what the
prober, the merges and the spots found, in this order (the first rule
wins): *dead* (every probed first article gone), *fake* (a film, episode
or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
@@ -369,106 +312,6 @@ processes, one NNTP connection each:
`spot-text`, and the spot), not on the catalog row; the catalog only says
`passworded`.
## Voyager 3: window indexing (design, 2026-10-25)
Branch `voyager-3` (a push to it only runs the tests; `rebuild` keeps
running on homelab until the reset). Goal: the whole history (~1,220 B
articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe
and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today
(144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is
≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources:
LOG.md 2026-10-24 (102).
What is wrong today: a release is written as soon as one range sees it,
updated by every later range that finds more parts, merged afterwards
(fragments, par2, names) and settled by wall-clock quiet time, which means
little in a backfill. That churn is most of the WAL (0.6 TB a day) and the
20 k write IOPS bursts, and leaves bloat. Half the rows are single-article
releases, a third of them one poster's image dump (27.7 M rows, LOG.md
(102)).
1. **Blocks.** A group's history is still fetched as ranges of 2 M
articles in chunks of 100 k (the spool, the corpus, the prefetcher stay),
but it is *leased* as blocks: consecutive ranges covering at least 30
days of posting time (`ranges.posted_low/high`), so the window below is
small against a block (≤ 3% of a block's releases touch an edge). A
small group is one block; boneless has a block a week or so. Blocks of
one group run in parallel on different workers, in any order.
2. **The window.** The worker reads a block in article order and keeps its
open files and releases in memory (counters, names, sizes; the segment
lists spooled to a file per block on the worker's disk). The *frontier*
is the median Date of the last 4,096 articles with a plausible date
(inside the block's span ± 7 days); Date headers alone aren't trusted
(clock skew, bogus years). A release is **finished** when the frontier
is 24 h past its last part (`WINDOW_HOURS`), or at the block's end:
written once, with its completeness decided then (every announced file
there; par2 decides later for those without a counter), never updated by
the header pass again. Incomplete releases are kept (user, 2026-10-24).
A fill posted after the window is a release of its own (the judge's
`repeated` rule already knows reposts). Releases still open at a block's
end are written flagged `edge`; the block merger (replaces the fragment
merger) joins the edge releases of adjacent blocks of a group by
identity key once both are done, and clears the flag.
The settler goes: `settled_at` is the finish. The **tail** keeps its
latency: a tail lease emits a release provisionally as today and the
window finalizes it once (two writes, not one per range).
3. **Albums.** Files of one poster in one owner group whose subjects differ
only by a number sequence, or share a directory-like prefix, posted
within the window, become one release (`source = 'album'`): its name the
shared template (or the prefix), its files kept by name so each stays
findable; at least two files, else the single stays a release. This is
for subjects that name no release (today's `post`/`stem` singles); a
poster's ten different cracks in a day stay ten releases.
4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` →
`posters(id)` (the table gets an id; 33 B → 4), `group_set_id` →
`group_sets` (the ~2,600 distinct group combinations; 85 B jsonb → 4),
`release_key` and `identity_key` as 20-byte `bytea` (41 B each → 20),
`source` a smallint. ~150 of ~430 B a row; the poster and key indexes
shrink with them (~12 GB of 50 today); `releases_category_newest_idx`
(5.3 GB, used once) goes, category browsing filters the newest index.
Today's 85 GB would be ~55. Cold tables move to the HDD tablespace:
`release_inspections` and `probe_candidates` (28 GB, written once, not
read by search); `probe_cache` is there already.
5. **Memory and disk at the edge.** The busiest group posts ~50 M articles
a day; at a file per ~50 articles that is ~1 M open files of ~100 B in
memory per block, and their segment lists (~25 B each) in the block's
spool file, read back when a release is finished. Per worker thread a
few hundred MB at worst.
6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
(30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
those 73% are random 32-hex tokens, 27% `[PRiVATE]` hashes, ~0.3% posts
with a readable name and no counter. The filter (crosspost, counterless)
loses nothing voyager 3 would index; albums work from filtered chunks
(the image dump's subjects carry "(1/1)"). Indexing without a counter at
all (obfuscated uploads, ~31% of all lines) would need bodies, and the
raw corpus (2,328 ranges) for a sweep.
7. **Where it lives.** The window, blocks and albums are in the Rust edge
(`edge/`), the batch store, the planner and leases; the TypeScript
RangeIndexer keeps the scanner (the corpus filter, tests) and drops its
parity with the Rust worker for the window (the `check.ts` comparison
covered the range path, which goes). New migrations rebuild the schema
from a reset, no in-place conversion of the 81 M rows.
8. **Test before the reset.** Replay the corpus (the raw ranges and the
captures, several B articles) with the new edge into a scratch database
on bulk: NVMe bytes per article (the one number that decides whether
4 TB is enough), the share of albums and incomplete releases, and a
sample of releases against today's. Speed is measured after the reset,
on the NVMe.
9. **The reset.** Tail first, then history by priority, read from the
corpus where stored. Survives: `probe_cache` (HDD, by message-id),
`spots`, `groups` and their priorities, `nntp_shares`, the corpus.
Recomputed: releases, chunks, inspections, candidates, par2 data,
verdicts, file hashes. Resets are paused (2026-10-12); this one waits
for the user's go-ahead, and for the new NVMe.
## Decisions
a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
@@ -483,18 +326,3 @@ f. Random-named releases (no file with a readable name) keep no chunk in
phase 2 names them (bodies) and fetches their segments again. The full
history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
g. The account's NNTP connections shared by priority (`nntp_shares`): the
coordinator (headers) before the prober, each between its minimum and
maximum, 5 kept for the spotter, the spot inspector, the plain
connection and reconnects. A process grows only into connections the
others closed, so the account never goes over its limit. Replaces the
fixed split (38 + 8), under which the prober stayed at 8 while history
was paused and the coordinator's connections idle (LOG.md 2026-10-18
(75)). **Accepted** 2026-10-18.
h. Idle NNTP connections closed: one with nothing to do for
`NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
(the coordinator's pool on its next sync, the plain one too; a dry
prober lane; the spotter between rounds; the spot inspector with no
spots). Eweka dropped the quiet ones and the next command failed
("Spot inspection failed: NNTP connection closed"), and idle ones took
the account's connections from downloads. **Accepted** 2026-10-22.
app/controllers/new_account_controller.ts
+26 -3
View File
@@ -1,13 +1,36 @@
import User from '#models/user'
import db from '@adonisjs/lucid/services/db'
import { signupValidator } from '#validators/user'
import type { HttpContext } from '@adonisjs/core/http'
import UserTransformer from '#transformers/user_transformer'
import AccessService from '#services/access_service'
class InviteGone extends Error {}
export default class NewAccountController {
async store({ request, serialize }: HttpContext) {
const { fullName, email, password } = await request.validateUsing(signupValidator)
async store({ request, response, serialize }: HttpContext) {
const { fullName, email, password, invite } = await request.validateUsing(signupValidator)
if (AccessService.signup === 'closed') return response.forbidden({ error: 'Signup is closed' })
if (AccessService.signup === 'invite' && !invite) return response.forbidden({ error: 'An invite code is needed to sign up' })
const user = await User.create({ fullName, email, password })
// The user and their invite together: an invite that's gone rolls the signup back.
const user = await db
.transaction(async (trx) => {
const created = await User.create({ fullName, email, password }, { client: trx })
if (AccessService.signup !== 'invite') return created
const gives = await AccessService.redeem(invite!, created.id, trx)
if (!gives) throw new InviteGone()
if (gives.tier) {
AccessService.grantTier(created, gives.tier, gives.tier_days)
await created.useTransaction(trx).save()
}
return created
})
.catch((error) => {
if (error instanceof InviteGone) return null
throw error
})
if (!user) return response.forbidden({ error: 'Unknown or used invite code' })
const token = await User.accessTokens.create(user)
return serialize({
app/controllers/newznab_controller.ts
+38 -13
View File
@@ -2,6 +2,7 @@ import type { HttpContext } from '@adonisjs/core/http'
import env from '#start/env'
import NzbService from '#services/nzb_service'
import SearchService from '#services/search_service'
import AccessService, { AccessDenied } from '#services/access_service'
import { capsXml, errorXml, expandCategories, resultsXml, type NewznabServer } from '#lib/newznab/index'
const LIMITS = { max: 100, default: 50 }
@@ -35,22 +36,37 @@ function int(value: unknown): number | undefined {
* Releases with readable names (or an inner name from the prober, which then is the title) that aren't known to miss files.
*/
export default class NewznabController {
async handle({ request, response }: HttpContext) {
async handle(ctx: HttpContext) {
const xml = (body: string, type = 'application/xml') => {
ctx.response.header('Content-Type', `${type}; charset=utf-8`)
return ctx.response.send(body)
}
try {
return await this.answer(ctx, xml)
} catch (error) {
if (error instanceof AccessDenied) return xml(errorXml(error.newznabCode, error.message))
throw error
}
}
private async answer(ctx: HttpContext, xml: (body: string, type?: string) => void) {
const { request, response } = ctx
const qs = request.qs()
const t = String(qs.t ?? '')
const server = this.server(request)
const xml = (body: string, type = 'application/xml') => {
response.header('Content-Type', `${type}; charset=utf-8`)
return response.send(body)
}
const apiKey = configuredApiKey()
if (apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
if (t === 'caps') return xml(capsXml(server, LIMITS))
// With accounts the key is a user's (AccessService); else the one configured, if any.
const grant = await AccessService.authorize(ctx, qs.apikey)
const apiKey = grant ? grant.user.apiKey : configuredApiKey()
if (!grant && apiKey && qs.apikey !== apiKey) return xml(errorXml(100, 'Incorrect user credentials'))
const server = this.server(request, apiKey)
if (t === 'caps') return xml(capsXml(server, LIMITS, { available: AccessService.accounts && AccessService.signup !== 'closed', open: AccessService.accounts && AccessService.signup === 'open' }))
if (t === 'get') {
const id = int(qs.id)
if (id === undefined) return xml(errorXml(200, 'Missing parameter (id)'))
// Unknown ids and releases past the tier's retention aren't counted.
if (!(await AccessService.checkRelease(grant, id))) return xml(errorXml(300, 'No such item'))
await AccessService.charge(grant, 'grab')
const nzb = await NzbService.build(id)
if (!nzb) return xml(errorXml(300, 'No such item'))
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
@@ -59,6 +75,7 @@ export default class NewznabController {
}
if (!(t in SEARCH_FUNCTIONS)) return xml(errorXml(202, `No such function (${t || 'none'})`))
await AccessService.charge(grant, 'api')
const requested = String(qs.cat ?? '')
.split(',')
@@ -79,7 +96,7 @@ export default class NewznabController {
categories,
readable: true,
notIncomplete: true,
maxAgeDays: int(qs.maxage),
maxAgeDays: AccessService.maxAgeDays(grant, int(qs.maxage)),
season: int(qs.season),
episode: int(qs.season) !== undefined ? int(qs.ep) : undefined,
sort: 'newest',
@@ -105,12 +122,20 @@ export default class NewznabController {
episode: tv ? `E${tv[2].padStart(2, '0')}` : undefined,
}
})
return xml(resultsXml(server, items, offset, result?.total ?? 0), 'application/rss+xml')
// What's left of the day, for clients that show it (newznab:apilimits).
const limits = grant && {
apiCurrent: grant.usage.apiHits,
apiMax: grant.limits.apiHits,
grabCurrent: grant.usage.grabs,
grabMax: grant.limits.grabs,
}
return xml(resultsXml(server, items, offset, result?.total ?? 0, limits || undefined), 'application/rss+xml')
}
private server(request: HttpContext['request']): NewznabServer {
/** Download links carry the caller's key. */
private server(request: HttpContext['request'], apiKey: string | undefined): NewznabServer {
const proto = request.header('x-forwarded-proto') ?? 'http'
const host = request.header('x-forwarded-host') ?? request.header('host') ?? 'localhost'
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey: configuredApiKey() }
return { title: 'Voyager 2', baseUrl: `${proto}://${host}`, apiKey }
}
}
app/controllers/profile_controller.ts
+12 -1
View File
@@ -1,8 +1,19 @@
import UserTransformer from '#transformers/user_transformer'
import type { HttpContext } from '@adonisjs/core/http'
import AccessService from '#services/access_service'
export default class ProfileController {
/** The user, their tier, Newznab key and what they used today. */
async show({ auth, serialize }: HttpContext) {
return serialize(UserTransformer.transform(auth.use('api').getUserOrFail()))
const user = auth.use('api').getUserOrFail()
return serialize({ user: UserTransformer.transform(user), access: await AccessService.summary(user) })
}
/** POST /api/v1/account/api-key: a new Newznab key (the old one stops working). */
async resetApiKey({ auth }: HttpContext) {
const user = auth.use('api').getUserOrFail()
user.apiKey = AccessService.newKey()
await user.save()
return { apiKey: user.apiKey }
}
}
app/controllers/releases_controller.ts
+24 -8
View File
@@ -1,5 +1,6 @@
import type { HttpContext } from '@adonisjs/core/http'
import NzbService from '#services/nzb_service'
import AccessService from '#services/access_service'
import SearchService, { type SortOrder } from '#services/search_service'
import { expandCategories } from '#lib/newznab/index'
@@ -14,13 +15,16 @@ function megabytes(value: unknown): number | undefined {
/** The catalog for the web interface (REBUILD.md, part 9). */
export default class ReleasesController {
/** GET /api/v1/releases?q=&cat=5000,2040&readable=1&complete=1&unlocked=1&min_size=&max_size=(MB)&source=&poster=&sort=&page=&per_page= */
async index({ request }: HttpContext) {
const qs = request.qs()
async index(ctx: HttpContext) {
const qs = ctx.request.qs()
const grant = await AccessService.authorize(ctx, qs.apikey)
await AccessService.charge(grant, 'api')
const categories = String(qs.cat ?? '')
.split(',')
.map((c) => Number(c.trim()))
.filter((c) => Number.isInteger(c) && c > 0)
return SearchService.search({
maxAgeDays: AccessService.maxAgeDays(grant),
q: typeof qs.q === 'string' ? qs.q : undefined,
categories: categories.length ? expandCategories(categories) : undefined,
readable: flag(qs.readable),
@@ -39,13 +43,20 @@ export default class ReleasesController {
}
/** GET /api/v1/releases/:id, with its files */
async show({ params, response }: HttpContext) {
const release = await SearchService.release(Number(params.id))
return release ?? response.notFound({ error: 'Release not found' })
async show(ctx: HttpContext) {
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
if (!(await AccessService.checkRelease(grant, Number(ctx.params.id)))) return ctx.response.notFound({ error: 'Release not found' })
await AccessService.charge(grant, 'api')
const release = await SearchService.release(Number(ctx.params.id))
return release ?? ctx.response.notFound({ error: 'Release not found' })
}
/** GET /api/v1/releases/:id/nzb */
async nzb({ params, response }: HttpContext) {
/** GET /api/v1/releases/:id/nzb (?apikey= for a plain link, as the web interface's) */
async nzb(ctx: HttpContext) {
const { params, response } = ctx
const grant = await AccessService.authorize(ctx, ctx.request.qs().apikey)
if (!(await AccessService.checkRelease(grant, Number(params.id)))) return response.notFound({ error: 'Release not found' })
await AccessService.charge(grant, 'grab')
const nzb = await NzbService.build(Number(params.id))
if (!nzb) return response.notFound({ error: 'Release not found' })
const filename = `${nzb.name.replace(/[^\w.\-()[\] ]+/g, '_').slice(0, 180) || 'release'}.nzb`
@@ -54,7 +65,12 @@ export default class ReleasesController {
return response.send(nzb.xml)
}
/** GET /api/v1/stats */
/** GET /api/v1/access: whether the web interface needs an account, and how one is made. */
async access() {
return { mode: AccessService.mode, signup: AccessService.signup }
}
/** GET /api/v1/stats (open: what the index holds, not what's in it) */
async stats() {
return SearchService.stats()
}
app/exceptions/handler.ts
+3
View File
@@ -8,6 +8,9 @@ export default class HttpExceptionHandler extends ExceptionHandler {
*/
protected debug = !app.inProduction
/** A refused request (AccessService: no key, a limit reached) is an answer, not a fault. */
protected ignoreCodes = ['E_ACCESS_DENIED']
/**
* The method is used for handling errors and returning
* response to the client
app/models/user.ts
+8
View File
@@ -1,4 +1,6 @@
import { randomBytes } from 'node:crypto'
import { UserSchema } from '#database/schema'
import { beforeCreate } from '@adonisjs/lucid/orm'
import hash from '@adonisjs/core/services/hash'
import { compose } from '@adonisjs/core/helpers'
import { withAuthFinder } from '@adonisjs/auth/mixins/lucid'
@@ -8,6 +10,12 @@ export default class User extends compose(UserSchema, withAuthFinder(hash)) {
static accessTokens = DbAccessTokensProvider.forModel(User)
declare currentAccessToken?: AccessToken
/** The Newznab key (AccessService), known on the model right away rather than from the column's default. */
@beforeCreate()
static assignApiKey(user: User) {
user.apiKey ??= randomBytes(16).toString('hex')
}
get initials() {
const [first, last] = this.fullName ? this.fullName.split(' ') : this.email.split('@')
if (first && last) {
app/services/access_service.ts
+171
View File
@@ -0,0 +1,171 @@
import { randomBytes } from 'node:crypto'
import { DateTime } from 'luxon'
import db from '@adonisjs/lucid/services/db'
import { Exception } from '@adonisjs/core/exceptions'
import type { HttpContext } from '@adonisjs/core/http'
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
import User from '#models/user'
import { ACCESS, SIGNUP, TIERS, type TierLimits } from '#config/access'
export type Use = 'api' | 'grab'
/** Newznab's error codes for access (the catalog answers with the HTTP status). */
const STATUS = { 100: 401, 101: 403, 102: 403, 500: 429, 501: 429 } as const
/** Why a request is refused; the catalog's answer (Newznab catches it for its own). */
export class AccessDenied extends Exception {
constructor(
readonly newznabCode: keyof typeof STATUS,
message: string
) {
super(message, { status: STATUS[newznabCode], code: 'E_ACCESS_DENIED' })
}
async handle(error: this, ctx: HttpContext) {
ctx.response.status(error.status).send({ error: error.message, code: error.newznabCode })
}
}
export interface Usage {
apiHits: number
grabs: number
}
/** A caller let in: who, their tier as it is now, and what they used today. */
export interface Grant {
user: User
tier: string
limits: TierLimits
usage: Usage
}
const COLUMNS: Record<Use, string> = { api: 'api_hits', grab: 'grabs' }
const TODAY = `(now() AT TIME ZONE 'UTC')::date`
const DAY_MS = 86_400_000
/** Accounts, tiers and their daily limits (config/access.ts). */
export default class AccessService {
/** From config/access.ts; tests switch it. */
static mode: 'open' | 'accounts' = ACCESS
static signup: 'open' | 'invite' | 'closed' = SIGNUP
static get accounts() {
return this.mode === 'accounts'
}
static newKey() {
return randomBytes(16).toString('hex')
}
/** The user's tier as it is now: one that ran out (or isn't configured) is free. */
static tierOf(user: User): { tier: string; limits: TierLimits } {
const expired = user.tierUntil !== null && user.tierUntil.toMillis() <= Date.now()
const tier = !expired && TIERS[user.tier] ? user.tier : 'free'
return { tier, limits: TIERS[tier] }
}
/** The user a request names: the web interface's token, else an apikey (Newznab, the catalog's NZB links). */
static async caller(ctx: HttpContext, apikey?: unknown): Promise<User | null> {
const guard = ctx.auth.use('api')
if (guard.isAuthenticated || (await guard.check())) return guard.user!
if (typeof apikey === 'string' && /^[0-9a-f]{32}$/.test(apikey)) return User.findBy('apiKey', apikey)
return null
}
/** Lets a request in (counting nothing yet); null when access is open. */
static async authorize(ctx: HttpContext, apikey?: unknown): Promise<Grant | null> {
if (!this.accounts) return null
const user = await this.caller(ctx, apikey)
if (!user) throw new AccessDenied(100, 'Incorrect user credentials')
if (user.suspended) throw new AccessDenied(101, 'Account suspended')
return { user, ...this.tierOf(user), usage: await this.usage(user.id) }
}
/** Counts one use against the grant's tier, or refuses it once the day's limit is reached. */
static async charge(grant: Grant | null, use: Use) {
if (!grant) return
const usage = await this.count(grant.user.id, use, use === 'api' ? grant.limits.apiHits : grant.limits.grabs)
if (!usage) throw use === 'api' ? new AccessDenied(500, 'Request limit reached') : new AccessDenied(501, 'Download limit reached')
grant.usage = usage
}
/**
* Adds one use to today's row unless the limit is reached (null then): one
* statement, so concurrent requests can't both take the last one.
*/
static async count(userId: number, use: Use, limit: number | null): Promise<Usage | null> {
if (limit !== null && limit <= 0) return null
const column = COLUMNS[use]
const { rows } = await db.rawQuery(
`INSERT INTO usage (user_id, day, ${column}) VALUES (?, ${TODAY}, 1)
ON CONFLICT (user_id, day) DO UPDATE SET ${column} = usage.${column} + 1
${limit === null ? '' : `WHERE usage.${column} < ?`}
RETURNING api_hits, grabs`,
limit === null ? [userId] : [userId, limit]
)
return rows[0] ? { apiHits: rows[0].api_hits, grabs: rows[0].grabs } : null
}
static async usage(userId: number): Promise<Usage> {
const { rows } = await db.rawQuery(`SELECT api_hits, grabs FROM usage WHERE user_id = ? AND day = ${TODAY}`, [userId])
return { apiHits: rows[0]?.api_hits ?? 0, grabs: rows[0]?.grabs ?? 0 }
}
/** A search's age bound: the asked one within the tier's retention. */
static maxAgeDays(grant: Grant | null, requested?: number): number | undefined {
const retention = grant?.limits.retentionDays
if (!retention) return requested
return requested ? Math.min(requested, retention) : retention
}
/** Refuses a release posted before the tier's retention (one without a date is let through). */
static checkPosted(grant: Grant | null, postedAt: Date | string | null) {
const retention = grant?.limits.retentionDays
if (!retention || !postedAt) return
if (new Date(postedAt).getTime() < Date.now() - retention * DAY_MS) {
throw new AccessDenied(102, `Older than your tier's retention (${retention} days)`)
}
}
/** Whether a release exists, refusing it when it's older than the tier sees; asked before an NZB is counted. */
static async checkRelease(grant: Grant | null, id: number): Promise<boolean> {
const release = await db.from('releases').where('id', id).select('posted_at').first()
if (!release) return false
this.checkPosted(grant, release.posted_at)
return true
}
/** Takes an unused invite for a new user: what it gives, or null when it's unknown or used. */
static async redeem(code: string, userId: number, trx: TransactionClientContract) {
const { rows } = await trx.rawQuery(
`UPDATE invites SET used_by = ?, used_at = now() WHERE code = ? AND used_by IS NULL AND used_at IS NULL RETURNING tier, tier_days`,
[userId, code]
)
return (rows[0] as { tier: string | null; tier_days: number | null } | undefined) ?? null
}
/**
* Gives a user a tier for `days` (null: without end). The same tier still
* running is extended from where it ends, so a renewal paid early loses nothing.
*/
static grantTier(user: User, tier: string, days: number | null) {
if (!TIERS[tier]) throw new Error(`No such tier: ${tier} (config/access.ts has ${Object.keys(TIERS).join(', ')})`)
const now = DateTime.utc()
const running = user.tier === tier && user.tierUntil && user.tierUntil > now ? user.tierUntil : now
user.tier = tier
user.tierUntil = days === null ? null : running.plus({ days })
}
/** What a profile shows of the account's access. */
static async summary(user: User) {
const { tier, limits } = this.tierOf(user)
return {
mode: this.mode,
tier,
tierUntil: tier === user.tier ? user.tierUntil : null,
apiKey: user.apiKey,
limits,
usage: await this.usage(user.id),
}
}
}
app/services/batch_store.ts
+8 -41
View File
@@ -1,20 +1,8 @@
import db from '@adonisjs/lucid/services/db'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
import ChunkPartitions from '#services/chunk_partitions'
import { decodeChunk } from '#lib/chunks/index'
import { SIDECAR } from '#services/sidecar_folder'
/**
* A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
* fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
*/
function sidecarOnly(r: ReleaseChunk): boolean | null {
if (r.files > 3 || r.bytes >= 1_000_000) return null
const files = decodeChunk(r.payload)
return files.length > 0 && files.every((f) => SIDECAR.test(f.name))
}
export class StaleBatch extends Error {}
@@ -237,9 +225,8 @@ class BatchStore {
last_article = greatest(r.last_article, a.last_article),
readable = r.readable OR a.readable,
has_par2 = r.has_par2 OR a.has_par2,
-- As it stands (the counter's files all found); settling makes it final. Without a counter, what its par2
-- set decided stays (app/services/par2_completeness.ts).
complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN r.complete ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
-- As it stands (the counter's files all found); settling makes it final.
complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN NULL ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
last_seen_at = now()
FROM a WHERE r.id = a.id`,
[JSON.stringify(updates)]
@@ -248,21 +235,13 @@ class BatchStore {
}
const fresh = part.filter((_, idx) => !joined.has(idx))
if (fresh.length > 0) {
// The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
// delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
// taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
const { rows: counts } = await trx.rawQuery(
`SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
THEN substr(release_key, length(identity_key) + 2)::int END) AS n
FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
[[...new Set(fresh.map((r) => r.key))]]
)
// The first upload of a key keeps the key; later ones are key:2, key:3, …
const { rows: counts } = await trx.rawQuery(`SELECT identity_key, count(*) AS n FROM releases WHERE identity_key = ANY(?) GROUP BY 1`, [[...new Set(fresh.map((r) => r.key))]])
const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
const numbered = (r: ReleaseChunk) => {
const n = (taken.get(r.key) ?? 0) + 1
taken.set(r.key, n)
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r) }
}
// Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
@@ -273,10 +252,10 @@ class BatchStore {
for (let attempt = 0; inserts.length > 0; attempt++) {
if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
const { rows } = await trx.rawQuery(
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete)
SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)
ON CONFLICT (release_key) DO NOTHING
RETURNING id, release_key`,
[groupId, JSON.stringify(inserts)]
@@ -326,18 +305,6 @@ class BatchStore {
.ignore()
.returning('id')
out.candidates += added.length
// A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
// Files added to a settled release (a later range of its upload): the settler made its candidates claimable
// already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
if (added.length > 0) {
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
[added.map((a: { id: string | number }) => String(a.id))]
)
}
}
return out
app/services/corpus.ts
+7 -37
View File
@@ -1,24 +1,11 @@
import { access, mkdir, readFile, readdir, rename, rm, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { deflate as deflateCb, gzip as gzipCb, unzip } from 'node:zlib'
import { unzip } from 'node:zlib'
import { promisify } from 'node:util'
import coordinatorConfig from '#config/coordinator'
import { filterOverviews } from '#lib/edge/index'
import type { ChunkSpan } from '#services/spool'
const inflate = promisify(unzip)
const deflate = promisify(deflateCb)
const gzip = promisify(gzipCb)
/** The largest inflated chunk the filter reads as one string (V8's limit is 0x1fffffe8 characters). */
const MAX_FILTER_BYTES = 256 * 1024 * 1024
/** What a filtered chunk was filtered with (CORPUS_FILTER): the rule set and the group priorities of its lease. */
export interface CorpusFilter {
ruleVersion: number
priorities: Record<string, number>
}
/** The rules a filtered chunk leaves out (lib/edge/corpus_filter.ts), in its range.json. */
export const FILTERED = ['crosspost', 'obfuscated']
export interface CorpusRange {
group: string
@@ -32,10 +19,7 @@ export interface CorpusRange {
* every chunk is there, the layout edge-bench reads (edge/fixtures). The
* grouping sweeps replay them with each build's rules instead of waiting for
* a re-index; the prefetcher reads a stored chunk instead of fetching it
* again. Off without CORPUS_DIR. Filtered (CORPUS_FILTER; LOG.md 2026-10-23
* (101)): the lines the edge worker only counts are left out, ~half of them,
* and each chunk's `.drops` beside it (gzip JSON, ChunkDrops) lists their
* article numbers by rule; its range.json says `filtered`.
* again. Off without CORPUS_DIR.
*/
class Corpus {
get dir(): string | undefined {
@@ -77,42 +61,28 @@ class Corpus {
/**
* Stores a chunk that inflates (a reply cut short isn't kept), and the
* range.json once the range is complete (`total` chunks). With `filter`,
* without the lines the edge worker drops on their own, compressed again
* as the news server's (zlib), with its `.drops` written first.
* range.json once the range is complete (`total` chunks).
*/
async write(range: CorpusRange, span: ChunkSpan, data: Buffer, total: number, filter?: CorpusFilter) {
async write(range: CorpusRange, span: ChunkSpan, data: Buffer, total: number) {
if (!this.dir) return
let raw: Buffer
try {
raw = await inflate(data)
await inflate(data)
} catch {
return
}
const dir = this.rangeDir(range)
const file = this.path(range, span)
await mkdir(dir, { recursive: true })
let stored = data
// Past V8's longest string (0x1fffffe8) the text can't be filtered; such a chunk is stored as it came (the
// worker drops the same lines itself). Seen once: a chunk inflating past 512 MB (LOG.md 2026-10-24 (102)).
if (filter && raw.length < MAX_FILTER_BYTES) {
// Latin-1 both ways: every byte back as it was.
const { kept, drops } = filterOverviews(raw.toString('latin1'), range.group, filter.priorities, filter.ruleVersion)
stored = await deflate(Buffer.from(kept, 'latin1'), { level: 9 })
await writeFile(`${file}.drops.part`, await gzip(JSON.stringify(drops)))
await rename(`${file}.drops.part`, `${file}.drops`)
}
await writeFile(`${file}.part`, stored)
await writeFile(`${file}.part`, data)
await rename(`${file}.part`, file)
const chunks = (await readdir(dir)).filter((f) => f.endsWith('.gz')).length
if (chunks >= total) await writeFile(join(dir, 'range.json'), JSON.stringify(filter ? { ...range, filtered: FILTERED } : range))
if (chunks >= total) await writeFile(join(dir, 'range.json'), JSON.stringify(range))
}
/** A chunk a worker couldn't inflate after all: not served again, and the range not complete. */
async removeChunk(range: CorpusRange, span: ChunkSpan) {
if (!this.dir) return
await rm(this.path(range, span), { force: true })
await rm(`${this.path(range, span)}.drops`, { force: true })
await rm(join(this.rangeDir(range), 'range.json'), { force: true })
}
}
app/services/disk_space.ts
-40
View File
@@ -1,40 +0,0 @@
import { statfs } from 'node:fs/promises'
import logger from '@adonisjs/core/services/logger'
/** How often free space is measured. */
const CHECK_MS = 60_000
/** Resumed only this much above the floor: not paused and resumed at every flush. */
const MARGIN_GB = 1
/**
* Free space on the filesystem holding `path`, against a floor: low below it, again fine MARGIN_GB above it. In a
* container `/` is an overlay on the host's Docker root, which on homelab is the NVMe the catalog and the WAL are on.
* The prober filled it ~0.8–1.1 GB an hour at ~200 articles/s, and nothing stopped it at the 10 GB floor (LOG.md
* 2026-10-18 (77)). Off when the floor is 0.
*/
export class DiskSpace {
private checkedAt = 0
private low = false
constructor(
private what: string,
private path: string,
private floorGb: number,
private free: (path: string) => Promise<number> = async (p) => {
const s = await statfs(p)
return s.bavail * s.bsize
},
private checkMs = CHECK_MS
) {}
async short(): Promise<boolean> {
if (!this.floorGb) return false
if (Date.now() - this.checkedAt < this.checkMs) return this.low
this.checkedAt = Date.now()
const gb = (await this.free(this.path)) / 1e9
const was = this.low
this.low = gb < this.floorGb + (was ? MARGIN_GB : 0)
if (this.low !== was) logger.info(`${gb.toFixed(1)} GB free (floor ${this.floorGb} GB): ${this.what} ${this.low ? 'paused' : 'resumed'}.`)
return this.low
}
}
app/services/group_dates.ts
-138
View File
@@ -1,138 +0,0 @@
import type { Overview, XoverClient } from '#lib/nntp/index'
/** When an article of a group was posted: the median Date of a slice of articles at `article`. */
export interface DateSample {
article: number
postedAt: Date
}
/** Before this, a Date header is a broken clock (the oldest retention is 2008). */
const OLDEST = Date.UTC(1995, 0, 1)
const DAY = 86_400_000
/**
* The median of the overviews' Date headers that parse and are plausible (1995 … a day from `now`), with at least
* `min` of them; null otherwise (an expired or empty slice).
*/
export function medianDate(overviews: Overview[], now = new Date(), min = 3): Date | null {
const latest = now.getTime() + DAY
const times = overviews
.map(([, fields]) => Date.parse(fields.date ?? ''))
.filter((t) => t >= OLDEST && t <= latest)
.sort((a, b) => a - b)
if (times.length < min) return null
return new Date(times[Math.floor(times.length / 2)])
}
/** Where a group is sampled: `count` article numbers spread over low … high, the first at low and the last `size` before high. */
export function samplePoints(low: number, high: number, count: number, size: number): number[] {
const last = Math.max(low, high - size + 1)
if (count <= 1 || last <= low) return [low]
const points = new Set<number>()
for (let i = 0; i < count; i++) points.add(Math.round(low + ((last - low) * i) / (count - 1)))
return [...points].sort((a, b) => a - b)
}
/**
* The samples as a posting-time curve: in article order, a sample later than both of the next two by over a day
* (a slice of one poster's wrong clock) is dropped, then each date is at least its predecessor's (articles arrive
* in number order, bar stragglers and old crossposts).
*/
export function monotone(samples: DateSample[]): DateSample[] {
const sorted = [...samples].sort((a, b) => a.article - b.article)
const kept = sorted.filter((s, i) => {
const next = sorted.slice(i + 1, i + 3)
return next.length < 2 || next.some((n) => n.postedAt.getTime() + DAY >= s.postedAt.getTime())
})
let floor = Number.NEGATIVE_INFINITY
return kept.map((s) => {
floor = Math.max(floor, s.postedAt.getTime())
return { article: s.article, postedAt: new Date(floor) }
})
}
/**
* When `article` was posted by the group's samples (monotone, in article order): interpolated between the two around
* it, the first or last sample's date beyond them; null without samples.
*/
export function estimateDate(samples: DateSample[], article: number): Date | null {
if (samples.length === 0) return null
if (article <= samples[0].article) return samples[0].postedAt
const last = samples[samples.length - 1]
if (article >= last.article) return last.postedAt
let lo = 0
let hi = samples.length - 1
while (hi - lo > 1) {
const mid = (lo + hi) >> 1
if (samples[mid].article <= article) lo = mid
else hi = mid
}
const a = samples[lo]
const b = samples[hi]
const f = (article - a.article) / (b.article - a.article)
return new Date(a.postedAt.getTime() + f * (b.postedAt.getTime() - a.postedAt.getTime()))
}
/** How many samples a group gets: one per `every` articles of its span, at least `min` and at most `max`. */
export function sampleCount(
low: number,
high: number,
every: number,
min = 16,
max = 1000
): number {
return Math.min(max, Math.max(min, Math.ceil((high - low) / every) + 1))
}
/**
* Samples a group over one connection: GROUP, then an XOVER of `size` articles at each of `count` points (a slice
* with too few articles that parse, expired or sparse, is widened to 32 × size once). The curve (monotone) and the
* server's span.
*/
export async function sampleGroup(
use: <T>(work: (client: XoverClient) => Promise<T>) => Promise<T>,
name: string,
options: { every: number; size: number; maxFailures?: number },
now = new Date()
): Promise<{
samples: DateSample[]
low: number
high: number
commands: number
failed: string[]
}> {
const info = await use((client) => client.group(name))
const samples: DateSample[] = []
const failed: string[] = []
let commands = 1
// Each point on a connection of its own (GROUP again): a reply that times out (Eweka stalls some requests for
// minutes; 16 groups never got through on one connection) costs that point, not the group.
for (const point of samplePoints(
info.low,
info.high,
sampleCount(info.low, info.high, options.every),
options.size
)) {
if (failed.length >= (options.maxFailures ?? 3)) break
try {
const date = await use(async (client) => {
await client.group(name)
commands += 2
const slice = medianDate(
await client.xover(point, Math.min(point + options.size - 1, info.high)),
now
)
if (slice) return slice
commands++
return medianDate(
await client.xover(point, Math.min(point + options.size * 32 - 1, info.high)),
now
)
})
if (date) samples.push({ article: point, postedAt: date })
} catch (error: any) {
failed.push(`${point}: ${error.message}`)
}
}
return { samples: monotone(samples), low: info.low, high: info.high, commands, failed }
}
app/services/leases.ts
+9 -63
View File
@@ -41,12 +41,7 @@ class Leases {
AND l.heartbeat_at >= now() - make_interval(secs => ?)) < ${MAX_LEASES_PER_GROUP}
ORDER BY priority, id LIMIT 1 FOR UPDATE SKIP LOCKED)
RETURNING r.id, r.group_id, r.start_id, r.end_id, r.next_id`,
[
nodeId,
rules.version,
coordinatorConfig.leaseTtlSeconds,
coordinatorConfig.leaseTtlSeconds,
]
[nodeId, rules.version, coordinatorConfig.leaseTtlSeconds, coordinatorConfig.leaseTtlSeconds]
)
let { rows } = await take(true)
if (rows.length === 0) ({ rows } = await take(false))
@@ -54,38 +49,12 @@ class Leases {
if (!row) return null
await db.from('nodes').where('id', nodeId).update({ last_seen_at: new Date() })
const group = await db.from('groups').where('id', row.group_id).select('name').firstOrFail()
return {
range: {
id: Number(row.id),
group: group.name,
start: Number(row.start_id),
end: Number(row.end_id),
next: Number(row.next_id),
},
rules,
}
}
/**
* Ranges read to their end whose worker went quiet before finishing them: done. The lease query skips a range past
* its end, so one whose finish never came stayed leased for good (a gap range since 2026-10-16 14:03, during the lock
* pile-up; LOG.md 2026-10-17 (72)). Its batches are all stored; its spooled chunks go.
*/
async finishAbandoned(): Promise<number> {
const { rows } = await db.rawQuery(
`UPDATE ranges SET status = 'done', done_at = now()
WHERE status = 'leased' AND next_id > end_id AND heartbeat_at < now() - make_interval(secs => ?)
RETURNING id`,
[coordinatorConfig.leaseTtlSeconds]
)
for (const { id } of rows) await Spool.remove(Number(id))
return rows.length
return { range: { id: Number(row.id), group: group.name, start: Number(row.start_id), end: Number(row.end_id), next: Number(row.next_id) }, rules }
}
async leased(rangeId: number, nodeId: number) {
const range = await db.from('ranges').where('id', rangeId).first()
if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId)
throw new LeaseLost(`Range ${rangeId} is not leased to this node`)
if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId) throw new LeaseLost(`Range ${rangeId} is not leased to this node`)
return range
}
@@ -98,14 +67,8 @@ class Leases {
/** The worker reached the range's end: done, and its spooled chunks go. */
async finish(rangeId: number, nodeId: number) {
const range = await this.leased(rangeId, nodeId)
if (Number(range.next_id) <= Number(range.end_id))
throw new LeaseLost(
`Range ${rangeId} is at ${range.next_id}, not past its end ${range.end_id}`
)
await db
.from('ranges')
.where('id', rangeId)
.update({ status: 'done', done_at: new Date(), heartbeat_at: new Date() })
if (Number(range.next_id) <= Number(range.end_id)) throw new LeaseLost(`Range ${rangeId} is at ${range.next_id}, not past its end ${range.end_id}`)
await db.from('ranges').where('id', rangeId).update({ status: 'done', done_at: new Date(), heartbeat_at: new Date() })
await Spool.remove(rangeId)
}
@@ -116,19 +79,12 @@ class Leases {
*/
async corrupt(rangeId: number, nodeId: number, from: number) {
const range = await this.leased(rangeId, nodeId)
const span = Spool.chunksOf({ start: Number(range.start_id), end: Number(range.end_id) }).find(
(s) => s.from === from
)
const span = Spool.chunksOf({ start: Number(range.start_id), end: Number(range.end_id) }).find((s) => s.from === from)
if (span) {
await Spool.removeChunk(rangeId, span)
if (range.capture) {
const group = await db.from('groups').where('id', range.group_id).select('name').first()
await Corpus.removeChunk(
{ group: group.name, start: Number(range.start_id), end: Number(range.end_id) },
span
)
// Not complete in the corpus any more: the corpus fill fetches the chunk again.
await db.from('ranges').where('id', rangeId).update({ stored_at: null })
await Corpus.removeChunk({ group: group.name, start: Number(range.start_id), end: Number(range.end_id) }, span)
}
}
const plain: number[] = (range.plain_chunks ?? []).map(Number)
@@ -136,21 +92,11 @@ class Leases {
// refreshed yet) and spooled again (range 161820, LOG.md 2026-10-11), so a second try plain first.
if (plain.filter((p) => p === from).length >= 2) {
// Fetched plain (deflated here) and still not inflating: not the server's compression; set aside.
await db
.from('ranges')
.where('id', rangeId)
.update({
status: 'failed',
node_id: null,
error: `chunk ${from} doesn't inflate, also fetched plain`,
})
await db.from('ranges').where('id', rangeId).update({ status: 'failed', node_id: null, error: `chunk ${from} doesn't inflate, also fetched plain` })
return
}
// The server's compressed reply is cut short (the same bytes every time): fetched plain next time.
await db.rawQuery(
`UPDATE ranges SET status = 'planned', node_id = NULL, plain_chunks = array_append(plain_chunks, ?::bigint) WHERE id = ?`,
[from, rangeId]
)
await db.rawQuery(`UPDATE ranges SET status = 'planned', node_id = NULL, plain_chunks = array_append(plain_chunks, ?::bigint) WHERE id = ?`, [from, rangeId])
}
/** A stopping worker gives its range back; the next one resumes it. */
app/services/nntp_pool.ts
+13 -93
View File
@@ -2,18 +2,12 @@ import env from '#start/env'
import logger from '@adonisjs/core/services/logger'
import coordinatorConfig from '#config/coordinator'
import { XoverClient } from '#lib/nntp/index'
import type { NntpShare } from '#services/nntp_shares'
/** Connections wanted beyond the most in use at once since the last sync (the planner's, a hedge's). */
const SHARE_MARGIN = 2
/**
* The coordinator's header connections (REBUILD.md, decision e: workers
* have no NNTP account). At most NNTP_CONNECTIONS, reused; a connection
* that failed, or is older than NNTP_CONNECTION_MAX_MINUTES, is closed and
* replaced on the next use. Sharing the account (share()), at most its
* share: over it, connections close as they come free. One left free for
* NNTP_IDLE_CLOSE_SECONDS is closed on the next sync.
* replaced on the next use.
*/
class NntpPool {
private idle: XoverClient[] = []
@@ -23,43 +17,9 @@ class NntpPool {
private openedAt = new Map<XoverClient, number>()
/** When a connection gets closed once free: its max age ±25%, so they don't all reconnect at once. */
private retireAt = new Map<XoverClient, number>()
/** When an idle connection came free. */
private idleSince = new Map<XoverClient, number>()
private limit = coordinatorConfig.nntpConnections
/** Since the last sync: the most connections in use at once, and whether work waited for one (starved()). */
private peak = 0
private short = false
get size() {
return this.limit
}
/** Work that could use another connection didn't get one (the prefetcher with all its loops running). */
starved() {
this.short = true
}
/**
* Tells the share what the pool could use and has open, and takes the share's grant as its size. After work waited:
* twice its size (history going on gets its maximum in a few syncs; a tail round, done in seconds, doesn't take
* the prober's connections for one); else the most in use at once since the last sync and a margin.
*/
async share(share: NntpShare) {
const want = this.short || this.waiting.length > 0 ? Math.max(this.limit * 2, this.limit + SHARE_MARGIN) : this.peak + SHARE_MARGIN
this.short = false
this.peak = this.busy.size
const granted = await share.sync(want, this.open)
const was = this.limit
this.limit = granted
// Over the share: the idle ones close now, the busy ones once free (release()).
while (this.open > this.limit && this.idle.length > 0) this.retire(this.idle.pop()!)
this.closeIdle()
// Under it with work waiting: connections for it.
while (this.waiting.length > 0 && this.open < this.limit) {
const next = this.waiting.shift()!
this.acquire().then(next, () => this.waiting.unshift(next))
}
return { was, granted }
return coordinatorConfig.nntpConnections
}
async use<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
@@ -84,7 +44,6 @@ class NntpPool {
private async run<T>(client: XoverClient, work: (client: XoverClient) => Promise<T>): Promise<T> {
this.busy.add(client)
this.peak = Math.max(this.peak, this.busy.size)
let broken = false
try {
return await work(client)
@@ -97,28 +56,9 @@ class NntpPool {
}
}
/** Closes the connections free for nntpIdleCloseMs, the plain one too. */
private closeIdle() {
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs <= 0) return
const before = Date.now() - idleMs
for (const client of this.idle.filter((c) => (this.idleSince.get(c) ?? Infinity) <= before)) {
this.idle.splice(this.idle.indexOf(client), 1)
this.retire(client)
}
if (this.plain && this.plainPending === 0 && this.plainUsedAt <= before) {
void this.plain.then((c) => c.close()).catch(() => {})
this.plain = null
}
}
private async acquire(): Promise<XoverClient> {
// The one used last: the others stay free long enough to close.
const ready = this.idle.pop()
if (ready) {
this.idleSince.delete(ready)
return ready
}
if (ready) return ready
if (this.open < this.size) {
this.open++
try {
@@ -144,12 +84,15 @@ class NntpPool {
}
private release(client: XoverClient, broken: boolean) {
if (this.open > this.limit) {
this.retire(client)
return
}
if (broken || Date.now() >= (this.retireAt.get(client) ?? Infinity)) {
this.retire(client)
this.open--
this.openedAt.delete(client)
this.retireAt.delete(client)
try {
client.close()
} catch (error: any) {
logger.debug(`Closing a failed NNTP connection: ${error.message}`)
}
// Someone waiting gets a fresh connection.
const next = this.waiting.shift()
if (next) this.acquire().then(next, () => this.waiting.unshift(next))
@@ -157,30 +100,12 @@ class NntpPool {
}
const next = this.waiting.shift()
if (next) next(client)
else {
this.idle.push(client)
this.idleSince.set(client, Date.now())
}
}
private retire(client: XoverClient) {
this.open--
this.openedAt.delete(client)
this.retireAt.delete(client)
this.idleSince.delete(client)
try {
client.close()
} catch (error: any) {
logger.debug(`Closing an NNTP connection: ${error.message}`)
}
else this.idle.push(client)
}
/** Closes every connection, also those in use: their commands fail. */
private plain: Promise<XoverClient> | null = null
private plainQueue: Promise<unknown> = Promise.resolve()
/** Commands queued or running on the plain connection, and when the last one ended. */
private plainPending = 0
private plainUsedAt = 0
/**
* One connection without XFEATURE COMPRESS GZIP, opened when first needed,
@@ -188,7 +113,6 @@ class NntpPool {
* cuts short (Leases.corrupt). Its replies are deflated here.
*/
async usePlain<T>(work: (client: XoverClient) => Promise<T>): Promise<T> {
this.plainPending++
const run = this.plainQueue.then(async () => {
this.plain ??= XoverClient.connect({
host: env.get('NNTP_HOST')!,
@@ -208,10 +132,7 @@ class NntpPool {
throw error
}
})
this.plainQueue = run.catch(() => {}).finally(() => {
this.plainPending--
this.plainUsedAt = Date.now()
})
this.plainQueue = run.catch(() => {})
return run
}
@@ -222,7 +143,6 @@ class NntpPool {
this.busy.clear()
this.openedAt.clear()
this.retireAt.clear()
this.idleSince.clear()
this.open = 0
}
}
app/services/nntp_shares.ts
-90
View File
@@ -1,90 +0,0 @@
import db from '@adonisjs/lucid/services/db'
import coordinatorConfig from '#config/coordinator'
/** A share whose process hasn't synced for this long is gone: its connections closed with it. */
const STALE_SECONDS = 60
/** Syncs one at a time (pg_advisory_xact_lock). */
const LOCK = 2_026_101_875
export interface Share {
name: string
priority: number
/** Kept whatever the others want (as far as the total goes, by priority). */
min: number
/** What it could use now. */
want: number
/** Connections it has open (or may have: a lane without one yet counts). */
open: number
}
/** What each share should have: its minimum first, then by priority what it wants, up to the total. */
export function targets(total: number, shares: Share[]): Map<string, number> {
const ordered = [...shares].sort((a, b) => b.priority - a.priority || a.name.localeCompare(b.name))
const target = new Map<string, number>()
let left = total
for (const s of ordered) {
const t = Math.min(s.min, left)
target.set(s.name, t)
left -= t
}
for (const s of ordered) {
const more = Math.min(Math.max(s.want - target.get(s.name)!, 0), left)
target.set(s.name, target.get(s.name)! + more)
left -= more
}
return target
}
/**
* The connections `self` may have: its target, but no more than the others leave over. Another share holds what it
* has open or was granted, whichever is more (it may open up to its grant until its next sync tells it otherwise),
* so the account never goes over the total: a share grows only into what the others closed or gave up.
*/
export function grant(total: number, self: Share, others: (Share & { granted: number })[]): number {
const target = targets(total, [self, ...others]).get(self.name)!
const held = others.reduce((sum, o) => sum + Math.max(o.open, o.granted), 0)
return Math.max(0, Math.min(target, total - held))
}
/**
* A process's share of the NNTP account's connections (LOG.md 2026-10-18 (75)): the coordinator and the prober had a
* fixed split, and with history paused the coordinator's 38 stood idle while the prober's 8 were its limit. Each
* syncs every nntpShares.everyMs (what it wants and has open) and gets what it may open; one wanting less closes
* connections as they come free, and the other grows as those show up closed.
*/
export class NntpShare {
granted = 0
constructor(
readonly name: string,
private priority: number,
readonly min: number,
readonly max: number
) {}
async sync(want: number, open: number): Promise<number> {
const self: Share = { name: this.name, priority: this.priority, min: this.min, want: Math.min(Math.max(want, this.min), this.max), open }
const total = coordinatorConfig.nntpShares.total
this.granted = await db.transaction(async (trx) => {
await trx.rawQuery(`SELECT pg_advisory_xact_lock(?)`, [LOCK])
const { rows } = await trx.rawQuery(
`SELECT name, priority, min, want, open, granted FROM nntp_shares WHERE name <> ? AND heartbeat_at > now() - make_interval(secs => ?)`,
[this.name, STALE_SECONDS]
)
const granted = grant(total, self, rows)
await trx.rawQuery(
`INSERT INTO nntp_shares (name, priority, min, want, open, granted, heartbeat_at) VALUES (?, ?, ?, ?, ?, ?, now())
ON CONFLICT (name) DO UPDATE SET priority = excluded.priority, min = excluded.min, want = excluded.want,
open = excluded.open, granted = excluded.granted, heartbeat_at = now()`,
[this.name, this.priority, this.min, self.want, open, granted]
)
return granted
})
return this.granted
}
/** Stopping: its connections are closed, the others needn't wait for the row to go stale. */
async leave() {
await db.from('nntp_shares').where('name', this.name).delete()
}
}
app/services/nzb_service.ts
+5 -25
View File
@@ -3,26 +3,15 @@ import db from '@adonisjs/lucid/services/db'
import { decodeChunk, type ChunkFile } from '#lib/chunks/index'
import { buildNzb } from '#lib/nzb/index'
/** Where a known password came from, the most trusted first: the spot's NZB meta, its title, its text. */
const PASSWORD_SOURCES = ['nzb', 'spot-title', 'spot-text']
/**
* A release's files and NZB (REBUILD.md, part 9), from its chunks in the
* chunk store: every chunk decoded, its files in name order. A file stored
* twice (the same name and first article, e.g. a range read again) is taken
* once. A known password (release_passwords) goes into the NZB's meta, where
* SABnzbd and NZBGet take it for unpacking.
* once.
*/
class NzbService {
/** The release's known passwords, the most trusted first. */
async passwords(releaseId: number): Promise<{ password: string; source: string }[]> {
const rows = await db.from('release_passwords').where('release_id', releaseId).orderBy('created_at').select('password', 'source')
const rank = (source: string) => (PASSWORD_SOURCES.indexOf(source) + PASSWORD_SOURCES.length + 1) % (PASSWORD_SOURCES.length + 1)
return rows.sort((a, b) => rank(a.source) - rank(b.source)).map((r) => ({ password: r.password, source: r.source }))
}
async files(releaseId: number, client: any = db): Promise<ChunkFile[]> {
const chunks = await client.from('release_chunks').where('release_id', releaseId).orderBy('id').select('payload')
async files(releaseId: number): Promise<ChunkFile[]> {
const chunks = await db.from('release_chunks').where('release_id', releaseId).orderBy('id').select('payload')
const files = new Map<string, ChunkFile>()
for (const chunk of chunks) {
for (const file of decodeChunk(chunk.payload)) files.set(`${file.name}\n${file.segments[0]?.[1] ?? ''}`, file)
@@ -33,11 +22,10 @@ class NzbService {
async build(releaseId: number): Promise<{ name: string; xml: string } | null> {
const release = await db.from('releases').where('id', releaseId).select('name', 'source').first()
if (!release) return null
const password = (await this.passwords(releaseId))[0]?.password
// A release a spot made of its own (no files matched the header pass's): the spot's NZB as read.
if (release.source === 'spot') {
const spot = await db.from('spots').where('release_id', releaseId).whereNotNull('nzb').select('nzb').first()
if (spot) return { name: release.name, xml: withPassword(gunzipSync(spot.nzb).toString('utf8'), password) }
if (spot) return { name: release.name, xml: gunzipSync(spot.nzb).toString('utf8') }
}
const xml = buildNzb(
(await this.files(releaseId)).map((f) => ({
@@ -47,18 +35,10 @@ class NzbService {
groups: f.groups,
segments: f.segments.map(([number, id, size]) => ({ number, id, size })),
})),
{ title: release.name, password }
{ title: release.name }
)
return { name: release.name, xml }
}
}
/** A read NZB with the password in its meta, when it has none (the spot's title or text named it, not its NZB). */
export function withPassword(xml: string, password: string | undefined): string {
if (!password || /<meta\s+type=["']password["']/i.test(xml)) return xml
const meta = `<meta type="password">${password.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;')}</meta>`
if (/<head\b[^>]*>/i.test(xml)) return xml.replace(/<head\b[^>]*>/i, (head) => `${head}${meta}`)
return xml.replace(/<nzb\b[^>]*>/i, (nzb) => `${nzb}<head>${meta}</head>`)
}
export default new NzbService()
app/services/par2_completeness.ts
-53
View File
@@ -1,53 +0,0 @@
import NzbService from '#services/nzb_service'
/** A file a par2 set describes (par2_files). */
export interface SetFile {
name: string
length: number
hash16k: string
}
const base = (name: string) => name.replace(/^.*[\\/]/, '').toLowerCase()
/**
* Whether a release holds every file its par2 sets describe (REBUILD.md, part 8): for one without an [i/N] counter,
* whose completeness the settler leaves unknown. A described file is there when a file of the release has its name,
* or a probed file of it its first 16 KiB and size (file_hashes). An upload renamed after its par2 was made (no
* described name among its files) is complete when it has as many files besides par2 as the set describes, and as
* many bytes (its articles' bytes, a little over the files' with yEnc).
*/
export function par2Complete(set: SetFile[], files: { name: string; size: number }[], hashes: { hash16k: string; size: number }[]): boolean {
const names = new Set(files.map((f) => base(f.name)))
const hashed = new Set(hashes.map((h) => `${h.hash16k}:${h.size}`))
const byName = set.filter((f) => names.has(base(f.name))).length
if (set.every((f) => names.has(base(f.name)) || hashed.has(`${f.hash16k}:${f.length}`))) return true
if (byName > 0) return false
const data = files.filter((f) => !/\.par2$/i.test(f.name))
return data.length >= set.length && data.reduce((n, f) => n + f.size, 0) >= set.reduce((n, f) => n + f.length, 0)
}
/**
* Decides `complete` for the settled releases among `releaseIds` that have no counter and a par2 set with its file
* list (par2_files; the pruner drops it once the set is done, so it decides before). How many it decided. `client`:
* a transaction that moved chunks (a merge) reads them as moved.
*/
export async function decidePar2Completeness(client: any, releaseIds: (number | string)[]): Promise<number> {
if (releaseIds.length === 0) return 0
const { rows: releases } = await client.rawQuery(
`SELECT id FROM releases WHERE id = ANY(?) AND file_total IS NULL AND settled_at IS NOT NULL`,
[releaseIds.map(String)]
)
let decided = 0
for (const { id } of releases) {
const { rows: set } = await client.rawQuery(
`SELECT DISTINCT f.file_id, f.name, f.length::float8 AS length, f.hash16k FROM par2_sets s JOIN par2_files f ON f.set_id = s.set_id WHERE s.release_id = ?`,
[id]
)
if (set.length === 0) continue
const { rows: hashes } = await client.rawQuery(`SELECT hash16k, size::float8 AS size FROM file_hashes WHERE release_id = ?`, [id])
const files = (await NzbService.files(Number(id), client)).map((f) => ({ name: f.name, size: f.segments.reduce((n, [, , bytes]) => n + bytes, 0) }))
await client.rawQuery(`UPDATE releases SET complete = ? WHERE id = ?`, [par2Complete(set, files, hashes), id])
decided++
}
return decided
}
app/services/par2_merger.ts
+52 -233
View File
@@ -2,12 +2,8 @@ import db from '@adonisjs/lucid/services/db'
import logger from '@adonisjs/core/services/logger'
import { par2ReleaseName } from '#lib/release/index'
import { looksRandom } from '#lib/release/random'
import { SidecarFolder, SIDECAR } from '#services/sidecar_folder'
import { randomPoster } from '#lib/release/identity'
import NzbService from '#services/nzb_service'
import { SidecarFolder } from '#services/sidecar_folder'
import { Par2Pruner } from '#services/par2_pruner'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
import { decidePar2Completeness } from '#services/par2_completeness'
/** Releases of a set further apart than this (posted) aren't one upload: a repost is merged separately (decision d). */
const MAX_SPREAD_MS = 48 * 3600 * 1000
@@ -19,45 +15,21 @@ const MAX_HOLDERS = 4
/** Members with more files than this many times the set's, plus SLACK_FILES (its par2 volumes), are more than one upload. */
const MAX_FILES_PER_SET_FILE = 3
const SLACK_FILES = 10
/**
* A par2 file held by more file rows than this (the same first 16 KiB and size) is a common file, not one upload's:
* it matches nothing. One such file was in 190,380 uploads, and a set naming it loaded every one of them; the merger
* ran out of heap every few minutes (LOG.md 2026-10-20 (85)). The judge calls 20 releases repeated.
*/
const COMMON_FILE = 20
/**
* The name pass: a set's own poster's releases in its group posted this close to it are looked at; at most
* NAME_CANDIDATES of them (a poster's uploads of a day).
*/
const NAME_SPREAD_MS = 12 * 3600 * 1000
const NAME_CANDIDATES = 200
/** Files a set's upload carries besides those it describes: sidecars, its par2 files, a cover or preview. */
const besideSet = (name: string) => SIDECAR.test(name) || /\.(par2|jpe?g|png)$/i.test(name)
/** Between rounds. */
const ROUND_MS = 60_000
/** A window looks back this far past the previous one's end (rows committed after it was read). */
/** A round looks back this far past the previous one's start (rows committed while it ran). */
const OVERLAP_MS = 60_000
/**
* A round goes through what's new in windows of this much (created_at), its place kept after each: a round behind by
* two days of the probe backlog (3.5M sets) never ended, and every restart started it over (LOG.md 2026-10-20 (85)).
*/
const WINDOW_MS = 60 * 60_000
/** The sidecar fold also runs this often inside a long par2 pass: its queue grows by the settler's pace. */
const FOLD_EVERY_MS = 5 * 60_000
/** Between par2_files prunes (complete and judged sets' file lists). */
const PRUNE_EVERY_MS = 60 * 60_000
/** A prune's share of the merger's time: the rest goes on in the next one. */
const PRUNE_BUDGET_MS = 2 * 60_000
/** The merger's row in worker_cursors: the end of its last whole window, kept over a restart. */
/** The merger's row in worker_cursors: the start of its last whole round, kept over a restart. */
const CURSOR = 'merger'
export interface MergeRound {
sets: number
merges: number
releases: number
/** The name pass: merges, releases in them. */
nameMerges: number
nameReleases: number
/** The inner-name pass: stems checked, merges, releases in them. */
stems: number
stemMerges: number
@@ -72,25 +44,6 @@ export interface MergeRound {
*/
const INNER_STEM = `substring(i.result->>'yencName' from '^([A-Za-z0-9]{16,64})\\.')`
/**
* A file name as the name pass compares it: without its folder, case, spaces and punctuation, its UTF-8 read back when
* it was read as Latin-1 ("Vermächtnis" posted, "Vermächtnis" in the par2).
*/
export function fileKey(name: string): string {
let n = name.replace(/^.*[\\/]/, '')
if (/[\u00c2-\u00f4][\u0080-\u00bf]/.test(n)) {
const read = Buffer.from(n, 'latin1').toString('utf8')
if (!read.includes('\ufffd')) n = read
}
return n.normalize('NFC').toLowerCase().replace(/[^\p{L}\p{N}]+/gu, '')
}
/** The releases holding a par2 file f (its first 16 KiB and size), none when it's a common file (COMMON_FILE). */
const HOLDERS = `SELECT c.release_id FROM (
SELECT l.release_id, count(*) OVER () AS holders
FROM (SELECT h.release_id FROM file_hashes h WHERE h.hash16k = f.hash16k AND h.size = f.length LIMIT ${COMMON_FILE + 1}) l
) c WHERE c.holders <= ${COMMON_FILE}`
interface Member {
id: number
release_key: string
@@ -142,12 +95,11 @@ export class Par2Merger {
// First, at start too: a prune needn't wait for a long round.
if (!this.stopped && Date.now() - prunedAt >= PRUNE_EVERY_MS) {
prunedAt = Date.now()
const pruned = await new Par2Pruner().round(PRUNE_BUDGET_MS)
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged, ${pruned.final} final.`)
const pruned = await new Par2Pruner().round()
if (pruned.rows) logger.info(`Pruned ${pruned.rows} par2 file rows: ${pruned.complete} complete sets, ${pruned.judged} judged.`)
}
const round = await this.round()
if (round.merges > 0) logger.info(`Merged ${round.releases} releases into ${round.merges} by par2 set (${round.sets} sets checked).`)
if (round.nameMerges > 0) logger.info(`Merged ${round.nameReleases} releases into ${round.nameMerges} by their par2 set's file names.`)
if (round.stemMerges > 0) logger.info(`Merged ${round.stemReleases} releases into ${round.stemMerges} by inner name (${round.stems} stems checked).`)
if (round.folded?.folded) logger.info(`Folded ${round.folded.folded} sidecars into their uploads (${round.folded.sidecars} sidecar-only of ${round.folded.looked} small releases).`)
} catch (error: any) {
@@ -157,18 +109,14 @@ export class Par2Merger {
}
}
/**
* The sets and inner stems with something new since the last round (a par2 found, a hash of one of their files, a
* lone inspection), a window at a time.
*/
/** The sets with something new since the last round: a par2 found, or a hash of one of their files. */
async round(): Promise<MergeRound> {
// A Date keeps milliseconds, now() microseconds: rounded up, a row made at now() is before the end.
const { rows: now } = await db.rawQuery(`SELECT now() + interval '1 millisecond' AS at`)
const end = new Date(now[0].at)
const { rows: now } = await db.rawQuery(`SELECT now() AS at`)
if (!this.since) {
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
this.since = saved ? new Date(saved.at) : new Date(0)
}
const since = new Date(this.since.getTime() - OVERLAP_MS)
// First: sidecars (.nfo, .sfv, .nzb) apart from their upload join it. A queue, not since: it needn't wait for the
// par2 pass (hours when that starts over).
const folded = await this.folder.round()
@@ -182,164 +130,52 @@ export class Par2Merger {
foldedAt = Date.now()
if (more.folded) logger.info(`Folded ${more.folded} sidecars into their uploads (${more.sidecars} sidecar-only of ${more.looked} small releases).`)
}
const round: MergeRound = { sets: 0, merges: 0, releases: 0, nameMerges: 0, nameReleases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
while (this.since < end) {
const { rows } = await db.rawQuery(
`SELECT set_id FROM par2_sets WHERE created_at >= ?
UNION
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size WHERE h.created_at >= ?`,
[since, since]
)
const round: MergeRound = { sets: rows.length, merges: 0, releases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded }
for (const { set_id: setId } of rows) {
if (this.stopped) return round
const from = await this.firstNew(new Date(this.since.getTime() - OVERLAP_MS), end)
const until = from ? new Date(Math.min(from.getTime() + WINDOW_MS, end.getTime())) : end
if (from) {
const { touched, sets } = await this.setsToMerge(from, until)
round.sets += touched
for (const setId of sets) {
if (this.stopped) return round
await foldAgain()
// One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
const merged = await this.mergeSet(setId.trim()).catch((error) => {
logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.merges++
round.releases += merged + 1
}
}
// Every set the window found, by its files' names: readable files have no hashes (the files of one upload under
// names that differ: scene tracks apart from their album's par2, "Vermaechtnis" beside "Vermächtnis"; LOG.md
// 2026-10-23 (97)).
const { rows: found } = await db.rawQuery(`SELECT DISTINCT set_id FROM par2_sets WHERE created_at >= ? AND created_at <= ?`, [from, until])
for (const { set_id: setId } of found) {
if (this.stopped) return round
const merged = await this.mergeByName(setId.trim()).catch((error) => {
logger.error(`Name merge of set ${setId.trim()} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.nameMerges++
round.nameReleases += merged + 1
}
}
const { rows: stems } = await db.rawQuery(
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i
WHERE i.kind = 'lone' AND i.created_at >= ? AND i.created_at <= ? AND ${INNER_STEM} IS NOT NULL`,
[from, until]
)
for (const { stem } of stems) {
if (this.stopped) return round
if (!looksRandom(stem)) continue
round.stems++
const merged = await this.mergeStem(stem).catch((error) => {
logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.stemMerges++
round.stemReleases += merged + 1
}
}
await foldAgain()
// One set that fails doesn't hold up the others (nor the next round, which would start from the same point).
const merged = await this.mergeSet(setId.trim()).catch((error) => {
logger.error(`Par2 merge of set ${setId.trim()} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.merges++
round.releases += merged + 1
}
// Only after a whole window: a restart halfway goes on from the window before.
this.since = until
await db.rawQuery(
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, this.since]
)
}
const { rows: stems } = await db.rawQuery(
`SELECT DISTINCT ${INNER_STEM} AS stem FROM release_inspections i WHERE i.kind = 'lone' AND i.created_at >= ? AND ${INNER_STEM} IS NOT NULL`,
[since]
)
for (const { stem } of stems) {
if (this.stopped) return round
if (!looksRandom(stem)) continue
round.stems++
const merged = await this.mergeStem(stem).catch((error) => {
logger.error(`Inner-name merge of ${stem} failed: ${error.message}`)
return 0
})
if (merged > 0) {
round.stemMerges++
round.stemReleases += merged + 1
}
}
// Only after a whole round: a restart halfway goes on from the round before.
this.since = now[0].at
await db.rawQuery(
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, this.since]
)
return round
}
/** When the first row a window reads was made, from `from` up to `end`; null when none (a gap, the nightly pause). */
private async firstNew(from: Date, end: Date): Promise<Date | null> {
const { rows } = await db.rawQuery(
`SELECT least(
(SELECT min(created_at) FROM par2_sets WHERE created_at >= ?),
(SELECT min(created_at) FROM file_hashes WHERE created_at >= ?),
(SELECT min(created_at) FROM release_inspections WHERE created_at >= ?)) AS at`,
[from, from, from]
)
const at = rows[0].at ? new Date(rows[0].at) : null
return at && at <= end ? at : null
}
/**
* The sets a window touched (a par2 found, a hash of one of their files), and those among them a merge could
* change: held or matched by two releases or more. Most have every file in one release already (96% of a backlog
* hour's 147k): mergeSet needn't look at them.
*/
private async setsToMerge(from: Date, until: Date): Promise<{ touched: number; sets: string[] }> {
const { rows } = await db.rawQuery(
`WITH touched AS MATERIALIZED (
SELECT set_id FROM par2_sets WHERE created_at >= ? AND created_at <= ?
UNION
SELECT f.set_id FROM file_hashes h JOIN par2_files f ON f.hash16k = h.hash16k AND f.length = h.size
WHERE h.created_at >= ? AND h.created_at <= ?
)
SELECT (SELECT count(*) FROM touched) AS touched, array(
SELECT t.set_id FROM touched t
WHERE EXISTS (
SELECT 1 FROM (
SELECT s.release_id FROM par2_sets s WHERE s.set_id = t.set_id
UNION ALL
SELECT m.release_id FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) m WHERE f.set_id = t.set_id
) u HAVING count(DISTINCT u.release_id) >= 2)
) AS sets`,
[from, until, from, until]
)
return { touched: Number(rows[0].touched), sets: rows[0].sets }
}
/**
* The releases a set's file names find (mergeByName without merging): its newest holder and the holders near it,
* and the releases of its poster in its group posted within NAME_SPREAD_MS whose files the set all names, besides
* sidecars, par2 and images. Null when there's nothing to merge. Not a holder further off: the same par2 posted
* again years later ("Auto Bild Klassik Februar 02-2014": 2014 and 2026) is a repost (decision d).
*/
async nameMatches(trx: any, setId: string): Promise<Map<string, Member> | null> {
const { rows: all } = await trx.rawQuery(
`SELECT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable, r.poster FROM par2_sets s JOIN releases r ON r.id = s.release_id
WHERE s.set_id = ? ORDER BY r.posted_at DESC NULLS LAST, r.id`,
[setId]
)
const anchor: Member | undefined = all[0]
// A random poster (one per file) bounds nothing: those uploads are the hash pass's.
if (!anchor?.poster || !anchor.posted_at || randomPoster(anchor.poster)) return null
const holders = all.filter(
(h: Member) => h.group_id === anchor.group_id && h.posted_at && Math.abs(new Date(h.posted_at).getTime() - new Date(anchor.posted_at!).getTime()) <= NAME_SPREAD_MS
)
const { rows: described } = await trx.rawQuery(`SELECT name FROM par2_files WHERE set_id = ?`, [setId])
if (described.length === 0) return null
const names = new Set(described.map((d: { name: string }) => fileKey(d.name)))
const at = new Date(anchor.posted_at).getTime()
const { rows: candidates } = await trx.rawQuery(
`SELECT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable, r.poster FROM releases r
WHERE left(r.poster, 200) = left(?, 200) AND r.poster = ? AND r.group_id = ? AND r.posted_at BETWEEN ? AND ? AND r.id <> ALL(?)
AND r.files <= ?
ORDER BY left(r.poster, 200), r.posted_at DESC NULLS LAST, r.id DESC LIMIT ?`,
// More files than the set describes, with its sidecars, can't all be named in it.
[anchor.poster, anchor.poster, anchor.group_id, new Date(at - NAME_SPREAD_MS), new Date(at + NAME_SPREAD_MS), holders.map((h: Member) => h.id), described.length + SLACK_FILES, NAME_CANDIDATES]
)
const found = new Map<string, Member>(holders.map((h: Member) => [String(h.id), h]))
let files = holders.reduce((n: number, h: Member) => n + Number(h.files), 0)
for (const c of candidates) {
const own = await NzbService.files(Number(c.id), trx)
const content = own.filter((f) => !besideSet(f.name))
if (content.length === 0 || !content.every((f) => names.has(fileKey(f.name)))) continue
found.set(String(c.id), c)
files += Number(c.files)
}
if (found.size < 2) return null
// No more files than the set describes, with its par2 volumes and sidecars (as the hash pass).
if (files > MAX_FILES_PER_SET_FILE * names.size + SLACK_FILES) return null
return found
}
/** Merges the releases whose files one set names (nameMatches); how many went into another. */
async mergeByName(setId: string): Promise<number> {
return db.transaction(async (trx) => {
const found = await this.nameMatches(trx, setId)
return found ? this.merge(trx, found, setId, 'name') : 0
})
}
/** Merges the releases whose lone files' inner names share a random stem; how many went into another. */
async mergeStem(stem: string): Promise<number> {
return db.transaction(async (trx) => {
@@ -384,7 +220,7 @@ export class Par2Merger {
const near = (r: Member) => r.group_id === anchor.group_id && (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
const { rows: matches } = await trx.rawQuery(
`SELECT f.file_id, r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable
FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) h JOIN releases r ON r.id = h.release_id
FROM par2_files f JOIN file_hashes h ON h.hash16k = f.hash16k AND h.size = f.length JOIN releases r ON r.id = h.release_id
WHERE f.set_id = ?`,
[setId]
)
@@ -414,7 +250,7 @@ export class Par2Merger {
* Merges the found releases into the one with the most files (named by the par2 set's files when none
* is readable); how many went into it. Each merge is kept in release_merges, with its set or stem.
*/
private async merge(trx: any, found: Map<string, Member>, key: string, via: 'par2' | 'inner' | 'sidecar' | 'name'): Promise<number> {
private async merge(trx: any, found: Map<string, Member>, key: string, via: 'par2' | 'inner' | 'sidecar'): Promise<number> {
{
// Locked in id order (the batch store and other rounds take the same rows).
const { rows: locked } = await trx.rawQuery(`SELECT id FROM releases WHERE id = ANY(?) ORDER BY id FOR UPDATE`, [[...found.keys()]])
@@ -429,25 +265,9 @@ export class Par2Merger {
SELECT id, release_key, ?, ?, files, ? FROM releases WHERE id = ANY(?)`,
[target.id, key, via, others]
)
// A candidate of an article the target, or another member of lower id, has already (the same file in two releases)
// isn't moved: the name pass met them (every round of 3 sets failed on the unique key; LOG.md 2026-10-23 (99)).
await trx.rawQuery(
`DELETE FROM probe_candidates p WHERE p.release_id = ANY(?)
AND (EXISTS (SELECT 1 FROM probe_candidates t WHERE t.release_id = ? AND t.kind = p.kind AND t.message_id = p.message_id)
OR EXISTS (SELECT 1 FROM probe_candidates o WHERE o.release_id = ANY(?) AND o.release_id < p.release_id AND o.kind = p.kind AND o.message_id = p.message_id))`,
[others, target.id, others]
)
for (const table of ['release_chunks', 'probe_candidates', 'release_inspections', 'file_hashes']) {
await trx.rawQuery(`UPDATE ${table} SET release_id = ? WHERE release_id = ANY(?)`, [target.id, others])
}
// An unsettled member's candidates into a settled target: the settler won't come by again to make them claimable.
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE r.id = ? AND r.settled_at IS NOT NULL AND pc.release_id = r.id AND pc.status = 'waiting' AND NOT pc.ready`,
[target.id]
)
// The members' 7z tails together: those below the highest volume go.
await pruneSevenZipTails(trx, [target.id])
// A set held by the target already, or by two of the others (a par2 and its repost), isn't held twice (every
// round failed on the primary key from 20:28 UTC on 2026-10-12; LOG.md 2026-10-12 (30)).
await trx.rawQuery(
@@ -475,13 +295,12 @@ export class Par2Merger {
WHERE t.id = ?`,
[name ?? '', others, target.id]
)
// Completeness as the settler decides it, once settled; without a counter, by the par2 sets it now holds.
// Completeness as the settler decides it, once settled.
await trx.rawQuery(
`UPDATE releases SET complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END WHERE id = ? AND settled_at IS NOT NULL`,
`UPDATE releases SET complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END WHERE id = ? AND settled_at IS NOT NULL`,
[target.id]
)
await trx.rawQuery(`DELETE FROM releases WHERE id = ANY(?)`, [others])
await decidePar2Completeness(trx, [target.id])
return others.length
}
}
app/services/par2_pruner.ts
+43 -75
View File
@@ -1,107 +1,75 @@
import db from '@adonisjs/lucid/services/db'
import { decidePar2Completeness } from '#services/par2_completeness'
/** A set this long past its last holder's par2 has had its merges: one posted later is a repost (decision d). */
const SETTLED_MS = 2 * 86400_000
/** Sets per statement. */
const BATCH = 20_000
/** The sweep's row in worker_cursors: the last set_id it looked at (key); past the last set it starts over. */
/** The complete pass's row in worker_cursors: the newest holder's created_at of the last set it looked at. */
const CURSOR = 'par2-prune'
/** The merger's row in worker_cursors: the end of its last whole window. */
const MERGER = 'merger'
/** The merger's windows look back this far (Par2Merger's OVERLAP_MS). */
const MERGER_OVERLAP_MS = 60_000
/** Verdicts whose releases needn't merge any more; locked isn't one (a password found takes it back). */
const JUNK = ['dead', 'fake', 'repeated']
/** Probe candidates whose results a merge reads: a file's hash (lone), a set's file list (par2). */
const MERGE_KINDS = ['lone', 'par2']
export interface PruneRound {
complete: number
judged: number
final: number
rows: number
}
/**
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60), 2026-10-20 (87)). A set's file list
* matches file hashes to releases and names an unreadable upload once; it's done when
* Drops par2_files rows the merge can't use any more (LOG.md 2026-10-16 (60)): the table only grew, 35M rows and
* 13.6 GB at 1.8M sets. A set's file list matches file hashes to releases and names an unreadable upload once; it's
* done when
* - complete: SETTLED_MS after its last holder's par2, every file it describes has its hash in a release holding the
* set (all of it merged into one);
* - judged: every release holding it has a junk verdict (dead, fake, repeated);
* - final: SETTLED_MS after its last holder's par2, every holder is final (release_final: a run of done ranges holds
* it with hours to spare, so no more of its upload will be read), the candidates of the ranges around it are probed,
* and the merger has seen every hash of its files. A file still missing then is one no probe will hash: most sets
* describe a readable upload's files, which aren't lone candidates (9 in 10 sets were kept for that, 19 GB).
* - judged: every release holding it has a junk verdict (dead, fake, repeated).
* par2_sets stays (the judge's unusable rule, the merge's holders). A set comes up again only with a new holder: its
* par2 reposted, whose probe writes the rows again; without rows its merge finds no files and the count check holds.
* The sweep goes through the sets with rows by set_id and starts over at the end: a set kept now (a holder at the edge
* of a run, its ranges' probes not done yet) is looked at again on the next pass.
*/
export class Par2Pruner {
/** Sets per batch (BATCH; smaller in the tests). */
constructor(private batch = BATCH) {}
/**
* Batches until a pass is done or `budgetMs` is spent (the cursor is saved per batch, the next round goes on from
* it): run from the merger's loop, a first round over two days of sets held the merges back for hours (LOG.md
* 2026-10-17 (72)).
*/
async round(budgetMs = Infinity): Promise<PruneRound> {
const deadline = Date.now() + budgetMs
const round: PruneRound = { complete: 0, judged: 0, final: 0, rows: 0 }
const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
let after: string = cursors[0]?.key ?? ''
// Ranges done since the oldest merge-kind candidate still unprobed: their releases' hashes and file lists may be
// still to come. Hashes since the merger's last window: it hasn't matched them yet.
const { rows: marks } = await db.rawQuery(
`SELECT coalesce((SELECT min(created_at) FROM probe_candidates WHERE status IN ('waiting', 'taken') AND kind = ANY(?)), now()) AS probed,
coalesce((SELECT at FROM worker_cursors WHERE name = ?), to_timestamp(0)) - make_interval(secs => ?) AS merged,
now() - make_interval(secs => ?) AS settled`,
[MERGE_KINDS, MERGER, MERGER_OVERLAP_MS / 1000, SETTLED_MS / 1000]
)
const { probed, merged, settled } = marks[0]
async round(): Promise<PruneRound> {
const round: PruneRound = { complete: 0, judged: 0, rows: 0 }
const saved = await db.from('worker_cursors').where('name', CURSOR).first()
let since: Date = saved ? new Date(saved.at) : new Date(0)
const until = new Date(Date.now() - SETTLED_MS)
for (;;) {
const { rows: sets } = await db.rawQuery(
`WITH batch AS MATERIALIZED (SELECT DISTINCT set_id FROM par2_files WHERE set_id > ? ORDER BY set_id LIMIT ?),
recent AS MATERIALIZED (SELECT group_id, posted_low, posted_high FROM ranges WHERE status = 'done' AND done_at >= ?),
unseen AS MATERIALIZED (SELECT DISTINCT hash16k, size FROM file_hashes WHERE created_at >= ?)
SELECT b.set_id, CASE
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
AND NOT EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = b.set_id AND NOT EXISTS (
SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
WHERE h.hash16k = f.hash16k AND h.size = f.length)) THEN 'complete'
WHEN NOT EXISTS (SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts v ON v.release_id = o.release_id AND v.rule = ANY(?)
WHERE o.set_id = b.set_id AND v.release_id IS NULL) THEN 'judged'
WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
AND NOT EXISTS (SELECT 1 FROM par2_sets s JOIN releases r ON r.id = s.release_id WHERE s.set_id = b.set_id AND (
release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) IS NOT TRUE
OR EXISTS (SELECT 1 FROM recent x WHERE x.group_id = r.group_id
AND x.posted_low <= coalesce(r.last_posted_at, r.posted_at) + interval '6 hours' AND x.posted_high >= r.posted_at - interval '6 hours')))
AND NOT EXISTS (SELECT 1 FROM par2_files f JOIN unseen u ON u.hash16k = f.hash16k AND u.size = f.length WHERE f.set_id = b.set_id)
THEN 'final' END AS reason
FROM batch b ORDER BY b.set_id`,
[after, this.batch, probed, merged, settled, JUNK, settled]
`SELECT set_id, max(created_at) AS at FROM par2_sets GROUP BY set_id
HAVING max(created_at) > ? AND max(created_at) <= ? ORDER BY 2 LIMIT ?`,
[since, until, BATCH]
)
const done = (reason: keyof PruneRound) => sets.filter((s: { reason: string }) => s.reason === reason).map((s: { set_id: string }) => s.set_id)
for (const reason of ['complete', 'judged', 'final'] as const) {
const ids = done(reason)
round[reason] += ids.length
// The holders' completeness while the file lists are there (app/services/par2_completeness.ts).
if (reason !== 'judged' && ids.length > 0) {
const { rows: holders } = await db.rawQuery(`SELECT DISTINCT release_id FROM par2_sets WHERE set_id = ANY(?::char(32)[])`, [ids])
await decidePar2Completeness(db, holders.map((h: { release_id: string }) => h.release_id))
}
round.rows += await this.drop(ids)
}
// Past the last set: the next batch starts the next pass.
after = sets.length < this.batch ? '' : sets.at(-1).set_id
if (sets.length === 0) break
const ids = sets.map((s: { set_id: string }) => s.set_id)
const { rows: done } = await db.rawQuery(
`SELECT c.set_id FROM unnest(?::char(32)[]) AS c(set_id)
WHERE EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id)
AND NOT EXISTS (
SELECT 1 FROM par2_files f WHERE f.set_id = c.set_id AND NOT EXISTS (
SELECT 1 FROM file_hashes h JOIN par2_sets s ON s.release_id = h.release_id AND s.set_id = f.set_id
WHERE h.hash16k = f.hash16k AND h.size = f.length))`,
[ids]
)
round.complete += done.length
round.rows += await this.drop(done.map((s: { set_id: string }) => s.set_id))
since = sets.at(-1).at
await db.rawQuery(
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
[CURSOR, after]
`INSERT INTO worker_cursors (name, at) VALUES (?, ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at`,
[CURSOR, since]
)
if (after === '' || Date.now() > deadline) break
if (sets.length < BATCH) break
}
// Judged: no cursor (verdicts come and go); only sets with rows left.
const { rows: judged } = await db.rawQuery(
`SELECT DISTINCT s.set_id FROM release_verdicts v JOIN par2_sets s ON s.release_id = v.release_id
WHERE v.rule = ANY(?)
AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
AND NOT EXISTS (
SELECT 1 FROM par2_sets o LEFT JOIN release_verdicts ov ON ov.release_id = o.release_id AND ov.rule = ANY(?)
WHERE o.set_id = s.set_id AND ov.release_id IS NULL)`,
[JUNK, JUNK]
)
round.judged = judged.length
const judgedIds = judged.map((s: { set_id: string }) => s.set_id)
for (let i = 0; i < judgedIds.length; i += BATCH) round.rows += await this.drop(judgedIds.slice(i, i + BATCH))
return round
}
app/services/prefetcher.ts
+28 -154
View File
@@ -3,8 +3,7 @@ import logger from '@adonisjs/core/services/logger'
import coordinatorConfig from '#config/coordinator'
import NntpPool from '#services/nntp_pool'
import Spool, { type ChunkSpan } from '#services/spool'
import Corpus, { type CorpusFilter } from '#services/corpus'
import RuleSets from '#services/rule_sets'
import Corpus from '#services/corpus'
import CatalogSpace from '#services/catalog_space'
import { hedge, type Request, type XoverClient } from '#lib/nntp/index'
@@ -73,13 +72,7 @@ export class Prefetcher {
private writingNow = 0
private sampler?: NodeJS.Timeout
/** Requests in flight: what, since when, on which connection (the report names the oldest). */
private inFlight = new Set<{
rangeId: number
from: number
to: number
since: number
client: XoverClient
}>()
private inFlight = new Set<{ rangeId: number; from: number; to: number; since: number; client: XoverClient }>()
/** Capture ranges found complete in the corpus (not looked at again). */
private stored = new Set<number>()
@@ -138,8 +131,6 @@ export class Prefetcher {
.catch((error) => logger.warn(`Prefetching: ${error.message}`))
.finally(() => this.running--)
}
// Work left with every loop running: the pool's share could be bigger.
if (!this.stopped && this.running >= this.loops && this.peek()) NntpPool.starved()
}
stop() {
@@ -154,9 +145,6 @@ export class Prefetcher {
/** One connection's loop: chunks until there is nothing left to fetch. */
private async work() {
while (!this.stopped) {
// The pool's share got smaller: fewer loops (waiting on the pool, they'd count as work short of a connection).
// Two leaving at once is one too many for a tick (TICK_MS), whose loop comes back.
if (this.running > this.loops) return
const next = this.take()
if (!next) return
const { target, span } = next
@@ -169,14 +157,10 @@ export class Prefetcher {
}
const corpusRange = { group: target.group, start: target.start, end: target.end }
// A range in the corpus is read from disk (a reset re-indexes it without the news server).
let data =
target.capture && !target.corpusOnly ? await Corpus.read(corpusRange, span) : null
let data = target.capture && !target.corpusOnly ? await Corpus.read(corpusRange, span) : null
const stored = data !== null
try {
if (!data)
data = target.plain.has(span.from)
? await NntpPool.usePlain(fetch)
: await this.fetchHedged(target.id, target.group, span)
if (!data) data = target.plain.has(span.from) ? await NntpPool.usePlain(fetch) : await this.fetchHedged(target.id, target.group, span)
} finally {
this.fetchingNow--
}
@@ -186,8 +170,7 @@ export class Prefetcher {
this.stats.chunks++
this.stats.articles += span.to - span.from + 1
this.stats.fetchMs += Date.now() - fetchedAt
if (target.capture)
await Corpus.write(corpusRange, span, data, target.total, await this.corpusFilter())
if (target.capture) await Corpus.write(corpusRange, span, data, target.total)
}
if (target.corpusOnly) {
this.stats.captured++
@@ -204,8 +187,7 @@ export class Prefetcher {
`UPDATE ranges SET spooled_chunks = spooled_chunks + 1 WHERE id = ? RETURNING spooled_chunks`,
[target.id]
)
if (Number(rows[0]?.spooled_chunks) >= target.total)
await db.from('ranges').where('id', target.id).update({ spooled_at: new Date() })
if (Number(rows[0]?.spooled_chunks) >= target.total) await db.from('ranges').where('id', target.id).update({ spooled_at: new Date() })
} finally {
this.writingNow--
this.stats.writeMs += Date.now() - writtenAt
@@ -227,10 +209,7 @@ export class Prefetcher {
*/
private async fetchHedged(rangeId: number, group: string, span: ChunkSpan): Promise<Buffer> {
let age: number | undefined
const request = (
start: (work: (client: XoverClient) => Promise<Buffer>) => Promise<Buffer> | null,
first: boolean
): Request<Buffer> | null => {
const request = (start: (work: (client: XoverClient) => Promise<Buffer>) => Promise<Buffer> | null, first: boolean): Request<Buffer> | null => {
let client: XoverClient | undefined
let cancelled = false
const done = start(async (c) => {
@@ -262,9 +241,7 @@ export class Prefetcher {
data = await c.xoverRaw(span.from, span.to)
} catch (error) {
if (!late) throw error
throw new Error(
`Chunk ${span.from}–${span.to} of range ${rangeId} stalled (under ${coordinatorConfig.nntpStallBytesPerMinute / 1000} kB in a minute, ${Math.round(c.buffered / 1000)} kB in); fetching it again`
)
throw new Error(`Chunk ${span.from}–${span.to} of range ${rangeId} stalled (under ${coordinatorConfig.nntpStallBytesPerMinute / 1000} kB in a minute, ${Math.round(c.buffered / 1000)} kB in); fetching it again`)
} finally {
clearInterval(watch)
this.inFlight.delete(entry)
@@ -307,8 +284,7 @@ export class Prefetcher {
/** The longest-running request: range, chunk, how long, how much came in. */
private oldest() {
let oldest:
{ rangeId: number; from: number; to: number; since: number; client: XoverClient } | undefined
let oldest: { rangeId: number; from: number; to: number; since: number; client: XoverClient } | undefined
for (const r of this.inFlight) if (!oldest || r.since < oldest.since) oldest = r
if (!oldest) return 'none in flight'
return `oldest request ${Math.round((Date.now() - oldest.since) / 1000)} s (range ${oldest.rangeId}, ${oldest.from}–${oldest.to}, ${Math.round(oldest.client.buffered / 1000)} kB in)`
@@ -333,12 +309,7 @@ export class Prefetcher {
const { chunks, articles, fetchMs, hedged, hedgeWon, byAge, samples, requests } = this.stats
if (chunks > 0) {
const ages = byAge
.map((b, i) =>
`${i === 0 ? '<' : `${AGE_BUCKETS[i - 1]}–`}${AGE_BUCKETS[i] === Infinity ? '' : AGE_BUCKETS[i]} min ${b.slow}/${b.requests}`.replace(
'– min',
'+ min'
)
)
.map((b, i) => `${i === 0 ? '<' : `${AGE_BUCKETS[i - 1]}–`}${AGE_BUCKETS[i] === Infinity ? '' : AGE_BUCKETS[i]} min ${b.slow}/${b.requests}`.replace('– min', '+ min'))
.join(', ')
logger.info(
`Prefetched ${chunks} chunks in ${seconds.toFixed(0)} s: ${Math.round(articles / seconds).toLocaleString('en')} articles/s, ` +
@@ -364,18 +335,13 @@ export class Prefetcher {
}
private peek(): boolean {
return this.targets.some(
(t) => this.open(t) && t.missing.some((s) => !this.claimed.has(`${t.id}:${s.from}`))
)
return this.targets.some((t) => this.open(t) && t.missing.some((s) => !this.claimed.has(`${t.id}:${s.from}`)))
}
/** Whether the target is under its range's and its group's limit of connections. */
private open(target: Target) {
const { perRange, perGroup } = coordinatorConfig.spool
return (
(perRange <= 0 || (this.perRange.get(target.id) ?? 0) < perRange) &&
(perGroup <= 0 || (this.perGroup.get(target.group) ?? 0) < perGroup)
)
return (perRange <= 0 || (this.perRange.get(target.id) ?? 0) < perRange) && (perGroup <= 0 || (this.perGroup.get(target.group) ?? 0) < perGroup)
}
private count(target: Target, by: number) {
@@ -406,58 +372,26 @@ export class Prefetcher {
this.sizedAt = Date.now()
this.spoolBytes = await Spool.size()
}
const mine = (q: any) =>
this.shard.count > 1 ? q.whereRaw('r.id % ? = ?', [this.shard.count, this.shard.index]) : q
const rows = (q: any) =>
mine(
q
.from('ranges as r')
.join('groups as g', 'g.id', 'r.group_id')
.select(
'r.id',
'g.name as group',
'r.start_id',
'r.end_id',
'r.next_id',
'r.plain_chunks',
'r.capture'
)
)
const mine = (q: any) => (this.shard.count > 1 ? q.whereRaw('r.id % ? = ?', [this.shard.count, this.shard.index]) : q)
const rows = (q: any) => mine(q.from('ranges as r').join('groups as g', 'g.id', 'r.group_id').select('r.id', 'g.name as group', 'r.start_id', 'r.end_id', 'r.next_id', 'r.plain_chunks', 'r.capture'))
// Leased ranges, checked against the spool's files whatever spooled_at says: a worker waits for them.
const leased = await rows(db.query()).where('r.status', 'leased').orderBy('r.leased_at')
// Planned ranges ahead: those started already, then new ones up to SPOOL_AHEAD_PER_LEASE per lease.
const ahead = Math.max(
coordinatorConfig.spool.minAhead,
coordinatorConfig.spool.aheadPerLease * leased.length
)
const started = await rows(db.query())
.where('r.status', 'planned')
.where('r.spooled_chunks', '>', 0)
.whereNull('r.spooled_at')
.orderBy([{ column: 'r.priority' }, { column: 'r.id' }])
const [{ ready }] = await mine(
db.from('ranges as r').where('r.status', 'planned').where('r.spooled_chunks', '>', 0)
).count('* as ready')
const ahead = Math.max(coordinatorConfig.spool.minAhead, coordinatorConfig.spool.aheadPerLease * leased.length)
const started = await rows(db.query()).where('r.status', 'planned').where('r.spooled_chunks', '>', 0).whereNull('r.spooled_at').orderBy([{ column: 'r.priority' }, { column: 'r.id' }])
const [{ ready }] = await mine(db.from('ranges as r').where('r.status', 'planned').where('r.spooled_chunks', '>', 0)).count('* as ready')
const room = this.spoolBytes < coordinatorConfig.spool.maxBytes
// With at most perRange connections a range, enough ranges still being fetched for all loops: fully
// spooled ones count as ready but need no connection (30 of 40 busy on small ranges; LOG.md 2026-10-10 (2)).
// 8 more than that: ranges with a chunk or two left take fewer than perRange (with +2, 3 of 40 loops idle).
const { perRange } = coordinatorConfig.spool
const short =
perRange > 0 ? Math.ceil(this.loops / perRange) + 8 - (leased.length + started.length) : 0
const short = perRange > 0 ? Math.ceil(this.loops / perRange) + 8 - (leased.length + started.length) : 0
const want = Math.max(ahead - Number(ready), short)
// The catalog at its cap on the NVMe: no history fetched ahead (CatalogSpace).
const full = await CatalogSpace.full()
const fresh =
room && want > 0
? await rows(db.query())
.where('r.status', 'planned')
.where('r.spooled_chunks', 0)
.whereRaw('r.next_id <= r.end_id')
.where((q: any) => (full ? q.where('r.priority', '<', 0) : q))
.orderBy([{ column: 'r.priority' }, { column: 'r.id' }])
.limit(want)
: []
const fresh = room && want > 0
? await rows(db.query()).where('r.status', 'planned').where('r.spooled_chunks', 0).whereRaw('r.next_id <= r.end_id').where((q: any) => (full ? q.where('r.priority', '<', 0) : q)).orderBy([{ column: 'r.priority' }, { column: 'r.id' }]).limit(want)
: []
const targets: Target[] = []
for (const row of [...leased, ...(room ? started : []), ...fresh]) {
const start = Number(row.start_id)
@@ -465,99 +399,39 @@ export class Prefetcher {
const total = Spool.chunksOf({ start, end }).length
const missing: ChunkSpan[] = []
for (const span of Spool.chunksOf({ start, end }, Number(row.next_id))) {
if (!this.claimed.has(`${row.id}:${span.from}`) && !(await Spool.has(Number(row.id), span)))
missing.push(span)
if (!this.claimed.has(`${row.id}:${span.from}`) && !(await Spool.has(Number(row.id), span))) missing.push(span)
}
if (missing.length > 0) {
targets.push({
id: Number(row.id),
group: row.group,
plain: new Set((row.plain_chunks ?? []).map(Number)),
missing,
total,
start,
end,
capture: row.capture === true,
})
targets.push({ id: Number(row.id), group: row.group, plain: new Set((row.plain_chunks ?? []).map(Number)), missing, total, start, end, capture: row.capture === true })
}
}
// Last, for loops with nothing else to do: capture ranges not stored yet, into the corpus only.
for (const target of await this.corpusTargets(rows, new Set(targets.map((t) => t.id))))
targets.push(target)
for (const target of await this.corpusTargets(rows, new Set(targets.map((t) => t.id)))) targets.push(target)
return targets
}
private filterRules: { at: number; filter: CorpusFilter } | null = null
/** The rules corpus chunks are filtered with (CORPUS_FILTER), read again once a minute; undefined when off. */
private async corpusFilter(): Promise<CorpusFilter | undefined> {
if (!coordinatorConfig.corpus.filter) return undefined
if (!this.filterRules || Date.now() - this.filterRules.at > 60_000) {
const rules = await RuleSets.current()
this.filterRules = {
at: Date.now(),
filter: { ruleVersion: rules.version, priorities: rules.priorities },
}
}
return this.filterRules.filter
}
/**
* Up to corpus.ahead capture ranges whose chunks aren't all in the corpus, by priority: planned ones not started
* (not leased soon), and done ones (a window of every group, ranges:capture, takes those indexed already too);
* as many of a group as its connections allow (perGroup / perRange): in id order the 16 were one group's, which
* took 4 of the loops and left the rest idle (30k articles/s; LOG.md 2026-10-23 (100)). A range found complete
* gets stored_at and isn't read again.
*/
/** Up to corpus.ahead planned capture ranges, not leased soon, whose chunks aren't all in the corpus. */
private async corpusTargets(rows: (q: any) => any, taken: Set<number>): Promise<Target[]> {
const { dir, ahead } = coordinatorConfig.corpus
if (!dir || ahead <= 0) return []
const { perRange, perGroup } = coordinatorConfig.spool
const ofGroup =
perGroup > 0 && perRange > 0 ? Math.max(1, Math.floor(perGroup / perRange)) : Infinity
const groups = new Map<string, number>()
const candidates = await rows(db.query())
.where('r.capture', true)
.whereNull('r.stored_at')
.where((q: any) =>
q
.where((p: any) => p.where('r.status', 'planned').where('r.spooled_chunks', 0))
.orWhere('r.status', 'done')
)
.orderBy([{ column: 'r.priority' }, { column: 'r.id' }])
.limit(ahead * 16)
const candidates = await rows(db.query()).where('r.status', 'planned').where('r.capture', true).where('r.spooled_chunks', 0).orderBy('r.id')
const targets: Target[] = []
for (const row of candidates) {
if (targets.length >= ahead) break
const id = Number(row.id)
if (taken.has(id) || this.stored.has(id)) continue
if ((groups.get(row.group) ?? 0) >= ofGroup) continue
const start = Number(row.start_id)
const end = Number(row.end_id)
const range = { group: row.group, start, end }
if (await Corpus.complete(range)) {
this.stored.add(id)
await db.from('ranges').where('id', id).update({ stored_at: new Date() })
continue
}
const chunks = Spool.chunksOf({ start, end })
const missing: ChunkSpan[] = []
for (const span of chunks)
if (!this.claimed.has(`${id}:${span.from}`) && !(await Corpus.has(range, span)))
missing.push(span)
for (const span of chunks) if (!this.claimed.has(`${id}:${span.from}`) && !(await Corpus.has(range, span))) missing.push(span)
if (missing.length > 0) {
targets.push({
id,
group: row.group,
plain: new Set((row.plain_chunks ?? []).map(Number)),
missing,
total: chunks.length,
start,
end,
capture: true,
corpusOnly: true,
})
groups.set(row.group, (groups.get(row.group) ?? 0) + 1)
targets.push({ id, group: row.group, plain: new Set((row.plain_chunks ?? []).map(Number)), missing, total: chunks.length, start, end, capture: true, corpusOnly: true })
}
}
return targets
app/services/prober.ts
+65 -143
View File
@@ -6,9 +6,6 @@ import coordinatorConfig from '#config/coordinator'
import { XoverClient } from '#lib/nntp/index'
import { YencError } from '#lib/yenc/index'
import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/index'
import { NntpShare } from '#services/nntp_shares'
import { DiskSpace } from '#services/disk_space'
import { decidePar2Completeness } from '#services/par2_completeness'
interface Candidate {
id: number
@@ -21,8 +18,7 @@ type Outcome = { candidate: Candidate; status: 'done' | 'missing' | 'error' | 'w
/**
* The order candidates are probed in: what the par2 merge needs first (a random-named file's own start, then the
* par2 sets and archives of releases without a readable name), then the rest (names and passwords of readable ones),
* then 7z listings (the end of a split 7z's last volume; queued from 2026-10-13, probed from 2026-10-23).
* par2 sets and archives of releases without a readable name), then the rest (names and passwords of readable ones).
*/
const TIERS: { kind: string; unreadable: boolean }[] = [
{ kind: 'lone', unreadable: false },
@@ -30,19 +26,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
{ kind: 'archive', unreadable: true },
{ kind: 'par2', unreadable: false },
{ kind: 'archive', unreadable: false },
{ kind: '7z-tail', unreadable: true },
{ kind: '7z-tail', unreadable: false },
]
const FLUSH_MS = 2000
/**
* JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
* and the whole flush failed with it (LOG.md 2026-10-18 (78)).
*/
function jsonb(value: unknown): string {
return JSON.stringify(value, (_key, v) => (typeof v === 'string' ? v.replaceAll('\0', '') : v))
}
/** Slices a claim takes from random points of the queue. */
const SLICES = 20
const REPORT_MS = 60_000
@@ -61,14 +46,6 @@ export class Prober {
private refilling: Promise<void> | null = null
private stopped = false
private stats = Prober.freshStats()
/** Lanes running (each one connection, once it has work); at most the share's grant. Counted apart from the set: a lane leaving takes itself off at once. */
private lanes = new Set<Promise<void>>()
private running = 0
/** The last lane to look found nothing to claim: the prober wants its minimum only. */
private dry = false
/** What it stores (inspections, par2 file lists, the WAL) fills the NVMe: no claims below PROBE_MIN_FREE_GB. */
private space = new DiskSpace('probing', '/', coordinatorConfig.probe.minFreeGb)
private share = new NntpShare('prober', coordinatorConfig.nntpShares.proberPriority, coordinatorConfig.probe.minConnections, coordinatorConfig.probe.connections)
/** Per minute: what was read, and where the lanes' time went (ms summed over lanes; flushes ms on their own). */
private static freshStats() {
@@ -79,7 +56,8 @@ export class Prober {
// A previous run's claims: one prober at a time.
const { rowCount } = await db.rawQuery(`UPDATE probe_candidates SET status = 'waiting' WHERE status = 'taken'`)
if (rowCount) logger.info(`Probing: ${rowCount} candidates from a previous run back in the queue.`)
logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
const connections = coordinatorConfig.probe.connections
logger.info(`Probing with ${connections} connections.`)
const flusher = (async () => {
while (!this.stopped || this.outcomes.length > 0) {
await sleep(FLUSH_MS)
@@ -89,26 +67,7 @@ export class Prober {
this.report()
}
})()
// Lanes as the share allows: the coordinator's history comes first (app/services/nntp_shares.ts).
while (!this.stopped) {
try {
const was = this.share.granted
const granted = await this.share.sync(this.dry ? 0 : this.share.max, this.running)
if (granted !== was) logger.info(`Probing: ${granted} connections (was ${was}).`)
while (this.running < granted) {
this.running++
const lane: Promise<void> = this.lane()
.catch((error) => logger.error(`Probing: a lane failed: ${error.message}`))
.finally(() => this.lanes.delete(lane))
this.lanes.add(lane)
}
} catch (error: any) {
logger.error(`Probing: NNTP share failed: ${error.message}`)
}
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !this.stopped; waited += 1000) await sleep(1000)
}
await Promise.all(this.lanes)
await this.share.leave().catch(() => {})
await Promise.all(Array.from({ length: connections }, () => this.lane()))
// What was claimed and not probed goes back.
for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
this.stopped = true
@@ -119,90 +78,68 @@ export class Prober {
this.stopped = true
}
/**
* One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure; done when
* there are more lanes than the share (its connection closed, the coordinator takes it on its next sync).
*/
/** One connection's loop: PROBE_PIPELINE articles at a time (pipelined), reconnecting after a failure. */
private async lane() {
let client: XoverClient | null = null
let usedAt = Date.now()
try {
while (!this.stopped) {
if (this.running > this.share.granted) break
const batch: Candidate[] = []
const waited = Date.now()
while (batch.length < coordinatorConfig.probe.pipeline) {
const candidate = await this.next()
if (!candidate) break
batch.push(candidate)
}
this.stats.waitMs += Date.now() - waited
this.dry = batch.length === 0
if (batch.length === 0) {
// Nothing to probe for a while: the connection closes (nntpIdleCloseMs), and opens again with work.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (client && idleMs > 0 && Date.now() - usedAt >= idleMs) {
client.close()
client = null
while (!this.stopped) {
const batch: Candidate[] = []
const waited = Date.now()
while (batch.length < coordinatorConfig.probe.pipeline) {
const candidate = await this.next()
if (!candidate) break
batch.push(candidate)
}
this.stats.waitMs += Date.now() - waited
if (batch.length === 0) {
this.stats.idleMs += 30_000
await sleep(30_000)
continue
}
try {
client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
port: env.get('NNTP_PORT', 563),
user: env.get('NNTP_USER'),
password: env.get('NNTP_PASSWORD'),
secure: env.get('NNTP_SECURE', true),
compress: false,
timeoutMs: coordinatorConfig.nntpTimeoutMs,
})
const fetched = Date.now()
const bodies = await client.bodies(batch.map((c) => c.messageId))
this.stats.fetchMs += Date.now() - fetched
const inspected = Date.now()
for (const [i, candidate] of batch.entries()) {
const body = bodies[i]
this.stats.articles++
if (body === null) {
// Taken down or expired: final.
this.outcomes.push({ candidate, status: 'missing' })
continue
}
this.stats.idleMs += 30_000
await sleep(30_000)
continue
}
try {
client ??= await XoverClient.connect({
host: env.get('NNTP_HOST')!,
port: env.get('NNTP_PORT', 563),
user: env.get('NNTP_USER'),
password: env.get('NNTP_PASSWORD'),
secure: env.get('NNTP_SECURE', true),
compress: false,
timeoutMs: coordinatorConfig.nntpTimeoutMs,
})
const fetched = Date.now()
const bodies = await client.bodies(batch.map((c) => c.messageId))
this.stats.fetchMs += Date.now() - fetched
const inspected = Date.now()
for (const [i, candidate] of batch.entries()) {
const body = bodies[i]
this.stats.articles++
if (body === null) {
// Taken down or expired: final.
this.outcomes.push({ candidate, status: 'missing' })
continue
}
this.stats.bytes += body.length
try {
this.outcomes.push({ candidate, status: 'done', inspection: inspect(body, candidate.kind) })
} catch (error: any) {
// Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
this.outcomes.push({ candidate, status: 'error' })
}
this.stats.bytes += body.length
try {
this.outcomes.push({ candidate, status: 'done', inspection: inspect(body) })
} catch (error: any) {
// Not yEnc, a broken one, or data a parser chokes on: a final answer, not a connection problem.
if (!(error instanceof YencError)) logger.warn(`Probing ${candidate.messageId}: ${error.message}`)
this.outcomes.push({ candidate, status: 'error' })
}
this.stats.inspectMs += Date.now() - inspected
usedAt = Date.now()
} catch (error: any) {
logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
client?.close()
client = null
await sleep(5000)
}
this.stats.inspectMs += Date.now() - inspected
} catch (error: any) {
logger.warn(`Probing ${batch[0].messageId} and ${batch.length - 1} more: ${error.message}`)
for (const candidate of batch) this.outcomes.push({ candidate, status: 'waiting' })
client?.close()
client = null
await sleep(5000)
}
} finally {
client?.close()
this.running--
}
}
/** Candidates claimed at a time: a few rounds of every lane's pipeline (a claim takes seconds; 400 lasted ~2 s at 37 lanes). */
private get batch() {
return Math.max(coordinatorConfig.probe.batch, this.running * coordinatorConfig.probe.pipeline * 4)
client?.close()
}
private async next(): Promise<Candidate | undefined> {
if (this.queue.length < this.batch / 2) {
if (this.queue.length < coordinatorConfig.probe.batch / 2) {
this.refilling ??= this.claim().finally(() => (this.refilling = null))
if (this.queue.length === 0) await this.refilling
}
@@ -225,24 +162,13 @@ export class Prober {
}
private async claimTiers() {
// The queue empties, the lanes go dry and the share shrinks to its minimum until there's room again.
if (await this.space.short()) return
for (const { kind, unreadable } of TIERS) {
const want = this.batch - this.queue.length
const want = coordinatorConfig.probe.batch - this.queue.length
if (want <= 0) return
// The starting points chosen here: in the query Postgres drew random() once for all slices.
// Over the claimable ones of this tier only: the rest of the queue isn't settled yet.
// Per value of unreadable, each an end of the index: over `unreadable >= false` Postgres walked every waiting
// entry of the kind, 2.5 minutes a claim at 28M archive candidates (LOG.md 2026-10-18 (74)).
const values = unreadable ? [true] : [false, true]
const ends = values
.map(() => `SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable = ?`)
.join(' UNION ALL ')
const [{ lo, hi }] = (
await db.rawQuery(
`SELECT min(lo) AS lo, max(hi) AS hi FROM (${ends}) e`,
values.flatMap((v) => [kind, v])
)
await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM probe_candidates WHERE status = 'waiting' AND ready AND kind = ? AND unreadable >= ?`, [kind, unreadable])
).rows
if (lo === null) continue
const starts = Array.from({ length: SLICES }, () => Number(lo) + Math.floor(Math.random() * (Number(hi) - Number(lo) + 1)))
@@ -283,18 +209,16 @@ export class Prober {
await db.transaction(async (trx) => {
await trx.rawQuery(
`UPDATE probe_candidates pc SET status = v.status FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, status text) WHERE pc.id = v.id`,
[jsonb(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
[JSON.stringify(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
)
// Final answers read from the news server, kept by message-id for the next catalog (a reset).
// One per message-id: the same article is a candidate of more than one release, and ON CONFLICT can't take a
// row twice in one statement.
const fresh = [...new Map(outcomes.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
const fresh = outcomes.filter((o) => !o.cached && o.status !== 'waiting')
if (fresh.length > 0) {
await trx.rawQuery(
`INSERT INTO probe_cache (message_id, status, result, version)
SELECT v.message_id, v.status, v.result, ? FROM jsonb_to_recordset(?::jsonb) AS v(message_id text, status text, result jsonb)
ON CONFLICT (message_id) DO UPDATE SET status = excluded.status, result = excluded.result, version = excluded.version, created_at = now()`,
[PROBE_VERSION, jsonb(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
[PROBE_VERSION, JSON.stringify(fresh.map((o) => ({ message_id: o.candidate.messageId, status: o.status, result: o.inspection ?? null })))]
)
}
const done = outcomes.filter((o) => o.inspection)
@@ -304,11 +228,9 @@ export class Prober {
SELECT v.release_id, v.candidate_id, v.kind, v.result FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, candidate_id bigint, kind text, result jsonb)
ON CONFLICT (release_id, candidate_id) DO UPDATE SET result = excluded.result, created_at = now()`,
// A par2 set's file list goes to par2_files, not into every inspection.
[jsonb(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
[JSON.stringify(done.map((o) => ({ release_id: o.candidate.releaseId, candidate_id: o.candidate.id, kind: o.candidate.kind, result: { ...o.inspection, par2: undefined } })))]
)
await this.storeForMerge(trx, done)
// A par2 index read: the release's completeness by its set, when it has no counter.
await decidePar2Completeness(trx, [...new Set(done.filter((o) => o.inspection!.par2).map((o) => o.candidate.releaseId))])
// Each touched release from all its inspections (earlier rounds may have stored others).
const ids = [...new Set(done.map((o) => o.candidate.releaseId))]
const { rows } = await trx.rawQuery(
@@ -330,7 +252,7 @@ export class Prober {
await trx.rawQuery(
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, passworded boolean, inner_name text) WHERE r.id = v.id`,
[jsonb(updates)]
[JSON.stringify(updates)]
)
})
}
@@ -343,7 +265,7 @@ export class Prober {
`INSERT INTO file_hashes (candidate_id, release_id, hash16k, size)
SELECT v.candidate_id, v.release_id, v.hash16k, v.size FROM jsonb_to_recordset(?::jsonb) AS v(candidate_id bigint, release_id bigint, hash16k text, size bigint)
ON CONFLICT (candidate_id) DO NOTHING`,
[jsonb(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
[JSON.stringify(hashed.map((o) => ({ candidate_id: o.candidate.id, release_id: o.candidate.releaseId, hash16k: o.inspection!.hash16k, size: o.inspection!.size })))]
)
}
for (const o of done) {
@@ -353,7 +275,7 @@ export class Prober {
`INSERT INTO par2_files (set_id, file_id, hash16k, length, name)
SELECT ?, v.file_id, v.hash16k, v.length, v.name FROM jsonb_to_recordset(?::jsonb) AS v(file_id text, hash16k text, length bigint, name text)
ON CONFLICT DO NOTHING`,
[set.setId, jsonb(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
[set.setId, JSON.stringify(set.files.map((f) => ({ file_id: f.fileId, hash16k: f.hash16k, length: f.length, name: f.name })))]
)
await trx.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files) VALUES (?, ?, ?) ON CONFLICT DO NOTHING`, [set.setId, o.candidate.releaseId, set.files.length])
}
app/services/release_judge.ts
+51 -181
View File
@@ -1,18 +1,10 @@
import db from '@adonisjs/lucid/services/db'
import logger from '@adonisjs/core/services/logger'
/** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
/** Raised when a rule changes: verdicts of an older version are judged again. */
export const JUDGE_VERSION = 1
/** Between rounds of the changed releases (judge_dirty, noted by triggers; migration 4900). */
const ROUND_MS = 2 * 60 * 1000
/**
* A full round (every release, each rule a scan of its table: 35–38 min) this often: for what no trigger notes. Time
* (unusable a day after settling, a release final once the ranges beside it are read) and fewer releases with a file
* after a merge (repeated).
*/
const FULL_MS = 24 * 60 * 60 * 1000
/** Changed releases judged at a time. */
const CHUNK = 50_000
/** Between rounds: each rule is a scan of its table. */
const ROUND_MS = 60 * 60 * 1000
/** A file's first 16 KiB and size in this many releases is one file posted again and again (test files, spam). */
const REPEATED_RELEASES = 20
@@ -32,15 +24,10 @@ export interface Rule {
name: string
/** Why, in a few words (the log, the web interface). */
says: string
/** SELECT of the release ids the rule judges; scoped, only those in judge_scope. */
sql: (scoped: boolean) => string
/** SELECT of the release ids the rule judges. */
sql: string
}
/** A join to the releases in scope (an incremental round), on the given release id column. */
const only = (scoped: boolean, column: string) => (scoped ? `JOIN judge_scope s ON s.release_id = ${column}` : '')
/**
* The junk judge's rules, in order: a release takes the first that matches (REBUILD.md, junk). Each runs on what
* the header pass, the prober, the merges and the spots know by now.
@@ -49,33 +36,26 @@ export const RULES: Rule[] = [
{
name: 'dead',
says: 'every probed first article is gone from the server (taken down or expired)',
sql: (scoped) =>
`SELECT c.release_id FROM probe_candidates c ${only(scoped, 'c.release_id')}
WHERE c.status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(c.status = 'missing')`,
sql: `SELECT release_id FROM probe_candidates WHERE status IN ('done', 'missing', 'error') GROUP BY 1 HAVING bool_and(status = 'missing')`,
},
{
name: 'fake',
says: 'named as a film, episode or music, but the content is a program',
// An archive whose entries are all executables, under a media name (its own or its entries'); or a file that is
// an executable by name or by its first bytes, under a media name.
sql: (scoped) => `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
WHERE i.kind IN ('archive', '7z-tail') AND jsonb_array_length(i.result->'names') > 0
sql: `SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
WHERE i.kind = 'archive' AND jsonb_array_length(i.result->'names') > 0
AND NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(i.result->'names') n WHERE n !~* '${EXECUTABLE}')
AND (r.name ~* '${MEDIA}' OR (i.result->>'names') ~* '${MEDIA}')
UNION
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id ${only(scoped, 'i.release_id')}
SELECT i.release_id FROM release_inspections i JOIN releases r ON r.id = i.release_id
WHERE i.kind = 'lone' AND ((i.result->>'yencName') ~* '${EXECUTABLE}' OR i.result->>'type' = 'exe')
AND (r.name ~* '${MEDIA}' OR (i.result->>'yencName') ~* '${MEDIA}')`,
},
{
name: 'repeated',
says: `the same file content in ${REPEATED_RELEASES}+ releases (test files, spam)`,
// Scoped: by the files of the releases in scope that are in REPEATED_RELEASES+ releases, counted once a chunk
// (judge_tipped: ReleaseJudge.changed, which first adds the other releases of a file that just tipped over).
sql: (scoped) =>
scoped
? `SELECT DISTINCT h.release_id FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id JOIN judge_tipped USING (hash16k, size)`
: `SELECT DISTINCT h.release_id FROM file_hashes h
sql: `SELECT DISTINCT h.release_id FROM file_hashes h
JOIN (SELECT hash16k, size FROM file_hashes GROUP BY 1, 2 HAVING count(DISTINCT release_id) >= ${REPEATED_RELEASES}) x USING (hash16k, size)`,
},
{
@@ -84,26 +64,19 @@ export const RULES: Rule[] = [
// "@_CelebMovie_@", "NudeCelebs@CelebMovie.com", "@_CelebWow_@", "@_CelebFX_@", "@_iLuvCelebs_@", "@nude-celebs", …:
// a celebrity's scraped pictures as a folder under a typed subject, the same files again and again under other
// titles (LOG.md 2026-10-16 (64)). Not Celeborn, CelebTA or other posters with "celeb" in their name.
// By the posters table's trigram index, each poster's releases then by releases_poster_newest_idx: 7 s, where the
// regex on all of releases was 45 s, every round (LOG.md 2026-10-19 (80)).
sql: (scoped) =>
scoped
? `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`
: `SELECT r.id FROM posters p JOIN releases r ON left(r.poster, 200) = p.poster
WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
sql: `SELECT r.id FROM releases r
WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
},
{
name: 'locked',
says: 'passworded, and no spot, NZB or title gives the password',
// releases_passworded_idx: a scan of all of releases was 30 s a round.
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')} WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
sql: `SELECT r.id FROM releases r WHERE r.passworded AND NOT EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id)`,
},
{
name: 'unusable',
says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
// Final (release_final): next to an unread range its par2 or the rest of its files may still come (LOG.md 2026-10-16 (67)).
sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
sql: `SELECT r.id FROM releases r
WHERE r.settled_at < now() - interval '1 day' AND NOT r.readable AND r.inner_name IS NULL
AND release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete)
AND NOT EXISTS (SELECT 1 FROM release_merges m WHERE m.into_id = r.id)
@@ -113,18 +86,15 @@ export const RULES: Rule[] = [
]
export interface JudgeRound {
/** Releases judged: every one (a full round), or the changed ones and those a hash of theirs tipped over. */
releases?: number
/** Per rule: releases it judges now (in scope), verdicts it gained, verdicts it lost. */
/** Per rule: releases it judges now, verdicts added, verdicts taken back. */
rules: Record<string, { matched: number; added: number; removed: number }>
}
/**
* The junk judge (REBUILD.md, junk; voyager 1's second filter, D42): after the header pass's rules (garbage
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Every few
* minutes the releases whose facts changed (judge_dirty) are judged by every rule; a full round judges every release
* daily. A release gets the verdict of the first rule in order that matches it; one that no longer matches loses its
* verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
* signatures, executables, …) a release is judged again once it's probed, by what its articles showed. Each
* round recomputes every rule: a release a rule matches gets a verdict (the first rule in order, over a later one's), one it no longer
* matches loses its verdict. Verdicts only hide (search leaves them out); nothing is dropped yet.
*/
export class ReleaseJudge {
private stopped = false
@@ -137,13 +107,9 @@ export class ReleaseJudge {
while (!this.stopped) {
const started = Date.now()
try {
if (await this.fullDue()) {
const round = await this.full()
if (!this.stopped) logger.info(`Judged all releases: ${describe(round)}.`)
} else {
const round = await this.changed()
if (round.releases! > 0) logger.info(`Judged ${round.releases!.toLocaleString('en')} changed releases: ${describe(round)}.`)
}
const round = await this.round()
const parts = Object.entries(round.rules).map(([n, r]) => `${n} ${r.matched} (+${r.added} −${r.removed})`)
logger.info(`Judged: ${parts.join(', ')}.`)
} catch (error: any) {
logger.error(`Judge round failed: ${error.message}`)
}
@@ -151,139 +117,43 @@ export class ReleaseJudge {
}
}
/** A full round is due: a day after the last, or the last was by another rule set. */
async fullDue(): Promise<boolean> {
const state = await db.from('judge_state').first()
return !state || state.version !== JUDGE_VERSION || Date.now() - new Date(state.full_at).getTime() >= FULL_MS
}
/**
* Every release by every rule, the matches gathered in judge_full (each rule its own statement: no transaction of
* half an hour), then applied at once. With dry, only counts. Stopped halfway, nothing is applied.
*/
async full(dry = false): Promise<JudgeRound> {
const started = new Date()
/** One round over every rule; with dry, only counts. */
async round(dry = false): Promise<JudgeRound> {
const round: JudgeRound = { rules: {} }
// Verdicts of an older rule set are judged again from scratch.
if (!dry) await db.from('release_verdicts').where('rule_version', '<>', JUDGE_VERSION).delete()
await db.rawQuery(`TRUNCATE judge_full`)
const judged = new Set<string>()
for (const rule of RULES) {
if (this.stopped) return round
const { rowCount } = await db.rawQuery(
`INSERT INTO judge_full (release_id, rule) SELECT id, ? FROM (${rule.sql(false)}) AS x(id) ON CONFLICT DO NOTHING`,
[rule.name]
)
round.rules[rule.name] = { matched: rowCount ?? 0, added: 0, removed: 0 }
}
if (dry || this.stopped) return round
await db.rawQuery(`ANALYZE judge_full`)
await db.transaction((trx) => this.apply(trx, 'judge_full', false, round))
await db.rawQuery(
`INSERT INTO judge_state (id, full_at, version) VALUES (1, ?, ?) ON CONFLICT (id) DO UPDATE SET full_at = excluded.full_at, version = excluded.version`,
[started, JUDGE_VERSION]
)
return round
}
/**
* The releases noted in judge_dirty since, CHUNK at a time, each chunk in a transaction (a failed one stays noted).
* A changed release can bring a file to REPEATED_RELEASES releases: the others with it are judged too (all but those
* already repeated, or dead or fake, which come first). Only when it's the chunk's releases that bring it there: a
* file in that many without them had its releases judged before.
*/
async changed(): Promise<JudgeRound> {
const round: JudgeRound = { releases: 0, rules: {} }
for (const rule of RULES) round.rules[rule.name] = { matched: 0, added: 0, removed: 0 }
let more = true
while (more && !this.stopped) {
await db.transaction(async (trx) => {
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
await trx.rawQuery(`DROP TABLE IF EXISTS judge_scope`)
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
await trx.rawQuery(`CREATE TEMP TABLE judge_scope (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
const { rows } = await trx.rawQuery(
`WITH taken AS (DELETE FROM judge_dirty WHERE ctid = ANY(ARRAY(SELECT ctid FROM judge_dirty LIMIT ${CHUNK})) RETURNING release_id),
scoped AS (INSERT INTO judge_scope SELECT DISTINCT release_id FROM taken ON CONFLICT DO NOTHING RETURNING 1)
SELECT (SELECT count(*) FROM taken) AS taken`
)
const taken = Number(rows[0].taken)
more = taken === CHUNK
if (taken === 0) return
// The chunk's files in REPEATED_RELEASES+ releases, from file_hashes_hash_size_release_idx alone (migration 5000).
await trx.rawQuery(`DROP TABLE IF EXISTS judge_tipped`)
await trx.rawQuery(
`CREATE TEMP TABLE judge_tipped (hash16k char(32), size bigint, just boolean NOT NULL, PRIMARY KEY (hash16k, size)) ON COMMIT DROP`
)
await trx.rawQuery(
`INSERT INTO judge_tipped
SELECT x.hash16k, x.size, c.outside < ${REPEATED_RELEASES}
FROM (SELECT DISTINCT h.hash16k, h.size FROM file_hashes h JOIN judge_scope s ON s.release_id = h.release_id) x
CROSS JOIN LATERAL (
SELECT count(DISTINCT f.release_id) AS total,
count(DISTINCT f.release_id) FILTER (WHERE NOT EXISTS (SELECT 1 FROM judge_scope s WHERE s.release_id = f.release_id)) AS outside
FROM file_hashes f WHERE f.hash16k = x.hash16k AND f.size = x.size) c
WHERE c.total >= ${REPEATED_RELEASES}`
)
await trx.rawQuery(
`INSERT INTO judge_scope SELECT DISTINCT f.release_id FROM file_hashes f JOIN judge_tipped t USING (hash16k, size)
WHERE t.just AND NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = f.release_id AND v.rule IN ('dead', 'fake', 'repeated'))
ON CONFLICT DO NOTHING`
)
await trx.rawQuery(`ANALYZE judge_scope`)
const { rows: scope } = await trx.rawQuery(`SELECT count(*) AS n FROM judge_scope`)
round.releases! += Number(scope[0].n)
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL) ON COMMIT DROP`)
for (const rule of RULES) {
const { rowCount } = await trx.rawQuery(
`INSERT INTO judged (release_id, rule) SELECT id, ? FROM (${rule.sql(true)}) AS x(id) ON CONFLICT DO NOTHING`,
if (this.stopped) break
const { rows } = await db.rawQuery(rule.sql)
const ids = rows.map((r: Record<string, unknown>) => String(Object.values(r)[0])).filter((id: string) => !judged.has(id))
for (const id of ids) judged.add(id)
const stats = { matched: ids.length, added: 0, removed: 0 }
if (!dry) {
await db.transaction(async (trx) => {
// Dropped first: inside an outer transaction (the tests) ON COMMIT DROP comes only at its end.
await trx.rawQuery(`DROP TABLE IF EXISTS judged`)
await trx.rawQuery(`CREATE TEMP TABLE judged (release_id bigint PRIMARY KEY) ON COMMIT DROP`)
for (let i = 0; i < ids.length; i += 50_000) {
await trx.rawQuery(`INSERT INTO judged SELECT unnest(?::bigint[]) ON CONFLICT DO NOTHING`, [ids.slice(i, i + 50_000)])
}
const removed = await trx.rawQuery(
`DELETE FROM release_verdicts v WHERE v.rule = ? AND NOT EXISTS (SELECT 1 FROM judged j WHERE j.release_id = v.release_id)`,
[rule.name]
)
round.rules[rule.name].matched += rowCount ?? 0
}
await trx.rawQuery(`ANALYZE judged`)
await this.apply(trx, 'judged', true, round)
})
const added = await trx.rawQuery(
`INSERT INTO release_verdicts (release_id, rule, rule_version)
SELECT j.release_id, ?, ? FROM judged j JOIN releases r ON r.id = j.release_id
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
WHERE release_verdicts.rule <> excluded.rule`,
[rule.name, JUDGE_VERSION]
)
stats.removed = removed.rowCount ?? 0
stats.added = added.rowCount ?? 0
})
}
round.rules[rule.name] = stats
}
return round
}
/**
* The matches (judged: release_id, rule; the first rule's each) into release_verdicts: verdicts no rule gives any
* more removed (scoped: of the releases in judge_scope only), changed ones written. Only the changes go through the
* insert: each match through its conflict check took 60–150 s a rule (LOG.md 2026-10-19 (80)).
*/
private async apply(trx: any, judged: string, scoped: boolean, round: JudgeRound) {
const tally = (rows: { rule: string; n: string }[], field: 'added' | 'removed') => {
for (const { rule, n } of rows) {
round.rules[rule] ??= { matched: 0, added: 0, removed: 0 }
round.rules[rule][field] += Number(n)
}
}
const { rows: removed } = await trx.rawQuery(
`WITH gone AS (DELETE FROM release_verdicts v
WHERE ${scoped ? 'v.release_id IN (SELECT release_id FROM judge_scope) AND' : ''}
NOT EXISTS (SELECT 1 FROM ${judged} j WHERE j.release_id = v.release_id)
RETURNING v.rule)
SELECT rule, count(*) AS n FROM gone GROUP BY 1`
)
tally(removed, 'removed')
const { rows: added } = await trx.rawQuery(
`WITH written AS (INSERT INTO release_verdicts (release_id, rule, rule_version)
SELECT j.release_id, j.rule, ? FROM ${judged} j LEFT JOIN release_verdicts v ON v.release_id = j.release_id
WHERE v.rule IS DISTINCT FROM j.rule AND EXISTS (SELECT 1 FROM releases r WHERE r.id = j.release_id)
ON CONFLICT (release_id) DO UPDATE SET rule = excluded.rule, rule_version = excluded.rule_version
WHERE release_verdicts.rule <> excluded.rule
RETURNING rule)
SELECT rule, count(*) AS n FROM written GROUP BY 1`,
[JUDGE_VERSION]
)
tally(added, 'added')
}
}
/** "dead 1,157,018 (+1,136 −712), …" for the log. */
function describe(round: JudgeRound): string {
return Object.entries(round.rules)
.map(([name, r]) => `${name} ${r.matched.toLocaleString('en')} (+${r.added.toLocaleString('en')} −${r.removed.toLocaleString('en')})`)
.join(', ')
}
app/services/search_service.ts
+1 -8
View File
@@ -57,8 +57,6 @@ export interface ReleaseSummary {
settled: boolean
/** The prober found encrypted archive entries or headers; null: not probed (yet). */
passworded: boolean | null
/** A password is known (release_passwords): it's in the NZB. */
passwordKnown: boolean
/** What the prober found inside (an archive entry, a par2 set's file, a yEnc name), when it says more than the name. */
innerName: string | null
}
@@ -165,7 +163,6 @@ class SearchService {
const [{ rows }, { rows: counted }] = await Promise.all([
db.rawQuery(
`SELECT r.id, r.name, r.source, r.poster, r.groups, r.posted_at, r.file_total, r.files, r.bytes, r.readable, r.complete, r.category, r.settled_at, r.passworded, r.inner_name,
EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id) AS password_known,
release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) AS final
FROM ${source.from} WHERE ${source.where} ORDER BY ${order} LIMIT ? OFFSET ?`,
[...source.bindings, ...bindings, perPage, offset]
@@ -195,7 +192,6 @@ class SearchService {
category: r.category,
settled: r.settled_at !== null,
passworded: r.passworded,
passwordKnown: r.password_known,
innerName: r.inner_name,
})),
}
@@ -205,11 +201,10 @@ class SearchService {
async release(id: number) {
const r = await db.from('releases').where('id', id).first()
if (!r) return null
const [files, { rows: inspections }, { rows: finality }, passwords] = await Promise.all([
const [files, { rows: inspections }, { rows: finality }] = await Promise.all([
NzbService.files(id),
db.rawQuery(`SELECT kind, result, created_at FROM release_inspections WHERE release_id = ? ORDER BY candidate_id`, [id]),
db.rawQuery(`SELECT release_final(?, ?, ?, ?, ?, ?) AS final`, [r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete]),
NzbService.passwords(id),
])
return {
id: Number(r.id),
@@ -231,8 +226,6 @@ class SearchService {
lastSeenAt: r.last_seen_at,
settledAt: r.settled_at,
passworded: r.passworded,
/** Known passwords (spots, their NZBs), the most trusted first; the first is in the NZB. */
passwords,
innerName: r.inner_name,
inspectedAt: r.inspected_at,
/** What the prober read from the release's articles (lib/probe Inspection), by probe kind. */
app/services/settler.ts
+5 -7
View File
@@ -15,8 +15,7 @@ const BATCH = 50_000
* Settles releases (REBUILD.md, part 8): once nothing has been added for
* QUIET_HOURS (QUIET_HISTORY_MINUTES once its last post is that old), its completeness is decided: complete when every file the
* [i/N] counter announced was found, unknown without a counter (phase 2
* decides those by their par2 sets: app/services/par2_completeness.ts).
* Phase 2 picks its work from settled releases.
* reads par2 sets for those). Phase 2 picks its work from settled releases.
*/
class Settler {
/** Up to `batches` batches of BATCH (all there are without). */
@@ -26,14 +25,13 @@ class Settler {
// Their probe candidates become claimable (Prober.claim), the unreadable ones first.
const { rows } = await db.rawQuery(
`WITH settled AS (
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END
UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN NULL ELSE files >= file_total END
WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
RETURNING id, readable, files, bytes, poster, sidecar_only),
RETURNING id, readable, files, bytes, poster),
marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
-- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
-- from it: the merger's sidecar fold looks at them.
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
-- Small ones may be an upload's .nfo or .nzb that came apart from it: the merger's sidecar fold looks at them.
queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
ON CONFLICT DO NOTHING)
SELECT count(*)::int AS n FROM settled`,
[QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
app/services/seven_zip_tails.ts
-22
View File
@@ -1,22 +0,0 @@
/**
* 7z keeps its listing at the end of the last volume, so a split 7z gets a `7z-tail` probe candidate: the last
* segment of its highest volume (lib/edge/range_indexer.ts). RAR lists its files in every volume's header, and
* its first volume's start (one `archive` candidate) is enough. The live tail reads an upload in pieces while
* it's posted, and each piece queues a tail for the last volume it saw so far (TsIgNBewuFbuwWy: 7, for
* .002–.008; LOG.md 2026-10-23 (94)); merges bring their members' tails together. Of the given releases'
* waiting tails, those below the release's highest volume go. Those of the highest volume stay, all of them:
* a piece that saw only part of it queued a segment that isn't its last.
*/
export async function pruneSevenZipTails(trx: any, releaseIds: (number | string)[]): Promise<number> {
if (releaseIds.length === 0) return 0
const result = await trx.rawQuery(
`DELETE FROM probe_candidates pc USING (
SELECT id, status, volume, max(volume) OVER (PARTITION BY release_id) AS highest
FROM (SELECT id, release_id, status, substring(file from '\\.7z\\.(\\d{1,4})$')::int AS volume
FROM probe_candidates WHERE release_id = ANY(?) AND kind = '7z-tail') t
) s
WHERE pc.id = s.id AND s.status = 'waiting' AND s.volume < s.highest`,
[releaseIds.map(String)]
)
return result.rowCount ?? 0
}
app/services/sidecar_folder.ts
+9 -7
View File
@@ -6,7 +6,7 @@ const BATCH = 2000
/** An upload's sidecar is posted with it: its release is looked for this far from it. */
const WINDOW_MINUTES = 10
/** The files an upload carries beside its content: an index, a checksum list, a scene release's own files. */
export const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
const SIDECAR = /\.(nfo|sfv|nzb|srr|srs|txt|m3u|m3u8|diz|md5)$/i
/** The words of a name worth comparing: 4+ letters or digits. */
function words(text: string): Set<string> {
@@ -53,17 +53,19 @@ export class SidecarFolder {
async round(): Promise<FoldRound> {
const round: FoldRound = { looked: 0, sidecars: 0, folded: 0 }
// A candidate whose release went (merged, deleted) leaves with its batch: a separate scan for those over the whole
// queue took 5+ min a round at 13M (LOG.md 2026-10-17 (72)).
const { rows: queued } = await db.rawQuery(
`SELECT f.release_id AS queued_id, r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
`SELECT r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
FROM fold_candidates f JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
[BATCH]
)
const done: number[] = queued.map((r: { queued_id: string }) => Number(r.queued_id))
const { rows: stale } = await db.rawQuery(
`SELECT f.release_id FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id WHERE r.id IS NULL LIMIT ?`,
[BATCH]
)
const done: number[] = stale.map((r: { release_id: string }) => Number(r.release_id))
for (const c of queued) {
if (c.id === null) continue
round.looked++
done.push(Number(c.id))
const { rows: chunks } = await db.rawQuery(`SELECT payload FROM release_chunks WHERE release_id = ?`, [c.id])
const files: ChunkFile[] = chunks.flatMap((x: { payload: Buffer }) => decodeChunk(x.payload))
if (files.length === 0 || !files.every((f) => SIDECAR.test(f.name))) continue
app/services/spot_inspector.ts
-16
View File
@@ -102,12 +102,6 @@ export class SpotInspector {
this.client = undefined
}
if (!round || round.inspected === 0) {
// Nothing to inspect for a while: closed (nntpIdleCloseMs), opened again with work.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs > 0 && IDLE_MS >= idleMs) {
this.client?.close()
this.client = undefined
}
for (let waited = 0; waited < IDLE_MS && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
}
}
@@ -294,12 +288,6 @@ export class SpotInspector {
.ignore()
await trx.from('releases').whereIn('id', [...perRelease.keys()]).update({ passworded: true })
}
// The spot's category for the ones whose names say nothing (migration 5200: spot_category).
await trx.rawQuery(`UPDATE releases SET category = c FROM (SELECT spot_category(?::integer, ?::jsonb) AS c) s WHERE id = ANY(?) AND category = 8010 AND c IS NOT NULL`, [
spot.category,
JSON.stringify(spot.subcats ?? {}),
[...perRelease.keys()],
])
const top = [...perRelease.entries()].sort((a, b) => b[1] - a[1])[0]
// The NZB is only kept for a release of the spot's own: the header pass's releases have their chunks.
await trx.from('spots').where('message_id', spotId).update({ matched: fileRelease.size, release_id: top ? top[0] : null, nzb: null })
@@ -353,8 +341,6 @@ export class SpotInspector {
readable: true,
passworded: spot.password ? true : null,
settled_at: new Date(),
// Kept when its title says nothing (the category trigger, migration 5200).
category: db.raw('coalesce(spot_category(?::integer, ?::jsonb), 8010)', [spot.category, JSON.stringify(spot.subcats ?? {})]),
})
.onConflict('release_key')
.ignore()
@@ -363,8 +349,6 @@ export class SpotInspector {
const releaseId = Number(row.id)
await db.transaction(async (trx) => {
await trx.table('release_spots').insert({ release_id: releaseId, spot_id: spotId, files: files.length }).onConflict(['release_id', 'spot_id']).ignore()
// Its poster for poster search, as the batch store adds its releases' (migration 4000).
if (spot.poster) await trx.rawQuery(`INSERT INTO posters (poster) VALUES (left(?, 200)) ON CONFLICT DO NOTHING`, [spot.poster])
if (spot.password) {
await trx.table('release_passwords').insert({ release_id: releaseId, password: spot.password, source: spot.password_source ?? 'spot-text', spot_id: spotId }).onConflict(['release_id', 'password']).ignore()
}
app/services/spotter.ts
-6
View File
@@ -60,12 +60,6 @@ export class Spotter {
this.client = undefined
}
const wait = coordinatorConfig.spots.everySeconds * 1000
// Quiet until the next round: closed (nntpIdleCloseMs), opened again then.
const idleMs = coordinatorConfig.nntpIdleCloseMs
if (idleMs > 0 && wait - (Date.now() - started) >= idleMs) {
this.client?.close()
this.client = undefined
}
for (let waited = Date.now() - started; waited < wait && !this.stopped; waited += 1000) await new Promise((r) => setTimeout(r, 1000))
}
}
app/validators/user.ts
+2
View File
@@ -14,6 +14,8 @@ export const signupValidator = vine.create({
email: email().unique({ table: 'users', column: 'email' }),
password: password(),
passwordConfirmation: password().sameAs('password'),
// Needed when SIGNUP is `invite` (config/access.ts).
invite: vine.string().trim().maxLength(32).optional(),
})
/**
commands/access_invite.ts
+37
View File
@@ -0,0 +1,37 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import AccessService from '#services/access_service'
import { TIERS } from '#config/access'
/** Makes invite codes (SIGNUP=invite), optionally with a tier for the new account, e.g. a sold VIP year. */
export default class AccessInvite extends BaseCommand {
static commandName = 'access:invite'
static description = 'Makes invite codes, optionally giving a tier (e.g. --tier=vip --days=365)'
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'How many codes', default: 1 })
declare count: number
@flags.string({ description: `Tier the new account gets (${Object.keys(TIERS).join(', ')})` })
declare tier?: string
@flags.number({ description: 'For how many days the tier holds (without: no end)' })
declare days?: number
@flags.string({ description: 'A note, e.g. who it is for or the payment' })
declare note?: string
async run() {
if (this.tier && !TIERS[this.tier]) {
this.logger.error(`No such tier: ${this.tier}`)
this.exitCode = 1
return
}
const codes = Array.from({ length: this.count }, () => AccessService.newKey())
await db.table('invites').multiInsert(
codes.map((code) => ({ code, tier: this.tier ?? null, tier_days: this.tier ? (this.days ?? null) : null, note: this.note ?? null }))
)
for (const code of codes) this.logger.log(code)
}
}
commands/access_suspend.ts
+28
View File
@@ -0,0 +1,28 @@
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import User from '#models/user'
/** Suspends a user (Newznab error 101, the catalog 403), e.g. for a shared key; --undo lifts it. */
export default class AccessSuspend extends BaseCommand {
static commandName = 'access:suspend'
static description = 'Suspends a user (or lifts it with --undo)'
static options: CommandOptions = { startApp: true }
@args.string({ description: 'The user’s email' })
declare email: string
@flags.boolean({ description: 'Lift the suspension' })
declare undo: boolean
async run() {
const user = await User.findBy('email', this.email)
if (!user) {
this.logger.error(`No user ${this.email}`)
this.exitCode = 1
return
}
user.suspended = !this.undo
await user.save()
this.logger.info(`${user.email}: ${user.suspended ? 'suspended' : 'active'}`)
}
}
commands/access_tier.ts
+32
View File
@@ -0,0 +1,32 @@
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import User from '#models/user'
import AccessService from '#services/access_service'
/** Gives a user a tier (a payment, by hand until one is taken on the site); a running one is extended. */
export default class AccessTier extends BaseCommand {
static commandName = 'access:tier'
static description = "Sets a user's tier, e.g. access:tier someone@example.com vip --days=365"
static options: CommandOptions = { startApp: true }
@args.string({ description: 'The user’s email' })
declare email: string
@args.string({ description: 'Tier (config/access.ts)' })
declare tier: string
@flags.number({ description: 'For how many days (without: no end); the same tier still running is extended' })
declare days?: number
async run() {
const user = await User.findBy('email', this.email)
if (!user) {
this.logger.error(`No user ${this.email}`)
this.exitCode = 1
return
}
AccessService.grantTier(user, this.tier, this.days ?? null)
await user.save()
this.logger.info(`${user.email}: ${user.tier} ${user.tierUntil ? `until ${user.tierUntil.toISO()}` : 'without end'}`)
}
}
commands/coordinator_run.ts
+1 -22
View File
@@ -6,9 +6,7 @@ import Planner from '#services/planner'
import FragmentMerger from '#services/fragment_merger'
import Settler from '#services/settler'
import RangeBlocks from '#services/range_blocks'
import Leases from '#services/leases'
import NntpPool from '#services/nntp_pool'
import { NntpShare } from '#services/nntp_shares'
import coordinatorConfig from '#config/coordinator'
const TICK_MS = 2000
@@ -44,7 +42,6 @@ export default class CoordinatorRun extends BaseCommand {
const [index, count] = (this.shard ?? '1/1').split('/').map(Number)
if (!(count >= 1 && index >= 1 && index <= count)) throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
const prefetcher = new Prefetcher({ index: index - 1, count })
const share = new NntpShare(count > 1 ? `coordinator ${index}/${count}` : 'coordinator', coordinatorConfig.nntpShares.coordinatorPriority, coordinatorConfig.nntpMinConnections, coordinatorConfig.nntpConnections)
const main = index === 1
let stopping = false
let stopped: () => void
@@ -57,21 +54,6 @@ export default class CoordinatorRun extends BaseCommand {
await done
})
this.logger.info(count > 1 ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).` : 'Coordinator started.')
// The pool's size: its share of the account's connections, the prober the rest (app/services/nntp_shares.ts).
// Before the first connection: until then it's NNTP_CONNECTIONS.
const first = await NntpPool.share(share)
this.logger.info(`NNTP share: ${first.granted} connections.`)
const sharing = (async () => {
while (!stopping) {
try {
const { was, granted } = await NntpPool.share(share)
if (granted !== was) this.logger.info(`NNTP share: ${granted} connections (was ${was}).`)
} catch (error: any) {
this.logger.error(`NNTP share failed: ${error.message}`)
}
for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !stopping; waited += TICK_MS) await sleep(TICK_MS)
}
})()
// Two loops: the prefetcher's ticks don't wait for merging or planning (a merge round
// of up to MERGE_BUDGET_MS left the connections idle once their targets were fetched).
const prefetching = (async () => {
@@ -88,8 +70,6 @@ export default class CoordinatorRun extends BaseCommand {
const blocks = (async () => {
while (main && !stopping) {
try {
const abandoned = await Leases.finishAbandoned()
if (abandoned > 0) this.logger.info(`Finished ${abandoned} ranges read to their end whose worker went quiet.`)
await RangeBlocks.rebuild()
} catch (error: any) {
this.logger.error(`Range blocks failed: ${error.message}`)
@@ -143,13 +123,12 @@ export default class CoordinatorRun extends BaseCommand {
}
await sleep(TICK_MS)
}
await Promise.all([prefetching, blocks, sharing])
await Promise.all([prefetching, blocks])
// The chunks in flight, for at most STOP_WAIT_MS (a connection the server holds back can take minutes); then closed.
const waitUntil = Date.now() + STOP_WAIT_MS
while (prefetcher.active > 0 && Date.now() < waitUntil) await sleep(200)
if (prefetcher.active > 0) this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
await NntpPool.close()
await share.leave().catch(() => {})
this.logger.info('Coordinator stopped.')
stopped!()
await this.terminate()
commands/corpus_filter.ts
-54
View File
@@ -1,54 +0,0 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import { access, readFile, readdir, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import db from '@adonisjs/lucid/services/db'
import Corpus, { FILTERED } from '#services/corpus'
import RuleSets from '#services/rule_sets'
import coordinatorConfig from '#config/coordinator'
/**
* Filters corpus chunks stored as the news server sent them (before CORPUS_FILTER, 2026-10-23): each chunk without a
* `.drops` of the planned capture ranges (--all: of every range in the corpus) written again as Corpus.write would.
*/
export default class CorpusFilterCommand extends BaseCommand {
static commandName = 'corpus:filter'
static description = 'Filters stored corpus chunks (planned capture ranges; --all: every range) as CORPUS_FILTER does'
static options: CommandOptions = { startApp: true }
@flags.boolean({ description: 'Every range in the corpus, also done ones (the grouping sweeps read them raw)' })
declare all: boolean
async run() {
const dir = coordinatorConfig.corpus.dir
if (!dir) return this.logger.error('CORPUS_DIR is not set')
const rules = await RuleSets.current()
const filter = { ruleVersion: rules.version, priorities: rules.priorities }
const query = db.from('ranges as r').join('groups as g', 'g.id', 'r.group_id').where('r.capture', true).select('g.name as group', 'r.start_id', 'r.end_id')
const ranges = await (this.all ? query : query.where('r.status', 'planned'))
let chunks = 0
let before = 0
let after = 0
for (const r of ranges) {
const range = { group: r.group, start: Number(r.start_id), end: Number(r.end_id) }
const rangeDir = join(dir, range.group, `${range.start}-${range.end}`)
const files = await readdir(rangeDir).catch(() => [] as string[])
const gz = files.filter((f) => f.endsWith('.gz'))
for (const name of gz) {
if (files.includes(`${name}.drops`)) continue
const m = /^\d{4}-(\d+)-(\d+)\.gz$/.exec(name)
if (!m) continue
const data = await readFile(join(rangeDir, name))
await Corpus.write(range, { from: Number(m[1]), to: Number(m[2]) }, data, Number.POSITIVE_INFINITY, filter)
before += data.length
after += (await readFile(join(rangeDir, name))).length
chunks++
}
// A complete range's range.json says it's filtered now (Corpus.write only writes it at the last chunk).
const json = join(rangeDir, 'range.json')
if (gz.length > 0 && (await access(json).then(() => true, () => false))) await writeFile(json, JSON.stringify({ ...range, filtered: FILTERED }))
if (chunks > 0 && chunks % 1000 < gz.length) this.logger.info(`${chunks} chunks filtered: ${(before / 1e9).toFixed(1)} → ${(after / 1e9).toFixed(1)} GB.`)
}
this.logger.info(`Done: ${chunks} chunks of ${ranges.length} ranges filtered, ${(before / 1e9).toFixed(1)} → ${(after / 1e9).toFixed(1)} GB.`)
}
}
commands/nntp_probe.ts
-105
View File
@@ -1,105 +0,0 @@
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import env from '#start/env'
import { XoverClient } from '#lib/nntp/index'
/**
* What the news server does with XOVER spans around an article that made a request hang (ranges:dates: one point
* per group timed out after 2 minutes, again on a fresh connection; LOG.md 2026-10-25 (105)). Each probe on a
* connection of its own with a short timeout: the span asked for, the reply, how many lines, how long. First the
* span itself and wider ones around it, then halves of it down to single articles, then a span past `high`.
*/
export default class NntpProbe extends BaseCommand {
static commandName = 'nntp:probe'
static description =
'Probes XOVER spans around an article that hangs, bisecting to the articles at fault'
static options: CommandOptions = { startApp: true }
@args.string({ description: 'Group' })
declare group: string
@args.string({ description: 'First article of the span that hung' })
declare article: string
@flags.number({ description: 'Articles in the span (default 32)' })
declare size?: number
@flags.number({ description: 'Seconds without data before a probe is given up (default 20)' })
declare timeout?: number
@flags.boolean({ description: 'Only the span itself and the wider ones, no bisection' })
declare noBisect: boolean
private timeoutMs = 20_000
private info = { count: 0, low: 0, high: 0 }
async run() {
this.timeoutMs = (this.timeout ?? 20) * 1000
const a = Number(this.article)
const size = this.size ?? 32
const client = await this.connect()
this.info = await client.group(this.group)
client.close()
this.logger.info(
`${this.group}: ${this.info.count} articles, ${this.info.low}–${this.info.high}.`
)
const span = (from: number, to: number) =>
[Math.max(this.info.low, from), Math.min(this.info.high, to)] as const
this.logger.info('The span and wider ones:')
await this.probe(...span(a, a + size - 1))
await this.probe(...span(a, a + size * 32 - 1))
await this.probe(...span(a - size * 32, a + size - 1))
await this.probe(...span(a - 50_000, a + 49_999))
this.logger.info('Past the end, and an empty span before it:')
await this.probe(this.info.high + 100, this.info.high + 131)
await this.probe(this.info.high + 1, this.info.high + 1)
if (this.noBisect) return
this.logger.info('Halves of the span:')
await this.bisect(a, Math.min(this.info.high, a + size - 1))
}
/** Halves of a span that hangs, down to single articles; a half that answers isn't split further. */
private async bisect(from: number, to: number) {
const hung = await this.probe(from, to)
if (!hung || from === to) return
const mid = Math.floor((from + to) / 2)
await this.bisect(from, mid)
await this.bisect(mid + 1, to)
}
/** One XOVER on a fresh connection; true when it timed out. */
private async probe(from: number, to: number): Promise<boolean> {
const startedAt = Date.now()
let client: XoverClient | null = null
try {
client = await this.connect()
await client.group(this.group)
const rows = await client.xover(from, to)
const t = client.lastTiming
this.logger.info(
` XOVER ${from}-${to}: ${rows.length} lines in ${Date.now() - startedAt} ms (reply ${t.replyMs.toFixed(0)}, body ${t.bodyMs.toFixed(0)})${rows.length > 0 ? `; first ${rows[0][0]}, last ${rows[rows.length - 1][0]}` : ''}`
)
return false
} catch (error: any) {
const timedOut = /timed out/i.test(error.message)
this.logger.warning(
` XOVER ${from}-${to}: ${error.message} after ${Date.now() - startedAt} ms${client ? `, ${client.buffered} bytes buffered` : ''}`
)
return timedOut
} finally {
client?.close()
}
}
private connect() {
return XoverClient.connect({
host: env.get('NNTP_HOST')!,
port: env.get('NNTP_PORT', 563),
user: env.get('NNTP_USER'),
password: env.get('NNTP_PASSWORD'),
secure: env.get('NNTP_SECURE', true),
compress: true,
timeoutMs: this.timeoutMs,
})
}
}
commands/par2_prune.ts
-17
View File
@@ -1,17 +0,0 @@
import { BaseCommand } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { Par2Pruner } from '#services/par2_pruner'
/** The par2 prune (Par2Pruner) in one go: a whole pass from the first set, instead of the merger's 2 minutes an hour. */
export default class Par2Prune extends BaseCommand {
static commandName = 'par2:prune'
static description = 'Drops the par2 file lists the merge can no longer use: one whole pass over every set, then exits'
static options: CommandOptions = { startApp: true }
async run() {
await db.from('worker_cursors').where('name', 'par2-prune').update({ key: '' })
const round = await new Par2Pruner().round()
this.logger.info(`Pruned ${round.rows} par2 file rows: ${round.complete} complete sets, ${round.judged} judged, ${round.final} final.`)
}
}
commands/ranges_capture.ts
-80
View File
@@ -1,80 +0,0 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
/** Bytes a filtered corpus article takes on disk (334 GB for 18.0B articles, 2026-10-25). */
const BYTES_PER_ARTICLE = 18.5
/**
* Marks every active group's ranges posted in a window (by ranges.est_date, from ranges:dates) `capture`, so the
* corpus fill fetches them: voyager 3's test data, a year of all groups at a time (REBUILD.md, corpus). Planned
* ranges not started get --priority (the fill's order, then the indexing's once there's room); done ones are
* fetched again into the corpus only. Ranges leased or half spooled are left out (their first chunks would be
* missing) and reported.
*/
export default class RangesCapture extends BaseCommand {
static commandName = 'ranges:capture'
static description =
'Marks the ranges posted in a window for the corpus: --from 2025-01-01 --to 2026-01-01'
static options: CommandOptions = { startApp: true }
@flags.string({ description: 'First day of the window (ISO date)', required: true })
declare from: string
@flags.string({ description: 'Day after the window (ISO date)', required: true })
declare to: string
@flags.number({
description: 'Priority for the planned ranges marked (default 0: before the history)',
})
declare priority?: number
@flags.boolean({ description: 'Only report what the window holds' })
declare dryRun: boolean
async run() {
const from = new Date(this.from)
const to = new Date(this.to)
if (Number.isNaN(from.getTime()) || Number.isNaN(to.getTime()) || to <= from)
return this.logger.error('--from and --to: ISO dates, from before to')
const priority = this.priority ?? 0
// A range covers est_date up to the next range's; the last range of a group up to now.
const hits = `
WITH spans AS (
SELECT r.id, r.status, r.capture, r.spooled_chunks, r.end_id - r.start_id + 1 AS n, r.est_date AS d0,
lead(r.est_date) OVER (PARTITION BY r.group_id ORDER BY r.start_id) AS d1
FROM ranges r JOIN groups g ON g.id = r.group_id WHERE g.active AND r.est_date IS NOT NULL
), hit AS (SELECT * FROM spans WHERE d0 < ? AND coalesce(d1, now()) > ?)`
const { rows } = await db.rawQuery(
`${hits}
SELECT CASE WHEN capture THEN 'marked already' WHEN status = 'planned' AND spooled_chunks = 0 THEN 'planned'
WHEN status = 'done' THEN 'done' ELSE 'in flight' END AS kind,
count(*) AS ranges, round(sum(n) / 1e9, 2) AS articles
FROM hit GROUP BY 1 ORDER BY 1`,
[to, from]
)
const fetch = rows
.filter((r: any) => r.kind === 'planned' || r.kind === 'done')
.reduce((sum: number, r: any) => sum + Number(r.articles), 0)
this.logger.info(`${this.from} to ${this.to}:`)
for (const r of rows)
this.logger.info(` ${r.kind}: ${r.ranges} ranges, ${r.articles}B articles`)
this.logger.info(
`To fetch: ${fetch.toFixed(2)}B articles, ~${((fetch * 1e9 * BYTES_PER_ARTICLE) / 1e12).toFixed(2)} TB in the corpus.`
)
if (this.dryRun) return
const { rows: marked } = await db.rawQuery(
`${hits}
UPDATE ranges r SET capture = true,
priority = CASE WHEN r.status = 'planned' THEN ? ELSE r.priority END
FROM hit WHERE r.id = hit.id AND NOT hit.capture
AND (hit.status = 'done' OR (hit.status = 'planned' AND hit.spooled_chunks = 0))
RETURNING r.status`,
[to, from, priority]
)
const planned = marked.filter((r: any) => r.status === 'planned').length
this.logger.info(
`Marked ${planned} planned ranges at priority ${priority} and ${marked.length - planned} done ranges.`
)
}
}
commands/ranges_dates.ts
-190
View File
@@ -1,190 +0,0 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import coordinatorConfig from '#config/coordinator'
import NntpPool from '#services/nntp_pool'
import { NntpShare } from '#services/nntp_shares'
import { estimateDate, sampleGroup, type DateSample } from '#services/group_dates'
/**
* When each active group's articles were posted (group_dates, sampled from the news server) and from that every
* range's estimated posting date (ranges.est_date), so ranges can be chosen by era: ranges:capture marks a
* posting-time window of every group for the corpus (REBUILD.md, corpus). A group gets a sample every --every
* articles (16 … 1000): boneless (161B articles) ~320, three weeks apart; ~15k XOVER commands of 32 articles in all.
*/
export default class RangesDates extends BaseCommand {
static commandName = 'ranges:dates'
static description =
"Samples when each active group's articles were posted and estimates every range's date"
static options: CommandOptions = { startApp: true }
@flags.string({ description: 'Only groups whose name matches this SQL LIKE pattern' })
declare group?: string
@flags.number({ description: 'Articles between two samples of a group (default 500M)' })
declare every?: number
@flags.number({
description:
'Connections, a share of the account beside the coordinator and the prober (default 6)',
})
declare connections?: number
@flags.boolean({ description: 'Only groups without samples yet' })
declare missing: boolean
@flags.boolean({
description: 'No sampling: estimate the ranges from the samples stored already',
})
declare estimateOnly: boolean
async run() {
try {
if (!this.estimateOnly) await this.sample()
await this.estimate()
await this.report()
} finally {
await NntpPool.close()
}
}
private groups(missing = this.missing) {
const q = db.from('groups').where('active', true).select('id', 'name').orderBy('priority')
if (this.group) q.where('name', 'like', this.group)
if (missing)
q.whereNotExists((sub) =>
sub.from('group_dates').whereRaw('group_dates.group_id = groups.id')
)
return q
}
private async sample() {
const every = this.every ?? 500_000_000
const connections = this.connections ?? 6
const groups = await this.groups()
const share = new NntpShare(
'dates',
coordinatorConfig.nntpShares.coordinatorPriority + 1,
1,
connections
)
const first = await NntpPool.share(share)
const sharing = setInterval(
() =>
NntpPool.share(share).catch((error) => this.logger.warning(`NNTP share: ${error.message}`)),
coordinatorConfig.nntpShares.everyMs
)
this.logger.info(
`Sampling ${groups.length} groups over ${first.granted} connections, a sample every ${every / 1e6}M articles.`
)
const startedAt = Date.now()
let done = 0
let commands = 0
let samples = 0
const queue = [...groups]
const worker = async () => {
for (let g = queue.shift(); g; g = queue.shift()) {
try {
const result = await sampleGroup((work) => NntpPool.use(work), g.name, {
every,
size: 32,
})
for (const f of result.failed) this.logger.warning(`${g.name} at ${f}`)
await db.transaction(async (trx) => {
await trx.from('group_dates').where('group_id', g.id).delete()
if (result.samples.length > 0)
await trx.table('group_dates').multiInsert(
result.samples.map((s) => ({
group_id: g.id,
article: s.article,
posted_at: s.postedAt,
}))
)
})
commands += result.commands
samples += result.samples.length
if (result.samples.length < 2)
this.logger.warning(
`${g.name}: ${result.samples.length} samples over ${result.low}–${result.high}.`
)
} catch (error: any) {
this.logger.warning(`${g.name}: ${error.message}`)
}
done++
if (done % 50 === 0)
this.logger.info(
`${done} of ${groups.length} groups: ${samples} samples, ${commands} commands, ${((Date.now() - startedAt) / 1000).toFixed(0)} s.`
)
}
}
try {
await Promise.all(
Array.from({ length: Math.max(1, Math.min(connections, groups.length)) }, worker)
)
} finally {
clearInterval(sharing)
}
this.logger.info(
`Sampled ${groups.length} groups: ${samples} samples, ${commands} commands, ${((Date.now() - startedAt) / 1000).toFixed(0)} s.`
)
}
/** Every range of the groups sampled gets est_date from the group's curve (1000 a statement). */
private async estimate() {
const groups = await this.groups(false).whereExists((sub) =>
sub.from('group_dates').whereRaw('group_dates.group_id = groups.id')
)
let updated = 0
for (const g of groups) {
const rows = await db
.from('group_dates')
.where('group_id', g.id)
.orderBy('article')
.select('article', 'posted_at')
const samples: DateSample[] = rows.map((r) => ({
article: Number(r.article),
postedAt: new Date(r.posted_at),
}))
const ranges = await db
.from('ranges')
.where('group_id', g.id)
.orderBy('start_id')
.select('id', 'start_id')
for (let i = 0; i < ranges.length; i += 1000) {
const batch = ranges.slice(i, i + 1000)
const values = batch.map(() => '(?::bigint, ?::timestamptz)').join(', ')
const bindings = batch.flatMap((r) => [
Number(r.id),
estimateDate(samples, Number(r.start_id))!.toISOString(),
])
await db.rawQuery(
`UPDATE ranges AS r SET est_date = v.d FROM (VALUES ${values}) AS v(id, d) WHERE r.id = v.id`,
bindings
)
updated += batch.length
}
}
this.logger.info(`${updated} ranges of ${groups.length} groups dated.`)
}
/** The planned history by posting year: what a window would take. */
private async report() {
const { rows } = await db.rawQuery(`
SELECT extract(year FROM est_date)::int AS year, count(*) AS ranges, count(DISTINCT group_id) AS groups,
round(sum(end_id - start_id + 1) / 1e9, 1) AS articles,
round(sum(end_id - start_id + 1) FILTER (WHERE status = 'done') / 1e9, 1) AS done,
round(sum(end_id - start_id + 1) FILTER (WHERE capture) / 1e9, 1) AS capture
FROM ranges WHERE est_date IS NOT NULL GROUP BY 1 ORDER BY 1`)
const undatedRows = await db.rawQuery(
`SELECT count(*) AS undated FROM ranges WHERE est_date IS NULL`
)
const undated = undatedRows.rows[0].undated
this.logger.info(
`By estimated posting year (B articles; done: indexed, capture: marked for the corpus); ${undated} ranges undated:`
)
for (const r of rows)
this.logger.info(
` ${r.year}: ${String(r.ranges).padStart(7)} ranges of ${String(r.groups).padStart(3)} groups, ${String(r.articles).padStart(6)}B articles, ${r.done}B done, ${r.capture}B capture`
)
}
}
commands/releases_judge.ts
+3 -3
View File
@@ -5,10 +5,10 @@ import { ReleaseJudge, RULES } from '#services/release_judge'
/** The junk judge (REBUILD.md, junk): releases judged after their probes; verdicts hide them from search. */
export default class ReleasesJudge extends BaseCommand {
static commandName = 'releases:judge'
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable): the changed ones every few minutes, all daily'
static description = 'Judges releases by what their probes showed (dead, fake, repeated, locked, unusable), hourly'
static options: CommandOptions = { startApp: true, staysAlive: true }
@flags.boolean({ description: 'One full round, then exit' })
@flags.boolean({ description: 'One round, then exit' })
declare once: boolean
@flags.boolean({ description: 'With --once: count only, write no verdicts' })
@@ -17,7 +17,7 @@ export default class ReleasesJudge extends BaseCommand {
async run() {
const judge = new ReleaseJudge()
if (this.once) {
const round = await judge.full(this.dry)
const round = await judge.round(this.dry)
for (const rule of RULES) {
const r = round.rules[rule.name]
if (r) this.logger.info(`${rule.name}: ${r.matched} (+${r.added} −${r.removed}) — ${rule.says}`)
commands/releases_merge_names.ts
-65
View File
@@ -1,65 +0,0 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { Par2Merger } from '#services/par2_merger'
/**
* The par2 merger's name pass over sets found before it was built (2026-10-23): every set with its file list, newest
* first, by set row. --dry-run counts what it would merge and shows examples, merging nothing.
*/
export default class ReleasesMergeNames extends BaseCommand {
static commandName = 'releases:merge-names'
static description = "Merges releases by their par2 set's file names, for sets found before the name pass; --dry-run shows only"
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'Sets to look at (newest first); all without' })
declare sets: number
@flags.boolean({ description: 'Count and show, merge nothing' })
declare dryRun: boolean
async run() {
const merger = new Par2Merger()
// Keyset on (newest created_at, set_id): sets stored in one batch share created_at, and the timestamp stays text
// so microseconds survive the round trip.
let before = ['9999-12-31', '']
let looked = 0
let merges = 0
let releases = 0
const examples: string[] = []
while (this.sets === undefined || looked < this.sets) {
const { rows } = await db.rawQuery(
`SELECT set_id, at::text AS at FROM (
SELECT s.set_id, max(s.created_at) AS at FROM par2_sets s WHERE EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
GROUP BY s.set_id) x
WHERE (at, set_id) < (?::timestamptz, ?) ORDER BY at DESC, set_id DESC LIMIT ${this.dryRun ? 50 : 500}`,
before
)
if (rows.length === 0) break
for (const { set_id: setId } of rows) {
looked++
if (this.dryRun) {
const found = await db.transaction(async (trx) => merger.nameMatches(trx, setId.trim()))
if (found) {
merges++
releases += found.size
if (examples.length < 15) {
const { rows: names } = await db.rawQuery(`SELECT id, name, files FROM releases WHERE id = ANY(?) ORDER BY id`, [[...found.keys()]])
examples.push(names.map((n: any) => `${n.name.slice(0, 50)} [${n.files}]`).join(' + '))
}
}
} else {
const merged = await merger.mergeByName(setId.trim())
if (merged > 0) {
merges++
releases += merged + 1
}
}
}
before = [rows.at(-1).at, rows.at(-1).set_id]
if (this.dryRun || looked % 20_000 < 500) this.logger.info(`${looked} sets looked at: ${releases} releases ${this.dryRun ? 'would merge' : 'merged'} into ${merges}.`)
}
this.logger.info(`Done: ${looked} sets looked at, ${releases} releases ${this.dryRun ? 'would merge' : 'merged'} into ${merges}.`)
for (const e of examples) this.logger.info(` ${e.slice(0, 300)}`)
}
}
commands/releases_par2_complete.ts
-38
View File
@@ -1,38 +0,0 @@
import { BaseCommand } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { decidePar2Completeness } from '#services/par2_completeness'
/** Releases per batch. */
const BATCH = 1000
/**
* Decides completeness by par2 set (app/services/par2_completeness.ts) for the releases that had their par2 read
* before it was built (2026-10-23): every holder of a set whose file list is still there, by release id. Once.
*/
export default class ReleasesPar2Complete extends BaseCommand {
static commandName = 'releases:par2-complete'
static description = 'Decides completeness by their par2 sets for settled releases without a counter, then exits'
static options: CommandOptions = { startApp: true }
async run() {
let after = 0
let looked = 0
let decided = 0
for (;;) {
const { rows } = await db.rawQuery(
`SELECT DISTINCT s.release_id FROM par2_sets s
WHERE s.release_id > ? AND EXISTS (SELECT 1 FROM par2_files f WHERE f.set_id = s.set_id)
ORDER BY s.release_id LIMIT ?`,
[after, BATCH]
)
if (rows.length === 0) break
const ids = rows.map((r: { release_id: string }) => Number(r.release_id))
decided += await decidePar2Completeness(db, ids)
looked += ids.length
after = ids.at(-1)!
if (looked % 50_000 < BATCH) this.logger.info(`${looked} releases with a par2 set looked at, ${decided} decided (up to id ${after}).`)
}
this.logger.info(`Done: ${looked} releases with a par2 set looked at, ${decided} decided.`)
}
}
commands/releases_recategorize.ts
-92
View File
@@ -1,92 +0,0 @@
import { BaseCommand, flags } from '@adonisjs/core/ace'
import type { CommandOptions } from '@adonisjs/core/types/ace'
import db from '@adonisjs/lucid/services/db'
import { setTimeout as sleep } from 'node:timers/promises'
import { DiskSpace } from '#services/disk_space'
/** Its place in worker_cursors (the last release id done), for a run that was stopped. */
const CURSOR = 'recategorize'
/**
* The categories of releases from before migration 5200 (release_category_for: the fixed name rules, the inner
* name, the groups; then the spots'), in batches by id: only rows whose category changes are written, each a new row
* version in a 70 GB table on the NVMe, so it waits while the filesystem is under --floor-gb free (autovacuum makes
* the dead versions reusable) and goes on from its cursor when run again.
*/
export default class ReleasesRecategorize extends BaseCommand {
static commandName = 'releases:recategorize'
static description = 'Recomputes release categories (name, inner name, groups, spots) in batches; --dry-run counts only'
static options: CommandOptions = { startApp: true }
@flags.number({ description: 'Release ids per batch' })
declare batch: number
@flags.number({ description: 'Stop after this id (a sample with --dry-run)' })
declare toId: number
@flags.number({ description: 'Wait while the NVMe has less free (GB)' })
declare floorGb: number
@flags.boolean({ description: 'Count what would change, write nothing' })
declare dryRun: boolean
@flags.boolean({ description: 'Start from the first id, not the cursor' })
declare restart: boolean
async run() {
const batch = this.batch ?? 20_000
const space = new DiskSpace('recategorizing', '/', this.floorGb ?? 14)
const { rows: ends } = await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM releases`)
const hi = Math.min(Number(ends[0].hi), this.toId ?? Infinity)
const { rows: cursors } = await db.rawQuery(`SELECT key FROM worker_cursors WHERE name = ?`, [CURSOR])
let after = this.restart || this.dryRun || !cursors[0] ? Number(ends[0].lo) - 1 : Number(cursors[0].key)
const changes = new Map<string, number>()
let changed = 0
let looked = 0
const started = Date.now()
while (after < hi) {
while (!this.dryRun && (await space.short())) await sleep(60_000)
const to = Math.min(after + batch, hi)
const fresh = `release_category_for(name, inner_name, groups, NULL)`
if (this.dryRun) {
const { rows } = await db.rawQuery(
`SELECT category AS was, ${fresh} AS now, count(*)::int AS n FROM releases WHERE id > ? AND id <= ? GROUP BY 1, 2`,
[after, to]
)
for (const r of rows) {
looked += r.n
if (r.was === r.now) continue
changed += r.n
changes.set(`${r.was} → ${r.now}`, (changes.get(`${r.was} → ${r.now}`) ?? 0) + r.n)
}
} else {
const result = await db.rawQuery(
`UPDATE releases r SET category = x.c FROM (SELECT id, ${fresh} AS c FROM releases WHERE id > ? AND id <= ?) x
WHERE r.id = x.id AND r.category IS DISTINCT FROM x.c`,
[after, to]
)
changed += result.rowCount ?? 0
await db.rawQuery(
`INSERT INTO worker_cursors (name, at, key) VALUES (?, now(), ?) ON CONFLICT (name) DO UPDATE SET at = excluded.at, key = excluded.key`,
[CURSOR, String(to)]
)
}
after = to
if (Math.floor(after / 5_000_000) !== Math.floor((after - batch) / 5_000_000)) {
this.logger.info(`Up to id ${after} of ${hi}: ${changed} changed${this.dryRun ? ` of ${looked}` : ''} (${Math.round((Date.now() - started) / 1000)} s).`)
}
}
if (this.dryRun) {
this.logger.info(`Would change ${changed} of ${looked}:`)
for (const [change, n] of [...changes].sort((a, b) => b[1] - a[1]).slice(0, 30)) this.logger.info(` ${change}: ${n}`)
return
}
// The spots' categories for the linked releases whose names say nothing.
const spots = await db.rawQuery(
`UPDATE releases r SET category = spot_category(s.category, s.subcats)
FROM release_spots rs JOIN spots s ON s.message_id = rs.spot_id
WHERE r.id = rs.release_id AND r.category = 8010 AND spot_category(s.category, s.subcats) IS NOT NULL`
)
this.logger.info(`Done: ${changed} releases recategorized, ${spots.rowCount ?? 0} by their spots.`)
}
}
config/access.ts
+31
View File
@@ -0,0 +1,31 @@
import env from '#start/env'
/**
* Who may use the catalog and the Newznab API.
*
* `open` (the default): as before, one key for Newznab (NEWZNAB_API_KEY, or
* none) and an open web catalog, no limits. `accounts`: every request names a
* user (a Newznab `apikey`, or the web interface's token), whose tier sets what
* they may do per UTC day and how far back they see.
*/
export const ACCESS = env.get('ACCESS', 'open')
/** Who may sign up: anyone, with an unused invite code, or no one. With accounts, invites unless set. */
export const SIGNUP = env.get('SIGNUP', ACCESS === 'accounts' ? 'invite' : 'open')
export interface TierLimits {
/** Searches and release details per day (Newznab t=search & co., the catalog); null: no limit. */
apiHits: number | null
/** NZBs per day; null: no limit. */
grabs: number | null
/** How many days back releases are found and fetched; null: everything. */
retentionDays: number | null
}
/** A tier that runs out (users.tier_until) falls back to `free`. */
export const TIERS: Record<string, TierLimits> = {
free: { apiHits: 100, grabs: 10, retentionDays: 365 },
vip: { apiHits: 10_000, grabs: 1_000, retentionDays: null },
// Staff and friends.
unlimited: { apiHits: null, grabs: null, retentionDays: null },
}
config/coordinator.ts
+2 -32
View File
@@ -31,21 +31,8 @@ export default {
},
/** A leased range without a heartbeat for this long goes back to the queue. */
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
/** NNTP connections the coordinator fetches headers with: at most (its share, nntpShares), and at least while sharing. */
/** NNTP connections the coordinator fetches headers with. */
nntpConnections: env.get('NNTP_CONNECTIONS', 4),
nntpMinConnections: env.get('NNTP_MIN_CONNECTIONS', 8),
/**
* The account's connections, shared by the coordinator and the prober by priority (app/services/nntp_shares.ts):
* all of them less the reserved ones (the spotter's and the spot inspector's 1 each, the coordinator's plain one,
* and 2 for a hedged request's or a reconnect's new connection while the old one closes).
*/
nntpShares: {
total: env.get('NNTP_ACCOUNT_CONNECTIONS', 50) - env.get('NNTP_RESERVED_CONNECTIONS', 5),
everyMs: 15_000,
/** Headers before probes: history takes what it can use, the prober the rest. */
coordinatorPriority: 2,
proberPriority: 1,
},
/**
* A reply with no data for this long fails, and its connection is replaced.
* Eweka sometimes stops answering on long-lived connections (all 40 quiet,
@@ -81,24 +68,13 @@ export default {
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
*/
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
/**
* A connection with nothing to do for this long is closed, and opened again
* when there's work. 0: kept. Eweka drops connections that stay quiet, and
* the next command on one failed ("Spot inspection failed: NNTP connection
* closed"); idle ones also take the account's connections from downloads.
*/
nntpIdleCloseMs: env.get('NNTP_IDLE_CLOSE_SECONDS', 60) * 1000,
probe: {
/**
* At most (its share, nntpShares): NNTP connections the prober reads articles with, besides the
* NNTP connections the prober reads articles with, besides the
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
*/
connections: env.get('PROBE_CONNECTIONS', 6),
/** Of them, always (its share's minimum; nntpShares). */
minConnections: env.get('PROBE_MIN_CONNECTIONS', 2),
/** No claims while the filesystem under `/` (the NVMe, in the container) has less free (app/services/disk_space.ts). 0: off. */
minFreeGb: env.get('PROBE_MIN_FREE_GB', 0),
/** Candidates claimed at a time. */
batch: 400,
/** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
@@ -131,12 +107,6 @@ export default {
* ranges waited behind it for hours (1,400 marked, none stored, 34 of 40 connections idle; LOG.md 2026-10-12 (27)).
*/
ahead: env.get('CORPUS_AHEAD', 16),
/**
* Stored without the lines the edge worker drops on their own (crossposts another indexed group owns, obfuscated
* subjects; ~48% of the indexed ranges' headers), what was left out in each chunk's `.drops`: twice the history
* in the same space (LOG.md 2026-10-23 (101)). Off: as the news server sent them.
*/
filter: env.get('CORPUS_FILTER', true),
},
spool: {
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
database/migrations/1770000003300_access.ts
+49
View File
@@ -0,0 +1,49 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Accounts with limits (config/access.ts): a Newznab key and a tier per user,
* invites, and what each user used per day.
*/
export default class extends BaseSchema {
async up() {
this.schema.alterTable('users', (table) => {
// Existing users get a key of their own (a volatile default is evaluated per row).
table.string('api_key', 32).notNullable().unique().defaultTo(this.raw(`md5(random()::text || clock_timestamp()::text)`))
table.string('tier', 16).notNullable().defaultTo('free')
// Null: the tier doesn't run out.
table.timestamp('tier_until', { useTz: true }).nullable()
table.boolean('suspended').notNullable().defaultTo(false)
})
this.schema.createTable('invites', (table) => {
table.string('code', 32).primary()
// What the invite gives (e.g. a sold VIP invite); null: the free tier.
table.string('tier', 16).nullable()
table.integer('tier_days').nullable()
table.string('note').nullable()
table.integer('used_by').nullable().references('id').inTable('users').onDelete('SET NULL')
table.timestamp('used_at', { useTz: true }).nullable()
table.timestamp('created_at', { useTz: true }).notNullable().defaultTo(this.now())
})
this.schema.createTable('usage', (table) => {
table.integer('user_id').notNullable().references('id').inTable('users').onDelete('CASCADE')
// A UTC day.
table.date('day').notNullable()
table.integer('api_hits').notNullable().defaultTo(0)
table.integer('grabs').notNullable().defaultTo(0)
table.primary(['user_id', 'day'])
})
}
async down() {
this.schema.dropTable('usage')
this.schema.dropTable('invites')
this.schema.alterTable('users', (table) => {
table.dropColumn('api_key')
table.dropColumn('tier')
table.dropColumn('tier_until')
table.dropColumn('suspended')
})
}
}
database/migrations/1770000004200_releases_sidecar_only.ts
-16
View File
@@ -1,16 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* releases.sidecar_only: set by the batch store when it inserts a small release (at most 3 files, under 1 MB) whose
* files are all sidecars (.nfo, .sfv, .nzb, …). The settler queues only those for the sidecar fold: it queued every
* small release, 13.5M (single images mostly), and the fold's queue scans held the merger back (LOG.md 2026-10-17 (72)).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`ALTER TABLE releases ADD COLUMN IF NOT EXISTS sidecar_only boolean`)
}
async down() {
this.schema.raw(`ALTER TABLE releases DROP COLUMN IF EXISTS sidecar_only`)
}
}
database/migrations/1770000004300_worker_cursors_key.ts
-16
View File
@@ -1,16 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* worker_cursors.key: a tie-breaker after `at`. The par2 pruner's cursor was a timestamp alone, read back to the
* millisecond: 20,000 sets of one prober flush share created_at 2026-10-15 14:38:39.883984, and every round took the
* same batch again (LOG.md 2026-10-17 (73)). Now (at to the microsecond, set_id).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`ALTER TABLE worker_cursors ADD COLUMN IF NOT EXISTS key text`)
}
async down() {
this.schema.raw(`ALTER TABLE worker_cursors DROP COLUMN IF EXISTS key`)
}
}
database/migrations/1770000004400_nntp_shares.ts
-26
View File
@@ -1,26 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* nntp_shares: the account's connections shared by the processes that read articles (app/services/nntp_shares.ts).
* Each process's row says what it could use and has open, and what it may open; a row without a heartbeat for a
* minute is a process gone (its connections closed with it). The split was fixed per process: with history paused
* the coordinator's 38 stood idle while the prober's 8 were the limit (LOG.md 2026-10-18 (75)).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`
CREATE TABLE IF NOT EXISTS nntp_shares (
name text PRIMARY KEY,
priority integer NOT NULL,
min integer NOT NULL,
want integer NOT NULL,
open integer NOT NULL,
granted integer NOT NULL,
heartbeat_at timestamptz NOT NULL DEFAULT now()
)`)
}
async down() {
this.schema.raw(`DROP TABLE IF EXISTS nntp_shares`)
}
}
database/migrations/1770000004500_probe_cache_pkey_on_nvme.ts
-26
View File
@@ -1,26 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* probe_cache's primary key in the catalog's tablespace (default_tablespace: nvme on homelab); the table stays on the
* HDD. Each claim looks its candidates up by message_id (`message_id = ANY(…)`): with the index on the HDD that took
* 1.27 s a claim, 48 s of each minute, the prober's lanes waiting on claims (LOG.md 2026-10-18 (76)). ~97% aren't in it,
* so only the index is read. Homelab moved it by hand (prober stopped); a no-op where it's there already, or without a
* default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE c.relname = 'probe_cache_pkey' AND coalesce(t.spcname, 'pg_default') <> ?`,
[target]
)
if (rows.length > 0) await db.rawQuery(`ALTER INDEX probe_cache_pkey SET TABLESPACE ${target}`)
})
}
async down() {}
}
database/migrations/1770000004600_release_inspections_on_hdd.ts
-12
View File
@@ -1,12 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Was: release_inspections' rows on the HDD (pg_default), for the NVMe's room during the probe backlog (LOG.md
* 2026-10-18 (79)). The judge's fake rule reads all of it every round: on the HDD that took 17–18 min, was ~3
* (LOG.md 2026-10-19 (81)). Migration 4800 moves it back; nothing here now.
*/
export default class extends BaseSchema {
async up() {}
async down() {}
}
database/migrations/1770000004700_judge_indexes.ts
-22
View File
@@ -1,22 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The junk judge's locked rule by an index of the passworded releases (1.2M of 74M), not a scan of all of releases
* every round (30 s; LOG.md 2026-10-19 (80)). And the posters of the spot inspector's own releases (source `spot`),
* which it didn't add to posters: poster search, and the judge's scraped rule by posters, missed them.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS releases_passworded_idx ON releases (id) WHERE passworded`)
this.schema.raw(
`INSERT INTO posters (poster) SELECT DISTINCT left(s.poster, 200) FROM spots s
JOIN releases r ON r.id = s.release_id AND r.source = 'spot' WHERE s.poster IS NOT NULL ON CONFLICT DO NOTHING`
)
}
async down() {
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS releases_passworded_idx`)
}
}
database/migrations/1770000004800_release_inspections_on_nvme.ts
-25
View File
@@ -1,25 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* release_inspections back in the catalog's tablespace (default_tablespace: nvme on homelab), where migration 4600 had
* moved it to the HDD: the judge's fake rule reads all of it every round, 17–18 min on the HDD, ~3 on the NVMe (LOG.md
* 2026-10-19 (81)). Homelab moved it by hand (prober, merger and judge stopped); a no-op where it's there already, or
* without a default tablespace (tests).
*/
export default class extends BaseSchema {
async up() {
this.defer(async (db) => {
const { rows: setting } = await db.rawQuery(`SELECT current_setting('default_tablespace') AS target`)
const target = setting[0].target
if (!target) return
const { rows } = await db.rawQuery(
`SELECT 1 FROM pg_class c LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE c.relname = 'release_inspections' AND c.relkind = 'r' AND coalesce(t.spcname, 'pg_default') <> ?`,
[target]
)
if (rows.length > 0) await db.rawQuery(`ALTER TABLE release_inspections SET TABLESPACE ${target}`)
})
}
async down() {}
}
database/migrations/1770000004900_judge_dirty.ts
-119
View File
@@ -1,119 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* The junk judge by what changed (LOG.md 2026-10-19 (83)): a full round over every release took 35–38 min of each
* hour, longer with every inspection the prober added. Triggers note each release whose facts a rule reads changed
* (judge_dirty: its probes' statuses, inspections, file hashes, passwords, spots, par2 sets, merges, its own judged
* columns); the judge judges those every few minutes, and every release in a daily full round (judge_state: when the
* last one was, by which rule set), its matches gathered in judge_full. judge_dirty is unlogged: what a crash loses
* the next full round judges.
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_dirty (release_id bigint NOT NULL)`)
this.schema.raw(`CREATE UNLOGGED TABLE IF NOT EXISTS judge_full (release_id bigint PRIMARY KEY, rule varchar(16) NOT NULL)`)
this.schema.raw(
`CREATE TABLE IF NOT EXISTS judge_state (id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1), full_at timestamptz NOT NULL, version integer NOT NULL)`
)
// The verdicts are an hourly full round's: the next full one a day from now.
this.schema.raw(`INSERT INTO judge_state (full_at, version) VALUES (now(), 1) ON CONFLICT DO NOTHING`)
// Rows of a table with release_id: the releases of the rows inserted, deleted, or updated (both, when moved).
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_by_release_id() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF TG_OP IN ('INSERT', 'UPDATE') THEN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM new_rows WHERE release_id IS NOT NULL;
END IF;
IF TG_OP IN ('UPDATE', 'DELETE') THEN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT release_id FROM old_rows WHERE release_id IS NOT NULL;
END IF;
RETURN NULL;
END $$`)
// probe_candidates: only a status that counts for the dead rule, or a candidate moved by a merge (not the claims'
// 'taken', the settler's ready).
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_probes() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
INSERT INTO judge_dirty (release_id)
SELECT DISTINCT unnest(ARRAY[n.release_id, o.release_id]) FROM new_rows n JOIN old_rows o ON o.id = n.id
WHERE n.release_id IS DISTINCT FROM o.release_id
OR (n.status IS DISTINCT FROM o.status AND (n.status IN ('done', 'missing', 'error') OR o.status IN ('done', 'missing', 'error')));
RETURN NULL;
END $$`)
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_merges() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
INSERT INTO judge_dirty (release_id) SELECT DISTINCT into_id FROM new_rows;
RETURN NULL;
END $$`)
this.schema.raw(`
CREATE OR REPLACE FUNCTION judge_dirty_releases() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO judge_dirty (release_id) SELECT id FROM new_rows;
ELSIF TG_OP = 'DELETE' THEN
INSERT INTO judge_dirty (release_id) SELECT id FROM old_rows;
ELSE
INSERT INTO judge_dirty (release_id) VALUES (NEW.id);
END IF;
RETURN NULL;
END $$`)
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets']) {
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON ${table}`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON ${table}`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON ${table}`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON ${table} REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON ${table} REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON ${table} REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
}
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON probe_candidates`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON probe_candidates`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON probe_candidates REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_probes()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON probe_candidates REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON release_merges`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON release_merges REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_merges()`
)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_insert ON releases`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_update ON releases`)
this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_delete ON releases`)
this.schema.raw(
`CREATE TRIGGER judge_dirty_insert AFTER INSERT ON releases REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
)
this.schema.raw(
`CREATE TRIGGER judge_dirty_delete AFTER DELETE ON releases REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`
)
// Per row, and only when a column a rule reads changed: the tail and the settler update releases all the time.
this.schema.raw(
`CREATE TRIGGER judge_dirty_update AFTER UPDATE ON releases FOR EACH ROW
WHEN (OLD.name IS DISTINCT FROM NEW.name OR OLD.poster IS DISTINCT FROM NEW.poster OR OLD.source IS DISTINCT FROM NEW.source
OR OLD.file_total IS DISTINCT FROM NEW.file_total OR OLD.passworded IS DISTINCT FROM NEW.passworded
OR OLD.settled_at IS DISTINCT FROM NEW.settled_at OR OLD.readable IS DISTINCT FROM NEW.readable
OR OLD.inner_name IS DISTINCT FROM NEW.inner_name OR OLD.group_id IS DISTINCT FROM NEW.group_id
OR OLD.first_article IS DISTINCT FROM NEW.first_article OR OLD.last_article IS DISTINCT FROM NEW.last_article
OR OLD.posted_at IS DISTINCT FROM NEW.posted_at OR OLD.last_posted_at IS DISTINCT FROM NEW.last_posted_at
OR OLD.complete IS DISTINCT FROM NEW.complete)
EXECUTE FUNCTION judge_dirty_releases()`
)
}
async down() {
for (const table of ['release_inspections', 'file_hashes', 'release_passwords', 'release_spots', 'par2_sets', 'probe_candidates', 'release_merges', 'releases']) {
for (const event of ['insert', 'update', 'delete']) this.schema.raw(`DROP TRIGGER IF EXISTS judge_dirty_${event} ON ${table}`)
}
this.schema.raw(`DROP FUNCTION IF EXISTS judge_dirty_by_release_id(), judge_dirty_probes(), judge_dirty_merges(), judge_dirty_releases()`)
this.schema.raw(`DROP TABLE IF EXISTS judge_dirty, judge_full, judge_state`)
}
}
database/migrations/1770000004900_range_dates.ts
-27
View File
@@ -1,27 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* When a group's articles were posted, sampled from the news server (ranges:dates; app/services/group_dates.ts), and
* from those each range's estimated posting date, so ranges can be chosen by era: ranges:capture --from --to marks a
* posting-time window of every group for the corpus (voyager 3's test data; REBUILD.md, corpus). `stored_at`: the
* range is complete in the corpus (the prefetcher saw its range.json), so the corpus fill skips it without looking.
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`
CREATE TABLE IF NOT EXISTS group_dates (
group_id integer NOT NULL REFERENCES groups (id) ON DELETE CASCADE,
article bigint NOT NULL,
posted_at timestamptz NOT NULL,
PRIMARY KEY (group_id, article)
)`)
this.schema.raw(`ALTER TABLE ranges ADD COLUMN IF NOT EXISTS est_date timestamptz`)
this.schema.raw(`ALTER TABLE ranges ADD COLUMN IF NOT EXISTS stored_at timestamptz`)
}
async down() {
this.schema.raw(`ALTER TABLE ranges DROP COLUMN IF EXISTS stored_at`)
this.schema.raw(`ALTER TABLE ranges DROP COLUMN IF EXISTS est_date`)
this.schema.raw(`DROP TABLE IF EXISTS group_dates`)
}
}
database/migrations/1770000005000_file_hashes_hash_size_release.ts
-21
View File
@@ -1,21 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* file_hashes by (hash16k, size, release_id), in place of (hash16k): how many releases have a file, from the index
* alone. The judge counts that for every file of each changed release (repeated); with the row fetched for size and
* release_id per entry, a file posted in 100k releases cost 100k heap reads every round (24–32 s a chunk; LOG.md
* 2026-10-19 (84)). The merger and pruner join on (hash16k, size) too.
*/
export default class extends BaseSchema {
static disableTransactions = true
async up() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_size_release_idx ON file_hashes (hash16k, size, release_id)`)
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_idx`)
}
async down() {
this.schema.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS file_hashes_hash_idx ON file_hashes (hash16k)`)
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS file_hashes_hash_size_release_idx`)
}
}
database/migrations/1770000005100_late_candidates_ready.ts
-17
View File
@@ -1,17 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Probe candidates added to a release after it settled (a later range of its upload, a merge into it) stayed not
* ready: the settler marks a release's candidates only when it settles it (LOG.md 2026-10-20 (87)). 93,642 waited
* since 2026-10-13 (17k lone, 31k par2, 45k archive). The batch store and the merge mark them now; these are made
* claimable once.
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(`
UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.status = 'waiting' AND NOT pc.ready AND r.id = pc.release_id AND r.settled_at IS NOT NULL`)
}
async down() {}
}
database/migrations/1770000005200_release_category_trigger.ts
-128
View File
@@ -1,128 +0,0 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* A release's category from more than its name (REBUILD.md, part 9; LOG.md 2026-10-23 (95)). Migration 1200 made it
* a generated column of release_category(name), and knex took the `?` in its regexes for bindings: the function held
* `blu-$1ray`, `web-$2dl`, `h\.$326[45]`, so BluRay, WEB-DL and H.264 names without a resolution were Misc. Now:
* - release_category(name), fixed (no `?` in any regex here: `{0,1}`) and with XXX, audiobooks, console and PC games,
* and TV codes the old one missed (`S01e07[`, `S04E01E02`);
* - the inner name the prober found (an obfuscated release's real name) when the name says nothing;
* - else a category given before (a spot's, below), else the groups' (erotica, sounds, ebooks, …);
* - a spot's category (spot_category: its head category and subcategories), set by the spot inspector.
* `category` becomes a plain column (DROP EXPRESSION keeps the values, no rewrite) set by a trigger; existing rows are
* recategorized by `releases:recategorize` in batches, not here (a rewrite of the 70 GB table).
*/
export default class extends BaseSchema {
async up() {
this.schema.raw(String.raw`
CREATE OR REPLACE FUNCTION release_category(name text) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
SELECT CASE
-- XXX first: its names carry years, resolutions and episode-like codes too.
WHEN name ~* '(^|[^a-z0-9])xxx([^a-z0-9]|$)' THEN
CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 6045
WHEN name ~* '720p|1080p|x26[45]|h\.{0,1}26[45]|hevc' THEN 6040
WHEN name ~* 'imageset|picset|(^|[._ -])pics([._ -]|$)' THEN 6060 ELSE 6070 END
-- TV: S01E02 (also S01e02[, S04E01E02), 1x02, or a season pack (S01).
WHEN name ~* '(^|[^a-z0-9])(s[0-9]{1,2}[ ._-]{0,1}e[0-9]{1,3}|[0-9]{1,2}x[0-9]{2})([^0-9]|$)' OR name ~* '(^|[._ -])s[0-9]{1,2}([._ -]|$)' THEN
CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 5045
WHEN name ~* '720p|1080p' THEN 5040 ELSE 5030 END
-- Movies: a year plus a source or quality marker.
WHEN name ~* '(^|[._ (-])(19|20)[0-9]{2}([._ )-]|$)'
AND name ~* '(480p|576p|720p|1080p|2160p|blu-{0,1}ray|web-{0,1}dl|webrip|bdrip|brrip|dvdrip|hdtv|x26[45]|h\.{0,1}26[45]|hevc|remux)' THEN
CASE WHEN name ~* '2160p|(^|[._ -])(uhd|4k)([._ -]|$)' THEN 2045
WHEN name ~* '720p|1080p|blu-{0,1}ray' THEN 2040 ELSE 2030 END
-- Audio: audiobooks, lossless, mp3, or scene music names (-WEB-2026-GROUP).
WHEN name ~* '(audiobook|audioboek|luisterboek|h(oe|ö)rbuch)' THEN 3030
WHEN name ~* '(^|[._ -])flac([._ -]|$)' THEN 3040
WHEN name ~* '(^|[._ -])mp3([._ -]|$)' OR name ~* '-(WEB|CD|CDM|CDS|EP|VINYL|VLS|SAT|FM|DAB)-(19|20)[0-9]{2}-[A-Za-z0-9]+$' THEN 3010
WHEN name ~* '(^|[._ -])(epub|ebook|e-book|mobi|azw3)([._ -]|$)' THEN 7020
-- Games: console tags, PC scene groups and repackers.
WHEN name ~* '(^|[._ -])(nsw|ps4|ps5|xboxone|xbox360|wiiu|3ds|nds)([._ -]|$)' THEN 1000
WHEN name ~* '-(codex|plaza|skidrow|reloaded|flt|doge|tenoke|rune|empress|razor1911|tinyiso|darksiders|simplex)$' OR name ~* '(fitgirl|dodi|elamigos)[._ -]{0,1}repack' THEN 4050
ELSE 8010
END
$$`)
this.schema.raw(String.raw`
CREATE OR REPLACE FUNCTION release_group_category(groups jsonb) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
SELECT CASE
WHEN g ~* '(erotica|xxx|porn|erotic|nude|naked)' THEN CASE WHEN g ~* '(pictures|pics|images|photos)' THEN 6060 ELSE 6070 END
WHEN g ~* 'anime' THEN 5070
WHEN g ~* 'audiobook' THEN 3030
WHEN g ~* '(sounds|mp3|music)' THEN CASE WHEN g ~* '(lossless|flac)' THEN 3040 ELSE 3010 END
WHEN g ~* 'comics' THEN 7030
WHEN g ~* 'e-{0,1}books{0,1}' THEN 7020
WHEN g ~* '(games|nintendo|xbox|playstation|wii)' THEN 1000
WHEN g ~* '(^|[. ])(tv|hdtv|teevee|tvseries)([. ]|$)' THEN 5000
WHEN g ~* '(^|[. ])(movies|dvd|bluray|dvdr)([. ]|$)' THEN 2000
ELSE 8010
END
FROM (SELECT coalesce(string_agg(value, ' '), '') AS g FROM jsonb_array_elements_text(CASE WHEN jsonb_typeof(groups) = 'array' THEN groups ELSE '[]' END)) x
$$`)
// The name's, else the inner name's, else one given before (a spot's), else the groups'.
this.schema.raw(`
CREATE OR REPLACE FUNCTION release_category_for(name text, inner_name text, groups jsonb, given integer) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
SELECT CASE WHEN n <> 8010 THEN n WHEN i <> 8010 THEN i WHEN coalesce(given, 8010) <> 8010 THEN given ELSE release_group_category(groups) END
FROM (SELECT release_category(name) AS n, coalesce(release_category(inner_name), 8010) AS i) x
$$`)
// Whether a spot's subcategories of one type letter ({"a": [9], "z": [0]}) hold one of `vals`.
this.schema.raw(`
CREATE OR REPLACE FUNCTION spot_has(subcats jsonb, letter text, vals integer[]) RETURNS boolean IMMUTABLE LANGUAGE sql AS $$
SELECT coalesce((SELECT bool_or((e #>> '{}')::integer = ANY(vals)) FROM jsonb_array_elements(
CASE WHEN jsonb_typeof(subcats->letter) = 'array' THEN subcats->letter ELSE '[]' END) e), false)
$$`)
// A spot's category (lib/spotnet/categories.ts) in Newznab's; null for pictures and the unknown.
this.schema.raw(`
CREATE OR REPLACE FUNCTION spot_category(category integer, subcats jsonb) RETURNS integer IMMUTABLE LANGUAGE sql AS $$
SELECT CASE category
WHEN 0 THEN CASE
WHEN spot_has(subcats, 'z', '{3}') THEN CASE WHEN spot_has(subcats, 'a', '{15}') THEN 6045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 6040
WHEN spot_has(subcats, 'a', '{12,13}') THEN 6060 ELSE 6070 END
WHEN spot_has(subcats, 'z', '{1}') THEN CASE WHEN spot_has(subcats, 'a', '{15}') THEN 5045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 5040 ELSE 5030 END
WHEN spot_has(subcats, 'z', '{2}') OR spot_has(subcats, 'a', '{5,11}') THEN 7020
WHEN spot_has(subcats, 'z', '{4}') THEN NULL
ELSE CASE WHEN spot_has(subcats, 'a', '{15}') THEN 2045 WHEN spot_has(subcats, 'a', '{4,6,7,8,9}') THEN 2040 ELSE 2030 END
END
WHEN 1 THEN CASE WHEN spot_has(subcats, 'z', '{3}') THEN 3030 WHEN spot_has(subcats, 'a', '{2,4,7,8}') OR spot_has(subcats, 'c', '{8}') THEN 3040 ELSE 3010 END
WHEN 2 THEN CASE WHEN spot_has(subcats, 'a', '{0,1,2}') THEN 4050 ELSE 1000 END
WHEN 3 THEN 4000
END
$$`)
this.schema.raw(`
CREATE OR REPLACE FUNCTION releases_set_category() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
NEW.category := release_category_for(NEW.name, NEW.inner_name, NEW.groups, CASE WHEN TG_OP = 'UPDATE' THEN OLD.category ELSE NEW.category END);
RETURN NEW;
END
$$`)
// ACCESS EXCLUSIVE for a moment, while the workers write: a few tries instead of a queue behind them.
this.schema.raw(`
DO $$
BEGIN
FOR attempt IN 1..30 LOOP
BEGIN
PERFORM set_config('lock_timeout', '5s', true);
IF EXISTS (SELECT 1 FROM information_schema.columns WHERE table_name = 'releases' AND column_name = 'category' AND is_generated = 'ALWAYS') THEN
ALTER TABLE releases ALTER COLUMN category DROP EXPRESSION;
END IF;
DROP TRIGGER IF EXISTS releases_category ON releases;
CREATE TRIGGER releases_category BEFORE INSERT OR UPDATE OF name, inner_name, groups ON releases
FOR EACH ROW EXECUTE FUNCTION releases_set_category();
RETURN;
EXCEPTION WHEN lock_not_available THEN
PERFORM pg_sleep(2);
END;
END LOOP;
RAISE EXCEPTION 'releases: no lock for the category trigger in 30 tries';
END
$$`)
}
async down() {
this.schema.raw(`DROP TRIGGER IF EXISTS releases_category ON releases`)
this.schema.raw(`DROP FUNCTION IF EXISTS releases_set_category()`)
this.schema.raw(`DROP FUNCTION IF EXISTS spot_category(integer, jsonb)`)
this.schema.raw(`DROP FUNCTION IF EXISTS spot_has(jsonb, text, integer[])`)
this.schema.raw(`DROP FUNCTION IF EXISTS release_category_for(text, text, jsonb, integer)`)
this.schema.raw(`DROP FUNCTION IF EXISTS release_group_category(jsonb)`)
}
}
database/schema.ts
+41 -1
View File
@@ -87,6 +87,25 @@ export class GroupSchema extends BaseModel {
declare tail: boolean
}
export class InviteSchema extends BaseModel {
static $columns = ['code', 'createdAt', 'note', 'tier', 'tierDays', 'usedAt', 'usedBy'] as const
$columns = InviteSchema.$columns
@column({ isPrimary: true })
declare code: string
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()
declare note: string | null
@column()
declare tier: string | null
@column()
declare tierDays: number | null
@column.dateTime()
declare usedAt: DateTime | null
@column()
declare usedBy: number | null
}
export class NodeAccessTokenSchema extends BaseModel {
static $columns = ['abilities', 'createdAt', 'expiresAt', 'hash', 'id', 'lastUsedAt', 'name', 'tokenableId', 'type', 'updatedAt'] as const
$columns = NodeAccessTokenSchema.$columns
@@ -399,9 +418,24 @@ export class RuleSetSchema extends BaseModel {
declare version: number
}
export class UsageSchema extends BaseModel {
static $columns = ['apiHits', 'day', 'grabs', 'userId'] as const
$columns = UsageSchema.$columns
@column()
declare apiHits: number
@column.date()
declare day: DateTime
@column()
declare grabs: number
@column({ isPrimary: true })
declare userId: number
}
export class UserSchema extends BaseModel {
static $columns = ['createdAt', 'email', 'fullName', 'id', 'password', 'updatedAt'] as const
static $columns = ['apiKey', 'createdAt', 'email', 'fullName', 'id', 'password', 'suspended', 'tier', 'tierUntil', 'updatedAt'] as const
$columns = UserSchema.$columns
@column()
declare apiKey: string
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()
@@ -412,6 +446,12 @@ export class UserSchema extends BaseModel {
declare id: number
@column({ serializeAs: null })
declare password: string
@column()
declare suspended: boolean
@column()
declare tier: string
@column.dateTime()
declare tierUntil: DateTime | null
@column.dateTime({ autoCreate: true, autoUpdate: true })
declare updatedAt: DateTime | null
}
deploy/compose.yaml
+7 -13
View File
@@ -115,14 +115,11 @@ services:
<<: *app-env
NNTP_CONNECTIONS: ${NNTP_CONNECTIONS:-38}
# Phase 2's prober: reads the probe queue's articles over connections of its
# own. The account has 50 (NNTP_ACCOUNT_CONNECTIONS): 5 reserved (the
# coordinator's plain one, the spotter's and the spot inspector's 1 each, 2
# to spare: a hedged request closes its connection and opens another; with
# 10 the spot inspector got 502 Too many connections), the other 45 shared
# by the coordinator (NNTP_MIN_CONNECTIONS–NNTP_CONNECTIONS) and the prober
# (PROBE_MIN_CONNECTIONS–PROBE_CONNECTIONS), headers first
# (app/services/nntp_shares.ts).
# Phase 2's prober: reads the probe queue's articles over PROBE_CONNECTIONS
# connections of its own. The account has 50: the coordinator's 38 (+ its
# plain one), the prober's 8, the spotter's and the spot inspector's 1 each,
# 2 to spare (a hedged request closes its connection and opens another)
# (with 10 the spot inspector got 502 Too many connections).
prober:
<<: *app
command: ['node', 'ace', 'probe:run']
@@ -130,9 +127,6 @@ services:
environment:
<<: *app-env
PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-8}
# No claims under this much free on the NVMe (its overlay root): 2 GB above the 4 GB floor (2026-10-24), for
# what the tail, the merger and the judge write meanwhile.
PROBE_MIN_FREE_GB: ${PROBE_MIN_FREE_GB:-6}
# The par2 merge: releases whose files one par2 set describes (what the prober read) become one. No NNTP.
merger:
@@ -140,8 +134,8 @@ services:
command: ['node', 'ace', 'releases:merge']
volumes: []
# The junk judge: releases judged by what their probes showed (dead, fake, repeated, locked, unusable), the changed
# ones every 2 minutes, all daily; verdicts hide them from search. No NNTP.
# The junk judge: releases judged hourly by what their probes showed (dead, fake, repeated, locked, unusable);
# verdicts hide them from search. No NNTP.
judge:
<<: *app
command: ['node', 'ace', 'releases:judge']
edge/src/indexer.rs
+1 -40
View File
@@ -915,17 +915,12 @@ fn candidates_of(key: &str, files: &[Vec<Seg>], upload: usize) -> Vec<Candidate>
let c = |kind, s: &Seg| Candidate { release_key: key.to_string(), upload, kind, message_id: s.message_id.clone(), file: s.file.name.clone() };
for f in files {
let name = &f[0].file.name;
let volume = names::seven_zip_volume(name);
if let Some(volume) = volume.filter(|&v| v > 1) {
if let Some(volume) = names::seven_zip_volume(name).filter(|&v| v > 1) {
if tail.is_none_or(|(v, _)| volume > v) {
tail = Some((volume, f));
}
continue;
}
// An unsplit 7z of more than one segment: its listing is in its last one.
if volume == Some(0) && f.len() > 1 && tail.is_none() {
tail = Some((0, f));
}
if names::is_archive_start(name) {
out.push(c("archive", &f[0]));
} else if names::is_par2(name) && !names::is_later_volume(name) {
@@ -970,40 +965,6 @@ mod tests {
assert_eq!(batch.releases[0].key, batch.releases[1].key);
}
#[test]
fn archive_candidates_as_in_the_typescript() {
let mut priorities = FxHashMap::default();
priorities.insert(GROUP.to_string(), 1);
let mut ix = RangeIndexer::new(GROUP, 1, 10_000, 1, Rules::defaults(priorities), 5);
let file = |at: i64, release: &str, name: &str, n: usize, parts: i64| -> Vec<String> {
(0..parts).map(|i| line(at + i, &format!(r#"{release} [{n}/3] - "{name}" yEnc ({}/{parts})"#, i + 1), 0)).collect()
};
let mut lines = Vec::new();
// A split 7z: its start, and the end of its highest volume.
lines.extend(file(100, "Split.Seven", "Split.Seven.7z.001", 1, 2));
lines.extend(file(110, "Split.Seven", "Split.Seven.7z.003", 3, 2));
lines.extend(file(120, "Split.Seven", "Split.Seven.7z.002", 2, 2));
// An unsplit 7z of 3 segments: its start and its end.
lines.extend(file(200, "Whole.Seven", "Whole.Seven.7z", 1, 3));
// A ZIP: its start.
lines.extend(file(300, "Some.Zip", "Some.Zip.zip", 1, 2));
ix.add_text(&lines.join("\r\n"), 10_001);
ix.finish();
let batch = ix.take();
let mut got: Vec<(&str, &str, &str)> = batch.candidates.iter().map(|c| (c.kind, c.file.as_str(), c.message_id.as_str())).collect();
got.sort();
assert_eq!(
got,
vec![
("7z-tail", "Split.Seven.7z.003", "u111@test"),
("7z-tail", "Whole.Seven.7z", "u202@test"),
("archive", "Some.Zip.zip", "u300@test"),
("archive", "Split.Seven.7z.001", "u100@test"),
("archive", "Whole.Seven.7z", "u200@test"),
]
);
}
#[test]
fn copies_and_fills_as_in_the_typescript() {
let hour = 3_600_000i64;
edge/src/names.rs
+5 -11
View File
@@ -40,11 +40,11 @@ static SUFFIXES: LazyLock<Vec<(Regex, &'static str)>> = LazyLock::new(|| {
(re(r"(?i)\.7z(\.[0-9]{3})?$"), ""),
(re(r"(?i)\.zip(\.[0-9]{3})?$"), ""),
// JS: (?<=\.(7z|…))\.\d{3}$ — no lookbehind here, so the extension is matched and put back.
(re(r"(?i)(\.(?:7z|zip|rar|tar|mkv|mp4|avi|m4v|ts|m2ts|iso|img|bin|vob|wmv|mov|mpg|mpeg|divx|webm|flv|m2v|mts|r[0-9]{2,3}))\.[0-9]{2,3}$"), "$1"),
(re(r"(?i)(\.(?:7z|zip|rar|tar|mkv|mp4|avi|m4v|ts|m2ts|iso|img|bin|vob|r[0-9]{2,3}))\.[0-9]{2,3}$"), "$1"),
(re(r"(?i)\.(nfo|sfv|nzb|srr|srs|txt|jpg|png|md5|sha1|srt|sub|idx|ass|ssa|m3u|m3u8|cue|log)$"), ""),
(re(r"(?i)\.tar$"), ""),
(re(r"(?i)\.(zst|gz|xz|bz2)$"), ""),
(re(r"(?i)\.(mkv|mp4|avi|m4v|ts|iso|img|bin|flac|mp3|epub|pdf|wmv|mov|mpg|mpeg)$"), ""),
(re(r"(?i)\.(mkv|mp4|avi|m4v|ts|iso|img|bin|flac|mp3|epub|pdf)$"), ""),
]
});
@@ -147,13 +147,10 @@ fn strip_volume_digits(s: &[u8]) -> Option<usize> {
((2..=3).contains(&d) && n > d && s[n - d - 1] == b'.').then(|| n - d - 1)
}
const SPLIT_EXTS: [&[u8]; 23] = [
b"7z", b"zip", b"rar", b"tar", b"mkv", b"mp4", b"avi", b"m4v", b"ts", b"m2ts", b"iso", b"img", b"bin", b"vob", b"wmv", b"mov", b"mpg", b"mpeg", b"divx", b"webm", b"flv", b"m2v",
b"mts",
];
const SPLIT_EXTS: [&[u8]; 14] = [b"7z", b"zip", b"rar", b"tar", b"mkv", b"mp4", b"avi", b"m4v", b"ts", b"m2ts", b"iso", b"img", b"bin", b"vob"];
const SIDE_EXTS: [&[u8]; 19] = [b"nfo", b"sfv", b"nzb", b"srr", b"srs", b"txt", b"jpg", b"png", b"md5", b"sha1", b"srt", b"sub", b"idx", b"ass", b"ssa", b"m3u", b"m3u8", b"cue", b"log"];
const PACK_EXTS: [&[u8]; 4] = [b"zst", b"gz", b"xz", b"bz2"];
const MEDIA_EXTS: [&[u8]; 16] = [b"mkv", b"mp4", b"avi", b"m4v", b"ts", b"iso", b"img", b"bin", b"flac", b"mp3", b"epub", b"pdf", b"wmv", b"mov", b"mpg", b"mpeg"];
const MEDIA_EXTS: [&[u8]; 12] = [b"mkv", b"mp4", b"avi", b"m4v", b"ts", b"iso", b"img", b"bin", b"flac", b"mp3", b"epub", b"pdf"];
/// SUFFIXES[rule] once: the stem's new length when it matches.
fn strip_suffix_rule(s: &[u8], rule: usize) -> Option<usize> {
@@ -1033,7 +1030,7 @@ static SEVEN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.7z$"));
static PART1: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.part0*1\.rar$"));
static RAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.rar$"));
static PART: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.part([0-9]+)\.rar$"));
static BIN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(bin|zip)$"));
static BIN: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.bin$"));
static OLD_RAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.r[0-9]{2,3}$"));
static PAR2_VOL: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.vol[0-9]+[+-][0-9]+\.par2$"));
pub static PAR2: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.par2$"));
@@ -1329,9 +1326,6 @@ mod tests {
assert_eq!(file_stem("x.part07.rar"), "x");
assert_eq!(file_stem("Movie.H.264.mkv.001"), "Movie.H.264");
assert_eq!(file_stem("a.7z.002"), "a");
assert_eq!(file_stem("Morning Lovers.wmv.058"), "Morning Lovers");
assert_eq!(file_stem("clip.mov.12"), "clip");
assert_eq!(file_stem_re("Morning Lovers.wmv.058"), "Morning Lovers");
}
#[test]
lib/edge/corpus_filter.ts
-78
View File
@@ -1,78 +0,0 @@
import { SegmentScanner, scanOverviewText } from './scan.js'
/**
* What a corpus chunk left out (REBUILD.md, corpus; LOG.md 2026-10-23 (101)): the overview lines the edge worker only
* counts and never indexes, decided per line as RangeIndexer.line does: a crosspost an indexed group with a lower
* priority number owns (by owner), and a subject that names no file part (obfuscated). Article numbers ascending,
* stored as differences. Release-level rules (garbage, per-article, executable, poster) judge files put together
* across chunks, so they aren't applied here.
*/
export interface ChunkDrops {
version: 1
/** The rule set (rule_sets.version) and group priorities the decision was made with. */
ruleVersion: number
priorities: Record<string, number>
/** Overview lines in the chunk, and kept. */
lines: number
kept: number
/** By owning group, article numbers as differences (the first from 0). */
crosspost: Record<string, number[]>
obfuscated: number[]
}
const deltas = (numbers: number[]) => numbers.map((n, i) => (i === 0 ? n : n - numbers[i - 1]))
/** The article numbers back from ChunkDrops differences. */
export function undelta(differences: number[]): number[] {
let at = 0
return differences.map((d) => (at += d))
}
/**
* A chunk's overview text without the lines the edge worker drops on their own, and what was dropped. Lines kept
* byte for byte (with their line ends); lines the worker skips uncounted (no number or message-id) are kept.
*/
export function filterOverviews(text: string, group: string, priorities: Record<string, number>, ruleVersion: number): { kept: string; drops: ChunkDrops } {
const scanner = new SegmentScanner()
const crosspost = new Map<string, number[]>()
const obfuscated: number[] = []
const dropped = new Set<number>()
let lines = 0
scanOverviewText(text, (n, subject, from, date, messageId, bytes, xref) => {
lines++
if (!Number.isFinite(n) || !messageId) return
const owner = xref ? scanner.owner(xref, group, priorities) : group
if (owner !== group) {
const list = crosspost.get(owner)
if (list) list.push(n)
else crosspost.set(owner, [n])
dropped.add(n)
} else if (!scanner.segment(n, subject, from, date, messageId, bytes, xref, group)) {
obfuscated.push(n)
dropped.add(n)
}
})
const out: string[] = []
let start = 0
while (start < text.length) {
let end = text.indexOf('\n', start)
end = end === -1 ? text.length : end + 1
const tab = text.indexOf('\t', start)
const n = tab !== -1 && tab < end ? Number.parseInt(text.slice(start, tab), 10) : Number.NaN
if (!dropped.has(n)) out.push(text.slice(start, end))
start = end
}
const sorted = (list: number[]) => [...list].sort((a, b) => a - b)
return {
kept: out.join(''),
drops: {
version: 1,
ruleVersion,
priorities: Object.fromEntries(Object.entries(priorities).filter(([name]) => crosspost.has(name) || name === group)),
lines,
kept: lines - dropped.size,
crosspost: Object.fromEntries([...crosspost].map(([owner, list]) => [owner, deltas(sorted(list))])),
obfuscated: deltas(sorted(obfuscated)),
},
}
}
lib/edge/index.ts
-1
View File
@@ -5,4 +5,3 @@ export type { Lease } from './lease.js'
export { mergeFragments } from './merge.js'
export type { EdgeSegment, FileMeta } from './scan.js'
export { throttle } from './throttle.js'
export { filterOverviews, undelta, type ChunkDrops } from './corpus_filter.js'
lib/edge/range_indexer.ts
-2
View File
@@ -434,8 +434,6 @@ export function candidatesOf(key: string, files: AggregatedFile<EdgeSegment>[],
if (!tail || volume > tail.volume) tail = { volume, file: f }
continue
}
// An unsplit 7z of more than one segment: its listing is in its last one.
if (volume === 0 && f.segments.length > 1 && !tail) tail = { volume, file: f }
if (isArchiveStart(name)) out.push({ releaseKey: key, upload, kind: 'archive', messageId: f.segments[0].messageId, file: name })
else if (/\.par2$/i.test(name) && !isLaterVolume(name)) out.push({ releaseKey: key, upload, kind: 'par2', messageId: f.segments[0].messageId, file: name })
}
lib/edge/scan.ts
-2
View File
@@ -47,8 +47,6 @@ interface Counter {
function findCounter(subject: string, after: number): Counter | null {
let open = subject.length
while (true) {
// lastIndexOf('(', -1) searches from 0 again: a "(" at the start that isn't a counter looped forever.
if (open === 0) return null
open = subject.lastIndexOf('(', open - 1)
if (open <= after) return null
let i = open + 1
lib/files/volumes.ts
+2 -3
View File
@@ -1,11 +1,10 @@
/**
* Files whose first segment says what an archive holds: named first RAR
* volumes (.part01.rar, .rar), 7z (.7z, .7z.001: its type; the listing is at
* its end, the 7z-tail), ZIP (its first entries) and `.bin`, an extension
* volumes (.part01.rar, .rar), 7z (.7z, .7z.001) and `.bin`, an extension
* uploaders use for archive volumes too (sniffed by content).
*/
export function isArchiveStart(name: string): boolean {
if (/\.(bin|zip)$/i.test(name)) return true
if (/\.bin$/i.test(name)) return true
const volume = sevenZipVolume(name)
if (volume === 0 || volume === 1) return true
return /\.part0*1\.rar$/i.test(name) || (/\.rar$/i.test(name) && !/\.part\d+\.rar$/i.test(name))
lib/newznab/categories.ts
+5 -7
View File
@@ -1,12 +1,10 @@
/** Newznab categories this indexer assigns (release_category_for() and spot_category() in SQL, migration …5200). */
/** Newznab categories this indexer assigns (release_category() in SQL, migration …1200). */
export const CATEGORIES: { id: number; name: string; subcats: { id: number; name: string }[] }[] = [
{ id: 1000, name: 'Console', subcats: [] },
{ id: 2000, name: 'Movies', subcats: [{ id: 2030, name: 'SD' }, { id: 2040, name: 'HD' }, { id: 2045, name: 'UHD' }] },
{ id: 3000, name: 'Audio', subcats: [{ id: 3010, name: 'MP3' }, { id: 3030, name: 'Audiobook' }, { id: 3040, name: 'Lossless' }] },
{ id: 4000, name: 'PC', subcats: [{ id: 4050, name: 'Games' }] },
{ id: 5000, name: 'TV', subcats: [{ id: 5030, name: 'SD' }, { id: 5040, name: 'HD' }, { id: 5045, name: 'UHD' }, { id: 5070, name: 'Anime' }] },
{ id: 6000, name: 'XXX', subcats: [{ id: 6040, name: 'x264' }, { id: 6045, name: 'UHD' }, { id: 6060, name: 'ImgSet' }, { id: 6070, name: 'Other' }] },
{ id: 7000, name: 'Books', subcats: [{ id: 7020, name: 'Ebook' }, { id: 7030, name: 'Comics' }] },
{ id: 3000, name: 'Audio', subcats: [{ id: 3010, name: 'MP3' }, { id: 3040, name: 'Lossless' }] },
{ id: 4000, name: 'PC', subcats: [] },
{ id: 5000, name: 'TV', subcats: [{ id: 5030, name: 'SD' }, { id: 5040, name: 'HD' }, { id: 5045, name: 'UHD' }] },
{ id: 7000, name: 'Books', subcats: [{ id: 7020, name: 'Ebook' }] },
{ id: 8000, name: 'Other', subcats: [{ id: 8010, name: 'Misc' }] },
]
lib/newznab/index.ts
+1 -1
View File
@@ -2,4 +2,4 @@
* Newznab API formatting, independent of AdonisJS. From voyager 1 (its CONCEPT.md D23).
*/
export { CATEGORIES, categoryLabel, expandCategories, parentCategory } from './categories.js'
export { capsXml, errorXml, resultsXml, type NewznabItem, type NewznabServer } from './xml.js'
export { capsXml, errorXml, resultsXml, type ApiLimits, type NewznabItem, type NewznabServer } from './xml.js'
lib/newznab/xml.ts
+24 -4
View File
@@ -27,7 +27,7 @@ export interface NewznabServer {
apiKey?: string
}
/** Newznab error codes: 100 credentials, 200/201 parameters, 202 function, 300 item. */
/** Newznab error codes: 100 credentials, 101 suspended, 102 privileges, 200/201 parameters, 202 function, 300 item, 500/501 request/download limit. */
/** Spotnet UBB markup (or the HTML of old spots) as plain text: [br] as line breaks, other tags dropped. */
export function ubbToText(ubb: string): string {
return ubb
@@ -44,11 +44,23 @@ export function errorXml(code: number, description: string): string {
.end({ prettyPrint: true })
}
export function capsXml(server: NewznabServer, limits: { max: number; default: number }): string {
/** A caller's day so far and its limits (null: none), as nZEDb-style indexers report them. */
export interface ApiLimits {
apiCurrent: number
apiMax: number | null
grabCurrent: number
grabMax: number | null
}
export function capsXml(
server: NewznabServer,
limits: { max: number; default: number },
registration = { available: false, open: false }
): string {
const caps = create({ version: '1.0', encoding: 'UTF-8' }).ele('caps')
caps.ele('server', { version: '1.0', title: server.title, strapline: 'Usenet indexer', url: `${server.baseUrl}/` })
caps.ele('limits', { max: String(limits.max), default: String(limits.default) })
caps.ele('registration', { available: 'no', open: 'no' })
caps.ele('registration', { available: registration.available ? 'yes' : 'no', open: registration.open ? 'yes' : 'no' })
const searching = caps.ele('searching')
searching.ele('search', { available: 'yes', supportedParams: 'q' })
searching.ele('tv-search', { available: 'yes', supportedParams: 'q,season,ep' })
@@ -63,7 +75,7 @@ export function capsXml(server: NewznabServer, limits: { max: number; default: n
return caps.end({ prettyPrint: true })
}
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number): string {
export function resultsXml(server: NewznabServer, items: NewznabItem[], offset: number, total: number, limits?: ApiLimits): string {
const rss = create({ version: '1.0', encoding: 'UTF-8' }).ele('rss', {
'version': '2.0',
'xmlns:atom': 'http://www.w3.org/2005/Atom',
@@ -74,6 +86,14 @@ export function resultsXml(server: NewznabServer, items: NewznabItem[], offset:
channel.ele('description').txt(`${server.title} API results`)
channel.ele('link').txt(`${server.baseUrl}/`)
channel.ele('newznab:response', { offset: String(offset), total: String(total) })
if (limits) {
channel.ele('newznab:apilimits', {
apicurrent: String(limits.apiCurrent),
...(limits.apiMax !== null && { apimax: String(limits.apiMax) }),
grabcurrent: String(limits.grabCurrent),
...(limits.grabMax !== null && { grabmax: String(limits.grabMax) }),
})
}
const key = server.apiKey ? `&apikey=${encodeURIComponent(server.apiKey)}` : ''
for (const item of items) {
lib/probe/inspect.ts
+3 -25
View File
@@ -2,8 +2,6 @@ import { createHash } from 'node:crypto'
import { decodeSegment, YencCrcError, YencError, type DecodedSegment } from '#lib/yenc/index'
import { isPar2, parsePar2 } from '#lib/par2/index'
import { parseRarHeaders } from '#lib/rar/index'
import { parseSevenZipTail } from '#lib/sevenzip/index'
import { parseZipStart } from '#lib/zip/index'
import { sniffContentType, type ContentType } from '#lib/content/index'
/** What one probed article showed (REBUILD.md, part 6; stored per release and kind). */
@@ -57,11 +55,8 @@ function largestFirst(items: { name: string; size: number }[]): string[] {
.map((i) => i.name.slice(0, MAX_NAME_LENGTH))
}
/**
* One article's body: its yEnc header, what the first bytes are, and for par2, RAR, ZIP and 7z what's inside.
* Throws when it isn't yEnc. `kind` is the probe candidate's: a 7z-tail's article ends the archive.
*/
export function inspect(body: Buffer, kind?: string): Inspection {
/** One article's body: its yEnc header, what the first bytes are, and for par2 and RAR what's inside. Throws when it isn't yEnc. */
export function inspect(body: Buffer): Inspection {
const segment = decodeBody(body)
const result: Inspection = { yencName: segment.meta.begin.name.slice(0, MAX_NAME_LENGTH), size: segment.meta.begin.size, type: sniffContentType(segment.data) }
// The file's first 16 KiB, or all of a smaller file.
@@ -90,23 +85,6 @@ export function inspect(body: Buffer, kind?: string): Inspection {
result.passworded = info.headersEncrypted || files.some((e) => e.encrypted)
}
}
if (result.type === 'zip') {
const start = parseZipStart(segment.data)
if (start) {
result.entries = start.files.length
result.names = largestFirst(start.files)
result.passworded = start.encrypted
}
}
// 7z keeps its listing at the end: the last segment of the last volume (a 7z-tail), or a small archive whole.
if ((kind === '7z-tail' || result.type === '7z') && segment.offset + segment.data.length === result.size) {
const listing = parseSevenZipTail(segment.data)
if (listing) {
result.entries = listing.files.length
if (listing.files.length > 0) result.names = largestFirst(listing.files)
result.passworded = listing.encrypted
}
}
return result
}
@@ -119,7 +97,7 @@ export function inspect(body: Buffer, kind?: string): Inspection {
export function innerName(inspections: { kind: string; result: Inspection }[], releaseName = ''): string | null {
const of = (kind: string) => inspections.filter((i) => i.kind === kind).map((i) => i.result)
const content = (kind: string) => of(kind).flatMap((i) => i.names ?? []).find((n) => !PACKAGING.test(n))
const pick = content('archive') ?? content('7z-tail') ?? content('par2') ?? of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
const pick = content('archive') ?? content('par2') ?? of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
const name = pick?.replace(/^.*[\\/]/, '').slice(0, 255)
// The release's own name with an extension says nothing new (a random-named file's yEnc name is mostly its random name).
return name && !name.toLowerCase().startsWith(releaseName.toLowerCase() || '\0') ? name : null
lib/release/identity.ts
+3 -4
View File
@@ -40,13 +40,12 @@ const SUFFIXES = [
/\.7z(\.\d{3})?$/i,
/\.zip(\.\d{3})?$/i,
// Split volumes (x.mkv.001), not codec names (H.264, H.265).
// Also a split rar volume ("rld-gtre.r13.002": 1,620 single files of usenet-town's posts), and other video
// ("Morning Lovers.wmv.001" to ".058" were 58 releases; LOG.md 2026-10-23 (97)).
/(?<=\.(7z|zip|rar|tar|mkv|mp4|avi|m4v|ts|m2ts|iso|img|bin|vob|wmv|mov|mpg|mpeg|divx|webm|flv|m2v|mts|r\d{2,3}))\.\d{2,3}$/i,
// Also a split rar volume ("rld-gtre.r13.002": 1,620 single files of usenet-town's posts).
/(?<=\.(7z|zip|rar|tar|mkv|mp4|avi|m4v|ts|m2ts|iso|img|bin|vob|r\d{2,3}))\.\d{2,3}$/i,
/\.(nfo|sfv|nzb|srr|srs|txt|jpg|png|md5|sha1|srt|sub|idx|ass|ssa|m3u|m3u8|cue|log)$/i,
/\.tar$/i,
/\.(zst|gz|xz|bz2)$/i,
/\.(mkv|mp4|avi|m4v|ts|iso|img|bin|flac|mp3|epub|pdf|wmv|mov|mpg|mpeg)$/i,
/\.(mkv|mp4|avi|m4v|ts|iso|img|bin|flac|mp3|epub|pdf)$/i,
]
/** Strips per-file suffixes (volume numbers, archive and par2 extensions) from a filename. */
lib/sevenzip/index.ts
-2
View File
@@ -1,2 +0,0 @@
export { parseSevenZipTail, type SevenZipListing } from '#lib/sevenzip/parse'
export { lzmaDecode, LzmaError } from '#lib/sevenzip/lzma'
lib/sevenzip/lzma.ts
-221
View File
@@ -1,221 +0,0 @@
/**
* An LZMA decoder (raw stream, as a 7z coder stores it: the properties apart, the unpacked size known), after the
* LZMA SDK's reference decoder (LzmaSpec.cpp). For 7z headers: those are small, so the whole output is one buffer.
*/
const MODEL_BITS = 11
const BIT_MODEL_TOTAL = 1 << MODEL_BITS
const MOVE_BITS = 5
const TOP = 1 << 24
const POS_BITS_MAX = 4
const STATES = 12
const LEN_TO_POS_STATES = 4
const ALIGN_BITS = 4
const END_POS_MODEL_INDEX = 14
const FULL_DISTANCES = 1 << (END_POS_MODEL_INDEX >> 1)
const MATCH_MIN_LEN = 2
export class LzmaError extends Error {}
const probs = (n: number) => new Uint16Array(n).fill(BIT_MODEL_TOTAL >>> 1)
class RangeDecoder {
private range = 0xffffffff
private code = 0
constructor(
private input: Uint8Array,
private pos: number
) {
if (this.byte() !== 0) throw new LzmaError('LZMA: bad stream start')
for (let i = 0; i < 4; i++) this.code = ((this.code << 8) | this.byte()) >>> 0
if (this.code === this.range) throw new LzmaError('LZMA: corrupt stream')
}
private byte(): number {
if (this.pos >= this.input.length) throw new LzmaError('LZMA: input ended')
return this.input[this.pos++]
}
private normalize() {
if (this.range < TOP) {
this.range = (this.range << 8) >>> 0
this.code = ((this.code << 8) | this.byte()) >>> 0
}
}
direct(bits: number): number {
let result = 0
do {
this.range >>>= 1
this.code = (this.code - this.range) >>> 0
const t = this.code >>> 31 ? 0xffffffff : 0
this.code = (this.code + (this.range & t)) >>> 0
this.normalize()
result = ((result << 1) | (t ? 0 : 1)) >>> 0
} while (--bits)
return result
}
bit(p: Uint16Array, i: number): number {
const v = p[i]
const bound = ((this.range >>> MODEL_BITS) * v) >>> 0
let symbol: number
if (this.code < bound) {
p[i] = v + ((BIT_MODEL_TOTAL - v) >>> MOVE_BITS)
this.range = bound
symbol = 0
} else {
p[i] = v - (v >>> MOVE_BITS)
this.code = (this.code - bound) >>> 0
this.range = (this.range - bound) >>> 0
symbol = 1
}
this.normalize()
return symbol
}
tree(p: Uint16Array, offset: number, bits: number): number {
let m = 1
for (let i = 0; i < bits; i++) m = (m << 1) + this.bit(p, offset + m)
return m - (1 << bits)
}
reverseTree(p: Uint16Array, offset: number, bits: number): number {
let m = 1
let symbol = 0
for (let i = 0; i < bits; i++) {
const bit = this.bit(p, offset + m)
m = (m << 1) + bit
symbol |= bit << i
}
return symbol
}
}
class LenDecoder {
private choice = probs(2)
private low = probs((1 << POS_BITS_MAX) << 3)
private mid = probs((1 << POS_BITS_MAX) << 3)
private high = probs(1 << 8)
decode(rc: RangeDecoder, posState: number): number {
if (rc.bit(this.choice, 0) === 0) return rc.tree(this.low, posState << 3, 3)
if (rc.bit(this.choice, 1) === 0) return 8 + rc.tree(this.mid, posState << 3, 3)
return 16 + rc.tree(this.high, 0, 8)
}
}
/** Decodes `size` bytes of a raw LZMA stream with its 5 property bytes (lc/lp/pb, dictionary size). */
export function lzmaDecode(properties: Uint8Array, input: Uint8Array, size: number): Buffer {
if (properties.length < 5) throw new LzmaError('LZMA: short properties')
let d = properties[0]
if (d >= 9 * 5 * 5) throw new LzmaError('LZMA: bad properties')
const lc = d % 9
d = Math.floor(d / 9)
const lp = d % 5
const pb = Math.floor(d / 5)
const dictSize = Math.max(properties[1] | (properties[2] << 8) | (properties[3] << 16) | (properties[4] << 24), 1 << 12) >>> 0
const out = Buffer.alloc(size)
let pos = 0
const rc = new RangeDecoder(input, 0)
const literals = probs(0x300 << (lc + lp))
const posSlot = probs(LEN_TO_POS_STATES << 6)
const align = probs(1 << ALIGN_BITS)
const posDecoders = probs(1 + FULL_DISTANCES - END_POS_MODEL_INDEX)
const isMatch = probs(STATES << POS_BITS_MAX)
const isRep = probs(STATES)
const isRepG0 = probs(STATES)
const isRepG1 = probs(STATES)
const isRepG2 = probs(STATES)
const isRep0Long = probs(STATES << POS_BITS_MAX)
const lenDecoder = new LenDecoder()
const repLenDecoder = new LenDecoder()
const distance = (len: number): number => {
const lenState = Math.min(len, LEN_TO_POS_STATES - 1)
const slot = rc.tree(posSlot, lenState << 6, 6)
if (slot < 4) return slot
const directBits = (slot >>> 1) - 1
let dist = ((2 | (slot & 1)) * 2 ** directBits) >>> 0
if (slot < END_POS_MODEL_INDEX) {
dist = (dist + rc.reverseTree(posDecoders, dist - slot, directBits)) >>> 0
} else {
dist = (dist + rc.direct(directBits - ALIGN_BITS) * 2 ** ALIGN_BITS) >>> 0
dist = (dist + rc.reverseTree(align, 0, ALIGN_BITS)) >>> 0
}
return dist
}
let state = 0
let rep0 = 0
let rep1 = 0
let rep2 = 0
let rep3 = 0
while (pos < size) {
const posState = pos & ((1 << pb) - 1)
if (rc.bit(isMatch, (state << POS_BITS_MAX) + posState) === 0) {
const prev = pos > 0 ? out[pos - 1] : 0
const base = 0x300 * (((pos & ((1 << lp) - 1)) << lc) + (prev >>> (8 - lc)))
let symbol = 1
if (state >= 7) {
let match = out[pos - rep0 - 1]
do {
const matchBit = (match >>> 7) & 1
match <<= 1
const bit = rc.bit(literals, base + ((1 + matchBit) << 8) + symbol)
symbol = (symbol << 1) | bit
if (matchBit !== bit) break
} while (symbol < 0x100)
}
while (symbol < 0x100) symbol = (symbol << 1) | rc.bit(literals, base + symbol)
out[pos++] = symbol - 0x100
state = state < 4 ? 0 : state < 10 ? state - 3 : state - 6
continue
}
let len: number
if (rc.bit(isRep, state) !== 0) {
if (pos === 0) throw new LzmaError('LZMA: repeat before any output')
if (rc.bit(isRepG0, state) === 0) {
if (rc.bit(isRep0Long, (state << POS_BITS_MAX) + posState) === 0) {
state = state < 7 ? 9 : 11
out[pos] = out[pos - rep0 - 1]
pos++
continue
}
} else {
let dist: number
if (rc.bit(isRepG1, state) === 0) {
dist = rep1
} else {
if (rc.bit(isRepG2, state) === 0) {
dist = rep2
} else {
dist = rep3
rep3 = rep2
}
rep2 = rep1
}
rep1 = rep0
rep0 = dist
}
len = repLenDecoder.decode(rc, posState)
state = state < 7 ? 8 : 11
} else {
rep3 = rep2
rep2 = rep1
rep1 = rep0
len = lenDecoder.decode(rc, posState)
state = state < 7 ? 7 : 10
rep0 = distance(len)
// The end marker: allowed only where the output is complete, which the loop already ended at.
if (rep0 === 0xffffffff) throw new LzmaError('LZMA: end marker before the end')
if (rep0 >= dictSize || rep0 >= pos) throw new LzmaError('LZMA: distance out of range')
}
len += MATCH_MIN_LEN
if (len > size - pos) throw new LzmaError('LZMA: match past the end')
for (let i = 0; i < len; i++, pos++) out[pos] = out[pos - rep0 - 1]
}
return out
}
lib/sevenzip/parse.ts
-350
View File
@@ -1,350 +0,0 @@
import { crc32 } from 'node:zlib'
import { lzmaDecode } from '#lib/sevenzip/lzma'
/**
* What a 7z archive's header says (its listing): read from the archive's end, where 7z keeps it, so the last
* segment of the last volume is enough (the `7z-tail` probe; REBUILD.md, phase 2). The header there is mostly
* an encoded one: a pointer to the real header, LZMA-packed just before it, or AES-encrypted (`7z -mhe`), which
* hides the names and means a password.
*/
export interface SevenZipListing {
/** Files, not directories, with their unpacked size (0 for an empty one). */
files: { name: string; size: number }[]
/** Entries or the header encrypted (AES). */
encrypted: boolean
/** The header itself encrypted: no names known. */
headerEncrypted: boolean
}
const AES = '06f10701'
const LZMA = '030101'
/** A header bigger than this isn't unpacked (headers list names; 64 MiB is ~a million files). */
const MAX_HEADER = 64 * 1024 * 1024
const ID = {
end: 0x00,
header: 0x01,
archiveProperties: 0x02,
additionalStreamsInfo: 0x03,
mainStreamsInfo: 0x04,
filesInfo: 0x05,
packInfo: 0x06,
unpackInfo: 0x07,
subStreamsInfo: 0x08,
size: 0x09,
crc: 0x0a,
folder: 0x0b,
codersUnpackSize: 0x0c,
numUnpackStream: 0x0d,
emptyStream: 0x0e,
emptyFile: 0x0f,
name: 0x11,
encodedHeader: 0x17,
dummy: 0x19,
}
class Malformed extends Error {}
interface Coder {
id: string
properties: Uint8Array
inStreams: number
outStreams: number
}
interface Folder {
coders: Coder[]
bindOut: Set<number>
unpackSizes: number[]
crc?: number
}
interface StreamsInfo {
packPos: number
packSizes: number[]
folders: Folder[]
/** Per folder, how many files' streams it holds, and every stream's size in order. */
streamsPerFolder: number[]
streamSizes: number[]
}
class Reader {
constructor(
private buf: Uint8Array,
public pos = 0
) {}
get done() {
return this.pos >= this.buf.length
}
byte(): number {
if (this.pos >= this.buf.length) throw new Malformed('7z: header ended')
return this.buf[this.pos++]
}
bytes(n: number): Uint8Array {
if (n < 0 || this.pos + n > this.buf.length) throw new Malformed('7z: header ended')
const out = this.buf.subarray(this.pos, this.pos + n)
this.pos += n
return out
}
/** 7z's variable-length number: the first byte's high bits say how many bytes follow. */
number(): number {
const first = this.byte()
let mask = 0x80
let value = 0
for (let i = 0; i < 8; i++) {
if ((first & mask) === 0) {
value += (first & (mask - 1)) * 2 ** (8 * i)
break
}
value += this.byte() * 2 ** (8 * i)
mask >>>= 1
}
if (value > Number.MAX_SAFE_INTEGER) throw new Malformed('7z: number too large')
return value
}
uint32(): number {
const b = this.bytes(4)
return (b[0] | (b[1] << 8) | (b[2] << 16) | (b[3] << 24)) >>> 0
}
bits(n: number): boolean[] {
const out: boolean[] = []
let byte = 0
for (let i = 0; i < n; i++) {
if (i % 8 === 0) byte = this.byte()
out.push((byte & (0x80 >>> i % 8)) !== 0)
}
return out
}
/** "All defined" or a bit vector, then a CRC per defined item. */
digests(n: number): (number | undefined)[] {
const allDefined = this.byte() !== 0
const defined = allDefined ? Array(n).fill(true) : this.bits(n)
return defined.map((d) => (d ? this.uint32() : undefined))
}
expect(id: number) {
if (this.byte() !== id) throw new Malformed(`7z: expected property ${id}`)
}
}
/** Counts that come from the data: a garbage position makes huge ones, which shouldn't allocate. */
function count(n: number, max = 1_000_000): number {
if (n > max) throw new Malformed('7z: implausible count')
return n
}
function readFolder(r: Reader): Folder {
const coders: Coder[] = []
const numCoders = count(r.number(), 64)
if (numCoders === 0) throw new Malformed('7z: folder without coders')
for (let i = 0; i < numCoders; i++) {
const flags = r.byte()
if (flags & 0x80) throw new Malformed('7z: alternative coder methods')
const id = Buffer.from(r.bytes(flags & 0x0f)).toString('hex')
const complex = (flags & 0x10) !== 0
const inStreams = complex ? count(r.number(), 64) : 1
const outStreams = complex ? count(r.number(), 64) : 1
const properties = flags & 0x20 ? r.bytes(count(r.number(), 1 << 16)) : new Uint8Array(0)
coders.push({ id, properties, inStreams, outStreams })
}
const totalOut = coders.reduce((n, c) => n + c.outStreams, 0)
const totalIn = coders.reduce((n, c) => n + c.inStreams, 0)
const bindOut = new Set<number>()
for (let i = 0; i < totalOut - 1; i++) {
r.number()
bindOut.add(r.number())
}
const packed = totalIn - (totalOut - 1)
if (packed < 1) throw new Malformed('7z: folder without packed streams')
if (packed > 1) for (let i = 0; i < packed; i++) r.number()
return { coders, bindOut, unpackSizes: [] }
}
/** The folder's output: the one out stream no bind pair consumes. */
function unpackSize(folder: Folder): number {
for (let i = folder.unpackSizes.length - 1; i >= 0; i--) if (!folder.bindOut.has(i)) return folder.unpackSizes[i]
return 0
}
function readStreamsInfo(r: Reader): StreamsInfo {
const info: StreamsInfo = { packPos: 0, packSizes: [], folders: [], streamsPerFolder: [], streamSizes: [] }
let id = r.byte()
if (id === ID.packInfo) {
info.packPos = r.number()
const n = count(r.number())
for (id = r.byte(); id !== ID.end; id = r.byte()) {
if (id === ID.size) for (let i = 0; i < n; i++) info.packSizes.push(r.number())
else if (id === ID.crc) r.digests(n)
else throw new Malformed('7z: unexpected property in pack info')
}
id = r.byte()
}
if (id === ID.unpackInfo) {
r.expect(ID.folder)
const n = count(r.number())
if (r.byte() !== 0) throw new Malformed('7z: external folders')
for (let i = 0; i < n; i++) info.folders.push(readFolder(r))
r.expect(ID.codersUnpackSize)
for (const f of info.folders) for (const c of f.coders) for (let i = 0; i < c.outStreams; i++) f.unpackSizes.push(r.number())
for (id = r.byte(); id !== ID.end; id = r.byte()) {
if (id === ID.crc) r.digests(n).forEach((crc, i) => (info.folders[i].crc = crc))
else throw new Malformed('7z: unexpected property in unpack info')
}
id = r.byte()
}
info.streamsPerFolder = info.folders.map(() => 1)
if (id === ID.subStreamsInfo) {
id = r.byte()
if (id === ID.numUnpackStream) {
info.streamsPerFolder = info.folders.map(() => count(r.number()))
id = r.byte()
}
const hasSizes = id === ID.size
info.folders.forEach((f, i) => {
const n = info.streamsPerFolder[i]
if (n === 0) return
let sum = 0
for (let j = 0; j < n - 1 && hasSizes; j++) {
const size = r.number()
info.streamSizes.push(size)
sum += size
}
info.streamSizes.push(unpackSize(f) - sum)
})
if (hasSizes) id = r.byte()
while (id !== ID.end) {
if (id === ID.crc) {
// CRCs for the streams whose folder had none (a folder of one stream with a CRC has its own).
const unknown = info.folders.reduce((n, f, i) => n + (info.streamsPerFolder[i] === 1 && f.crc !== undefined ? 0 : info.streamsPerFolder[i]), 0)
r.digests(unknown)
} else {
throw new Malformed('7z: unexpected property in substreams info')
}
id = r.byte()
}
id = r.byte()
} else {
info.streamSizes = info.folders.map(unpackSize)
}
if (id !== ID.end) throw new Malformed('7z: streams info not ended')
return info
}
function readFilesInfo(r: Reader, streams: StreamsInfo | null): { name: string; size: number }[] {
const n = count(r.number())
let emptyStream: boolean[] = Array(n).fill(false)
let emptyFile: boolean[] = []
let names: string[] = []
for (let id = r.byte(); id !== ID.end; id = r.byte()) {
const size = r.number()
const end = r.pos + size
if (id === ID.emptyStream) {
emptyStream = r.bits(n)
} else if (id === ID.emptyFile) {
emptyFile = r.bits(emptyStream.filter(Boolean).length)
} else if (id === ID.name) {
if (r.byte() !== 0) throw new Malformed('7z: external names')
const text = Buffer.from(r.bytes(size - 1)).toString('utf16le')
names = text.split('\0').slice(0, n)
} else if (id !== ID.dummy && id > 0x19) {
throw new Malformed('7z: unknown file property')
}
r.pos = end
}
if (names.length !== n) throw new Malformed('7z: names missing')
const out: { name: string; size: number }[] = []
let stream = 0
let empty = 0
for (let i = 0; i < n; i++) {
if (!emptyStream[i]) {
out.push({ name: names[i], size: streams?.streamSizes[stream++] ?? 0 })
continue
}
// An empty stream is a directory unless it's marked an empty file.
if (emptyFile[empty++]) out.push({ name: names[i], size: 0 })
}
return out
}
function readHeader(r: Reader): SevenZipListing {
let id = r.byte()
if (id === ID.archiveProperties) {
for (let type = r.byte(); type !== 0; type = r.byte()) r.bytes(r.number())
id = r.byte()
}
if (id === ID.additionalStreamsInfo) {
readStreamsInfo(r)
id = r.byte()
}
let streams: StreamsInfo | null = null
if (id === ID.mainStreamsInfo) {
streams = readStreamsInfo(r)
id = r.byte()
}
let files: { name: string; size: number }[] = []
if (id === ID.filesInfo) {
files = readFilesInfo(r, streams)
id = r.byte()
}
if (id !== ID.end) throw new Malformed('7z: header not ended')
const encrypted = (streams?.folders ?? []).some((f) => f.coders.some((c) => c.id === AES))
return { files, encrypted, headerEncrypted: false }
}
/**
* The header that ends exactly at `end` in `data`, starting at `start`: a plain one, or an encoded one whose packed
* data lies just before it (where 7z writes it). Throws when it isn't one.
*/
function headerAt(data: Uint8Array, start: number, end: number): SevenZipListing {
const r = new Reader(data.subarray(0, end), start)
const id = r.byte()
if (id === ID.header) {
const listing = readHeader(r)
if (!r.done) throw new Malformed('7z: bytes after the header')
return listing
}
if (id !== ID.encodedHeader) throw new Malformed('7z: not a header')
const info = readStreamsInfo(r)
if (!r.done) throw new Malformed('7z: bytes after the encoded header')
if (info.folders.length !== 1 || info.packSizes.length < 1) throw new Malformed('7z: unexpected encoded header')
const folder = info.folders[0]
if (folder.coders.some((c) => c.id === AES)) return { files: [], encrypted: true, headerEncrypted: true }
if (folder.coders.length !== 1 || folder.coders[0].id !== LZMA) throw new Malformed(`7z: header coder ${folder.coders[0].id} not supported`)
const packSize = info.packSizes[0]
const size = unpackSize(folder)
if (packSize > start || size > MAX_HEADER) throw new Malformed('7z: packed header not in the data')
const header = lzmaDecode(folder.coders[0].properties, data.subarray(start - packSize, start), size)
if (folder.crc !== undefined && crc32(header) >>> 0 !== folder.crc) throw new Malformed('7z: header CRC mismatch')
return headerAt(header, 0, header.length)
}
/**
* The listing of the 7z archive that `data` is the end of (the last segment of its last volume, or a whole small
* archive); null when no header ends there.
*/
export function parseSevenZipTail(data: Uint8Array): SevenZipListing | null {
const end = data.length
// An encoded header is a few dozen bytes; a plain one (7z -mhc=off) can be large.
for (let start = end - 2; start >= Math.max(0, end - 1024); start--) {
if (data[start] !== ID.encodedHeader || data[start + 1] !== ID.packInfo) continue
try {
return headerAt(data, start, end)
} catch {
// Not a header here (any byte can look like the start of one): the next position.
}
}
for (let start = end - 2; start >= 0; start--) {
if (data[start] !== ID.header || ![ID.archiveProperties, ID.additionalStreamsInfo, ID.mainStreamsInfo, ID.filesInfo, ID.end].includes(data[start + 1])) continue
try {
return headerAt(data, start, end)
} catch {
// Not a header here (any byte can look like the start of one): the next position.
}
}
return null
}
lib/zip/index.ts
-1
View File
@@ -1 +0,0 @@
export { parseZipStart, type ZipStart } from '#lib/zip/parse'
lib/zip/parse.ts
-50
View File
@@ -1,50 +0,0 @@
/**
* The entries a ZIP's start shows: its local file headers, one before each entry's data, as far as the data
* holds them (the first segment of a .zip; REBUILD.md, phase 2). The full list is in the central directory at
* the end; the start names the first entries and says whether they're encrypted, as a RAR's first volume does.
*/
export interface ZipStart {
files: { name: string; size: number }[]
encrypted: boolean
}
const LOCAL = 0x04034b50
/** Traditional PKWARE encryption (bit 0 of the flags), or WinZip AES (method 99). */
const ENCRYPTED_FLAG = 0x1
const DATA_DESCRIPTOR_FLAG = 0x8
const AES_METHOD = 99
const MAX_ENTRIES = 10_000
export function parseZipStart(data: Buffer): ZipStart | null {
if (data.length < 30 || data.readUInt32LE(0) !== LOCAL) return null
const result: ZipStart = { files: [], encrypted: false }
let pos = 0
while (pos + 30 <= data.length && data.readUInt32LE(pos) === LOCAL && result.files.length < MAX_ENTRIES) {
const flags = data.readUInt16LE(pos + 6)
const method = data.readUInt16LE(pos + 8)
let compressed = data.readUInt32LE(pos + 18)
let size = data.readUInt32LE(pos + 22)
const nameLength = data.readUInt16LE(pos + 26)
const extraLength = data.readUInt16LE(pos + 28)
const nameEnd = pos + 30 + nameLength
if (nameEnd > data.length) break
// Bit 11: UTF-8 names; others are mostly CP437, read as latin1.
const name = data.toString(flags & 0x800 ? 'utf8' : 'latin1', pos + 30, nameEnd)
// Zip64: the real sizes in extra field 0x0001.
for (let e = nameEnd; e + 4 <= Math.min(nameEnd + extraLength, data.length); ) {
const id = data.readUInt16LE(e)
const length = data.readUInt16LE(e + 2)
if (id === 0x0001 && e + 4 + length <= data.length && length >= 16) {
size = Number(data.readBigUInt64LE(e + 4))
compressed = Number(data.readBigUInt64LE(e + 12))
}
e += 4 + length
}
if (flags & ENCRYPTED_FLAG || method === AES_METHOD) result.encrypted = true
if (!name.endsWith('/')) result.files.push({ name, size })
// Sizes after the data (a streamed zip): where the next header starts isn't known.
if (flags & DATA_DESCRIPTOR_FLAG) break
pos = nameEnd + extraLength + compressed
}
return result.files.length > 0 || result.encrypted ? result : null
}
public/app.css
+126
View File
@@ -891,3 +891,129 @@ kbd {
margin-top: 4px;
color: var(--faint);
}
/* --- Account (ACCESS=accounts) ------------------------------------------ */
.account-button {
height: 32px;
padding: 0 12px;
font-size: 13px;
}
.account-button .tier {
color: var(--accent);
text-transform: uppercase;
font-size: 11px;
letter-spacing: 0.04em;
}
.dialog {
width: min(420px, calc(100vw - 32px));
padding: 0;
border: 1px solid var(--border);
border-radius: var(--radius);
background: var(--surface);
color: var(--text);
box-shadow: var(--shadow);
}
.dialog::backdrop {
background: rgba(10, 14, 20, 0.35);
}
.dialog-head {
display: flex;
align-items: center;
justify-content: space-between;
padding: 14px 18px;
border-bottom: 1px solid var(--border);
}
.dialog-head h2 {
margin: 0;
font-size: 16px;
}
.dialog-body {
padding: 18px;
}
.form {
display: grid;
gap: 12px;
}
.form label {
display: grid;
gap: 4px;
font-size: 13px;
color: var(--muted);
}
.form input {
height: 38px;
padding: 0 10px;
border: 1px solid var(--border);
border-radius: 8px;
background: var(--bg);
color: var(--text);
font-size: 14px;
}
.form input:focus {
outline: 2px solid var(--accent-soft);
border-color: var(--accent);
}
.form .button {
justify-content: center;
}
.form-switch {
margin: 4px 0 0;
font-size: 13px;
color: var(--muted);
text-align: center;
}
.form-error {
margin: 0;
color: var(--bad);
font-size: 13px;
}
.api-key {
display: flex;
gap: 8px;
align-items: center;
margin-bottom: 18px;
}
.api-key code {
flex: 1;
min-width: 0;
padding: 8px 10px;
border-radius: 8px;
background: var(--bg);
font-family: var(--mono);
font-size: 12.5px;
overflow-wrap: anywhere;
}
.meter {
height: 6px;
margin-top: 4px;
border-radius: 3px;
background: var(--border);
overflow: hidden;
}
.meter span {
display: block;
height: 100%;
background: var(--accent);
}
.meter-full span {
background: var(--bad);
}
public/app.js
+195 -28
View File
@@ -17,21 +17,12 @@ const SOURCE_LABELS = {
subject: 'Subject',
post: 'Post id',
}
/** Where a known password came from (release_passwords.source). */
const PASSWORD_SOURCES = {
nzb: "from the spot's NZB",
'spot-title': "from the spot's title",
'spot-text': "from the spot's text",
}
/** Newznab categories (lib/newznab/categories.ts). */
const CATEGORIES = [
[1000, 'Console', []],
[2000, 'Movies', [[2030, 'SD'], [2040, 'HD'], [2045, 'UHD']]],
[3000, 'Audio', [[3010, 'MP3'], [3030, 'Audiobook'], [3040, 'Lossless']]],
[4000, 'PC', [[4050, 'Games']]],
[5000, 'TV', [[5030, 'SD'], [5040, 'HD'], [5045, 'UHD'], [5070, 'Anime']]],
[6000, 'XXX', [[6040, 'x264'], [6045, 'UHD'], [6060, 'ImgSet'], [6070, 'Other']]],
[7000, 'Books', [[7020, 'Ebook'], [7030, 'Comics']]],
[3000, 'Audio', [[3010, 'MP3'], [3040, 'Lossless']]],
[5000, 'TV', [[5030, 'SD'], [5040, 'HD'], [5045, 'UHD']]],
[7000, 'Books', [[7020, 'Ebook']]],
[8000, 'Other', [[8010, 'Misc']]],
]
function categoryLabel(id) {
@@ -60,6 +51,26 @@ const state = {
let requestId = 0
let searchAbort = null
/** Signing in (GET /api/v1/access): `open` needs none; with `accounts` the catalog takes a token. */
const account = {
mode: 'open',
signup: 'open',
token: storage('get', 'token'),
/** The profile's access: tier, limits, today's usage, the Newznab key. */
access: null,
}
/** localStorage, which a private window may refuse. */
function storage(action, key, value) {
try {
if (action === 'get') return localStorage.getItem(key)
if (action === 'set') localStorage.setItem(key, value)
else localStorage.removeItem(key)
} catch {
return null
}
}
// --- Formatting ------------------------------------------------------------
function formatSize(bytes) {
@@ -156,12 +167,26 @@ function syncControls() {
// --- API -------------------------------------------------------------------
async function api(path, { signal } = {}) {
const response = await fetch(path, { headers: { accept: 'application/json' }, signal })
if (!response.ok) throw new Error(`${response.status} ${response.statusText}`)
async function api(path, { signal, method = 'GET', body } = {}) {
const headers = { accept: 'application/json' }
if (account.token) headers.authorization = `Bearer ${account.token}`
if (body) headers['content-type'] = 'application/json'
const response = await fetch(path, { method, headers, signal, body: body && JSON.stringify(body) })
if (!response.ok) {
const error = await response.json().catch(() => ({}))
// A token that no longer works: sign in again.
if (response.status === 401 && account.mode === 'accounts' && !path.startsWith('/api/v1/auth/')) signOut()
throw new Error(error.error ?? error.errors?.[0]?.message ?? `${response.status} ${response.statusText}`)
}
return response.json()
}
/** A release's NZB as a plain link: with accounts it carries the key (a link can't send the token). */
function nzbHref(id) {
const key = account.mode === 'accounts' && account.access?.apiKey
return `/api/v1/releases/${id}/nzb${key ? `?apikey=${encodeURIComponent(key)}` : ''}`
}
function searchParams(page) {
const params = new URLSearchParams({ sort: state.sort, page: String(page), per_page: '50' })
if (state.q) params.set('q', state.q)
@@ -229,17 +254,16 @@ function releaseRow(release) {
if (release.complete) badges.append(badge(files, 'good', 'Complete'))
else if (release.fileTotal) badges.append(badge(`${release.fileCount}/${release.fileTotal}`, 'warn', 'Files present / announced'))
else badges.append(badge(files))
if (release.passworded && release.passwordKnown) badges.append(badge('Password known', 'warn', 'Its archive is encrypted; the password is in the NZB'))
else if (release.passworded) badges.append(badge('Password', 'bad', 'Its archive is encrypted: a password is needed'))
if (release.passworded) badges.append(badge('Password', 'bad', 'Its archive is encrypted: a password is needed'))
if (!release.settled) badges.append(badge('Collecting', 'accent', 'Recently seen: more files may still arrive'))
if (release.category && release.category !== 8010) badges.append(badge(categoryLabel(release.category), '', 'Category (from the name, the inner name, a spot or the groups)'))
if (release.category && release.category !== 8010) badges.append(badge(categoryLabel(release.category), '', 'Category (from the name)'))
if (release.source && release.source !== 'stem' && release.source !== 'subject') {
badges.append(badge(SOURCE_LABELS[release.source] ?? release.source, 'accent', 'How this release was recognised'))
}
row.querySelector('.release-size').textContent = formatSize(release.size)
const nzb = row.querySelector('.nzb')
nzb.href = `/api/v1/releases/${release.id}/nzb`
nzb.href = nzbHref(release.id)
row.querySelector('.release-main').addEventListener('click', () => openRelease(release.id, true))
return row
}
@@ -274,6 +298,8 @@ async function loadResults({ append = false } = {}) {
(shown < result.total ? ` · showing ${number.format(shown)}` : '')
$('#empty').hidden = result.total !== 0
$('#more').hidden = shown >= result.total && !result.totalCapped
// The day's count in the account button (the profile isn't counted).
if (account.access) loadProfile().catch(() => {})
} catch (error) {
if (id !== requestId) return
$('#list').classList.remove('stale')
@@ -309,7 +335,7 @@ function renderDetail(release) {
const actions = el(
'div',
{ class: 'actions' },
el('a', { class: 'button button-primary', href: `/api/v1/releases/${release.id}/nzb` },
el('a', { class: 'button button-primary', href: nzbHref(release.id) },
el('span', { text: 'Download NZB' })),
el('button', {
class: 'button',
@@ -357,12 +383,7 @@ function renderDetail(release) {
if (!release.settledAt) {
notices.push(el('div', { class: 'notice notice-warn', text: 'Recently seen: more files of this release may still arrive. The NZB has what is indexed so far.' }))
}
const passwords = release.passwords ?? []
if (passwords.length) {
notices.push(el('div', { class: 'notice notice-warn' },
el('span', { text: `${passwords.length === 1 ? 'Password' : 'Passwords'} (the first is in the NZB): ` }),
...passwords.map((p, i) => el('span', {}, i ? ', ' : '', el('code', { text: p.password }), ` (${PASSWORD_SOURCES[p.source] ?? p.source})`))))
} else if (release.passworded) {
if (release.passworded) {
notices.push(el('div', { class: 'notice notice-bad', text: 'Passworded: its archive has encrypted entries or headers. The password is not in the index.' }))
}
if (!release.readable && !release.innerName) {
@@ -433,8 +454,151 @@ function closeRelease(push = false) {
document.querySelector(`.release[data-id="${id}"] .release-main`)?.focus()
}
// --- Account (ACCESS=accounts) ---------------------------------------------
function renderAccountButton() {
const button = $('#account')
button.hidden = account.mode !== 'accounts'
if (!account.access) return button.replaceChildren('Sign in')
const { tier, usage, limits } = account.access
const of = (used, max) => (max === null ? number.format(used) : `${number.format(used)}/${number.format(max)}`)
button.title = 'Searches and NZBs today'
button.replaceChildren(el('span', { class: 'tier', text: tier }), ` ${of(usage.apiHits, limits.apiHits)} · ${of(usage.grabs, limits.grabs)} NZB`)
}
async function loadProfile() {
const { data } = await api('/api/v1/account/profile')
account.access = data.access
renderAccountButton()
}
/** Signed in with a new token: the profile, then the search that was waiting. */
async function signedIn(token) {
account.token = token
storage('set', 'token', token)
await loadProfile()
$('#account-dialog').close()
loadResults()
if (state.release) openRelease(state.release)
}
function signOut() {
if (account.token) api('/api/v1/account/logout', { method: 'POST' }).catch(() => {})
account.token = null
account.access = null
storage('remove', 'token')
renderAccountButton()
closeRelease()
$('#list').replaceChildren()
$('#more').hidden = true
$('#empty').hidden = true
$('#summary').textContent = 'Sign in to search the index.'
openAccount('signin')
}
function field(label, props) {
return el('label', {}, label, el('input', props))
}
function accountForm(view) {
const signup = view === 'signup'
const error = el('p', { class: 'form-error', role: 'alert', hidden: true })
const form = el(
'form',
{ class: 'form' },
field('Email', { name: 'email', type: 'email', autocomplete: 'email', required: true }),
field('Password', { name: 'password', type: 'password', autocomplete: signup ? 'new-password' : 'current-password', required: true, minlength: signup ? '8' : null }),
signup ? field('Password again', { name: 'passwordConfirmation', type: 'password', autocomplete: 'new-password', required: true }) : null,
signup && account.signup === 'invite' ? field('Invite code', { name: 'invite', autocomplete: 'off', spellcheck: 'false', required: true }) : null,
error,
el('button', { class: 'button button-primary', type: 'submit', text: signup ? 'Create account' : 'Sign in' })
)
form.addEventListener('submit', async (event) => {
event.preventDefault()
const values = Object.fromEntries(new FormData(form))
error.hidden = true
try {
const path = signup ? '/api/v1/auth/signup' : '/api/v1/auth/login'
const { data } = await api(path, { method: 'POST', body: signup ? { fullName: null, ...values } : values })
await signedIn(data.token)
} catch (failure) {
error.textContent = failure.message
error.hidden = false
}
})
const other = signup
? el('p', { class: 'form-switch' }, 'Have an account? ', el('button', { class: 'link-button', type: 'button', text: 'Sign in', onclick: () => openAccount('signin') }))
: account.signup !== 'closed'
? el('p', { class: 'form-switch' }, 'New here? ', el('button', { class: 'link-button', type: 'button', text: 'Create an account', onclick: () => openAccount('signup') }))
: null
return [form, other]
}
function meter(used, max) {
if (max === null) return null
const share = Math.min(1, used / Math.max(1, max))
return el('div', { class: `meter${used >= max ? ' meter-full' : ''}` }, el('span', { style: `width: ${Math.round(share * 100)}%` }))
}
function profileView() {
const { tier, tierUntil, limits, usage, apiKey } = account.access
const of = (used, max) => (max === null ? `${number.format(used)} (no limit)` : `${number.format(used)} of ${number.format(max)}`)
const key = el('code', { text: apiKey })
return [
el('dl', { class: 'facts' },
fact('Tier', tierUntil ? `${tier} until ${new Date(tierUntil).toLocaleDateString()}` : tier),
fact('Searches today', el('div', {}, of(usage.apiHits, limits.apiHits), meter(usage.apiHits, limits.apiHits))),
fact('NZBs today', el('div', {}, of(usage.grabs, limits.grabs), meter(usage.grabs, limits.grabs))),
fact('Retention', limits.retentionDays ? `${number.format(limits.retentionDays)} days` : 'Everything')),
...section('Newznab API key', 'for Prowlarr, Sonarr, Radarr', el('div', { class: 'api-key' },
key,
el('button', { class: 'button', type: 'button', text: 'Copy', onclick: async (event) => {
await navigator.clipboard?.writeText(account.access.apiKey)
event.target.textContent = 'Copied'
setTimeout(() => (event.target.textContent = 'Copy'), 1200)
} }),
el('button', { class: 'button', type: 'button', text: 'New key', title: 'The old key stops working', onclick: async () => {
if (!confirm('Make a new key? Apps using the old one stop working until they get the new one.')) return
const { apiKey: fresh } = await api('/api/v1/account/api-key', { method: 'POST' })
account.access.apiKey = fresh
key.textContent = fresh
} }))),
el('p', { class: 'form-switch' }, `URL for your apps: ${location.origin}`),
el('button', { class: 'button', type: 'button', text: 'Sign out', onclick: () => signOut() }),
]
}
async function openAccount(view) {
const dialog = $('#account-dialog')
if (view === 'profile') await loadProfile().catch(() => {})
if (view === 'profile' && !account.access) return
$('#account-title').textContent = { signin: 'Sign in', signup: 'Create an account', profile: 'Your account' }[view]
$('#account-body').replaceChildren(...(view === 'profile' ? profileView() : accountForm(view)))
if (!dialog.open) dialog.showModal()
dialog.querySelector('input')?.focus()
}
/** With accounts, nothing is searched before signing in. */
async function startAccess() {
try {
Object.assign(account, await api('/api/v1/access'))
} catch {
// An older server: open.
}
if (account.mode !== 'accounts') return true
if (account.token) await loadProfile().catch(() => {})
renderAccountButton()
if (account.access) return true
$('#summary').textContent = 'Sign in to search the index.'
if (!$('#account-dialog').open) openAccount('signin')
return false
}
// --- Wiring ----------------------------------------------------------------
$('#account').addEventListener('click', () => openAccount(account.access ? 'profile' : 'signin'))
$('#account-close').addEventListener('click', () => $('#account-dialog').close())
$('#q').addEventListener('input', (event) => {
state.q = event.target.value.trim()
scheduleSearch()
@@ -510,5 +674,8 @@ window.addEventListener('popstate', () => {
readUrl()
syncControls()
loadStats()
loadResults()
if (state.release) openRelease(state.release)
startAccess().then((ready) => {
if (!ready) return
loadResults()
if (state.release) openRelease(state.release)
})
public/index.html
+11
View File
@@ -14,8 +14,19 @@
<span>Voyager 2</span>
</a>
<dl class="stats" id="stats" aria-label="Index statistics"></dl>
<button type="button" class="button account-button" id="account" hidden></button>
</header>
<dialog class="dialog" id="account-dialog" aria-labelledby="account-title">
<div class="dialog-head">
<h2 id="account-title"></h2>
<button type="button" class="icon-button" id="account-close" aria-label="Close">
<svg aria-hidden="true" viewBox="0 0 24 24"><path d="M6 6l12 12M18 6 6 18" /></svg>
</button>
</div>
<div class="dialog-body" id="account-body"></div>
</dialog>
<main class="layout">
<section class="search" aria-label="Search">
<div class="search-row">
start/env.ts
+4 -7
View File
@@ -43,20 +43,19 @@ export default await Env.create(new URL('../', import.meta.url), {
NNTP_PASSWORD: Env.schema.string.optional(),
NNTP_SECURE: Env.schema.boolean.optional(),
NNTP_CONNECTIONS: Env.schema.number.optional(),
NNTP_MIN_CONNECTIONS: Env.schema.number.optional(),
NNTP_ACCOUNT_CONNECTIONS: Env.schema.number.optional(),
NNTP_RESERVED_CONNECTIONS: Env.schema.number.optional(),
NNTP_TIMEOUT_MS: Env.schema.number.optional(),
NNTP_HEDGE_AFTER_SECONDS: Env.schema.number.optional(),
NNTP_HEDGE_CONNECTIONS: Env.schema.number.optional(),
NNTP_CONNECTION_MAX_MINUTES: Env.schema.number.optional(),
NNTP_IDLE_CLOSE_SECONDS: Env.schema.number.optional(),
CORPUS_FILTER: Env.schema.boolean.optional(),
NNTP_STALL_KB_PER_MINUTE: Env.schema.number.optional(),
// Newznab API key (empty: no key needed)
NEWZNAB_API_KEY: Env.schema.secret.optional(),
// Access (config/access.ts): open (one key, no limits) or accounts (a key per user, tiers with daily limits)
ACCESS: Env.schema.enum.optional(['open', 'accounts'] as const),
SIGNUP: Env.schema.enum.optional(['open', 'invite', 'closed'] as const),
// Coordinator
RANGE_ARTICLES: Env.schema.number.optional(),
TAIL_EVERY_SECONDS: Env.schema.number.optional(),
@@ -79,8 +78,6 @@ export default await Env.create(new URL('../', import.meta.url), {
SPOOL_MAX_GB: Env.schema.number.optional(),
PREFETCH_PER_RANGE: Env.schema.number.optional(),
PROBE_CONNECTIONS: Env.schema.number.optional(),
PROBE_MIN_CONNECTIONS: Env.schema.number.optional(),
PROBE_MIN_FREE_GB: Env.schema.number.optional(),
HTTP_ROUTES: Env.schema.enum.optional(['all', 'node', 'public'] as const),
PREFETCH_PER_GROUP: Env.schema.number.optional(),
})
start/routes.ts
+6 -1
View File
@@ -45,13 +45,18 @@ router
router
.group(() => {
router.get('profile', [controllers.Profile, 'show'])
router.post('api-key', [controllers.Profile, 'resetApiKey'])
router.post('logout', [controllers.AccessTokens, 'destroy'])
})
.prefix('account')
.as('profile')
.use(middleware.auth())
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it); Newznab has its key.
// How the web interface signs in: open, or accounts with limits (config/access.ts).
router.get('access', [controllers.Releases, 'access'])
// The catalog (REBUILD.md, part 9): open, as voyager 1's (the web interface uses it), unless ACCESS
// is `accounts`: then a user's token or apikey, counted as Newznab's are (AccessService).
router.get('releases', [controllers.Releases, 'index'])
router.get('releases/:id', [controllers.Releases, 'show']).where('id', router.matchers.number())
router.get('releases/:id/nzb', [controllers.Releases, 'nzb']).where('id', router.matchers.number())
tests/fixtures/sevenzip/files-encrypted.tail
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/header-encrypted.tail
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/plain.tail
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/sample.bin
Vendored
-4017
View File
File diff suppressed because it is too large. Load diff
tests/fixtures/sevenzip/sample.lzma
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/split-last-volume.tail
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/tiny.7z
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/sevenzip/uncompressed-header.tail
Vendored
BIN
View File
Binary file not shown.
tests/functional/access.spec.ts
+155
View File
@@ -0,0 +1,155 @@
import { test } from '@japa/runner'
import { DateTime } from 'luxon'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import User from '#models/user'
import BatchStore from '#services/batch_store'
import AccessService from '#services/access_service'
import { TIERS } from '#config/access'
import { RangeIndexer } from '#lib/edge/index'
import { DEFAULT_RULES } from '#lib/rules/index'
import type { Overview } from '#lib/nntp/index'
const GROUP = 'alt.binaries.test'
function file(name: string, counter: string, parts: number, at: number): Overview[] {
return Array.from({ length: parts }, (_, i) => [at + i, {
'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
'message-id': `<${name}-${i + 1}@test>`, ':bytes': String(700000 + i), 'xref': `news.example ${GROUP}:${at + i}`,
}] as Overview)
}
/** Two releases: a show (recent) and a movie posted two years ago. */
async function catalog() {
const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
const [n] = await db.table('nodes').insert({ name: 'access-node' }).returning('id')
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: n.id }).returning('id')
const indexer = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
indexer.add([...file('Some.Show.S01E01.1080p.WEB.mkv', 'Some.Show.S01E01.1080p.WEB [1/1]', 2, 100), ...file('Old.Movie.2024.1080p.BluRay.mkv', 'Old.Movie.2024.1080p.BluRay [1/1]', 1, 300)], 1001)
indexer.finish()
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
const show = await db.from('releases').where('name', 'ilike', '%show%').firstOrFail()
const movie = await db.from('releases').where('name', 'ilike', '%movie%').firstOrFail()
await db.from('releases').where('id', movie.id).update({ posted_at: DateTime.now().minus({ years: 2 }).toJSDate() })
return { show: Number(show.id), movie: Number(movie.id) }
}
const user = (email = 'someone@example.com') => User.create({ email, password: 'secret-password' })
test.group('Access with accounts', (group) => {
const free = { ...TIERS.free }
group.each.setup(() => testUtils.db().withGlobalTransaction())
group.each.setup(() => {
AccessService.mode = 'accounts'
AccessService.signup = 'invite'
Object.assign(TIERS.free, { apiHits: 3, grabs: 1, retentionDays: 365 })
return () => {
AccessService.mode = 'open'
AccessService.signup = 'open'
Object.assign(TIERS.free, free)
}
})
test("Newznab takes a user's key, counts searches and grabs per day and keeps to the tier's retention", async ({ client, assert }) => {
const { show, movie } = await catalog()
const someone = await user()
assert.include((await client.get('/api?t=search&q=')).text(), 'code="100"')
assert.include((await client.get('/api?t=search&q=&apikey=0123456789abcdef0123456789abcdef')).text(), 'code="100"')
// caps isn't counted.
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), '<registration available="yes" open="no"/>')
const feed = (await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text()
// The free tier sees a year back: the movie isn't found.
assert.include(feed, 'Some Show S01E01')
assert.notInclude(feed, 'Old Movie')
assert.include(feed, `t=get&amp;id=${show}&amp;apikey=${someone.apiKey}`)
assert.include(feed, '<newznab:apilimits apicurrent="1" apimax="3" grabcurrent="0" grabmax="1"/>')
// Past the retention: refused, and not counted as a grab.
assert.include((await client.get(`/api?t=get&id=${movie}&apikey=${someone.apiKey}`)).text(), 'code="102"')
assert.include((await client.get(`/api?t=get&id=999999&apikey=${someone.apiKey}`)).text(), 'code="300"')
const nzb = await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)
assert.include(nzb.header('content-type'), 'application/x-nzb')
assert.include((await client.get(`/api?t=get&id=${show}&apikey=${someone.apiKey}`)).text(), 'code="501"')
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)
assert.include((await client.get(`/api?t=search&q=&apikey=${someone.apiKey}`)).text(), 'code="500"')
assert.deepEqual(await AccessService.usage(someone.id), { apiHits: 3, grabs: 1 })
// VIP sees everything; a tier that ran out is free again.
AccessService.grantTier(someone, 'vip', 30)
await someone.save()
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'Old Movie 2024')
someone.tierUntil = DateTime.now().minus({ days: 1 })
await someone.save()
assert.include((await client.get(`/api?t=movie&q=old movie&apikey=${someone.apiKey}`)).text(), 'code="500"')
someone.suspended = true
await someone.save()
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="101"')
})
test('the catalog needs a token or a key, and counts as Newznab does', async ({ client, assert }) => {
const { show, movie } = await catalog()
const someone = await user()
;(await client.get('/api/v1/releases')).assertStatus(401)
;(await client.get(`/api/v1/releases/${show}/nzb`)).assertStatus(401)
// Stats and how to sign in stay open.
;(await client.get('/api/v1/stats')).assertStatus(200)
assert.deepEqual((await client.get('/api/v1/access')).body(), { mode: 'accounts', signup: 'invite' })
const found = await client.get('/api/v1/releases').loginAs(someone)
found.assertStatus(200)
assert.deepEqual((found.body() as { releases: { id: number }[] }).releases.map((r) => r.id), [show])
;(await client.get(`/api/v1/releases/${movie}`).loginAs(someone)).assertStatus(403)
;(await client.get(`/api/v1/releases/${show}`).loginAs(someone)).assertStatus(200)
// A plain link (the web interface's download button) carries the key.
const nzb = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
nzb.assertStatus(200)
const over = await client.get(`/api/v1/releases/${show}/nzb?apikey=${someone.apiKey}`)
over.assertStatus(429)
assert.deepEqual(over.body(), { error: 'Download limit reached', code: 501 })
// The refused movie wasn't counted: a third search is left.
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(200)
;(await client.get('/api/v1/releases').loginAs(someone)).assertStatus(429)
const profile = ((await client.get('/api/v1/account/profile').loginAs(someone)).body() as any).data
assert.equal(profile.access.tier, 'free')
assert.equal(profile.access.apiKey, someone.apiKey)
assert.deepEqual(profile.access.usage, { apiHits: 3, grabs: 1 })
const reset = (await client.post('/api/v1/account/api-key').loginAs(someone)).body() as { apiKey: string }
assert.notEqual(reset.apiKey, someone.apiKey)
assert.include((await client.get(`/api?t=caps&apikey=${someone.apiKey}`)).text(), 'code="100"')
})
test('signup takes an unused invite, which may give a tier', async ({ client, assert }) => {
const form = (email: string, invite?: string) => ({ email, password: 'secret-password', passwordConfirmation: 'secret-password', fullName: null, invite })
await db.table('invites').insert([{ code: 'plain-invite' }, { code: 'vip-invite', tier: 'vip', tier_days: 365 }])
;(await client.post('/api/v1/auth/signup').json(form('a@example.com'))).assertStatus(403)
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'no-such-code'))).assertStatus(403)
// A refused invite leaves no account behind.
assert.isNull(await User.findBy('email', 'a@example.com'))
;(await client.post('/api/v1/auth/signup').json(form('a@example.com', 'plain-invite'))).assertStatus(200)
;(await client.post('/api/v1/auth/signup').json(form('b@example.com', 'plain-invite'))).assertStatus(403)
const vip = await client.post('/api/v1/auth/signup').json(form('c@example.com', 'vip-invite'))
vip.assertStatus(200)
const c = await User.findByOrFail('email', 'c@example.com')
assert.equal(c.tier, 'vip')
assert.closeTo(c.tierUntil!.diffNow('days').days, 365, 1)
assert.match(c.apiKey, /^[0-9a-f]{32}$/)
// Renewed early, the year is added to what's left.
AccessService.grantTier(c, 'vip', 365)
assert.closeTo(c.tierUntil!.diffNow('days').days, 730, 1)
AccessService.signup = 'closed'
;(await client.post('/api/v1/auth/signup').json(form('d@example.com', 'vip-invite'))).assertStatus(403)
})
})
tests/functional/batch_store.spec.ts
-59
View File
@@ -120,40 +120,6 @@ test.group('BatchStore', (group) => {
assert.equal(Number(chunks!.n), 4)
})
test('candidates added to a settled release (a later range of its upload) are claimable at once', async ({ assert }) => {
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E02.1080p.mkv', 2, 'Some.Show.S01E02.1080p [1/2]', '2026-10-05T10:00:00Z'))
// Settled before the next range came (the settler marks only what it settles).
await db.from('releases').update({ settled_at: new Date() })
await db.from('probe_candidates').update({ ready: true })
await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E02.1080p.par2', 1, 'Some.Show.S01E02.1080p [2/2]', '2026-10-05T10:20:00Z'))
const candidates = await db.from('probe_candidates').orderBy('id').select('kind', 'ready')
assert.deepEqual(candidates.map((c) => c.kind), ['par2'])
assert.isTrue(candidates.every((c) => c.ready))
assert.lengthOf(await releasesOf(), 1)
})
test('a split 7z read in pieces keeps the tail of its highest volume only; RAR gets none', async ({ assert }) => {
// As the live tail reads it while posted: each piece queues a tail for the last volume it saw so far.
await storeIn(GROUP, 1, [
...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
])
await storeIn(GROUP, 100_001, upload(100_010, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'))
// A lower volume posted late: its tail goes at once.
await storeIn(GROUP, 200_001, upload(200_010, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'))
await storeIn(GROUP, 300_001, upload(300_010, 'Some.Show.S01E03.part01.rar', 2, 'Some.Show.S01E03 [1/2]', '2026-10-23T09:00:00Z'))
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E03.part02.rar', 2, 'Some.Show.S01E03 [2/2]', '2026-10-23T09:02:00Z'))
const candidates = await db.from('probe_candidates').orderBy('file').select('kind', 'file')
assert.deepEqual(
candidates.map((c) => [c.kind, c.file]),
[
['archive', 'Some.Show.S01E03.part01.rar'],
['archive', 'TsIgNBewuFbuwWy.7z.001'],
['7z-tail', 'TsIgNBewuFbuwWy.7z.004'],
]
)
})
test('the same upload in another group (multi-posted) is a release of its own', async ({ assert }) => {
await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
@@ -180,29 +146,4 @@ test.group('BatchStore', (group) => {
const keys = (await releasesOf()).filter((r) => r.identity_key === first.identity_key).map((r) => r.release_key)
assert.deepEqual(keys, [first.identity_key, `${first.identity_key}:3`])
})
test('a re-upload is numbered after the highest number, not into the holes merges left', async ({ assert }) => {
for (let day = 0; day < 4; day++) {
await storeIn(GROUP, 1 + day * 100_000, upload(10 + day * 100_000, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', `2026-10-0${day * 2 + 1}T10:00:00Z`))
}
const [first] = await releasesOf()
// Merged away: key:2 and key:3 gone, key and key:4 left.
await db.from('releases').whereIn('release_key', [`${first.identity_key}:2`, `${first.identity_key}:3`]).delete()
await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-09T10:00:00Z'))
const keys = (await releasesOf()).map((r) => r.release_key.split(':')[1] ?? '1')
assert.deepEqual(keys, ['1', '4', '5'])
})
test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
await storeIn(GROUP, 1_000_000, [
...upload(1_000_000, 'Murdoch Mysteries S8D1.nzb', 1, '[00/80]', '2026-10-05T10:00:00Z'),
...upload(1_000_010, 'Some.Tool.v1.zip', 1, '[01/01]', '2026-10-05T10:00:00Z'),
...upload(1_000_020, 'Some.Show.S01E01.part01.rar', 2, '[02/30]', '2026-10-05T10:00:00Z'),
])
const flags = new Map((await db.from('releases').select('name', 'sidecar_only')).map((r) => [r.name, r.sidecar_only]))
assert.isTrue(flags.get('Murdoch Mysteries S8D1'))
assert.isFalse(flags.get('Some.Tool.v1'))
// Over 1 MB: not looked at.
assert.isNull(flags.get('Some.Show.S01E01'))
})
})
tests/functional/leases.spec.ts
-7
View File
@@ -31,12 +31,5 @@ test.group('Leases', (group) => {
// a1's worker went quiet: its lease is taken over (a2 alone counts).
await db.from('ranges').where('id', a1).update({ heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
assert.equal(await leased(5), a1)
// Read to its end, its worker gone before finishing: done. One still reading, or quiet but not at its end, stays.
await db.from('ranges').where('id', a2).update({ next_id: 201, heartbeat_at: new Date(Date.now() - 24 * 3600_000) })
await db.from('ranges').where('id', b1).update({ next_id: 101 })
assert.equal(await Leases.finishAbandoned(), 1)
assert.equal((await db.from('ranges').where('id', a2).firstOrFail()).status, 'done')
assert.equal((await db.from('ranges').where('id', b1).firstOrFail()).status, 'leased')
})
})
tests/functional/nntp_shares.spec.ts
-53
View File
@@ -1,53 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import coordinatorConfig from '#config/coordinator'
import { grant, targets, NntpShare } from '#services/nntp_shares'
test.group('NNTP shares', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('targets: minimums first, then by priority what each wants', ({ assert }) => {
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 0 }
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 0 }
// History running: the coordinator all it wants, the prober the rest.
assert.deepEqual(Object.fromEntries(targets(45, [prober, coordinator])), { coordinator: 38, prober: 7 })
// History paused: the coordinator its minimum (or what it uses), the prober the rest.
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 8 }])), { coordinator: 8, prober: 37 })
// Wanting more than its minimum doesn't take another's minimum.
assert.deepEqual(Object.fromEntries(targets(45, [prober, { ...coordinator, want: 60 }])), { coordinator: 43, prober: 2 })
})
test('grant: growing only into connections the others closed or gave up', ({ assert }) => {
const prober = { name: 'prober', priority: 1, min: 2, want: 40, open: 37, granted: 37 }
const coordinator = { name: 'coordinator', priority: 2, min: 8, want: 38, open: 8 }
// History came back: the prober still holds 37, the coordinator keeps its 8 for now.
assert.equal(grant(45, coordinator, [prober]), 8)
// The prober syncs: its target is 7, its grant goes down at once.
assert.equal(grant(45, { ...prober, open: 37 }, [{ ...coordinator, granted: 8 }]), 7)
// Its connections not closed yet: still held.
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 30 }]), 15)
// Closed: the coordinator gets all it wants.
assert.equal(grant(45, coordinator, [{ ...prober, granted: 7, open: 7 }]), 38)
})
test('sync: each process its row; a process gone (no heartbeat) holds nothing', async ({ assert }) => {
const total = coordinatorConfig.nntpShares.total
const coordinator = new NntpShare('coordinator', 2, 8, 38)
const prober = new NntpShare('prober', 1, 2, total)
assert.equal(await coordinator.sync(8, 0), 8)
assert.equal(await prober.sync(total, 0), total - 8)
// The coordinator wants all: the prober holds its grant until it has synced and closed them.
assert.equal(await coordinator.sync(38, 8), 8)
assert.equal(await prober.sync(total, total - 8), total - 38)
assert.equal(await coordinator.sync(38, 8), 8)
assert.equal(await prober.sync(total, total - 38), total - 38)
assert.equal(await coordinator.sync(38, 8), 38)
// The prober's process died: after a minute without a sync its share is free.
await db.from('nntp_shares').where('name', 'prober').update({ open: total, granted: total, heartbeat_at: new Date(Date.now() - 120_000) })
assert.equal(await coordinator.sync(38, 38), 38)
await coordinator.leave()
assert.equal(await prober.sync(total, 0), total)
})
})
tests/functional/par2_completeness.spec.ts
-67
View File
@@ -1,67 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import BatchStore from '#services/batch_store'
import { decidePar2Completeness, par2Complete } from '#services/par2_completeness'
import { RangeIndexer } from '#lib/edge/index'
import { DEFAULT_RULES } from '#lib/rules/index'
import type { Overview } from '#lib/nntp/index'
const GROUP = 'alt.binaries.test'
/** A file posted without an [i/N] counter: the release gets no file_total. */
function file(name: string, parts: number, at: number): Overview[] {
return Array.from({ length: parts }, (_, i) => [at + i, {
'subject': `"${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
'message-id': `<${name}-${i + 1}@test>`, ':bytes': '700000', 'xref': `news.example ${GROUP}:${at + i}`,
}] as Overview)
}
async function store(start: number, overviews: Overview[]) {
const g = (await db.from('groups').where('name', GROUP).first()) ?? (await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('*'))[0]
const [n] = await db.table('nodes').insert({ name: `n${start}` }).returning('id')
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: start, end_id: start + 999, next_id: start, status: 'leased', node_id: n.id }).returning('id')
const indexer = new RangeIndexer({ group: GROUP, start, end: start + 999, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
indexer.add(overviews, start + 1000)
indexer.finish()
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
}
const setFile = (name: string, length: number, hash16k = name.padEnd(32, '0').slice(0, 32)) => ({ name, length, hash16k })
test.group('par2 completeness', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('every described file by name or by hash; a renamed upload by its files and bytes', ({ assert }) => {
const set = [setFile('a.mkv', 1000), setFile('a.nfo', 10)]
const files = [{ name: 'a.mkv', size: 1100 }, { name: 'a.nfo', size: 12 }, { name: 'a.par2', size: 50 }]
assert.isTrue(par2Complete(set, files, []))
assert.isFalse(par2Complete(set, files.slice(0, 1), []))
// The nfo renamed, but its probe hashed it.
assert.isTrue(par2Complete(set, [files[0], { name: 'x.nfo', size: 12 }], [{ hash16k: set[1].hash16k, size: 10 }]))
// Renamed after the par2 was made (random names): two data files and enough bytes; one missing, or too small.
assert.isTrue(par2Complete(set, [{ name: 'Qx81.bin', size: 1100 }, { name: 'Zr02.bin', size: 12 }, { name: 'Qx.par2', size: 50 }], []))
assert.isFalse(par2Complete(set, [{ name: 'Qx81.bin', size: 1100 }, { name: 'Qx.par2', size: 50 }], []))
assert.isFalse(par2Complete(set, [{ name: 'Qx81.bin', size: 500 }, { name: 'Zr02.bin', size: 12 }], []))
})
test('decided once settled with a par2 set, and kept when more files come', async ({ assert }) => {
await store(1, [...file('Some.Upload.mkv', 3, 10), ...file('Some.Upload.par2', 1, 20)])
const release = await db.from('releases').firstOrFail()
assert.isNull(release.file_total)
await db.table('par2_files').multiInsert([setFile('Some.Upload.mkv', 2_000_000), setFile('Some.Upload.nfo', 100)].map((f, i) => ({ set_id: 's'.repeat(32), file_id: String(i).repeat(32), ...f })))
await db.table('par2_sets').insert({ set_id: 's'.repeat(32), release_id: release.id, files: 2 })
// Not settled yet: left alone.
assert.equal(await decidePar2Completeness(db, [release.id]), 0)
await db.from('releases').where('id', release.id).update({ settled_at: new Date() })
assert.equal(await decidePar2Completeness(db, [release.id]), 1)
assert.isFalse((await db.from('releases').where('id', release.id).firstOrFail()).complete)
// The nfo comes in a later range: complete, and the batch store's own update keeps it.
await store(1001, file('Some.Upload.nfo', 1, 1010))
await decidePar2Completeness(db, [release.id])
await store(2001, file('Some.Upload.sfv', 1, 2010))
const after = await db.from('releases').where('id', release.id).firstOrFail()
assert.equal(after.files, 4)
assert.isTrue(after.complete)
})
})
tests/functional/par2_merger.spec.ts
+1 -41
View File
@@ -49,7 +49,7 @@ test.group('Par2Merger', (group) => {
await db.table('par2_sets').insert({ set_id: setId, release_id: par2, files: 2 })
const round = await new Par2Merger().round()
assert.deepEqual(round, { sets: 1, merges: 1, releases: 3, nameMerges: 0, nameReleases: 0, stems: 0, stemMerges: 0, stemReleases: 0, folded: { looked: 0, sidecars: 0, folded: 0 } })
assert.deepEqual(round, { sets: 1, merges: 1, releases: 3, stems: 0, stemMerges: 0, stemReleases: 0, folded: { looked: 0, sidecars: 0, folded: 0 } })
const target = await db.from('releases').where('id', b).firstOrFail()
assert.equal(target.files, 4)
assert.equal(target.segments, 40)
@@ -78,44 +78,6 @@ test.group('Par2Merger', (group) => {
assert.equal((await new Par2Merger().round()).sets, 1)
})
test('a round behind by hours goes through them a window at a time, past the gaps', async ({ assert }) => {
const ago = (hours: number) => new Date(Date.now() - hours * 3600_000)
await db.table('par2_sets').multiInsert([
{ set_id: 'a1'.repeat(16), release_id: 1, files: 1, created_at: ago(30) },
{ set_id: 'a2'.repeat(16), release_id: 2, files: 1, created_at: ago(2.5) },
{ set_id: 'a3'.repeat(16), release_id: 3, files: 1, created_at: ago(0.5) },
])
await db.table('worker_cursors').insert({ name: 'merger', at: ago(48) })
assert.equal((await new Par2Merger().round()).sets, 3)
const saved = await db.from('worker_cursors').where('name', 'merger').firstOrFail()
assert.isAbove(new Date(saved.at).getTime(), ago(0.5).getTime())
})
test('a par2 file held by more than 20 files is a common one: it merges nothing', async ({ assert }) => {
const at = new Date('2022-07-08T03:46:00Z')
const ids: number[] = []
for (const key of ['dd000001', 'dd000002']) {
const [row] = await db
.table('releases')
.insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
.returning('id')
ids.push(Number(row.id))
}
// The par2 is held by the first; the second holds the file, and so do 20 releases elsewhere and years apart.
const setId = hash('1')
await db.table('par2_sets').insert({ set_id: setId, release_id: ids[0], files: 1 })
await db.table('par2_files').insert({ set_id: setId, file_id: hash('0'), hash16k: hash('9'), length: 30214, name: 'sample.nfo' })
await db.table('file_hashes').insert({ candidate_id: 4000, release_id: ids[1], hash16k: hash('9'), size: 30214 })
for (let i = 1; i <= 20; i++) await db.table('file_hashes').insert({ candidate_id: 4000 + i, release_id: 900000 + i, hash16k: hash('9'), size: 30214 })
const round = await new Par2Merger().round()
assert.deepEqual([round.sets, round.merges], [1, 0])
assert.equal(await new Par2Merger().mergeSet(setId), 0)
assert.equal((await db.from('releases').whereIn('id', ids).select('id')).length, 2)
// One holder fewer: the file is one upload's again.
await db.from('file_hashes').where('candidate_id', 4020).delete()
assert.equal(await new Par2Merger().mergeSet(setId), 1)
})
test('a set held by two of the releases merged is held once by the target', async ({ assert }) => {
const at = new Date('2022-07-08T03:46:00Z')
const ids: number[] = []
@@ -258,8 +220,6 @@ test.group('Par2Merger', (group) => {
// A picture isn't a sidecar; an nfo an hour later belongs to nothing here.
const picture = await release('cover', 1, null, at, ['cover.jpg'])
const late = await release('Murdoch Mysteries S8D1', 1, null, new Date(at.getTime() + 3600_000), ['Murdoch Mysteries S8D1.nfo'])
// Queued, its release gone meanwhile (merged): leaves the queue with its batch.
await db.table('fold_candidates').insert({ release_id: 987654321 })
const round = await new Par2Merger().round()
assert.deepEqual(round.folded, { looked: 5, sidecars: 2, folded: 1 })
tests/functional/par2_name_merge.spec.ts
-88
View File
@@ -1,88 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import BatchStore from '#services/batch_store'
import { Par2Merger, fileKey } from '#services/par2_merger'
import { RangeIndexer } from '#lib/edge/index'
import { DEFAULT_RULES } from '#lib/rules/index'
import type { Overview } from '#lib/nntp/index'
const GROUP = 'alt.binaries.test'
/** One file without a counter, from `poster`, posted `minutes` after the first. */
function file(name: string, at: number, minutes: number, poster = 'up <up@load.er>'): Overview[] {
const date = new Date(Date.parse('2026-10-20T10:00:00Z') + minutes * 60_000).toUTCString().replace('GMT', '+0000')
return [[at, { 'subject': `"${name}" yEnc (1/1)`, 'from': poster, 'date': date, 'message-id': `<${at}@test>`, ':bytes': '700000', 'xref': `news.example ${GROUP}:${at}` }] as Overview]
}
let stores = 0
async function store(overviews: Overview[]) {
stores++
const g = (await db.from('groups').where('name', GROUP).first()) ?? (await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('*'))[0]
const [n] = await db.table('nodes').insert({ name: `n${stores}` }).returning('id')
const [r] = await db.table('ranges').insert({ group_id: g.id, start_id: stores * 100_000, end_id: stores * 100_000 + 99_999, next_id: stores * 100_000, status: 'leased', node_id: n.id }).returning('id')
const indexer = new RangeIndexer({ group: GROUP, start: stores * 100_000, end: stores * 100_000 + 99_999, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
indexer.add(overviews, stores * 100_000 + 100_000)
indexer.finish()
await BatchStore.store(Number(r.id), Number(n.id), indexer.take())
}
test.group('Par2Merger / by file names', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('a name as compared: no case, spaces or punctuation, UTF-8 read as Latin-1 read back', ({ assert }) => {
assert.equal(fileKey('Some/Dir/Ein.dunkles.Vermächtnis - 1080p.MKV'), fileKey('Ein dunkles Vermächtnis 1080p.mkv'))
assert.notEqual(fileKey('Track One.flac'), fileKey('Track Two.flac'))
})
test("a set's poster's releases nearby whose files it all names join its release; others stay", async ({ assert }) => {
await store([
// The album's index files, its tracks apart (no counter: a release each), and the same poster's next album.
...file('Some.Album-WEB-2026.par2', 100_010, 0),
...file('Some.Album-WEB-2026.nfo', 100_011, 0),
...file('Track One.flac', 100_020, 1),
...file('Track Two.flac', 100_030, 2),
...file('Ein.dunkles.Vermächtnis.mkv', 100_040, 3),
...file('Another Song.flac', 100_050, 4),
// A track the set names, but posted by someone else, and one a day later.
...file('Track One.flac', 100_060, 5, 'other <o@x>'),
])
await store(file('Track Two.flac', 200_070, 24 * 60 + 10))
const ids = Object.fromEntries((await db.from('releases').select('id', 'name', 'poster', 'posted_at')).map((r) => [`${r.name}${r.poster.startsWith('other') ? ' (other)' : ''}${new Date(r.posted_at).getUTCDate() === 21 ? ' (later)' : ''}`, Number(r.id)]))
const index = ids['Some.Album-WEB-2026']
const setId = 'a'.repeat(32)
await db.table('par2_files').multiInsert(
['Track One.flac', 'Track Two.flac', 'Ein.dunkles.Vermächtnis.mkv'].map((name, i) => ({ set_id: setId, file_id: String(i).repeat(32), hash16k: 'f'.repeat(32), length: 600_000, name }))
)
await db.table('par2_sets').insert({ set_id: setId, release_id: index, files: 3 })
assert.equal(await new Par2Merger().mergeByName(setId), 3)
const left = (await db.from('releases').orderBy('name').select('name', 'files', 'poster')).map((r) => `${r.name} ${r.files}${r.poster.startsWith('other') ? ' (other)' : ''}`)
assert.includeMembers(left, ['Another Song 1', 'Track One 1 (other)', 'Track Two 1'])
assert.lengthOf(left, 4)
const merged = await db.from('release_merges').where('via', 'name').count('* as n')
assert.equal(Number(merged[0].n), 3)
// Nothing more to find.
assert.equal(await new Par2Merger().mergeByName(setId), 0)
})
test('not a holder years off (a repost of the par2); a member with an article the target has merges without it', async ({ assert }) => {
await store([...file('Some.Album-WEB-2026.par2', 100_010, 0), ...file('Track One.flac', 100_020, 1)])
// The same par2 posted again a year later, in a range of its own.
await store(file('Some.Album-WEB-2026.par2', 200_010, 365 * 24 * 60))
const releases = await db.from('releases').orderBy('posted_at').select('id', 'name', 'posted_at')
const [index, track, repost] = releases.map((r) => Number(r.id))
const setId = 'b'.repeat(32)
await db.table('par2_files').insert({ set_id: setId, file_id: '1'.repeat(32), hash16k: 'f'.repeat(32), length: 600_000, name: 'Track One.flac' })
await db.table('par2_sets').multiInsert([index, repost].map((id) => ({ set_id: setId, release_id: id, files: 1 })))
// The repost is the newest holder: nothing near it to join.
assert.equal(await new Par2Merger().mergeByName(setId), 0)
await db.from('par2_sets').where('release_id', repost).delete()
// Both releases hold a candidate of the same article.
await db.table('probe_candidates').multiInsert([index, track].map((id) => ({ release_id: id, kind: 'par2', message_id: '<same@x>', file: 'x.par2' })))
assert.equal(await new Par2Merger().mergeByName(setId), 1)
assert.lengthOf(await db.from('probe_candidates').where('message_id', '<same@x>'), 1)
assert.lengthOf(await db.from('releases').whereIn('id', [index, track]), 1)
assert.lengthOf(await db.from('releases').where('id', repost), 1)
})
})
tests/functional/par2_pruner.spec.ts
+5 -61
View File
@@ -53,72 +53,16 @@ test.group('Par2Pruner', (group) => {
const locked = await set('e', e, 2, null)
await db.table('release_verdicts').insert({ release_id: e, rule: 'locked', rule_version: 1 })
// Final: its holder held by a run of done ranges with hours to spare, nothing hashed (a readable upload's files).
const f = await release()
const final = await set('f', f, 2, null)
const posted = new Date('2024-03-01T12:00:00Z')
const [group] = await db.table('groups').insert({ name: 'alt.binaries.pruned', priority: 1 }).returning('id')
const groupId = Number(group.id)
await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100 })
const [range] = await db
.table('ranges')
.insert({ group_id: groupId, start_id: 1, end_id: 10000, next_id: 10001, status: 'done', done_at: old, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z') })
.returning('id')
await db.table('range_blocks').insert({ group_id: groupId, start_id: 1, end_id: 10000, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z'), at_group_start: false })
// A lone candidate unprobed since before the range was done: its hash may still match. Kept.
await db.table('probe_candidates').insert({ id: 9001, release_id: b, kind: 'lone', message_id: '<9001@x>', file: 'x', status: 'waiting', created_at: new Date(old.getTime() - 3600_000) })
// Out of budget: one batch of two sets (a, b), the next round goes on.
assert.deepEqual(await new Par2Pruner(2).round(0), { complete: 1, judged: 0, final: 0, rows: 2 })
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 1, final: 0, rows: 2 })
const round = await new Par2Pruner().round()
assert.deepEqual(round, { complete: 1, judged: 1, rows: 4 })
assert.equal(await count(complete), 0)
assert.equal(await count(apart), 2)
assert.equal(await count(recent), 2)
assert.equal(await count(dead), 0)
assert.equal(await count(locked), 2)
assert.equal(await count(final), 2)
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '5')
// Probed: final. The next pass starts over from the first set.
await db.from('probe_candidates').where('id', 9001).update({ status: 'done' })
assert.deepEqual(await new Par2Pruner(2).round(), { complete: 0, judged: 0, final: 1, rows: 2 })
assert.equal(await count(final), 0)
assert.equal((await db.from('par2_sets').count('* as n'))[0].n, '6')
// A hash of its files since the merger's last window: the merger hasn't matched it yet. Kept.
const g = await release()
const unseen = await set('g', g, 1, null)
await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100 })
await db.table('worker_cursors').insert({ name: 'merger', at: new Date(Date.now() - 3600_000) })
await db.table('probe_candidates').insert({ id: 9002, release_id: await release(), kind: 'lone', message_id: '<9002@x>', file: 'x', status: 'done' })
await db.table('file_hashes').insert({ candidate_id: 9002, release_id: 1, hash16k: 'g0'.padEnd(32, '0'), size: 1000 })
assert.equal((await new Par2Pruner().round()).final, 0)
// Past the hash by more than the merger's overlap.
await db.from('worker_cursors').where('name', 'merger').update({ at: new Date(Date.now() + 120_000) })
assert.equal((await new Par2Pruner().round()).final, 1)
assert.equal(await count(unseen), 0)
assert.isNotNull(range)
})
test('sets of one created_at, a batch each: every one looked at', async ({ assert }) => {
// One prober flush gives thousands of sets the same created_at (to the microsecond).
const flush = new Date('2026-10-15T14:38:39.883Z')
let candidate = 7000
for (const letter of ['p', 'q', 'r']) {
const key = `pr${letter}`.padEnd(40, '0')
const [row] = await db.table('releases').insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }).returning('id')
const id = Number(row.id)
const setId = letter.repeat(32)
const h = `${letter}0`.padEnd(32, '0')
await db.table('par2_files').insert({ set_id: setId, file_id: h, hash16k: h, length: 1000, name: 'f.rar' })
candidate++
await db.table('probe_candidates').insert({ id: candidate, release_id: id, kind: 'lone', message_id: `<${candidate}@x>`, file: 'x', status: 'done' })
await db.table('file_hashes').insert({ candidate_id: candidate, release_id: id, hash16k: h, size: 1000 })
await db.rawQuery(`INSERT INTO par2_sets (set_id, release_id, files, created_at) VALUES (?, ?, 1, ?::timestamptz + interval '984 microseconds')`, [setId, id, flush])
}
const round = await new Par2Pruner(1).round()
assert.equal(round.complete, 3)
assert.equal((await db.from('par2_files').whereIn('set_id', ['p'.repeat(32), 'q'.repeat(32), 'r'.repeat(32)]).count('* as n'))[0].n, '0')
// The pass ended: the next one starts from the first set.
assert.equal((await db.from('worker_cursors').where('name', 'par2-prune').firstOrFail()).key, '')
// Again: the cursor is past them, nothing more.
assert.deepEqual(await new Par2Pruner().round(), { complete: 0, judged: 0, rows: 0 })
})
})
tests/functional/prober.spec.ts
-45
View File
@@ -1,45 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
import { Prober } from '#services/prober'
test.group('Prober', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('a claim takes each tier: the readable tiers also take unreadable candidates, a kind with only one of them too', async ({ assert }) => {
const [row] = await db
.table('releases')
.insert({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 })
.returning('id')
const candidate = (kind: string, unreadable: boolean, ready = true) => ({
release_id: row.id, kind, message_id: `<${kind}-${unreadable}-${ready}@x>`, file: kind, ready, unreadable,
})
// One per kind, so each tier's id range is a single id (the slices' starts aren't random then).
await db.table('probe_candidates').multiInsert([candidate('lone', true), candidate('par2', true), candidate('archive', false), candidate('7z-tail', false, false)])
const prober = new Prober() as any
await prober.claimTiers()
assert.sameMembers(prober.queue.map((c: { kind: string }) => c.kind), ['lone', 'par2', 'archive'])
const status = Object.fromEntries((await db.from('probe_candidates').select('kind', 'status')).map((c) => [c.kind, c.status]))
assert.deepEqual(status, { 'lone': 'taken', 'par2': 'taken', 'archive': 'taken', '7z-tail': 'waiting' })
})
test('a flush stores a name with a NUL in it, and one article that two releases have as a candidate', async ({ assert }) => {
const release = async (key: string) =>
Number((await db.table('releases').insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 }).returning('id'))[0].id)
const a = await release('fa')
const b = await release('fb')
const candidate = async (releaseId: number) =>
Number((await db.table('probe_candidates').insert({ release_id: releaseId, kind: 'archive', message_id: '<same@x>', file: 'x.rar', status: 'taken' }).returning('id'))[0].id)
const inspection = { yencName: 'x.rar', size: 1000, type: null, names: ['Movie\u0000.mkv'], entries: 1, passworded: false }
const prober = new Prober() as any
for (const releaseId of [a, b]) {
prober.outcomes.push({ candidate: { id: await candidate(releaseId), releaseId, kind: 'archive', messageId: '<same@x>' }, status: 'done', inspection })
}
await prober.flush()
assert.deepEqual((await db.from('probe_candidates').select('status')).map((c) => c.status), ['done', 'done'])
const cached = await db.from('probe_cache').where('message_id', '<same@x>').firstOrFail()
assert.deepEqual(cached.result.names, ['Movie.mkv'])
assert.equal((await db.from('release_inspections').count('* as n'))[0].n, '2')
})
})
tests/functional/release_category.spec.ts
-65
View File
@@ -1,65 +0,0 @@
import { test } from '@japa/runner'
import db from '@adonisjs/lucid/services/db'
import testUtils from '@adonisjs/core/services/test_utils'
const categoryOf = async (name: string) => Number((await db.rawQuery(`SELECT release_category(?) AS c`, [name])).rows[0].c)
test.group('Release categories', (group) => {
group.each.setup(() => testUtils.db().withGlobalTransaction())
test('by name: the sources and codes the first function lost or missed, XXX, audio, books, games', async ({ assert }) => {
const expected: [string, number][] = [
// Lost to knex's bindings in migration 1200 (`blu-$1ray`, `web-$2dl`, `h\.$326[45]`).
['Some.Movie.2020.BluRay.DTS-X', 2040],
['Some.Movie.2020.WEB-DL.DDP5', 2030],
['Some.Movie.2020.H.264', 2030],
['Some.Movie.2020.2160p.WEB-DL', 2045],
// TV codes it missed.
['Rizzoli & Isles S01e07[Mux XviD', 5030],
['Rush S04E01E02 WS PDTVXviD', 5030],
['Show.S01E02.1080p.WEB', 5040],
['Show.1x02.HDTV', 5030],
['Resolution 1920x1080 wallpapers', 8010],
['Studio.24.10.22.Some.Name.XXX.1080p.MP4-GRP', 6040],
['Studio.Some.Name.XXX.2160p', 6045],
['Some.Name.XXX.IMAGESET-GRP', 6060],
['Artist-Album-WEB-2026-GRP', 3010],
['Artist - Album (2020) FLAC', 3040],
['Author - Title (Audiobook)', 3030],
['Book.Title.EPUB', 7020],
['Game.Title-CODEX', 4050],
['Game.Title.NSW-VENOM', 1000],
['TsIgNBewuFbuwWy', 8010],
]
const got: [string, number][] = []
for (const [name] of expected) got.push([name, await categoryOf(name)])
assert.deepEqual(got, expected)
})
test("the name's, else the inner name's, else a given one (a spot's), else the groups'", async ({ assert }) => {
const row = (key: string, extra: Record<string, unknown>) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), source: 'stem', files: 1, groups: '["alt.binaries.misc"]', ...extra })
await db.table('releases').multiInsert([
row('a', { name: 'Show.S01E02.1080p.WEB' }),
row('b', { name: 'TsIgNBewuFbuwWy' }),
row('c', { name: 'TsIgNBewuFbuwWy', groups: '["alt.binaries.pictures.erotica.puffies"]' }),
// A spot's release of its own, inserted with the spot's category.
row('d', { name: 'Some spot title', category: 2040 }),
])
const byName = async () => Object.fromEntries((await db.from('releases').select('release_key', 'category')).map((r) => [r.release_key[0], r.category]))
assert.deepEqual(await byName(), { a: 5040, b: 8010, c: 6060, d: 2040 })
// The prober finds an inner name; a spot links a release (the spot inspector sets a Misc one's category).
await db.from('releases').where('release_key', 'b'.padEnd(40, '0')).update({ inner_name: 'Some.Movie.2020.1080p.BluRay.x264-GRP.mkv' })
await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ category: 5030 })
await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ inner_name: 'nothing.bin' })
assert.deepEqual(await byName(), { a: 5040, b: 2040, c: 5030, d: 2040 })
})
test("a spot's category in Newznab's", async ({ assert }) => {
const { rows } = await db.rawQuery(
`SELECT spot_category(0, '{"a":[9],"z":[0]}') AS movie_hd, spot_category(0, '{"a":[15],"z":[1]}') AS series_uhd, spot_category(0, '{"a":[0],"z":[3]}') AS erotica,
spot_category(0, '{"a":[5]}') AS epub, spot_category(0, '{"z":[4]}') AS picture, spot_category(1, '{"a":[8]}') AS flac, spot_category(1, '{"a":[0],"z":[3]}') AS audiobook,
spot_category(2, '{"a":[0]}') AS pc_game, spot_category(2, '{"a":[12]}') AS console, spot_category(3, '{}') AS app`
)
assert.deepEqual(rows[0], { movie_hd: 2040, series_uhd: 5045, erotica: 6070, epub: 7020, picture: null, flac: 3040, audiobook: 3030, pc_game: 4050, console: 1000, app: 4000 })
})
})
Loaded 100 of 115 files, more files were not shown because too many files have changed in this diff. Show more