Voyager 1 retired: a push to master no longer deploys

Its database was deleted on 2026-10-09; a deploy would start it on an
empty one. The deploy job is removed (tests still run), and homelab's
~/bin/deploy-usenet-indexer refuses voyager 1 deploys (voyager 2 deploys
from branch rebuild with deploy-rebuild).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitea/workflows/deploy.yaml

@@ -1,9 +1,8 @@
-# Tests every push; a push to master that passes is deployed to homelab.
+# Tests every push. Voyager 1 (this branch) is retired since 2026-10-11: its
-#
+# database was deleted on 2026-10-09, and a push to master no longer deploys.
-# Deploying: the job SSHes to homelab with DEPLOY_SSH_KEY. That key may only
+# Voyager 2 (branch rebuild) has its own workflow and deploy
-# run ~/bin/deploy-usenet-indexer (a forced command in authorized_keys),
+# (deploy/README.md there); homelab's ~/bin/deploy-usenet-indexer refuses
-# which fetches the commit from this repo, builds the image, runs the
+# voyager 1 deploys too.
-# migrations and recreates the app services (never postgres or redis).
 name: Test and deploy
 
 on:
@@ -63,19 +62,3 @@ jobs:
       - name: Test
         run: node ace test
 
-  deploy:
-    needs: test
-    if: github.ref == 'refs/heads/master'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Deploy to homelab
-        env:
-          DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
-          DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }}
-        run: |
-          install -m 700 -d ~/.ssh
-          printf '%s\n' "$DEPLOY_SSH_KEY" > ~/.ssh/deploy_key
-          chmod 600 ~/.ssh/deploy_key
-          printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > ~/.ssh/known_hosts
-          ssh -i ~/.ssh/deploy_key -o IdentitiesOnly=yes -o BatchMode=yes \
-            daan@10.206.0.175 "deploy ${{ github.sha }}"

REBUILD.md

@@ -1,125 +0,0 @@
-# Rebuild: header-first indexer (spec, draft for approval)
-
-Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
-Everything learned from articles (names from yEnc headers, par2, archive
-listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
-never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
-
-## Target
-
-7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
-`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
-main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
-dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
-≥ 25 B overview lines/day (~50 connections), so ~3 weeks.
-
-## Parts
-
-1. **Coordinator** (server). Plans each group's history as ranges of 2 M
-   articles, leases them to workers, tracks progress and retries. The
-   group list (owner rule) and the garbage signature table go with every
-   lease. Fetches XOVER for the workers (relay: workers have no NNTP
-   account; the gzip stream is passed on as is).
-
-   **Prefetch**: the coordinator fetches the next ranges ahead of the
-   workers, in chunks of 50 k lines and in order, into a spool on disk
-   (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
-   sequentially). It keeps about two ranges ahead per active lease, capped
-   by size (default 10 GB), and fetches with all its header connections all
-   the time, independent of how fast workers are. A lease goes to a range
-   that is already (partly) spooled; a worker reads its chunks from the
-   spool, waiting only for chunks still being fetched, and falls back to a
-   direct relay fetch for a range that isn't spooled. Spooled chunks are
-   deleted when their range's results are stored, and fetched again if a
-   range is leased again later.
-   Order: oldest era first (2008–2018 holds 63% of the readable files on
-   17% of the lines), then forward.
-
-2. **Edge worker** (node, TypeScript reusing `lib/`; Rust later only if
-   CPU-bound). Per lease: compressed chunks of 10–50 k lines, in order →
-   parse subjects → skip crossposts outside their owner group (of the
-   indexed groups in the article's Xref, the one with the lowest priority
-   number; see the group list) → group articles into files → files
-   into releases (`lib/release` rules) → drop garbage (random name +
-   dropped signature or tool) → set aside per-article random subjects and
-   obfuscated subjects (counted only; phase 2). Returns per batch: release
-   chunks, edge fragments, probe candidates, counters. Holds no state beyond
-   its lease.
-
-   **Probe candidates**: per release, the articles phase 2 should read, as
-   (release key, kind, message-id, file in its chunk): the first article of
-   the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
-   index (smallest `.par2`), and of a lone random-named file; for a 7z set
-   also its last volume (the listing is in the tail). At most a few per
-   release (~115 M releases → ~150–250 M articles), not one per file. Per-
-   article and obfuscated subjects get none (they need every article's body).
-
-3. **Chunk store** (bulk, append-only). One record per release per batch:
-   its files' names, sizes and segment lists, compactly encoded (template
-   codec for message-ids, zstd). Written once, never updated; a re-run of a
-   range replaces that range's chunks (no per-file dedup). ~3 TB.
-
-4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
-   name, poster, groups, posted_at, file count, size, segments, expected
-   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
-   aggregates once per batch (one upsert per release per batch), plus a
-   small name index for search. ~115 M rows, ~50–100 GB.
-
-5. **Rules and recovery**. What a worker skips isn't stored (message-ids
-   are the bulk: garbage alone would be ~2.5 TB); a skipped range is read
-   again instead (headers are cheap: ~53 B per overview line compressed).
-   - *Per range, per rule and signature*: how many articles and files the
-     rule dropped (garbage per (tool, group), per-article random subjects,
-     obfuscated subjects, and crossposts per owner group). A few rows per
-     range.
-   - *Rule versions*: every range records the version of the rule set it ran
-     with. Retracting or changing a rule makes the ranges where it matched
-     stale; the coordinator leases them again for that rule only, and the
-     worker returns just what the rule had dropped.
-   - *Sample*: ~1% of what each rule drops (by a hash of the release key) is
-     kept with full file details and probe candidates, so a rule can be
-     checked, and its losses measured in phase 2, before it is retracted.
-
-6. **Probe queue**. The candidates, in the catalog's database: one small row
-   each (release, kind, message-id, chunk reference), priority by the
-   release's state (settled first) and kind (random names before readable
-   ones). Phase 2 takes them from here; phase 1 only fills it.
-
-7. **Fragments**. Files crossing a range edge: kept by range and article
-   number; merged when the neighbouring range is done, then written as a
-   chunk. Never staged per segment.
-
-8. **Settling**. A release is settled when the ranges covering its posting
-   span are done and no chunk arrived for a day: completeness is computed
-   then (expected vs found files and segments). Phase 2 picks its work from
-   settled releases.
-
-9. **NZB and API**. NZB built from a release's chunks; a minimal Newznab
-   search (names) and get. Web UI later.
-
-## Group list
-
-Editable at any time: an added group gets its ranges planned, a removed one
-no new leases. Each group gets a priority number when it is added (the first
-list by size, later additions after it); a crosspost belongs to the indexed
-group in its Xref with the lowest number. So adding a group never takes
-articles from groups already done, and removing one re-reads only the ranges
-of other groups whose counts show crossposts skipped for it (part 5). Start:
-the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
-2 M articles; boneless alone 80 k).
-
-## Not in phase 1
-
-Inspection and archive walks, par2/SFV naming, spots, NZB imports, the
-after-probe garbage deletes, live indexing (the current indexer keeps
-running and does those meanwhile).
-
-## Decisions
-
-a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
-   month, on the HDD; files on disk only if writes still limit. **Accepted.**
-b. A separate database on the same server. **Accepted.**
-c. Group list: editable, owner by priority number (above). **Accepted.**
-d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
-e. Headers through the coordinator's relay, with prefetch (part 1).
-   **Accepted.**

commands/FilesMoveDetails.ts

@@ -2,6 +2,8 @@ import { BaseCommand } from '@adonisjs/core/ace'
 import { CommandOptions } from '@adonisjs/core/types/ace'
 import db from '@adonisjs/lucid/services/db'
 
+const BATCH = 50_000
+
 /**
  * One-off (LOG.md 2026-10-08 (14)): copies the content types and archive
  * listings still on `files` into file_details, a partition at a time.
@@ -23,13 +25,31 @@ export default class FilesMoveDetails extends BaseCommand {
     let total = 0
     for (const { relname } of partitions) {
       const started = Date.now()
-      const result = await db.rawQuery(
+      const { rows } = await db.rawQuery(`SELECT min(id) AS lo, max(id) AS hi FROM ${relname}`)
-        `INSERT INTO file_details (file_id, content_type, archive)
+      if (rows[0].lo === null) continue
-         SELECT id, content_type, archive FROM ${relname} WHERE content_type IS NOT NULL OR archive IS NOT NULL
+      let copied = 0
-         ON CONFLICT (file_id) DO NOTHING`
+      // Short statements in id order: one statement over a whole partition took 30 minutes, and the
-      )
+      // garbage filter deleted files meanwhile, which failed the foreign key (LOG.md 2026-10-08 (15)).
-      total += result.rowCount ?? 0
+      for (let lo = Number(rows[0].lo); lo <= Number(rows[0].hi); lo += BATCH) {
-      this.logger.info(`${relname}: ${result.rowCount} copied in ${Math.round((Date.now() - started) / 1000)} s.`)
+        for (let attempt = 1; ; attempt++) {
+          try {
+            const result = await db.rawQuery(
+              `INSERT INTO file_details (file_id, content_type, archive)
+               SELECT id, content_type, archive FROM ${relname}
+               WHERE id >= ? AND id < ? AND (content_type IS NOT NULL OR archive IS NOT NULL)
+               ON CONFLICT (file_id) DO NOTHING`,
+              [lo, lo + BATCH]
+            )
+            copied += result.rowCount ?? 0
+            break
+          } catch (error: any) {
+            // A file deleted while this batch ran: the batch again, without it.
+            if (error.code !== '23503' || attempt >= 5) throw error
+          }
+        }
+      }
+      total += copied
+      this.logger.info(`${relname}: ${copied} copied in ${Math.round((Date.now() - started) / 1000)} s.`)
     }
     this.logger.info(`${total} files' details copied.`)
   }