Daan MeijerandClaude Opus 5.5
3d0dc6ca1f
The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.
Also: corpus_sweep.py counts the marked ranges without psql's command tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
55 lines
2.1 KiB
Bash
Executable File
55 lines
2.1 KiB
Bash
Executable File
#!/bin/sh
|
|
# Deploys voyager 2 (branch rebuild) on homelab at one commit. Installed as
|
|
# ~/bin/deploy-usenet-rebuild (by hand: it doesn't update itself). Gitea CI
|
|
# runs it through the forced SSH command of the deploy key, as
|
|
# "deploy-rebuild <sha>" (~/bin/deploy-usenet-indexer passes that on); by
|
|
# hand: deploy-usenet-rebuild <sha>. The code comes from the Gitea repo, not
|
|
# from the CI job.
|
|
set -eu
|
|
|
|
REPO=https://gitea.exp.sforum.nl/daan/usenet-indexer.git
|
|
BRANCH=rebuild
|
|
ROOT=/mnt/bulk/usenet-rebuild
|
|
CACHE="$ROOT/.cache"
|
|
GIT="$CACHE/repo.git"
|
|
|
|
SHA="${1:-}"
|
|
case "$SHA" in
|
|
*[!0-9a-f]* | "") echo "usage: deploy-usenet-rebuild <40-character commit sha>" >&2; exit 2 ;;
|
|
esac
|
|
[ "${#SHA}" -eq 40 ] || { echo "usage: deploy-usenet-rebuild <40-character commit sha>" >&2; exit 2; }
|
|
|
|
mkdir -p "$CACHE"
|
|
# One deploy at a time; a second one waits.
|
|
exec 9>"$CACHE/deploy.lock"
|
|
flock 9
|
|
|
|
echo "== fetching $SHA"
|
|
[ -d "$GIT" ] || git clone --quiet --bare "$REPO" "$GIT"
|
|
git -C "$GIT" fetch --quiet --prune "$REPO" '+refs/heads/*:refs/heads/*'
|
|
git -C "$GIT" cat-file -e "$SHA^{commit}"
|
|
git -C "$GIT" merge-base --is-ancestor "$SHA" "refs/heads/$BRANCH" || { echo "$SHA is not on $BRANCH" >&2; exit 2; }
|
|
|
|
STAGE=$(mktemp -d)
|
|
trap 'rm -rf "$STAGE"' EXIT
|
|
git -C "$GIT" archive "$SHA" | tar -x -C "$STAGE"
|
|
rsync -a --delete "$STAGE"/ "$ROOT/src/"
|
|
install -m 644 "$ROOT/src/deploy/compose.yaml" "$ROOT/compose.yaml"
|
|
|
|
cd "$ROOT"
|
|
echo "== building images"
|
|
docker compose build migrate edge
|
|
echo "== running migrations"
|
|
docker compose run --rm --no-deps migrate
|
|
# Only the app services, never postgres. The worker first: one holding a
|
|
# stream keeps the node API from exiting.
|
|
echo "== recreating app services"
|
|
docker compose stop edge
|
|
docker compose up -d --no-deps node-api web coordinator prober merger
|
|
docker compose up -d --no-deps edge
|
|
echo "$SHA $(date -Is)" >> "$ROOT/deployed.log"
|
|
# The images this replaced (321 MB each; the root filesystem is the NVMe, with a 10 GB floor).
|
|
docker image prune -f --filter label=com.docker.compose.project=usenet-rebuild >/dev/null
|
|
docker compose ps --format '{{.Service}} {{.Status}}' node-api web coordinator prober merger edge
|
|
echo "== deployed $SHA"
|