Files
jorgecuadros-platform/.gitea/workflows/deploy-galactus.yml
T
rmancinasandClaude Opus 5 fa38ff581e fix(deploy): reclaim superseded images so the host stops filling up
Every build pushes a new api + web image and every deploy pulls both onto
galactus, but nothing ever removed the pair they replaced. That reached 63
images / 83.85GB, of which 79.26GB was unused, and filled the 98GB root
filesystem to 100%.

The symptom was not a disk alert. It was "re-import is broken": the
Operaciones REIMPORT job leads with a mysqldump safety backup, that write
had nowhere to go, and PIPEFAIL took the job down before it touched the
database. Nothing in the ops_jobs log pointed at the disk.

Prune runs last, after the verify step, because Docker refuses to prune an
image that a container references — the running stack is what protects the
release just shipped. `until` adds a grace window on top so a rollback
dispatch stays a stack swap instead of a re-pull, but note it filters on
image creation time rather than pull time, so it does NOT cover rolling
back to an old tag; the running-container rule is what does.

continue-on-error: housekeeping that fails leaves a fat host, not a broken
release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-19 20:32:52 -07:00

414 lines
21 KiB
YAML

# Manual PROD deploy to galactus — the office server, Portainer endpoint 3.
#
# galactus is STANDALONE Docker (`swarm: inactive`), so this workflow applies
# the compose files under deploy/galactus/, NOT the Swarm files in deploy/.
# .gitea/workflows/deploy.yml is the cubex/Swarm equivalent; the two are kept
# separate on purpose because plain compose silently ignores Swarm's `deploy:`
# keys rather than failing on them.
#
# This does NOT build. build.yml already built + pushed both images from one
# matrix run, so api and web at the same tag are always in step.
#
# Order of operations, and why:
# 1. db + minio (scope=full only) — the API depends on both.
# 2. pre-migrate backup dumped INSIDE the still-running OLD api container,
# so the file lands in the volume the Operaciones
# restore screen reads. Must precede the migration.
# 3. prisma migrate deploy forward-only. Prisma has no down-migrations; see
# docs/DEPLOY_AND_MIGRATIONS.md — expand/contract is
# the rule, the backup is the emergency lever.
# Done HERE so the schema moves while the OLD code is
# still serving. The api container ALSO migrates at
# start (docker/api-entrypoint.sh); `migrate deploy`
# is idempotent, so the second run is a no-op and the
# container is what covers a restart that never goes
# through this workflow at all.
# 4. app (api + web) the new images.
# 5. verify ask the running API what it actually is.
# 6. prune images reclaim the superseded api/web images. LAST, and
# after verify: Docker will not prune an image a
# container references, so the running stack is what
# protects the release we just shipped.
#
# Rollback = re-dispatch with an older `tag`. That rolls back CODE only; the
# schema stays forward. This is exactly why every schema change must be
# backward-compatible with the previous release.
#
# Prereqs (once):
# - Gitea repo secrets, galactus-specific (suffix _GALACTUS so the cubex
# secrets keep working side by side):
# PORTAINER_URL_GALACTUS https://100.103.77.46:9443
# PORTAINER_API_KEY_GALACTUS Portainer access token for galactus
# PORTAINER_ENDPOINT_ID_GALACTUS 3
# PORTAINER_APP_STACK_NAME_GALACTUS e.g. jorgecuadros-prod-app
# PORTAINER_DB_STACK_NAME_GALACTUS e.g. jorgecuadros-prod-db
# PORTAINER_MINIO_STACK_NAME_GALACTUS e.g. jorgecuadros-prod-minio
# DATABASE_URL_GALACTUS mysql://jorgecuadros:<pass>@<galactus>:3306/jorgecuadros
# APP_API_ORIGIN_GALACTUS browser-facing API URL
# APP_WEB_ORIGIN_GALACTUS web public origin (API CORS)
# APP_S3_ENDPOINT_GALACTUS server-side minio URL
# SESSION_SECRET_GALACTUS 64-hex (openssl rand -hex 32)
# MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
# MYSQL_PASSWORD / MYSQL_ROOT_PASSWORD
# Optional — outbound mail. Not needed to deploy; needed for
# /notificaciones to send anything at all (the image sets
# NODE_ENV=production, which disables MailService's stdout fallback, so
# a blank config fails every send loudly):
# SES_REGION e.g. us-west-2
# SES_FROM a VERIFIED SES sending identity
# SES_FROM_NAME display name, optional
# SES_ACCESS_KEY / SES_SECRET_KEY
# SES_CONFIGURATION_SET optional, for bounce/complaint events
# NOTIFICATION_ADMIN_EMAILS fallback only — the summary recipients
# are edited in the UI and stored in
# app_settings; this is what a deployment
# uses until somebody saves them there
# These are NOT galactus-specific (no _GALACTUS suffix) — one SES identity
# serves every deployment.
# - The runner (which lives on cubex) must be able to reach galactus:9443
# (Portainer). It should also reach galactus:3306 for step 3, but that is
# no longer load-bearing: dispatch with skip_migrate=true and the api
# container applies the migrations itself at start.
# - ONE-TIME, on a database that predates migration history (i.e. one built
# with `prisma db push`): baseline it before the first run, or step 3 fails
# with P3005 "database schema is not empty":
# npx prisma@5 migrate resolve --applied 0000_init \
# --schema packages/database/prisma/schema.prisma
name: Deploy to galactus
on:
workflow_dispatch:
inputs:
tag:
description: "Image tag to deploy (1.2.3 — no leading v — or sha-<short>, or latest)"
required: true
default: "latest"
scope:
description: "What to deploy"
type: choice
required: true
default: "app"
options:
- app
- full
bootstrap:
description: "First-ever deploy: allow the pre-migrate backup to be skipped when no API container exists yet"
type: boolean
required: false
default: false
skip_migrate:
description: "Skip the runner-side migrate step (safe: the api container migrates at start)"
type: boolean
required: false
default: false
env:
REGISTRY: git.mancinas.io
jobs:
deploy:
name: Deploy ${{ github.event.inputs.tag }} (${{ github.event.inputs.scope }})
runs-on: docker
container:
image: node:20-alpine
steps:
- name: Install tools
# openssl: prisma's migration engine picks its musl/openssl build at
# runtime and cannot resolve one without it.
run: apk add --no-cache openssl ca-certificates git
- uses: actions/checkout@v4
# An unset secret arrives as an empty string, and the deploy action then
# fails with "Input required and not supplied: token" — which names the
# action's input, not the secret you forgot. Check them up front and say
# exactly which ones are missing.
- name: Preflight — required secrets
env:
PORTAINER_URL_GALACTUS: ${{ secrets.PORTAINER_URL_GALACTUS }}
PORTAINER_API_KEY_GALACTUS: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
PORTAINER_ENDPOINT_ID_GALACTUS: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
PORTAINER_APP_STACK_NAME_GALACTUS: ${{ secrets.PORTAINER_APP_STACK_NAME_GALACTUS }}
PORTAINER_DB_STACK_NAME_GALACTUS: ${{ secrets.PORTAINER_DB_STACK_NAME_GALACTUS }}
PORTAINER_MINIO_STACK_NAME_GALACTUS: ${{ secrets.PORTAINER_MINIO_STACK_NAME_GALACTUS }}
DATABASE_URL_GALACTUS: ${{ secrets.DATABASE_URL_GALACTUS }}
SESSION_SECRET_GALACTUS: ${{ secrets.SESSION_SECRET_GALACTUS }}
APP_API_ORIGIN_GALACTUS: ${{ secrets.APP_API_ORIGIN_GALACTUS }}
APP_WEB_ORIGIN_GALACTUS: ${{ secrets.APP_WEB_ORIGIN_GALACTUS }}
APP_S3_ENDPOINT_GALACTUS: ${{ secrets.APP_S3_ENDPOINT_GALACTUS }}
MINIO_ROOT_USER: ${{ secrets.MINIO_ROOT_USER }}
MINIO_ROOT_PASSWORD: ${{ secrets.MINIO_ROOT_PASSWORD }}
MYSQL_PASSWORD: ${{ secrets.MYSQL_PASSWORD }}
MYSQL_ROOT_PASSWORD: ${{ secrets.MYSQL_ROOT_PASSWORD }}
# Not required — the app boots fine without mail. Warned about below,
# because the failure mode is remote: everything looks healthy until
# someone clicks "Ejecutar" and every send fails.
SES_REGION: ${{ secrets.SES_REGION }}
SES_FROM: ${{ secrets.SES_FROM }}
SES_ACCESS_KEY: ${{ secrets.SES_ACCESS_KEY }}
SES_SECRET_KEY: ${{ secrets.SES_SECRET_KEY }}
SCOPE: ${{ github.event.inputs.scope }}
run: |
REQUIRED="PORTAINER_URL_GALACTUS PORTAINER_API_KEY_GALACTUS
PORTAINER_ENDPOINT_ID_GALACTUS PORTAINER_APP_STACK_NAME_GALACTUS
DATABASE_URL_GALACTUS SESSION_SECRET_GALACTUS
APP_API_ORIGIN_GALACTUS APP_WEB_ORIGIN_GALACTUS
APP_S3_ENDPOINT_GALACTUS MINIO_ROOT_USER MINIO_ROOT_PASSWORD
MYSQL_ROOT_PASSWORD"
if [ "$SCOPE" = "full" ]; then
REQUIRED="$REQUIRED PORTAINER_DB_STACK_NAME_GALACTUS
PORTAINER_MINIO_STACK_NAME_GALACTUS MYSQL_PASSWORD"
fi
missing=""
for name in $REQUIRED; do
eval "value=\${$name}"
[ -z "$value" ] && missing="$missing $name"
done
if [ -n "$missing" ]; then
echo "::error::missing repo secrets:$missing"
echo "::error::set them under Settings > Actions > Secrets"
exit 1
fi
echo "all required secrets present for scope=$SCOPE"
# Mail is optional to deploy but not optional to work. Say so loudly
# rather than letting /notificaciones fail one send at a time.
mail_missing=""
for name in SES_REGION SES_FROM SES_ACCESS_KEY SES_SECRET_KEY; do
eval "value=\${$name}"
[ -z "$value" ] && mail_missing="$mail_missing $name"
done
if [ -n "$mail_missing" ]; then
echo "::warning::outbound mail is NOT configured, missing:$mail_missing"
echo "::warning::the deploy will succeed, but every notification and"
echo "::warning::renewal aviso will fail with 'El envío de correo no"
echo "::warning::está configurado.' See docs/MASS_EMAIL_NOTIFICATIONS.md"
fi
# --- full only: database ---------------------------------------------
- name: Deploy database stack
if: ${{ github.event.inputs.scope == 'full' }}
uses: cssnr/portainer-stack-deploy-action@v1
with:
url: ${{ secrets.PORTAINER_URL_GALACTUS }}
token: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
name: ${{ secrets.PORTAINER_DB_STACK_NAME_GALACTUS }}
file: deploy/galactus/jorgecuadros-db.compose.yml
type: file
standalone: true
endpoint: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
env_data: |
{
"MYSQL_SERVER_ID": "1",
"MYSQL_PORT": "3306",
"MYSQL_DATABASE": "jorgecuadros",
"MYSQL_USER": "jorgecuadros",
"MYSQL_PASSWORD": "${{ secrets.MYSQL_PASSWORD }}",
"MYSQL_ROOT_PASSWORD": "${{ secrets.MYSQL_ROOT_PASSWORD }}"
}
# --- full only: object storage ---------------------------------------
- name: Deploy minio stack
if: ${{ github.event.inputs.scope == 'full' }}
uses: cssnr/portainer-stack-deploy-action@v1
with:
url: ${{ secrets.PORTAINER_URL_GALACTUS }}
token: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
name: ${{ secrets.PORTAINER_MINIO_STACK_NAME_GALACTUS }}
file: deploy/galactus/jorgecuadros-minio.compose.yml
type: file
standalone: true
endpoint: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
env_data: |
{
"MINIO_API_PORT": "9000",
"MINIO_CONSOLE_PORT": "9001",
"MINIO_ROOT_USER": "${{ secrets.MINIO_ROOT_USER }}",
"MINIO_ROOT_PASSWORD": "${{ secrets.MINIO_ROOT_PASSWORD }}"
}
# --- restore point, taken while the OLD api container is still up ------
- name: Pre-migrate backup
env:
PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
DATABASE_URL: ${{ secrets.DATABASE_URL_GALACTUS }}
# The dump runs as root: --single-transaction issues FLUSH TABLES,
# which needs the global RELOAD privilege the application user
# deliberately does not have.
MYSQL_ROOT_PASSWORD: ${{ secrets.MYSQL_ROOT_PASSWORD }}
BACKUP_TAG: ${{ github.event.inputs.tag }}
ALLOW_MISSING_CONTAINER: ${{ github.event.inputs.bootstrap }}
# Portainer serves a self-signed certificate. Scoped to this step
# only, which does nothing but talk to Portainer.
NODE_TLS_REJECT_UNAUTHORIZED: "0"
run: node deploy/scripts/pre-migrate-backup.mjs
# --- schema, forward-only ---------------------------------------------
# Belt to the container's braces: this runs while the OLD code is still
# serving, which is the order expand/contract is designed around. The
# api container repeats it at start for the paths this step cannot
# reach (skip_migrate, a host reboot, a stack re-applied by hand).
- name: Apply database migrations
if: ${{ github.event.inputs.skip_migrate != 'true' }}
env:
DATABASE_URL: ${{ secrets.DATABASE_URL_GALACTUS }}
run: |
set -e
SCHEMA=packages/database/prisma/schema.prisma
npx --yes prisma@5 migrate status --schema "$SCHEMA" || true
if ! npx --yes prisma@5 migrate deploy --schema "$SCHEMA"; then
echo "::error::migrate deploy failed. If this is P3005 (schema not empty),"
echo "::error::the database predates migration history — baseline it once with:"
echo "::error:: npx prisma@5 migrate resolve --applied 0000_init --schema $SCHEMA"
exit 1
fi
# --- make sure the host actually has the images ------------------------
# The deploy action's `pull: true` does not reliably refresh an already
# cached moving tag. Pull explicitly, or a "successful" deploy can leave
# the host serving an older build of the same tag.
- name: Pull images
env:
PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
REGISTRY: ${{ env.REGISTRY }}
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
IMAGES: ${{ github.repository_owner }}/jorgecuadros-api,${{ github.repository_owner }}/jorgecuadros-web
TAG: ${{ github.event.inputs.tag }}
NODE_TLS_REJECT_UNAUTHORIZED: "0"
run: node deploy/scripts/pull-images.mjs
# --- always: the app (web + api) -------------------------------------
- name: Deploy app stack
uses: cssnr/portainer-stack-deploy-action@v1
with:
url: ${{ secrets.PORTAINER_URL_GALACTUS }}
token: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
name: ${{ secrets.PORTAINER_APP_STACK_NAME_GALACTUS }}
file: deploy/galactus/jorgecuadros-app.compose.yml
type: file
standalone: true
pull: true
endpoint: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
# NOTE: the block below is parsed as JSON — no comments inside it.
#
# API_ORIGIN is deliberately absent. The browser derives the API origin
# from the page it loaded (apps/web/src/lib/api.ts), so the deployment
# survives the box moving between the tailnet, the office LAN and a
# demo domain. Setting it here would pin it again and re-break an https
# front door with mixed active content. APP_API_ORIGIN_GALACTUS lives
# on only as the URL the verify step probes.
env_data: |
{
"APP_TAG": "${{ github.event.inputs.tag }}",
"API_PORT": "3001",
"WEB_PORT": "3000",
"S3_BUCKET": "jorgecuadros-documents",
"WEB_ORIGIN": "${{ secrets.APP_WEB_ORIGIN_GALACTUS }}",
"S3_ENDPOINT": "${{ secrets.APP_S3_ENDPOINT_GALACTUS }}",
"DATABASE_URL": "${{ secrets.DATABASE_URL_GALACTUS }}",
"SESSION_SECRET": "${{ secrets.SESSION_SECRET_GALACTUS }}",
"SESSION_COOKIE_SECURE": "false",
"OPS_DB_ADMIN_USER": "root",
"OPS_DB_ADMIN_PASSWORD": "${{ secrets.MYSQL_ROOT_PASSWORD }}",
"REPLICA_DB_HOST": "${{ secrets.REPLICA_DB_HOST }}",
"REPLICA_DB_USER": "${{ secrets.REPLICA_DB_USER }}",
"REPLICA_DB_PASS": "${{ secrets.REPLICA_DB_PASS }}",
"MINIO_ROOT_USER": "${{ secrets.MINIO_ROOT_USER }}",
"MINIO_ROOT_PASSWORD": "${{ secrets.MINIO_ROOT_PASSWORD }}",
"SES_REGION": "${{ secrets.SES_REGION }}",
"SES_FROM": "${{ secrets.SES_FROM }}",
"SES_FROM_NAME": "${{ secrets.SES_FROM_NAME }}",
"SES_ACCESS_KEY": "${{ secrets.SES_ACCESS_KEY }}",
"SES_SECRET_KEY": "${{ secrets.SES_SECRET_KEY }}",
"SES_CONFIGURATION_SET": "${{ secrets.SES_CONFIGURATION_SET }}",
"NOTIFICATION_ADMIN_EMAILS": "${{ secrets.NOTIFICATION_ADMIN_EMAILS }}"
}
# --- prove it ----------------------------------------------------------
- name: Verify running version
env:
API_ORIGIN: ${{ secrets.APP_API_ORIGIN_GALACTUS }}
WEB_ORIGIN: ${{ secrets.APP_WEB_ORIGIN_GALACTUS }}
WANT: ${{ github.event.inputs.tag }}
# A stack naming a tag is not proof the containers run it. Ask BOTH
# tiers what they are, and require them to be the same commit: api and
# web are built from one matrix run, so a difference can only mean one
# of them did not actually get replaced.
run: |
set -e
apk add --no-cache curl >/dev/null
# These secrets are CORS origin LISTS as far as the app is concerned
# (WEB_ORIGIN is comma-separated so one deployment can be reached by
# LAN IP, tailnet name and demo domain at once). A list is not a URL,
# so probe the FIRST entry — keep the runner-reachable origin first.
API_ORIGIN=${API_ORIGIN%%,*}
WEB_ORIGIN=${WEB_ORIGIN%%,*}
fetch_version() {
for i in $(seq 1 30); do
if curl -fsS "$1/version" > "$2"; then return 0; fi
echo "waiting for $1 ($i/30)..."
sleep 5
done
echo "::error::$1/version never answered"
return 1
}
fetch_version "$API_ORIGIN" /tmp/api.json
fetch_version "$WEB_ORIGIN" /tmp/web.json
cat /tmp/api.json; echo; cat /tmp/web.json; echo
API_SHA=$(node -e 'console.log(require("/tmp/api.json").gitSha)')
WEB_SHA=$(node -e 'console.log(require("/tmp/web.json").gitSha)')
API_VER=$(node -e 'console.log(require("/tmp/api.json").version)')
# Compare the COMMIT, not the version string: on a branch build both
# tiers report "master", so version equality proves nothing.
if [ "$API_SHA" != "$WEB_SHA" ]; then
echo "::error::api and web are different builds — api $API_SHA, web $WEB_SHA"
echo "::error::one of the images was not replaced; check the Pull images step"
exit 1
fi
echo "api and web agree: $API_SHA"
# A semver dispatch is additionally comparable to the tag itself:
# metadata-action's {{version}} turns tag v1.2.3 into image 1.2.3,
# while `latest` and `sha-*` report the branch or short sha instead.
case "$WANT" in
[0-9]*.[0-9]*.[0-9]*)
if [ "$API_VER" != "$WANT" ]; then
echo "::error::deployed $WANT but the API reports $API_VER"
exit 1
fi
echo "verified: running $API_VER"
;;
*)
echo "dispatched '$WANT'; tiers report '$API_VER' (not directly comparable)"
;;
esac
# --- housekeeping ------------------------------------------------------
# Runs LAST, and only after the verify step proved the new containers are
# up. See deploy/scripts/prune-images.mjs: Docker refuses to prune an
# image a container references, so "the stack is running" is what makes
# the current images safe. Pruning earlier would have nothing holding
# them.
#
# continue-on-error: reclaiming disk is not what the deploy is for. A
# prune that fails leaves a fat host, not a broken release.
- name: Prune unused images
continue-on-error: true
env:
PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
# Grace window. Keeps the previous few releases on disk so a rollback
# dispatch is a stack swap instead of a re-pull.
KEEP_HOURS: "168"
NODE_TLS_REJECT_UNAUTHORIZED: "0"
run: node deploy/scripts/prune-images.mjs