fix(deploy): reclaim superseded images so the host stops filling up

Every build pushes a new api + web image and every deploy pulls both onto
galactus, but nothing ever removed the pair they replaced. That reached 63
images / 83.85GB, of which 79.26GB was unused, and filled the 98GB root
filesystem to 100%.

The symptom was not a disk alert. It was "re-import is broken": the
Operaciones REIMPORT job leads with a mysqldump safety backup, that write
had nowhere to go, and PIPEFAIL took the job down before it touched the
database. Nothing in the ops_jobs log pointed at the disk.

Prune runs last, after the verify step, because Docker refuses to prune an
image that a container references — the running stack is what protects the
release just shipped. `until` adds a grace window on top so a rollback
dispatch stays a stack swap instead of a re-pull, but note it filters on
image creation time rather than pull time, so it does NOT cover rolling
back to an old tag; the running-container rule is what does.

continue-on-error: housekeeping that fails leaves a fat host, not a broken
release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-19 20:32:52 -07:00
co-authored by Claude Opus 5
parent c4213aa697
commit fa38ff581e
2 changed files with 131 additions and 0 deletions
+25
View File
@@ -25,6 +25,10 @@
# through this workflow at all.
# 4. app (api + web) the new images.
# 5. verify ask the running API what it actually is.
# 6. prune images reclaim the superseded api/web images. LAST, and
# after verify: Docker will not prune an image a
# container references, so the running stack is what
# protects the release we just shipped.
#
# Rollback = re-dispatch with an older `tag`. That rolls back CODE only; the
# schema stays forward. This is exactly why every schema change must be
@@ -386,3 +390,24 @@ jobs:
echo "dispatched '$WANT'; tiers report '$API_VER' (not directly comparable)"
;;
esac
# --- housekeeping ------------------------------------------------------
# Runs LAST, and only after the verify step proved the new containers are
# up. See deploy/scripts/prune-images.mjs: Docker refuses to prune an
# image a container references, so "the stack is running" is what makes
# the current images safe. Pruning earlier would have nothing holding
# them.
#
# continue-on-error: reclaiming disk is not what the deploy is for. A
# prune that fails leaves a fat host, not a broken release.
- name: Prune unused images
continue-on-error: true
env:
PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
# Grace window. Keeps the previous few releases on disk so a rollback
# dispatch is a stack swap instead of a re-pull.
KEEP_HOURS: "168"
NODE_TLS_REJECT_UNAUTHORIZED: "0"
run: node deploy/scripts/prune-images.mjs