fix(deploy): reclaim superseded images so the host stops filling up
Every build pushes a new api + web image and every deploy pulls both onto galactus, but nothing ever removed the pair they replaced. That reached 63 images / 83.85GB, of which 79.26GB was unused, and filled the 98GB root filesystem to 100%. The symptom was not a disk alert. It was "re-import is broken": the Operaciones REIMPORT job leads with a mysqldump safety backup, that write had nowhere to go, and PIPEFAIL took the job down before it touched the database. Nothing in the ops_jobs log pointed at the disk. Prune runs last, after the verify step, because Docker refuses to prune an image that a container references — the running stack is what protects the release just shipped. `until` adds a grace window on top so a rollback dispatch stays a stack swap instead of a re-pull, but note it filters on image creation time rather than pull time, so it does NOT cover rolling back to an old tag; the running-container rule is what does. continue-on-error: housekeeping that fails leaves a fat host, not a broken release. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,10 @@
|
||||
# through this workflow at all.
|
||||
# 4. app (api + web) the new images.
|
||||
# 5. verify ask the running API what it actually is.
|
||||
# 6. prune images reclaim the superseded api/web images. LAST, and
|
||||
# after verify: Docker will not prune an image a
|
||||
# container references, so the running stack is what
|
||||
# protects the release we just shipped.
|
||||
#
|
||||
# Rollback = re-dispatch with an older `tag`. That rolls back CODE only; the
|
||||
# schema stays forward. This is exactly why every schema change must be
|
||||
@@ -386,3 +390,24 @@ jobs:
|
||||
echo "dispatched '$WANT'; tiers report '$API_VER' (not directly comparable)"
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- housekeeping ------------------------------------------------------
|
||||
# Runs LAST, and only after the verify step proved the new containers are
|
||||
# up. See deploy/scripts/prune-images.mjs: Docker refuses to prune an
|
||||
# image a container references, so "the stack is running" is what makes
|
||||
# the current images safe. Pruning earlier would have nothing holding
|
||||
# them.
|
||||
#
|
||||
# continue-on-error: reclaiming disk is not what the deploy is for. A
|
||||
# prune that fails leaves a fat host, not a broken release.
|
||||
- name: Prune unused images
|
||||
continue-on-error: true
|
||||
env:
|
||||
PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
|
||||
PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
|
||||
PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
|
||||
# Grace window. Keeps the previous few releases on disk so a rollback
|
||||
# dispatch is a stack swap instead of a re-pull.
|
||||
KEEP_HOURS: "168"
|
||||
NODE_TLS_REJECT_UNAUTHORIZED: "0"
|
||||
run: node deploy/scripts/prune-images.mjs
|
||||
|
||||
Reference in New Issue
Block a user