fix(deploy): pull images explicitly, and detect api/web drift by commit

The first successful galactus deploy came up all-green while the web tier was
running a build from two commits earlier. The registry held web:latest from
3ff56e6; the host still had a web:latest cached from 4ee7ec7; the deploy
reported success and served the old one. The API was only current because it
had been pulled by hand during earlier debugging.

Two independent failures, both fixed here.

1. Images are not pulled. The deploy action's `pull: true` does not reliably
   refresh an already-cached moving tag on a standalone endpoint. Added a
   Pull images step (deploy/scripts/pull-images.mjs) that pulls each image
   through Portainer's Docker API with registry credentials and fails the
   deploy if a pull fails — note the endpoint answers 200 even when the pull
   errored, so the stream body has to be inspected, not just the status.

2. The drift check could not see it. Both the verify step and the web footer
   compared APP_VERSION, but on a branch build BOTH tiers report "master", so
   equality proved nothing. They now compare gitSha, which is the only field
   that differs between two builds of the same branch. api and web come from
   one matrix run, so a difference can only mean an image was not replaced.

   This needed a /version on the web tier too — previously its build identity
   was only readable by scraping window.__APP_BUILD__ out of the HTML.

pull-images.mjs builds the X-Registry-Auth header as URL-safe base64 WITH
padding: Node's "base64url" omits the padding and Portainer's Go decoder
rejects it with "Illegal base64 data at input byte N".

Verified against galactus: pulls both images, and exits non-zero on a
nonexistent tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitea/workflows/deploy-galactus.yml

@@ -212,6 +212,23 @@ jobs:
             exit 1
           fi
 
+      # --- make sure the host actually has the images ------------------------
+      # The deploy action's `pull: true` does not reliably refresh an already
+      # cached moving tag. Pull explicitly, or a "successful" deploy can leave
+      # the host serving an older build of the same tag.
+      - name: Pull images
+        env:
+          PORTAINER_URL: ${{ secrets.PORTAINER_URL_GALACTUS }}
+          PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY_GALACTUS }}
+          PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID_GALACTUS }}
+          REGISTRY: ${{ env.REGISTRY }}
+          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+          IMAGES: ${{ github.repository_owner }}/jorgecuadros-api,${{ github.repository_owner }}/jorgecuadros-web
+          TAG: ${{ github.event.inputs.tag }}
+          NODE_TLS_REJECT_UNAUTHORIZED: "0"
+        run: node deploy/scripts/pull-images.mjs
+
       # --- always: the app (web + api) -------------------------------------
       - name: Deploy app stack
         uses: cssnr/portainer-stack-deploy-action@v1
@@ -243,31 +260,53 @@ jobs:
       - name: Verify running version
         env:
           API_ORIGIN: ${{ secrets.APP_API_ORIGIN_GALACTUS }}
+          WEB_ORIGIN: ${{ secrets.APP_WEB_ORIGIN_GALACTUS }}
           WANT: ${{ github.event.inputs.tag }}
-        # The stack file naming a tag is not proof the container is running it —
+        # A stack naming a tag is not proof the containers run it. Ask BOTH
-        # a skipped pull or a cached layer can leave the old code up. Ask it.
+        # tiers what they are, and require them to be the same commit: api and
+        # web are built from one matrix run, so a difference can only mean one
+        # of them did not actually get replaced.
         run: |
           set -e
           apk add --no-cache curl >/dev/null
+          fetch_version() {
             for i in $(seq 1 30); do
-            if curl -fsS "$API_ORIGIN/version" > /tmp/version.json; then break; fi
+              if curl -fsS "$1/version" > "$2"; then return 0; fi
-            echo "waiting for API ($i/30)..."
+              echo "waiting for $1 ($i/30)..."
               sleep 5
             done
-          cat /tmp/version.json
+            echo "::error::$1/version never answered"
-          GOT=$(node -e 'console.log(require("/tmp/version.json").version)')
+            return 1
-          # Only a semver dispatch is directly comparable: metadata-action's
+          }
-          # {{version}} turns tag v1.2.3 into image 1.2.3, while `latest` and
+          fetch_version "$API_ORIGIN" /tmp/api.json
-          # `sha-*` report the branch or short sha instead.
+          fetch_version "$WEB_ORIGIN" /tmp/web.json
-          case "$WANT" in
+          cat /tmp/api.json; echo; cat /tmp/web.json; echo
-            [0-9]*.[0-9]*.[0-9]*)
+
-              if [ "$GOT" != "$WANT" ]; then
+          API_SHA=$(node -e 'console.log(require("/tmp/api.json").gitSha)')
-                echo "::error::deployed $WANT but the API reports $GOT"
+          WEB_SHA=$(node -e 'console.log(require("/tmp/web.json").gitSha)')
+          API_VER=$(node -e 'console.log(require("/tmp/api.json").version)')
+
+          # Compare the COMMIT, not the version string: on a branch build both
+          # tiers report "master", so version equality proves nothing.
+          if [ "$API_SHA" != "$WEB_SHA" ]; then
+            echo "::error::api and web are different builds — api $API_SHA, web $WEB_SHA"
+            echo "::error::one of the images was not replaced; check the Pull images step"
             exit 1
           fi
-              echo "verified: API is running $GOT"
+          echo "api and web agree: $API_SHA"
+
+          # A semver dispatch is additionally comparable to the tag itself:
+          # metadata-action's {{version}} turns tag v1.2.3 into image 1.2.3,
+          # while `latest` and `sha-*` report the branch or short sha instead.
+          case "$WANT" in
+            [0-9]*.[0-9]*.[0-9]*)
+              if [ "$API_VER" != "$WANT" ]; then
+                echo "::error::deployed $WANT but the API reports $API_VER"
+                exit 1
+              fi
+              echo "verified: running $API_VER"
               ;;
             *)
-              echo "dispatched '$WANT'; API reports '$GOT' (not directly comparable)"
+              echo "dispatched '$WANT'; tiers report '$API_VER' (not directly comparable)"
               ;;
           esac

.gitea/workflows/deploy.yml

@@ -221,6 +221,23 @@ jobs:
             exit 1
           fi
 
+      # --- make sure the host actually has the images ------------------------
+      # The deploy action's `pull: true` does not reliably refresh an already
+      # cached moving tag; without this a "successful" deploy can leave the host
+      # serving an older build of the same tag.
+      - name: Pull images
+        env:
+          PORTAINER_URL: ${{ secrets.PORTAINER_URL }}
+          PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY }}
+          PORTAINER_ENDPOINT_ID: ${{ secrets.PORTAINER_ENDPOINT_ID }}
+          REGISTRY: ${{ env.REGISTRY }}
+          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+          IMAGES: ${{ github.repository_owner }}/jorgecuadros-api,${{ github.repository_owner }}/jorgecuadros-web
+          TAG: ${{ github.event.inputs.tag }}
+          NODE_TLS_REJECT_UNAUTHORIZED: "0"
+        run: node deploy/scripts/pull-images.mjs
+
       # --- always: the app (web + api) -------------------------------------
       - name: Deploy app stack
         uses: cssnr/portainer-stack-deploy-action@v1
@@ -253,29 +270,46 @@ jobs:
       - name: Verify running version
         env:
           API_ORIGIN: ${{ secrets.APP_API_ORIGIN }}
+          WEB_ORIGIN: ${{ secrets.APP_WEB_ORIGIN }}
           WANT: ${{ github.event.inputs.tag }}
         run: |
           set -e
           apk add --no-cache curl >/dev/null
+          fetch_version() {
             for i in $(seq 1 30); do
-            if curl -fsS "$API_ORIGIN/version" > /tmp/version.json; then break; fi
+              if curl -fsS "$1/version" > "$2"; then return 0; fi
-            echo "waiting for API ($i/30)..."
+              echo "waiting for $1 ($i/30)..."
               sleep 5
             done
-          cat /tmp/version.json
+            echo "::error::$1/version never answered"
-          GOT=$(node -e 'console.log(require("/tmp/version.json").version)')
+            return 1
-          # Only a semver dispatch is directly comparable: metadata-action's
+          }
-          # {{version}} turns tag v1.2.3 into image 1.2.3, while `latest` and
+          fetch_version "$API_ORIGIN" /tmp/api.json
-          # `sha-*` report the branch or short sha instead.
+          fetch_version "$WEB_ORIGIN" /tmp/web.json
-          case "$WANT" in
+          cat /tmp/api.json; echo; cat /tmp/web.json; echo
-            [0-9]*.[0-9]*.[0-9]*)
+
-              if [ "$GOT" != "$WANT" ]; then
+          API_SHA=$(node -e 'console.log(require("/tmp/api.json").gitSha)')
-                echo "::error::deployed $WANT but the API reports $GOT"
+          WEB_SHA=$(node -e 'console.log(require("/tmp/web.json").gitSha)')
+          API_VER=$(node -e 'console.log(require("/tmp/api.json").version)')
+
+          # Compare the COMMIT, not the version string: on a branch build both
+          # tiers report "master", so version equality proves nothing.
+          if [ "$API_SHA" != "$WEB_SHA" ]; then
+            echo "::error::api and web are different builds — api $API_SHA, web $WEB_SHA"
+            echo "::error::one of the images was not replaced; check the Pull images step"
             exit 1
           fi
-              echo "verified: API is running $GOT"
+          echo "api and web agree: $API_SHA"
+
+          case "$WANT" in
+            [0-9]*.[0-9]*.[0-9]*)
+              if [ "$API_VER" != "$WANT" ]; then
+                echo "::error::deployed $WANT but the API reports $API_VER"
+                exit 1
+              fi
+              echo "verified: running $API_VER"
               ;;
             *)
-              echo "dispatched '$WANT'; API reports '$GOT' (not directly comparable)"
+              echo "dispatched '$WANT'; tiers report '$API_VER' (not directly comparable)"
               ;;
           esac

apps/web/src/app/version/route.ts

@@ -0,0 +1,18 @@
+import { NextResponse } from "next/server";
+import { readBuildInfoFromEnv } from "@/lib/build-info";
+
+// Read per request, never prerendered — the whole point is to report what THIS
+// running container is, and a baked answer would defeat that.
+export const dynamic = "force-dynamic";
+
+/**
+ * The web tier's counterpart to the API's GET /version.
+ *
+ * Without this, the only way to see what the web container is running was to
+ * scrape window.__APP_BUILD__ out of the HTML. The deploy workflow compares the
+ * two tiers' gitSha to catch a half-applied release, so it needs a stable,
+ * parseable answer from both sides.
+ */
+export function GET() {
+  return NextResponse.json({ service: "web", ...readBuildInfoFromEnv() });
+}

apps/web/src/components/AppShell.tsx

@@ -3,7 +3,7 @@
 import { useEffect, useRef, useState, type ReactNode } from "react";
 import { usePathname, useRouter } from "next/navigation";
 import Link from "next/link";
-import { getApiVersion, logout, me, updateUiScale } from "@/lib/api";
+import { getApiVersion, logout, me, updateUiScale, type ServiceVersion } from "@/lib/api";
 import { shortSha, webBuildInfo } from "@/lib/build-info";
 import { AuthContext, can } from "@/lib/abilities";
 import { ROLE_LABEL } from "@/lib/labels";
@@ -190,13 +190,13 @@ function NavMenu({
  */
 function BuildFooter() {
   const web = webBuildInfo();
-  const [api, setApi] = useState<string | null>(null);
+  const [api, setApi] = useState<ServiceVersion | null>(null);
 
   useEffect(() => {
     let alive = true;
     getApiVersion()
       .then((v) => {
-        if (alive) setApi(v.version);
+        if (alive) setApi(v);
       })
       .catch(() => {
         // The shell already redirects to /login when the API is unreachable;
@@ -207,7 +207,12 @@ function BuildFooter() {
     };
   }, []);
 
-  const mismatch = api !== null && api !== web.version;
+  // Compare the COMMIT, not the version string. On a branch build both tiers
+  // report APP_VERSION "master", so comparing versions cannot see drift — which
+  // is exactly how a stale web image once sat next to a current API with this
+  // footer showing nothing wrong. The sha is the only field that actually
+  // differs between two builds of the same branch.
+  const mismatch = api !== null && api.gitSha !== web.gitSha;
 
   return (
     <footer className="shell-footer">
@@ -216,8 +221,8 @@ function BuildFooter() {
         className="shell-footer-build"
         title={`web ${web.version} (${shortSha(web.gitSha)}) — ${web.buildDate}`}
       >
-        v{web.version}
+        v{web.version} · {shortSha(web.gitSha)}
-        {api !== null && (mismatch ? ` · API v${api}` : "")}
+        {mismatch && api ? ` · API ${shortSha(api.gitSha)}` : ""}
       </span>
       {mismatch && (
         <span className="shell-footer-warn" role="status">

deploy/scripts/pull-images.mjs

@@ -0,0 +1,104 @@
+#!/usr/bin/env node
+/**
+ * Pull the api + web images onto the target host before the stack is applied.
+ *
+ * This exists because the deploy action's `pull: true` does NOT reliably
+ * refresh an already-cached tag on a standalone endpoint. Observed on galactus
+ * 2026-07-30: the registry held web:latest built from 3ff56e6, the host still
+ * had a web:latest cached from an earlier commit, the deploy reported success,
+ * and the running container served the OLD build. A moving tag like `latest`
+ * makes this silent — the stack file names the same string either way, so
+ * nothing downstream notices.
+ *
+ * Pulling explicitly, and failing the deploy if a pull fails, makes "the image
+ * the host runs" a thing the workflow controls rather than hopes for.
+ *
+ * Required env:
+ *   PORTAINER_URL, PORTAINER_API_KEY, PORTAINER_ENDPOINT_ID
+ *   REGISTRY, REGISTRY_USERNAME, REGISTRY_PASSWORD
+ *   IMAGES   comma-separated repositories, e.g. "owner/api,owner/web"
+ *   TAG      the tag to pull
+ *
+ * TLS: Portainer here is self-signed; the caller sets
+ * NODE_TLS_REJECT_UNAUTHORIZED=0 for this step.
+ */
+
+function required(name) {
+  const v = process.env[name];
+  if (!v) {
+    console.error(`missing required env: ${name}`);
+    process.exit(1);
+  }
+  return v;
+}
+
+const PORTAINER_URL = required("PORTAINER_URL").replace(/\/+$/, "");
+const API_KEY = required("PORTAINER_API_KEY");
+const ENDPOINT_ID = required("PORTAINER_ENDPOINT_ID");
+const REGISTRY = required("REGISTRY");
+const USERNAME = required("REGISTRY_USERNAME");
+const PASSWORD = required("REGISTRY_PASSWORD");
+const IMAGES = required("IMAGES").split(",").map((s) => s.trim()).filter(Boolean);
+const TAG = required("TAG");
+
+const DOCKER = `${PORTAINER_URL}/api/endpoints/${ENDPOINT_ID}/docker`;
+
+// Docker wants the credentials as base64url'd JSON in a header. Node's
+// "base64url" encoding omits the `=` padding, which Portainer's Go decoder
+// rejects outright ("Illegal base64 data at input byte N"), so build the
+// URL-safe alphabet by hand and KEEP the padding.
+const REGISTRY_AUTH = Buffer.from(
+  JSON.stringify({
+    username: USERNAME,
+    password: PASSWORD,
+    serveraddress: REGISTRY,
+  }),
+)
+  .toString("base64")
+  .replace(/\+/g, "-")
+  .replace(/\//g, "_");
+
+async function pull(repository) {
+  const image = `${REGISTRY}/${repository}`;
+  const url =
+    `${DOCKER}/images/create` +
+    `?fromImage=${encodeURIComponent(image)}&tag=${encodeURIComponent(TAG)}`;
+
+  const res = await fetch(url, {
+    method: "POST",
+    headers: { "X-API-Key": API_KEY, "X-Registry-Auth": REGISTRY_AUTH },
+  });
+  const body = await res.text();
+  if (!res.ok) {
+    throw new Error(`pull ${image}:${TAG} -> HTTP ${res.status} ${body.slice(0, 300)}`);
+  }
+  // The endpoint streams newline-delimited JSON and answers 200 even when the
+  // pull itself failed — the failure only shows up as an {"error": ...} object
+  // in the stream, so the status code alone proves nothing.
+  const lines = body.split("\n").filter((l) => l.trim());
+  for (const line of lines) {
+    let obj;
+    try {
+      obj = JSON.parse(line);
+    } catch {
+      continue;
+    }
+    if (obj.error) {
+      throw new Error(`pull ${image}:${TAG} failed: ${obj.error}`);
+    }
+  }
+  const last = lines.length ? JSON.parse(lines[lines.length - 1]) : {};
+  console.log(`${image}:${TAG} — ${last.status ?? "pulled"}`);
+}
+
+async function main() {
+  for (const repository of IMAGES) {
+    await pull(repository);
+  }
+  console.log("all images pulled");
+}
+
+main().catch((err) => {
+  console.error(`image pull FAILED: ${err.message}`);
+  process.exit(1);
+});