feat(deploy): apply migrations at api container start

`prisma migrate deploy` ran in one place only: a workflow step on the Gitea
runner, which has to reach the target host's MySQL on 3306 directly. Two
paths went around it:

  - `skip_migrate=true`, the documented answer for when the runner cannot
    reach 3306, left the schema a release behind with nothing to catch it.
    The mismatch surfaced later as a column-not-found at runtime rather than
    as a failed deploy.
  - A container brought back by `restart: unless-stopped` after a host
    reboot, or a stack re-applied by hand in Portainer, never runs the
    workflow at all.

docker/api-entrypoint.sh becomes the api image's ENTRYPOINT: migrate, then
exec node. If the migration fails the container exits non-zero and the API
never listens — serving against a schema that does not match the code is
worse than being down, because the failures are partial and silent (a write
to a missing column breaks one feature while the rest looks healthy).

This does not replace the workflow step and is not a substitute for it. That
step still runs FIRST, while the old code is serving, which is the order
expand/contract migrations are designed around. `migrate deploy` is
idempotent, so on the normal path the container's run is a no-op query.

Behaviour:

  RUN_MIGRATIONS=false     skip and start anyway; plumbed through both app
                           stack files, for a schema moved by hand
  DATABASE_URL unset       refuse to start, and say why
  P1001 (unreachable)      retry, default 20 x 3s -- a cold db container, and
                           galactus's MagicDNS lookup right after a reboot
  anything else            exit at once; retrying a broken migration only
                           delays the same error. P3005 prints the
                           `migrate resolve --applied 0000_init` hint the
                           workflow step already printed.

Only P1001 retries, so a genuinely broken migration is not buried under a
minute of noise.

Both stacks are replicas: 1 and must stay so for an unrelated reason (the
servicios email sweep has no DB lock). The old comment claiming migrations
must not run per-container because "N replicas would race" is dropped: they
would not corrupt anything, since Prisma takes a database advisory lock and
the losers find nothing pending -- they would only each pay the wait.

The prisma CLI is already in the runtime layer (the image copies
/repo/node_modules wholesale), but which of the two plausible .bin paths
carries it is an implementation detail of pnpm's hoisted linker, so the
entrypoint accepts either and the Dockerfile asserts one exists at BUILD
time. A missing CLI breaks the image build, not a production boot.

Verified by running the entrypoint against stubbed prisma binaries: clean
run, P3005, P1001-to-exhaustion, P1001-then-recovery, RUN_MIGRATIONS=false,
missing DATABASE_URL, missing CLI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitea/workflows/deploy-galactus.yml

@@ -17,6 +17,12 @@
 #   3. prisma migrate deploy  forward-only. Prisma has no down-migrations; see
 #                           docs/DEPLOY_AND_MIGRATIONS.md — expand/contract is
 #                           the rule, the backup is the emergency lever.
+#                           Done HERE so the schema moves while the OLD code is
+#                           still serving. The api container ALSO migrates at
+#                           start (docker/api-entrypoint.sh); `migrate deploy`
+#                           is idempotent, so the second run is a no-op and the
+#                           container is what covers a restart that never goes
+#                           through this workflow at all.
 #   4. app (api + web)      the new images.
 #   5. verify               ask the running API what it actually is.
 #
@@ -55,10 +61,10 @@
 #                                     uses until somebody saves them there
 #    These are NOT galactus-specific (no _GALACTUS suffix) — one SES identity
 #    serves every deployment.
-#  - The runner (which lives on cubex) must be able to reach BOTH
-#    galactus:9443 (Portainer) and galactus:3306 (MySQL, for migrate deploy).
-#    If it cannot reach 3306, run the migration by hand from a host that can
-#    and dispatch with skip_migrate=true.
+#  - The runner (which lives on cubex) must be able to reach galactus:9443
+#    (Portainer). It should also reach galactus:3306 for step 3, but that is
+#    no longer load-bearing: dispatch with skip_migrate=true and the api
+#    container applies the migrations itself at start.
 #  - ONE-TIME, on a database that predates migration history (i.e. one built
 #    with `prisma db push`): baseline it before the first run, or step 3 fails
 #    with P3005 "database schema is not empty":
@@ -88,7 +94,7 @@ on:
         required: false
         default: false
       skip_migrate:
-        description: "Skip prisma migrate deploy (use when the runner cannot reach MySQL and you migrated by hand)"
+        description: "Skip the runner-side migrate step (safe: the api container migrates at start)"
         type: boolean
         required: false
         default: false
@@ -237,6 +243,10 @@ jobs:
         run: node deploy/scripts/pre-migrate-backup.mjs
 
       # --- schema, forward-only ---------------------------------------------
+      # Belt to the container's braces: this runs while the OLD code is still
+      # serving, which is the order expand/contract is designed around. The
+      # api container repeats it at start for the paths this step cannot
+      # reach (skip_migrate, a host reboot, a stack re-applied by hand).
       - name: Apply database migrations
         if: ${{ github.event.inputs.skip_migrate != 'true' }}
         env:

.gitea/workflows/deploy.yml

@@ -12,6 +12,7 @@
 # git tag v1.2.3 into image tag 1.2.3. Tag v1.2.3, dispatch 1.2.3.
 #
 # Order: db+minio (full only) -> pre-migrate backup -> prisma migrate deploy ->
+# (the api container also migrates at start; see docker/api-entrypoint.sh)
 # app -> verify the API reports the version you asked for. Rollback = dispatch
 # an older tag; that rolls back CODE only, never the schema, which is why every
 # schema change must be expand/contract. See docs/DEPLOY_AND_MIGRATIONS.md.
@@ -47,9 +48,11 @@
 #      # Database stack (full only)
 #      MYSQL_PASSWORD             app-user password (matches DATABASE_URL)
 #      MYSQL_ROOT_PASSWORD        mysql root password
-#  - the runner must reach BOTH Portainer (9443) and MySQL (3306) — the
-#    migration step connects to the database directly. If it cannot reach 3306,
-#    migrate by hand and dispatch with skip_migrate=true.
+#  - the runner must reach Portainer (9443). It should also reach MySQL (3306)
+#    for the migrate step, but that is no longer load-bearing: dispatch with
+#    skip_migrate=true and the api container applies the migrations itself at
+#    start (docker/api-entrypoint.sh). `migrate deploy` is idempotent, so the
+#    two never conflict.
 #  - ONE-TIME on a database built with `prisma db push` (i.e. every database
 #    that exists today): baseline it before the first run, or the migrate step
 #    fails with P3005 "database schema is not empty":
@@ -79,7 +82,7 @@ on:
         required: false
         default: false
       skip_migrate:
-        description: "Skip prisma migrate deploy (use when the runner cannot reach MySQL and you migrated by hand)"
+        description: "Skip the runner-side migrate step (safe: the api container migrates at start)"
         type: boolean
         required: false
         default: false

deploy/galactus/jorgecuadros-app.compose.yml

@@ -61,6 +61,11 @@ services:
       INGEST_DIR: /data/ingest
       BACKUP_DIR: /data/backups
       MIGRATION_ENV: prod
+      # The API applies pending Prisma migrations at container start, before
+      # Nest listens, and refuses to start if they fail (docker/api-entrypoint.sh).
+      # Set false ONLY when the schema is being moved by hand — the app will
+      # then boot against whatever schema it finds.
+      RUN_MIGRATIONS: ${RUN_MIGRATIONS:-true}
       # Credentials the "Operaciones" screen runs mysqldump/mysql as. NOT the
       # application user: --single-transaction needs the global RELOAD privilege
       # and the app user has only ALL ON jorgecuadros.*, so every backup, sync

deploy/jorgecuadros-app.stack.yml

@@ -41,6 +41,11 @@ services:
       INGEST_DIR: /data/ingest
       BACKUP_DIR: /data/backups
       MIGRATION_ENV: prod
+      # The API applies pending Prisma migrations at container start, before
+      # Nest listens, and refuses to start if they fail (docker/api-entrypoint.sh).
+      # Set false ONLY when the schema is being moved by hand — the app will
+      # then boot against whatever schema it finds.
+      RUN_MIGRATIONS: ${RUN_MIGRATIONS:-true}
       # Credentials the "Operaciones" screen runs mysqldump/mysql as. NOT the
       # application user: --single-transaction needs the global RELOAD privilege
       # and the app user has only ALL ON jorgecuadros.*, so every backup, sync

docker/api-entrypoint.sh

@@ -0,0 +1,102 @@
+#!/bin/sh
+# Apply pending Prisma migrations, then hand off to the API.
+#
+# WHY THE CONTAINER AND NOT THE DEPLOY WORKFLOW
+#
+# The workflow still has its own `prisma migrate deploy` step and that is not
+# redundant: it runs BEFORE the new images are pulled, i.e. while the OLD code
+# is still serving, which is the order expand/contract migrations are designed
+# around (see docs/DEPLOY_AND_MIGRATIONS.md). Doing it here as well closes the
+# gaps that step cannot:
+#
+#   - The runner has to reach MySQL directly. When it cannot, the deploy is run
+#     with `skip_migrate=true` and the schema silently does not move — the app
+#     then boots against a schema that is one release behind, which surfaces
+#     later as a column-not-found at runtime rather than as a failed deploy.
+#   - A container restarted by `restart: unless-stopped` after a host reboot,
+#     or a stack re-applied by hand in Portainer, never goes through the
+#     workflow at all.
+#
+# `migrate deploy` is idempotent, so running it in both places costs one
+# no-op query on the normal path.
+#
+# THE API DOES NOT START IF THE MIGRATION FAILS. That is deliberate: serving
+# against a schema that does not match the code is worse than being down,
+# because the failures it produces are partial and silent (a write to a column
+# that does not exist yet fails for one feature while the rest of the app looks
+# healthy). The container exits non-zero and Docker's restart policy retries.
+set -e
+
+SCHEMA=/repo/packages/database/prisma/schema.prisma
+
+log() { echo "[entrypoint] $*"; }
+
+if [ "${RUN_MIGRATIONS:-true}" != "true" ]; then
+  log "RUN_MIGRATIONS=${RUN_MIGRATIONS} — skipping migrations, starting the API"
+  exec "$@"
+fi
+
+if [ -z "${DATABASE_URL}" ]; then
+  log "DATABASE_URL is unset; cannot migrate." >&2
+  log "Set it, or set RUN_MIGRATIONS=false if you migrate out of band." >&2
+  exit 1
+fi
+
+# pnpm's hoisted linker normally puts the CLI in the root .bin, but the
+# workspace package keeps its own link too. Accept either rather than pinning
+# a layout detail of the installer — the Dockerfile asserts at build time that
+# one of these exists, so a missing CLI breaks the image build, not a deploy.
+PRISMA=""
+for candidate in /repo/node_modules/.bin/prisma \
+                 /repo/packages/database/node_modules/.bin/prisma; do
+  if [ -x "$candidate" ]; then
+    PRISMA="$candidate"
+    break
+  fi
+done
+if [ -z "$PRISMA" ]; then
+  log "prisma CLI not found in this image; cannot migrate." >&2
+  exit 1
+fi
+
+# Retry ONLY a connection failure (P1001). On a full bring-up the database
+# container can still be starting, and on galactus the API additionally has to
+# resolve the host's MagicDNS name — a lookup that is unreliable for the first
+# moments after a host reboot (see the dns block in the app compose file, and
+# docs/DEPLOY_AND_MIGRATIONS.md).
+#
+# Every other failure exits immediately. Retrying a migration that is actually
+# broken just delays the same error behind a minute of noise, and P3005 in
+# particular needs a human.
+attempt=1
+max="${MIGRATE_MAX_ATTEMPTS:-20}"
+delay="${MIGRATE_RETRY_SECONDS:-3}"
+
+while : ; do
+  log "prisma migrate deploy (attempt ${attempt}/${max})"
+  if output=$("$PRISMA" migrate deploy --schema "$SCHEMA" 2>&1); then
+    printf '%s\n' "$output"
+    log "migrations up to date"
+    break
+  fi
+  printf '%s\n' "$output" >&2
+
+  if ! printf '%s' "$output" | grep -q 'P1001'; then
+    log "migrate deploy FAILED — refusing to start the API." >&2
+    if printf '%s' "$output" | grep -q 'P3005'; then
+      log "P3005: the database has tables but no migration history. This is a" >&2
+      log "database that predates Prisma migrations. Baseline it ONCE with:" >&2
+      log "  npx prisma@5 migrate resolve --applied 0000_init --schema $SCHEMA" >&2
+    fi
+    exit 1
+  fi
+
+  if [ "$attempt" -ge "$max" ]; then
+    log "database unreachable after ${max} attempts — giving up." >&2
+    exit 1
+  fi
+  attempt=$((attempt + 1))
+  sleep "$delay"
+done
+
+exec "$@"

docker/api.Dockerfile

@@ -113,5 +113,21 @@ ENV APP_VERSION=$APP_VERSION \
     GIT_SHA=$GIT_SHA \
     BUILD_DATE=$BUILD_DATE
 
+# Pending migrations are applied at container start, before Nest listens —
+# see the header of the script for why this is done here as well as in the
+# deploy workflow. Asserted at BUILD time so a missing prisma CLI breaks the
+# image build rather than a production boot: the runtime layer copies
+# /repo/node_modules wholesale, and which of these two paths carries the bin
+# is an implementation detail of pnpm's hoisted linker.
+COPY docker/api-entrypoint.sh /usr/local/bin/api-entrypoint.sh
+RUN chmod +x /usr/local/bin/api-entrypoint.sh
+RUN for c in /repo/node_modules/.bin/prisma \
+             /repo/packages/database/node_modules/.bin/prisma; do \
+      if [ -x "$c" ]; then echo "prisma CLI found at $c"; exit 0; fi; \
+    done; \
+    echo "FATAL: prisma CLI is not in the runtime layer; api-entrypoint.sh cannot migrate" >&2; \
+    exit 1
+
 EXPOSE 3001
+ENTRYPOINT ["/usr/local/bin/api-entrypoint.sh"]
 CMD ["node", "apps/api/dist/main.js"]

docs/DEPLOY_AND_MIGRATIONS.md

@@ -56,9 +56,11 @@ workflow's last step fails if the API does not report the tag you dispatched.
    `pre-migrate-<tag>-<timestamp>.sql.gz`, which is exactly what the
    **Operaciones** admin screen lists and can restore. A dump taken on the CI
    runner would be unreachable by the only restore path the platform has.
-3. **`prisma migrate deploy`** — as a workflow *step*, never the container
-   `CMD`. If it were the CMD, N replicas would race each other applying the
-   same migration.
+3. **`prisma migrate deploy`** — as a workflow *step*, so the schema moves
+   while the OLD code is still serving, which is the order expand/contract is
+   designed around. **The api container repeats this at start** (below); the
+   command is idempotent, so on the normal path the container's run is a
+   no-op.
 4. **app** — the new api + web images.
 5. **Verify** — `GET /version` on the running API must report the dispatched
    tag.
@@ -141,6 +143,59 @@ Commit the generated `migrations/<timestamp>_add_foo/` directory. `db push` is
 now a local-scratch tool only — using it against a database with history
 desynchronises it from `_prisma_migrations`.
 
+If you hand-write a migration instead of generating one, check it against what
+Prisma would have produced before committing — a hand-written file that drifts
+from `schema.prisma` fails on the *next* deploy, not this one:
+
+```bash
+prisma migrate diff \
+  --from-schema-datamodel <schema.prisma at the previous commit> \
+  --to-schema-datamodel packages/database/prisma/schema.prisma --script
+```
+
+## Migrations also run at container start
+
+`docker/api-entrypoint.sh` is the api image's `ENTRYPOINT`. It runs
+`prisma migrate deploy` and only then `exec`s the API. **If the migration
+fails the container exits non-zero and the API never listens.**
+
+That is the point. Serving against a schema that does not match the code is
+worse than being down, because the failures are partial and silent — a write
+to a column that does not exist yet breaks one feature while the rest of the
+app looks healthy.
+
+This does not replace the workflow step, which still runs first and against
+the old code. It covers what that step cannot:
+
+- **`skip_migrate: true`.** Previously that left the schema behind with no
+  further safety net, and the mismatch surfaced later as a runtime error. Now
+  it just moves the migration into the container, so it is a safe choice when
+  the runner cannot reach MySQL.
+- **Restarts that never touch the workflow** — `restart: unless-stopped`
+  bringing the stack back after a host reboot, or a stack re-applied by hand
+  in Portainer.
+
+Behaviour worth knowing:
+
+| | |
+|---|---|
+| `RUN_MIGRATIONS=false` | Skip and start anyway. Plumbed through both app stack files. For when the schema is being moved by hand. |
+| `DATABASE_URL` unset | Refuses to start (it would have failed at Nest boot anyway, but this says why). |
+| Cannot reach the database (**P1001**) | Retries, default 20 × 3s. Covers a cold `db` container and galactus's MagicDNS lookup right after a reboot. `MIGRATE_MAX_ATTEMPTS` / `MIGRATE_RETRY_SECONDS` tune it. |
+| Any other failure | Exits at once. Retrying a broken migration only delays the same error; **P3005** additionally prints the `migrate resolve --applied 0000_init` hint. |
+
+**On replicas.** Both stacks are `replicas: 1` and must stay that way for an
+unrelated reason (the servicios email sweep has no DB lock — see the caveats
+below). If that ever changes, concurrent `migrate deploy` runs are safe on
+their own: Prisma takes a database advisory lock, so the others block and then
+find nothing pending. They would each pay the wait at startup, not corrupt
+anything.
+
+The prisma CLI has to be present in the runtime layer for any of this. The
+image copies `/repo/node_modules` wholesale so it already is, and the
+Dockerfile **asserts it at build time** — a missing CLI breaks the image
+build rather than a production boot.
+
 ## galactus vs cubex
 
 `galactus` is standalone Docker (Portainer endpoint **3**), `cubex` is a 3-node
@@ -319,9 +374,10 @@ backup does them (see `deploy/scripts/pre-migrate-backup.mjs`):
   Portainer serves a self-signed certificate. It is scoped to that one step,
   which talks to nothing but Portainer. Replacing the certificate and dropping
   the flag is the real fix.
-- The runner lives on cubex and must reach the target host's Portainer (9443)
-  **and** MySQL (3306). If it cannot reach 3306, run the migration by hand from
-  a host that can and dispatch with `skip_migrate: true`.
+- The runner lives on cubex and must reach the target host's Portainer (9443).
+  It should also reach MySQL (3306) for the migrate step, but that is no longer
+  load-bearing — dispatch with `skip_migrate: true` and the api container
+  applies the migrations itself at start.
 - `bootstrap: true` lets the pre-migrate backup be skipped when no API container
   exists yet. Use it for a first-ever deploy only — it is the one switch that
   lets a migration run with no restore point.