diff --git a/.gitea/workflows/deploy-galactus.yml b/.gitea/workflows/deploy-galactus.yml index 5178b20..c9edd34 100644 --- a/.gitea/workflows/deploy-galactus.yml +++ b/.gitea/workflows/deploy-galactus.yml @@ -17,6 +17,12 @@ # 3. prisma migrate deploy forward-only. Prisma has no down-migrations; see # docs/DEPLOY_AND_MIGRATIONS.md — expand/contract is # the rule, the backup is the emergency lever. +# Done HERE so the schema moves while the OLD code is +# still serving. The api container ALSO migrates at +# start (docker/api-entrypoint.sh); `migrate deploy` +# is idempotent, so the second run is a no-op and the +# container is what covers a restart that never goes +# through this workflow at all. # 4. app (api + web) the new images. # 5. verify ask the running API what it actually is. # @@ -55,10 +61,10 @@ # uses until somebody saves them there # These are NOT galactus-specific (no _GALACTUS suffix) — one SES identity # serves every deployment. -# - The runner (which lives on cubex) must be able to reach BOTH -# galactus:9443 (Portainer) and galactus:3306 (MySQL, for migrate deploy). -# If it cannot reach 3306, run the migration by hand from a host that can -# and dispatch with skip_migrate=true. +# - The runner (which lives on cubex) must be able to reach galactus:9443 +# (Portainer). It should also reach galactus:3306 for step 3, but that is +# no longer load-bearing: dispatch with skip_migrate=true and the api +# container applies the migrations itself at start. # - ONE-TIME, on a database that predates migration history (i.e. one built # with `prisma db push`): baseline it before the first run, or step 3 fails # with P3005 "database schema is not empty": @@ -88,7 +94,7 @@ on: required: false default: false skip_migrate: - description: "Skip prisma migrate deploy (use when the runner cannot reach MySQL and you migrated by hand)" + description: "Skip the runner-side migrate step (safe: the api container migrates at start)" type: boolean required: false default: false @@ -237,6 +243,10 @@ jobs: run: node deploy/scripts/pre-migrate-backup.mjs # --- schema, forward-only --------------------------------------------- + # Belt to the container's braces: this runs while the OLD code is still + # serving, which is the order expand/contract is designed around. The + # api container repeats it at start for the paths this step cannot + # reach (skip_migrate, a host reboot, a stack re-applied by hand). - name: Apply database migrations if: ${{ github.event.inputs.skip_migrate != 'true' }} env: diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index 19f697a..cb6d4f3 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -12,6 +12,7 @@ # git tag v1.2.3 into image tag 1.2.3. Tag v1.2.3, dispatch 1.2.3. # # Order: db+minio (full only) -> pre-migrate backup -> prisma migrate deploy -> +# (the api container also migrates at start; see docker/api-entrypoint.sh) # app -> verify the API reports the version you asked for. Rollback = dispatch # an older tag; that rolls back CODE only, never the schema, which is why every # schema change must be expand/contract. See docs/DEPLOY_AND_MIGRATIONS.md. @@ -47,9 +48,11 @@ # # Database stack (full only) # MYSQL_PASSWORD app-user password (matches DATABASE_URL) # MYSQL_ROOT_PASSWORD mysql root password -# - the runner must reach BOTH Portainer (9443) and MySQL (3306) — the -# migration step connects to the database directly. If it cannot reach 3306, -# migrate by hand and dispatch with skip_migrate=true. +# - the runner must reach Portainer (9443). It should also reach MySQL (3306) +# for the migrate step, but that is no longer load-bearing: dispatch with +# skip_migrate=true and the api container applies the migrations itself at +# start (docker/api-entrypoint.sh). `migrate deploy` is idempotent, so the +# two never conflict. # - ONE-TIME on a database built with `prisma db push` (i.e. every database # that exists today): baseline it before the first run, or the migrate step # fails with P3005 "database schema is not empty": @@ -79,7 +82,7 @@ on: required: false default: false skip_migrate: - description: "Skip prisma migrate deploy (use when the runner cannot reach MySQL and you migrated by hand)" + description: "Skip the runner-side migrate step (safe: the api container migrates at start)" type: boolean required: false default: false diff --git a/deploy/galactus/jorgecuadros-app.compose.yml b/deploy/galactus/jorgecuadros-app.compose.yml index 7d4ef5c..aec3038 100644 --- a/deploy/galactus/jorgecuadros-app.compose.yml +++ b/deploy/galactus/jorgecuadros-app.compose.yml @@ -61,6 +61,11 @@ services: INGEST_DIR: /data/ingest BACKUP_DIR: /data/backups MIGRATION_ENV: prod + # The API applies pending Prisma migrations at container start, before + # Nest listens, and refuses to start if they fail (docker/api-entrypoint.sh). + # Set false ONLY when the schema is being moved by hand — the app will + # then boot against whatever schema it finds. + RUN_MIGRATIONS: ${RUN_MIGRATIONS:-true} # Credentials the "Operaciones" screen runs mysqldump/mysql as. NOT the # application user: --single-transaction needs the global RELOAD privilege # and the app user has only ALL ON jorgecuadros.*, so every backup, sync diff --git a/deploy/jorgecuadros-app.stack.yml b/deploy/jorgecuadros-app.stack.yml index aecd0ce..6710214 100644 --- a/deploy/jorgecuadros-app.stack.yml +++ b/deploy/jorgecuadros-app.stack.yml @@ -41,6 +41,11 @@ services: INGEST_DIR: /data/ingest BACKUP_DIR: /data/backups MIGRATION_ENV: prod + # The API applies pending Prisma migrations at container start, before + # Nest listens, and refuses to start if they fail (docker/api-entrypoint.sh). + # Set false ONLY when the schema is being moved by hand — the app will + # then boot against whatever schema it finds. + RUN_MIGRATIONS: ${RUN_MIGRATIONS:-true} # Credentials the "Operaciones" screen runs mysqldump/mysql as. NOT the # application user: --single-transaction needs the global RELOAD privilege # and the app user has only ALL ON jorgecuadros.*, so every backup, sync diff --git a/docker/api-entrypoint.sh b/docker/api-entrypoint.sh new file mode 100644 index 0000000..c6bd9a1 --- /dev/null +++ b/docker/api-entrypoint.sh @@ -0,0 +1,102 @@ +#!/bin/sh +# Apply pending Prisma migrations, then hand off to the API. +# +# WHY THE CONTAINER AND NOT THE DEPLOY WORKFLOW +# +# The workflow still has its own `prisma migrate deploy` step and that is not +# redundant: it runs BEFORE the new images are pulled, i.e. while the OLD code +# is still serving, which is the order expand/contract migrations are designed +# around (see docs/DEPLOY_AND_MIGRATIONS.md). Doing it here as well closes the +# gaps that step cannot: +# +# - The runner has to reach MySQL directly. When it cannot, the deploy is run +# with `skip_migrate=true` and the schema silently does not move — the app +# then boots against a schema that is one release behind, which surfaces +# later as a column-not-found at runtime rather than as a failed deploy. +# - A container restarted by `restart: unless-stopped` after a host reboot, +# or a stack re-applied by hand in Portainer, never goes through the +# workflow at all. +# +# `migrate deploy` is idempotent, so running it in both places costs one +# no-op query on the normal path. +# +# THE API DOES NOT START IF THE MIGRATION FAILS. That is deliberate: serving +# against a schema that does not match the code is worse than being down, +# because the failures it produces are partial and silent (a write to a column +# that does not exist yet fails for one feature while the rest of the app looks +# healthy). The container exits non-zero and Docker's restart policy retries. +set -e + +SCHEMA=/repo/packages/database/prisma/schema.prisma + +log() { echo "[entrypoint] $*"; } + +if [ "${RUN_MIGRATIONS:-true}" != "true" ]; then + log "RUN_MIGRATIONS=${RUN_MIGRATIONS} — skipping migrations, starting the API" + exec "$@" +fi + +if [ -z "${DATABASE_URL}" ]; then + log "DATABASE_URL is unset; cannot migrate." >&2 + log "Set it, or set RUN_MIGRATIONS=false if you migrate out of band." >&2 + exit 1 +fi + +# pnpm's hoisted linker normally puts the CLI in the root .bin, but the +# workspace package keeps its own link too. Accept either rather than pinning +# a layout detail of the installer — the Dockerfile asserts at build time that +# one of these exists, so a missing CLI breaks the image build, not a deploy. +PRISMA="" +for candidate in /repo/node_modules/.bin/prisma \ + /repo/packages/database/node_modules/.bin/prisma; do + if [ -x "$candidate" ]; then + PRISMA="$candidate" + break + fi +done +if [ -z "$PRISMA" ]; then + log "prisma CLI not found in this image; cannot migrate." >&2 + exit 1 +fi + +# Retry ONLY a connection failure (P1001). On a full bring-up the database +# container can still be starting, and on galactus the API additionally has to +# resolve the host's MagicDNS name — a lookup that is unreliable for the first +# moments after a host reboot (see the dns block in the app compose file, and +# docs/DEPLOY_AND_MIGRATIONS.md). +# +# Every other failure exits immediately. Retrying a migration that is actually +# broken just delays the same error behind a minute of noise, and P3005 in +# particular needs a human. +attempt=1 +max="${MIGRATE_MAX_ATTEMPTS:-20}" +delay="${MIGRATE_RETRY_SECONDS:-3}" + +while : ; do + log "prisma migrate deploy (attempt ${attempt}/${max})" + if output=$("$PRISMA" migrate deploy --schema "$SCHEMA" 2>&1); then + printf '%s\n' "$output" + log "migrations up to date" + break + fi + printf '%s\n' "$output" >&2 + + if ! printf '%s' "$output" | grep -q 'P1001'; then + log "migrate deploy FAILED — refusing to start the API." >&2 + if printf '%s' "$output" | grep -q 'P3005'; then + log "P3005: the database has tables but no migration history. This is a" >&2 + log "database that predates Prisma migrations. Baseline it ONCE with:" >&2 + log " npx prisma@5 migrate resolve --applied 0000_init --schema $SCHEMA" >&2 + fi + exit 1 + fi + + if [ "$attempt" -ge "$max" ]; then + log "database unreachable after ${max} attempts — giving up." >&2 + exit 1 + fi + attempt=$((attempt + 1)) + sleep "$delay" +done + +exec "$@" diff --git a/docker/api.Dockerfile b/docker/api.Dockerfile index 8712a62..1efff14 100644 --- a/docker/api.Dockerfile +++ b/docker/api.Dockerfile @@ -113,5 +113,21 @@ ENV APP_VERSION=$APP_VERSION \ GIT_SHA=$GIT_SHA \ BUILD_DATE=$BUILD_DATE +# Pending migrations are applied at container start, before Nest listens — +# see the header of the script for why this is done here as well as in the +# deploy workflow. Asserted at BUILD time so a missing prisma CLI breaks the +# image build rather than a production boot: the runtime layer copies +# /repo/node_modules wholesale, and which of these two paths carries the bin +# is an implementation detail of pnpm's hoisted linker. +COPY docker/api-entrypoint.sh /usr/local/bin/api-entrypoint.sh +RUN chmod +x /usr/local/bin/api-entrypoint.sh +RUN for c in /repo/node_modules/.bin/prisma \ + /repo/packages/database/node_modules/.bin/prisma; do \ + if [ -x "$c" ]; then echo "prisma CLI found at $c"; exit 0; fi; \ + done; \ + echo "FATAL: prisma CLI is not in the runtime layer; api-entrypoint.sh cannot migrate" >&2; \ + exit 1 + EXPOSE 3001 +ENTRYPOINT ["/usr/local/bin/api-entrypoint.sh"] CMD ["node", "apps/api/dist/main.js"] diff --git a/docs/DEPLOY_AND_MIGRATIONS.md b/docs/DEPLOY_AND_MIGRATIONS.md index 0bf4031..29b4254 100644 --- a/docs/DEPLOY_AND_MIGRATIONS.md +++ b/docs/DEPLOY_AND_MIGRATIONS.md @@ -56,9 +56,11 @@ workflow's last step fails if the API does not report the tag you dispatched. `pre-migrate--.sql.gz`, which is exactly what the **Operaciones** admin screen lists and can restore. A dump taken on the CI runner would be unreachable by the only restore path the platform has. -3. **`prisma migrate deploy`** — as a workflow *step*, never the container - `CMD`. If it were the CMD, N replicas would race each other applying the - same migration. +3. **`prisma migrate deploy`** — as a workflow *step*, so the schema moves + while the OLD code is still serving, which is the order expand/contract is + designed around. **The api container repeats this at start** (below); the + command is idempotent, so on the normal path the container's run is a + no-op. 4. **app** — the new api + web images. 5. **Verify** — `GET /version` on the running API must report the dispatched tag. @@ -141,6 +143,59 @@ Commit the generated `migrations/_add_foo/` directory. `db push` is now a local-scratch tool only — using it against a database with history desynchronises it from `_prisma_migrations`. +If you hand-write a migration instead of generating one, check it against what +Prisma would have produced before committing — a hand-written file that drifts +from `schema.prisma` fails on the *next* deploy, not this one: + +```bash +prisma migrate diff \ + --from-schema-datamodel \ + --to-schema-datamodel packages/database/prisma/schema.prisma --script +``` + +## Migrations also run at container start + +`docker/api-entrypoint.sh` is the api image's `ENTRYPOINT`. It runs +`prisma migrate deploy` and only then `exec`s the API. **If the migration +fails the container exits non-zero and the API never listens.** + +That is the point. Serving against a schema that does not match the code is +worse than being down, because the failures are partial and silent — a write +to a column that does not exist yet breaks one feature while the rest of the +app looks healthy. + +This does not replace the workflow step, which still runs first and against +the old code. It covers what that step cannot: + +- **`skip_migrate: true`.** Previously that left the schema behind with no + further safety net, and the mismatch surfaced later as a runtime error. Now + it just moves the migration into the container, so it is a safe choice when + the runner cannot reach MySQL. +- **Restarts that never touch the workflow** — `restart: unless-stopped` + bringing the stack back after a host reboot, or a stack re-applied by hand + in Portainer. + +Behaviour worth knowing: + +| | | +|---|---| +| `RUN_MIGRATIONS=false` | Skip and start anyway. Plumbed through both app stack files. For when the schema is being moved by hand. | +| `DATABASE_URL` unset | Refuses to start (it would have failed at Nest boot anyway, but this says why). | +| Cannot reach the database (**P1001**) | Retries, default 20 × 3s. Covers a cold `db` container and galactus's MagicDNS lookup right after a reboot. `MIGRATE_MAX_ATTEMPTS` / `MIGRATE_RETRY_SECONDS` tune it. | +| Any other failure | Exits at once. Retrying a broken migration only delays the same error; **P3005** additionally prints the `migrate resolve --applied 0000_init` hint. | + +**On replicas.** Both stacks are `replicas: 1` and must stay that way for an +unrelated reason (the servicios email sweep has no DB lock — see the caveats +below). If that ever changes, concurrent `migrate deploy` runs are safe on +their own: Prisma takes a database advisory lock, so the others block and then +find nothing pending. They would each pay the wait at startup, not corrupt +anything. + +The prisma CLI has to be present in the runtime layer for any of this. The +image copies `/repo/node_modules` wholesale so it already is, and the +Dockerfile **asserts it at build time** — a missing CLI breaks the image +build rather than a production boot. + ## galactus vs cubex `galactus` is standalone Docker (Portainer endpoint **3**), `cubex` is a 3-node @@ -319,9 +374,10 @@ backup does them (see `deploy/scripts/pre-migrate-backup.mjs`): Portainer serves a self-signed certificate. It is scoped to that one step, which talks to nothing but Portainer. Replacing the certificate and dropping the flag is the real fix. -- The runner lives on cubex and must reach the target host's Portainer (9443) - **and** MySQL (3306). If it cannot reach 3306, run the migration by hand from - a host that can and dispatch with `skip_migrate: true`. +- The runner lives on cubex and must reach the target host's Portainer (9443). + It should also reach MySQL (3306) for the migrate step, but that is no longer + load-bearing — dispatch with `skip_migrate: true` and the api container + applies the migrations itself at start. - `bootstrap: true` lets the pre-migrate backup be skipped when no API container exists yet. Use it for a first-ever deploy only — it is the one switch that lets a migration run with no restore point.