jorgecuadros-platform/docs/DEPLOY_AND_MIGRATIONS.md

Releasing, deploying, and changing the schema

How a version gets from this repo onto a server, and the one rule that keeps rollbacks possible.

The short version

Dispatch Cut release from the Actions tab and pick patch, minor or major (or explicit plus a number). It stamps every package.json, commits chore(release): vX.Y.Z, tags, and pushes both refs in one go. It refuses a version that already exists as a tag, and refuses a no-op bump.

The equivalent by hand, if you would rather cut it locally:

pnpm version:set 1.2.0                       # stamp every package.json
git commit -am "chore(release): v1.2.0"
git tag v1.2.0 && git push origin master v1.2.0

Either way that push triggers .gitea/workflows/build.yml, which builds both images in one matrix run and publishes:

tag pushed	image tags produced
v1.2.0	1.2.0, 1.2, sha-<short>
push to master	master, sha-<short>, latest

Then dispatch a deploy from the Actions tab:

• galactus (office server, standalone Docker) — Deploy to galactus
• cubex (3-node Swarm) — Deploy to Portainer

The v is not part of the image tag. docker/metadata-action's {{version}} strips it. Git tag v1.2.0, dispatch 1.2.0. Dispatching v1.2.0 deploys nothing that exists.

Cut release needs a RELEASE_TOKEN secret — a Gitea PAT with write:repository. It does not use the built-in Actions token on purpose: whether a push made with that token re-triggers build.yml depends on the Gitea version, and a release that quietly publishes no images is worse than one that fails outright. If the build somehow does not start, build.yml has workflow_dispatch — run it against the new tag by hand.

Because api and web are built from one matrix run, they cannot drift at build time. They can drift at deploy time if a stack is applied with only one image moved — the web footer shows both versions and flags a mismatch, and the deploy workflow's last step fails if the API does not report the tag you dispatched.

What a deploy actually does

1. db + minio — scope: full only. Idempotent; data lives on named volumes.
2. Pre-migrate backup — deploy/scripts/pre-migrate-backup.mjs runs mysqldump inside the still-running old API container, via Portainer's Docker API. The file lands in that container's BACKUP_DIR volume as pre-migrate-<tag>-<timestamp>.sql.gz, which is exactly what the Operaciones admin screen lists and can restore. A dump taken on the CI runner would be unreachable by the only restore path the platform has.
3. prisma migrate deploy — as a workflow step, so the schema moves while the OLD code is still serving, which is the order expand/contract is designed around. The api container repeats this at start (below); the command is idempotent, so on the normal path the container's run is a no-op.
4. app — the new api + web images.
5. Verify — GET /version on the running API must report the dispatched tag.

Rollback is tag: 1.1.9 re-dispatched. That rolls back code only. The schema stays where it is. Which brings us to the rule.

The rule: expand / contract

Prisma has no down-migrations. There is no prisma migrate down, and there never will be. So a schema change that the previous release cannot tolerate turns a 30-second rollback into a restore-from-backup outage.

Every schema change must leave the previous release working. Split anything destructive across two releases:

	Release N (expand)	Release N+1 (contract)
Rename a column	add the new column, write to both, read the old	drop the old column
Drop a column	stop reading and writing it in code	drop it
Add a required column	add it nullable (or with a default), backfill	make it NOT NULL
Split a table	create the new table, dual-write	stop writing the old, drop it
Add an enum value	add the value; old code must not choke on unknowns	start emitting it

Ship N, let it soak, then ship N+1. If N has to be rolled back you just re-dispatch the old tag — the expanded schema still satisfies it.

Restoring from the pre-migrate dump is the emergency lever, not the routine path, and on galactus it is worse than it sounds: galactus is the replication master, DDL replicates through the binlog, and restoring the master from a dump diverges every replica. GTIDs will not line up and each replica needs a full re-seed. Assume a restore is a multi-hour, whole-topology event.

Migration history

packages/database/prisma/migrations/0000_init/ is a baseline. It is the full schema as it stood on 2026-07-30, generated with:

prisma migrate diff --from-empty \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script

Until then the schema had only ever been applied with prisma db push, so no history existed and the schema state was disconnected from the app version.

One-time, on every database that already exists

0000_init describes tables those databases already have, so migrate deploy would fail with P3005 "the database schema is not empty". Mark it applied instead of applying it — this writes a _prisma_migrations row and changes no data:

DATABASE_URL=<the database> npx prisma@5 migrate resolve \
  --applied 0000_init --schema packages/database/prisma/schema.prisma

Do this once per database (prod, dev, any local copy). Verify first that the live schema really does match the baseline — this should print an empty migration:

prisma migrate diff --from-url "$DATABASE_URL" \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script

If it prints actual statements, the live database has drifted from schema.prisma. Reconcile before baselining, or the first real migration will fail against a schema Prisma believes it already knows.

From here on

# edit schema.prisma, then:
pnpm --filter @jorgecuadros/database exec prisma migrate dev --name add_foo

Commit the generated migrations/<timestamp>_add_foo/ directory. db push is now a local-scratch tool only — using it against a database with history desynchronises it from _prisma_migrations.

If you hand-write a migration instead of generating one, check it against what Prisma would have produced before committing — a hand-written file that drifts from schema.prisma fails on the next deploy, not this one:

prisma migrate diff \
  --from-schema-datamodel <schema.prisma at the previous commit> \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script

Migrations also run at container start

docker/api-entrypoint.sh is the api image's ENTRYPOINT. It runs prisma migrate deploy and only then execs the API. If the migration fails the container exits non-zero and the API never listens.

That is the point. Serving against a schema that does not match the code is worse than being down, because the failures are partial and silent — a write to a column that does not exist yet breaks one feature while the rest of the app looks healthy.

This does not replace the workflow step, which still runs first and against the old code. It covers what that step cannot:

• skip_migrate: true. Previously that left the schema behind with no further safety net, and the mismatch surfaced later as a runtime error. Now it just moves the migration into the container, so it is a safe choice when the runner cannot reach MySQL.
• Restarts that never touch the workflow — restart: unless-stopped bringing the stack back after a host reboot, or a stack re-applied by hand in Portainer.

Behaviour worth knowing:

RUN_MIGRATIONS=false	Skip and start anyway. Plumbed through both app stack files. For when the schema is being moved by hand.
DATABASE_URL unset	Refuses to start (it would have failed at Nest boot anyway, but this says why).
Cannot reach the database (P1001)	Retries, default 20 × 3s. Covers a cold db container and galactus's MagicDNS lookup right after a reboot. MIGRATE_MAX_ATTEMPTS / MIGRATE_RETRY_SECONDS tune it.
Any other failure	Exits at once. Retrying a broken migration only delays the same error; P3005 additionally prints the migrate resolve --applied 0000_init hint.

On replicas. Both stacks are replicas: 1 and must stay that way for an unrelated reason (the servicios email sweep has no DB lock — see the caveats below). If that ever changes, concurrent migrate deploy runs are safe on their own: Prisma takes a database advisory lock, so the others block and then find nothing pending. They would each pay the wait at startup, not corrupt anything.

The prisma CLI has to be present in the runtime layer for any of this. The image copies /repo/node_modules wholesale so it already is, and the Dockerfile asserts it at build time — a missing CLI breaks the image build rather than a production boot.

galactus vs cubex

galactus is standalone Docker (Portainer endpoint 3), cubex is a 3-node Swarm (endpoint 2). They need different compose files because plain compose silently ignores Swarm's deploy: keys rather than erroring:

	Swarm (deploy/*.stack.yml)	standalone (deploy/galactus/*.compose.yml)
restart	deploy.restart_policy	restart: unless-stopped — without this nothing comes back after a host reboot
placement	node.labels.jorgecuadros_db == true	dropped, one host
ports	{mode: ingress} long syntax	"3306:3306"
depends_on	ignored by Swarm	honoured, with condition: service_healthy
volumes	named	named (unchanged — the pinning hazard was a Swarm problem)

Keep the two sets in sync when either changes.

On both hosts, cross-stack traffic goes over the host address, not compose service DNS: db, minio and app are three separate stacks, so three separate networks. DATABASE_URL and S3_ENDPOINT name the host and its published port. Do not "simplify" them to mysql:3306.

galactus is addressed by MagicDNS, and containers need help resolving it

galactus is Tailscale-only once it is installed in the office, so every URL names galactus.tail01aa2.ts.net. Its LAN IP is a DHCP lease and has already drifted once — never put a 192.168.4.x address in a secret.

Containers on galactus cannot resolve that name by default. The host runs systemd-resolved, whose 127.0.0.53 stub is unreachable from inside a container, so Docker falls back to the upstream resolver listed in /run/systemd/resolve/resolv.conf — the LAN router, which knows nothing about the tailnet. Routing to 100.x works fine; only the lookup fails, and the symptom is Prisma P1001 "can't reach database server" on a container that otherwise started cleanly.

deploy/galactus/jorgecuadros-app.compose.yml therefore pins the resolver:

dns:        [100.100.100.100]        # Tailscale's fixed anycast MagicDNS address
dns_search: [tail01aa2.ts.net]       # this tailnet's suffix

Both are overridable (TAILSCALE_DNS, TAILNET_SUFFIX) if the tailnet changes. Browser-facing origins need none of this — those names resolve on the client.

Replication

galactus's MySQL is the master; every other MySQL in the estate is a replica. Consequences that bite:

• server-id must be unique across the whole topology (prod 1, cubex dev 11). A duplicate breaks replication silently.
• GTID is on from first boot, so replicas attach with SOURCE_AUTO_POSITION=1.
• binlog_expire_logs_seconds is raised to 60 days in the galactus compose file (MYSQL_BINLOG_EXPIRE_SECONDS). MySQL 8.4 defaults to 30 days; a replica offline longer than the retention needs a full re-seed.

Still open, and not handled by anything in this repo:

• No replication user with REPLICATION SLAVE granted exists yet.
• Nothing sets read_only / super_read_only on the replicas, so a stray write to a replica will diverge it.
• The channel to the VPS crosses the public internet. It needs a tunnel or TLS — do not publish raw 3306.

Seeding the first sign-in account

A freshly migrated database has a schema and no users, so nobody can log in. prisma migrate deploy creates tables, never rows; nothing in the deploy path seeds an account, by design — creating an administrator should be a deliberate act, not a side effect of shipping code.

apps/api/scripts/seed-user.mjs ships inside the API image. On the target host:

docker exec -e SEED_PASSWORD='<a strong password>' \
  <api-container> node apps/api/scripts/seed-user.mjs

Defaults are admin@jorgecuadros.local / ChangeMe!2026 / role ADMIN, overridable with SEED_EMAIL, SEED_PASSWORD, SEED_NAME. Do not accept the default password on anything but a dev database — it is published in this repo's README. The script upserts by email, so re-running is safe, but it also resets the password of an existing account.

The session cookie and TLS

SESSION_COOKIE_SECURE controls the Secure flag on the session cookie. It defaults to on in production, and it must be explicitly "false" for a deployment served over plain HTTP.

This is not cosmetic. express-session silently declines to emit a Secure cookie over an unencrypted connection: no Set-Cookie header is sent at all, POST /auth/login still answers 200 with the user object, no session is established, every subsequent request gets 403, and the UI bounces back to /login in a loop. It looks like an auth bug and is really a transport mismatch.

galactus runs with SESSION_COOKIE_SECURE=false, which is acceptable only because it is reachable exclusively over Tailscale — WireGuard already encrypts the wire, so the cookie never crosses an untrusted network. Turn it back on the moment the app is served over TLS or reachable off-tailnet. Behind a TLS-terminating reverse proxy, set trust proxy on the Nest app instead of disabling the flag.

The MySQL client inside the API image

Alpine's mysql-client package is MariaDB's client, and it installs an empty /usr/lib/mariadb/plugin. It therefore cannot speak caching_sha2_password, which is MySQL 8.4's default and effectively only auth method, and every mysqldump/mysql call from the container fails with:

ERROR 1045: Plugin caching_sha2_password could not be loaded:
  ... /usr/lib/mariadb/plugin/caching_sha2_password.so: No such file or directory

mariadb-connector-c supplies that plugin and is installed in docker/api.Dockerfile for exactly this reason — do not drop it as an unused dependency. It affects far more than the deploy backup: the entire Operaciones panel (backup, restore, sync, re-import) shells out to these binaries, so without it none of those work in a container either. The feature had only ever been exercised with the API running on a developer machine, where the Oracle client is installed, which is why this went unnoticed until the first containerised deploy.

The Operaciones panel needs its own database login

The panel's four jobs all shell out to mysqldump/mysql, and they cannot do so as the application user. mysqldump --single-transaction issues FLUSH TABLES, which requires the global RELOAD privilege; the MySQL image grants the app user only ALL PRIVILEGES ON jorgecuadros.* plus USAGE ON *.*. --skip-lock-tables does not avoid it. BACKUP therefore failed outright, and SYNC and RE-IMPORT with it, because both take a safety backup first.

The API is given an admin login out of band rather than permanently elevating the user it serves requests as:

OPS_DB_ADMIN_USER=root
OPS_DB_ADMIN_PASSWORD=<MYSQL_ROOT_PASSWORD>

Both deploy workflows pass these into the app stack from the existing MYSQL_ROOT_PASSWORD secret. Host, port and database still come from DATABASE_URL — the override changes who logs in, never which server. With the pair unset the service falls back to the DATABASE_URL credentials and logs a warning, which is what local development wants.

Two more things the panel's dumps now do, for the same reasons the pre-migrate backup does them (see deploy/scripts/pre-migrate-backup.mjs):

• --set-gtid-purged=OFF, but only when the dumper supports it. galactus is the replication source with GTID on, so on a MySQL client this flag is what keeps every dump from embedding SET @@GLOBAL.GTID_PURGED and becoming unrestorable onto the server it came from — which is precisely what the restore screen exists to do. The panel, however, dumps from inside the API container, where Alpine's mysql-client is MariaDB's: there mysqldump is a shim over mariadb-dump, the flag does not exist, and passing it failed every backup with mysqldump: unknown variable 'set-gtid-purged=OFF'. So the panel probes mysqldump --help and passes the flag only if it is advertised, invoking mariadb-dump directly otherwise (MariaDB writes no GTID state unless asked with --gtid, so there is nothing to suppress). The pre-migrate backup keeps the flag unconditionally — it runs in a real mysql:8.4 image, not in the API container.
• set -o pipefail and a CREATE TABLE count. mysqldump | gzip reports gzip's exit status, and a mysqldump that dies on its first statement still produces a ~372-byte perfectly valid archive that passes gzip -t. Without both checks a failed backup was recorded as a successful one and listed as an ordinary restore point. A dump that fails now deletes its own output.

Known caveats in the deploy path

• The pre-migrate backup step sets NODE_TLS_REJECT_UNAUTHORIZED=0 because Portainer serves a self-signed certificate. It is scoped to that one step, which talks to nothing but Portainer. Replacing the certificate and dropping the flag is the real fix.
• The runner lives on cubex and must reach the target host's Portainer (9443). It should also reach MySQL (3306) for the migrate step, but that is no longer load-bearing — dispatch with skip_migrate: true and the api container applies the migrations itself at start.
• bootstrap: true lets the pre-migrate backup be skipped when no API container exists yet. Use it for a first-ever deploy only — it is the one switch that lets a migration run with no restore point.
• The API container sends mail on a timer. Two sweeps run inside it (renewal avisos, on by default at 06:00 America/Tijuana; the servicios run-all, off by default) — see MASS_EMAIL_NOTIFICATIONS.md. Two consequences for deploys: the cadence lives in app_settings, so it survives a redeploy and is not restored by rolling back an image, and running more than one API replica would double-fire the servicios sweep, which has no DB lock (the pólizas one does). Keep it single-replica.
• SES_* is optional to deploy — the preflight only warns — but the production image sets NODE_ENV=production, which disables the stdout dev fallback. A blank SES config therefore makes every send fail and log FAILED. The secrets were created 2026-08-02; the preflight warning on the next run is what confirms the names are right. Two things it cannot check: that SES_FROM is a verified identity in SES_REGION, and that the account is out of the SES sandbox (in sandbox, delivery is restricted to verified recipients, which would fail a real sweep while looking correctly configured).