# Releasing, deploying, and changing the schema How a version gets from this repo onto a server, and the one rule that keeps rollbacks possible. ## The short version Dispatch **Cut release** from the Actions tab and pick `patch`, `minor` or `major` (or `explicit` plus a number). It stamps every `package.json`, commits `chore(release): vX.Y.Z`, tags, and pushes both refs in one go. It refuses a version that already exists as a tag, and refuses a no-op bump. The equivalent by hand, if you would rather cut it locally: ```bash pnpm version:set 1.2.0 # stamp every package.json git commit -am "chore(release): v1.2.0" git tag v1.2.0 && git push origin master v1.2.0 ``` Either way that push triggers `.gitea/workflows/build.yml`, which builds **both** images in one matrix run and publishes: | tag pushed | image tags produced | | --- | --- | | `v1.2.0` | `1.2.0`, `1.2`, `sha-` | | push to `master` | `master`, `sha-`, `latest` | Then dispatch a deploy from the Actions tab: - **galactus** (office server, standalone Docker) — *Deploy to galactus* - **cubex** (3-node Swarm) — *Deploy to Portainer* > **The `v` is not part of the image tag.** `docker/metadata-action`'s > `{{version}}` strips it. Git tag `v1.2.0`, dispatch `1.2.0`. Dispatching > `v1.2.0` deploys nothing that exists. **Cut release needs a `RELEASE_TOKEN` secret** — a Gitea PAT with `write:repository`. It does not use the built-in Actions token on purpose: whether a push made with that token re-triggers `build.yml` depends on the Gitea version, and a release that quietly publishes no images is worse than one that fails outright. If the build somehow does not start, `build.yml` has `workflow_dispatch` — run it against the new tag by hand. Because api and web are built from one matrix run, they cannot drift at build time. They *can* drift at deploy time if a stack is applied with only one image moved — the web footer shows both versions and flags a mismatch, and the deploy workflow's last step fails if the API does not report the tag you dispatched. ## What a deploy actually does 1. **db + minio** — `scope: full` only. Idempotent; data lives on named volumes. 2. **Pre-migrate backup** — `deploy/scripts/pre-migrate-backup.mjs` runs `mysqldump` *inside the still-running old API container*, via Portainer's Docker API. The file lands in that container's `BACKUP_DIR` volume as `pre-migrate--.sql.gz`, which is exactly what the **Operaciones** admin screen lists and can restore. A dump taken on the CI runner would be unreachable by the only restore path the platform has. 3. **`prisma migrate deploy`** — as a workflow *step*, so the schema moves while the OLD code is still serving, which is the order expand/contract is designed around. **The api container repeats this at start** (below); the command is idempotent, so on the normal path the container's run is a no-op. 4. **app** — the new api + web images. 5. **Verify** — `GET /version` on the running API must report the dispatched tag. Rollback is `tag: 1.1.9` re-dispatched. **That rolls back code only.** The schema stays where it is. Which brings us to the rule. ## The rule: expand / contract Prisma has no down-migrations. There is no `prisma migrate down`, and there never will be. So a schema change that the *previous* release cannot tolerate turns a 30-second rollback into a restore-from-backup outage. **Every schema change must leave the previous release working.** Split anything destructive across two releases: | | Release N (expand) | Release N+1 (contract) | | --- | --- | --- | | Rename a column | add the new column, write to both, read the old | drop the old column | | Drop a column | stop reading and writing it in code | drop it | | Add a required column | add it nullable (or with a default), backfill | make it `NOT NULL` | | Split a table | create the new table, dual-write | stop writing the old, drop it | | Add an enum value | add the value; old code must not choke on unknowns | start emitting it | Ship N, let it soak, *then* ship N+1. If N has to be rolled back you just re-dispatch the old tag — the expanded schema still satisfies it. Restoring from the pre-migrate dump is the **emergency lever, not the routine path**, and on galactus it is worse than it sounds: galactus is the replication master, DDL replicates through the binlog, and restoring the master from a dump diverges every replica. GTIDs will not line up and each replica needs a full re-seed. Assume a restore is a multi-hour, whole-topology event. ## Migration history `packages/database/prisma/migrations/0000_init/` is a **baseline**. It is the full schema as it stood on 2026-07-30, generated with: ```bash prisma migrate diff --from-empty \ --to-schema-datamodel packages/database/prisma/schema.prisma --script ``` Until then the schema had only ever been applied with `prisma db push`, so no history existed and the schema state was disconnected from the app version. ### One-time, on every database that already exists `0000_init` describes tables those databases already have, so `migrate deploy` would fail with **P3005 "the database schema is not empty"**. Mark it applied instead of applying it — this writes a `_prisma_migrations` row and changes no data: ```bash DATABASE_URL= npx prisma@5 migrate resolve \ --applied 0000_init --schema packages/database/prisma/schema.prisma ``` Do this once per database (prod, dev, any local copy). Verify first that the live schema really does match the baseline — this should print an empty migration: ```bash prisma migrate diff --from-url "$DATABASE_URL" \ --to-schema-datamodel packages/database/prisma/schema.prisma --script ``` If it prints actual statements, the live database has drifted from `schema.prisma`. Reconcile *before* baselining, or the first real migration will fail against a schema Prisma believes it already knows. ### From here on ```bash # edit schema.prisma, then: pnpm --filter @jorgecuadros/database exec prisma migrate dev --name add_foo ``` Commit the generated `migrations/_add_foo/` directory. `db push` is now a local-scratch tool only — using it against a database with history desynchronises it from `_prisma_migrations`. If you hand-write a migration instead of generating one, check it against what Prisma would have produced before committing — a hand-written file that drifts from `schema.prisma` fails on the *next* deploy, not this one: ```bash prisma migrate diff \ --from-schema-datamodel \ --to-schema-datamodel packages/database/prisma/schema.prisma --script ``` ## Migrations also run at container start `docker/api-entrypoint.sh` is the api image's `ENTRYPOINT`. It runs `prisma migrate deploy` and only then `exec`s the API. **If the migration fails the container exits non-zero and the API never listens.** That is the point. Serving against a schema that does not match the code is worse than being down, because the failures are partial and silent — a write to a column that does not exist yet breaks one feature while the rest of the app looks healthy. This does not replace the workflow step, which still runs first and against the old code. It covers what that step cannot: - **`skip_migrate: true`.** Previously that left the schema behind with no further safety net, and the mismatch surfaced later as a runtime error. Now it just moves the migration into the container, so it is a safe choice when the runner cannot reach MySQL. - **Restarts that never touch the workflow** — `restart: unless-stopped` bringing the stack back after a host reboot, or a stack re-applied by hand in Portainer. Behaviour worth knowing: | | | |---|---| | `RUN_MIGRATIONS=false` | Skip and start anyway. Plumbed through both app stack files. For when the schema is being moved by hand. | | `DATABASE_URL` unset | Refuses to start (it would have failed at Nest boot anyway, but this says why). | | Cannot reach the database (**P1001**) | Retries, default 20 × 3s. Covers a cold `db` container and galactus's MagicDNS lookup right after a reboot. `MIGRATE_MAX_ATTEMPTS` / `MIGRATE_RETRY_SECONDS` tune it. | | Any other failure | Exits at once. Retrying a broken migration only delays the same error; **P3005** additionally prints the `migrate resolve --applied 0000_init` hint. | **On replicas.** Both stacks are `replicas: 1` and must stay that way for an unrelated reason (the servicios email sweep has no DB lock — see the caveats below). If that ever changes, concurrent `migrate deploy` runs are safe on their own: Prisma takes a database advisory lock, so the others block and then find nothing pending. They would each pay the wait at startup, not corrupt anything. The prisma CLI has to be present in the runtime layer for any of this. The image copies `/repo/node_modules` wholesale so it already is, and the Dockerfile **asserts it at build time** — a missing CLI breaks the image build rather than a production boot. ## galactus vs cubex `galactus` is standalone Docker (Portainer endpoint **3**), `cubex` is a 3-node Swarm (endpoint **2**). They need different compose files because **plain compose silently ignores Swarm's `deploy:` keys** rather than erroring: | | Swarm (`deploy/*.stack.yml`) | standalone (`deploy/galactus/*.compose.yml`) | | --- | --- | --- | | restart | `deploy.restart_policy` | `restart: unless-stopped` — **without this nothing comes back after a host reboot** | | placement | `node.labels.jorgecuadros_db == true` | dropped, one host | | ports | `{mode: ingress}` long syntax | `"3306:3306"` | | `depends_on` | ignored by Swarm | honoured, with `condition: service_healthy` | | volumes | named | named (unchanged — the pinning hazard was a Swarm problem) | Keep the two sets in sync when either changes. On both hosts, cross-stack traffic goes over the **host address**, not compose service DNS: db, minio and app are three separate stacks, so three separate networks. `DATABASE_URL` and `S3_ENDPOINT` name the host and its published port. Do not "simplify" them to `mysql:3306`. ### galactus is addressed by MagicDNS, and containers need help resolving it galactus is Tailscale-only once it is installed in the office, so every URL names `galactus.tail01aa2.ts.net`. Its LAN IP is a DHCP lease and has already drifted once — never put a `192.168.4.x` address in a secret. Containers on galactus cannot resolve that name by default. The host runs systemd-resolved, whose `127.0.0.53` stub is unreachable from inside a container, so Docker falls back to the upstream resolver listed in `/run/systemd/resolve/resolv.conf` — the LAN router, which knows nothing about the tailnet. Routing to `100.x` works fine; only the *lookup* fails, and the symptom is Prisma **P1001 "can't reach database server"** on a container that otherwise started cleanly. `deploy/galactus/jorgecuadros-app.compose.yml` therefore pins the resolver: ```yaml dns: [100.100.100.100] # Tailscale's fixed anycast MagicDNS address dns_search: [tail01aa2.ts.net] # this tailnet's suffix ``` Both are overridable (`TAILSCALE_DNS`, `TAILNET_SUFFIX`) if the tailnet changes. Browser-facing origins need none of this — those names resolve on the client. ## Replication galactus's MySQL is the **master**; every other MySQL in the estate is a replica. Consequences that bite: - `server-id` must be unique across the whole topology (prod `1`, cubex dev `11`). A duplicate breaks replication silently. - GTID is on from first boot, so replicas attach with `SOURCE_AUTO_POSITION=1`. - `binlog_expire_logs_seconds` is raised to 60 days in the galactus compose file (`MYSQL_BINLOG_EXPIRE_SECONDS`). MySQL 8.4 defaults to 30 days; a replica offline longer than the retention needs a full re-seed. Still open, and **not** handled by anything in this repo: - No replication user with `REPLICATION SLAVE` granted exists yet. - Nothing sets `read_only` / `super_read_only` on the replicas, so a stray write to a replica will diverge it. - The channel to the VPS crosses the public internet. It needs a tunnel or TLS — do not publish raw 3306. ## Seeding the first sign-in account A freshly migrated database has a schema and **no users**, so nobody can log in. `prisma migrate deploy` creates tables, never rows; nothing in the deploy path seeds an account, by design — creating an administrator should be a deliberate act, not a side effect of shipping code. `apps/api/scripts/seed-user.mjs` ships inside the API image. On the target host: ```bash docker exec -e SEED_PASSWORD='' \ node apps/api/scripts/seed-user.mjs ``` Defaults are `admin@jorgecuadros.local` / `ChangeMe!2026` / role `ADMIN`, overridable with `SEED_EMAIL`, `SEED_PASSWORD`, `SEED_NAME`. **Do not accept the default password on anything but a dev database** — it is published in this repo's README. The script upserts by email, so re-running is safe, but it also **resets the password of an existing account**. ## The session cookie and TLS `SESSION_COOKIE_SECURE` controls the `Secure` flag on the session cookie. It defaults to on in production, and it must be explicitly `"false"` for a deployment served over plain HTTP. This is not cosmetic. express-session silently declines to emit a `Secure` cookie over an unencrypted connection: no `Set-Cookie` header is sent at all, `POST /auth/login` still answers `200` with the user object, no session is established, every subsequent request gets `403`, and the UI bounces back to `/login` in a loop. It looks like an auth bug and is really a transport mismatch. galactus runs with `SESSION_COOKIE_SECURE=false`, which is acceptable **only** because it is reachable exclusively over Tailscale — WireGuard already encrypts the wire, so the cookie never crosses an untrusted network. Turn it back on the moment the app is served over TLS or reachable off-tailnet. Behind a TLS-terminating reverse proxy, set `trust proxy` on the Nest app instead of disabling the flag. ## The MySQL client inside the API image Alpine's `mysql-client` package is **MariaDB's** client, and it installs an empty `/usr/lib/mariadb/plugin`. It therefore cannot speak `caching_sha2_password`, which is MySQL 8.4's default and effectively only auth method, and every `mysqldump`/`mysql` call from the container fails with: ``` ERROR 1045: Plugin caching_sha2_password could not be loaded: ... /usr/lib/mariadb/plugin/caching_sha2_password.so: No such file or directory ``` `mariadb-connector-c` supplies that plugin and is installed in `docker/api.Dockerfile` for exactly this reason — do not drop it as an unused dependency. It affects far more than the deploy backup: the entire **Operaciones** panel (backup, restore, sync, re-import) shells out to these binaries, so without it none of those work in a container either. The feature had only ever been exercised with the API running on a developer machine, where the Oracle client is installed, which is why this went unnoticed until the first containerised deploy. ## The Operaciones panel needs its own database login The panel's four jobs all shell out to `mysqldump`/`mysql`, and they cannot do so as the application user. `mysqldump --single-transaction` issues `FLUSH TABLES`, which requires the **global** `RELOAD` privilege; the MySQL image grants the app user only `ALL PRIVILEGES ON jorgecuadros.*` plus `USAGE ON *.*`. `--skip-lock-tables` does not avoid it. BACKUP therefore failed outright, and SYNC and RE-IMPORT with it, because both take a safety backup first. The API is given an admin login out of band rather than permanently elevating the user it serves requests as: ``` OPS_DB_ADMIN_USER=root OPS_DB_ADMIN_PASSWORD= ``` Both deploy workflows pass these into the app stack from the existing `MYSQL_ROOT_PASSWORD` secret. Host, port and database still come from `DATABASE_URL` — the override changes *who logs in*, never *which server*. With the pair unset the service falls back to the `DATABASE_URL` credentials and logs a warning, which is what local development wants. Two more things the panel's dumps now do, for the same reasons the pre-migrate backup does them (see `deploy/scripts/pre-migrate-backup.mjs`): - **`--set-gtid-purged=OFF`, but only when the dumper supports it.** galactus is the replication *source* with GTID on, so on a MySQL client this flag is what keeps every dump from embedding `SET @@GLOBAL.GTID_PURGED` and becoming unrestorable onto the server it came from — which is precisely what the restore screen exists to do. The panel, however, dumps from *inside the API container*, where Alpine's `mysql-client` is MariaDB's: there `mysqldump` is a shim over `mariadb-dump`, the flag does not exist, and passing it failed every backup with `mysqldump: unknown variable 'set-gtid-purged=OFF'`. So the panel probes `mysqldump --help` and passes the flag only if it is advertised, invoking `mariadb-dump` directly otherwise (MariaDB writes no GTID state unless asked with `--gtid`, so there is nothing to suppress). The pre-migrate backup keeps the flag unconditionally — it runs in a real `mysql:8.4` image, not in the API container. - **`set -o pipefail` and a `CREATE TABLE` count.** `mysqldump | gzip` reports gzip's exit status, and a `mysqldump` that dies on its first statement still produces a ~372-byte perfectly valid archive that passes `gzip -t`. Without both checks a failed backup was recorded as a successful one and listed as an ordinary restore point. A dump that fails now deletes its own output. ## Known caveats in the deploy path - The pre-migrate backup step sets `NODE_TLS_REJECT_UNAUTHORIZED=0` because Portainer serves a self-signed certificate. It is scoped to that one step, which talks to nothing but Portainer. Replacing the certificate and dropping the flag is the real fix. - The runner lives on cubex and must reach the target host's Portainer (9443). It should also reach MySQL (3306) for the migrate step, but that is no longer load-bearing — dispatch with `skip_migrate: true` and the api container applies the migrations itself at start. - `bootstrap: true` lets the pre-migrate backup be skipped when no API container exists yet. Use it for a first-ever deploy only — it is the one switch that lets a migration run with no restore point. - **The API container sends mail on a timer.** Two sweeps run inside it (renewal avisos, on by default at 06:00 America/Tijuana; the servicios run-all, off by default) — see [`MASS_EMAIL_NOTIFICATIONS.md`](MASS_EMAIL_NOTIFICATIONS.md). Two consequences for deploys: the cadence lives in `app_settings`, so it **survives a redeploy and is not restored by rolling back an image**, and running more than one API replica would double-fire the servicios sweep, which has no DB lock (the pólizas one does). Keep it single-replica. - `SES_*` is optional to deploy — the preflight only warns — but the production image sets `NODE_ENV=production`, which disables the stdout dev fallback. A blank SES config therefore makes every send fail and log `FAILED`. The secrets were created 2026-08-02; the preflight warning on the next run is what confirms the names are right. Two things it cannot check: that `SES_FROM` is a **verified identity** in `SES_REGION`, and that the account is **out of the SES sandbox** (in sandbox, delivery is restricted to verified recipients, which would fail a real sweep while looking correctly configured).