jorgecuadros-platform/docs/DEPLOY_AND_MIGRATIONS.md

# Releasing, deploying, and changing the schema

How a version gets from this repo onto a server, and the one rule that keeps
rollbacks possible.

## The short version

Dispatch **Cut release** from the Actions tab and pick `patch`, `minor` or
`major` (or `explicit` plus a number). It stamps every `package.json`, commits
`chore(release): vX.Y.Z`, tags, and pushes both refs in one go. It refuses a
version that already exists as a tag, and refuses a no-op bump.

The equivalent by hand, if you would rather cut it locally:

```bash
pnpm version:set 1.2.0                       # stamp every package.json
git commit -am "chore(release): v1.2.0"
git tag v1.2.0 && git push origin master v1.2.0
```

Either way that push triggers `.gitea/workflows/build.yml`, which builds **both** images in
one matrix run and publishes:

| tag pushed | image tags produced |
| --- | --- |
| `v1.2.0` | `1.2.0`, `1.2`, `sha-<short>` |
| push to `master` | `master`, `sha-<short>`, `latest` |

Then dispatch a deploy from the Actions tab:

- **galactus** (office server, standalone Docker) — *Deploy to galactus*
- **cubex** (3-node Swarm) — *Deploy to Portainer*

> **The `v` is not part of the image tag.** `docker/metadata-action`'s
> `{{version}}` strips it. Git tag `v1.2.0`, dispatch `1.2.0`. Dispatching
> `v1.2.0` deploys nothing that exists.

**Cut release needs a `RELEASE_TOKEN` secret** — a Gitea PAT with
`write:repository`. It does not use the built-in Actions token on purpose:
whether a push made with that token re-triggers `build.yml` depends on the Gitea
version, and a release that quietly publishes no images is worse than one that
fails outright. If the build somehow does not start, `build.yml` has
`workflow_dispatch` — run it against the new tag by hand.

Because api and web are built from one matrix run, they cannot drift at build
time. They *can* drift at deploy time if a stack is applied with only one image
moved — the web footer shows both versions and flags a mismatch, and the deploy
workflow's last step fails if the API does not report the tag you dispatched.

## What a deploy actually does

1. **db + minio** — `scope: full` only. Idempotent; data lives on named volumes.
2. **Pre-migrate backup** — `deploy/scripts/pre-migrate-backup.mjs` runs
   `mysqldump` *inside the still-running old API container*, via Portainer's
   Docker API. The file lands in that container's `BACKUP_DIR` volume as
   `pre-migrate-<tag>-<timestamp>.sql.gz`, which is exactly what the
   **Operaciones** admin screen lists and can restore. A dump taken on the CI
   runner would be unreachable by the only restore path the platform has.
3. **`prisma migrate deploy`** — as a workflow *step*, so the schema moves
   while the OLD code is still serving, which is the order expand/contract is
   designed around. **The api container repeats this at start** (below); the
   command is idempotent, so on the normal path the container's run is a
   no-op.
4. **app** — the new api + web images.
5. **Verify** — `GET /version` on the running API must report the dispatched
   tag.

Rollback is `tag: 1.1.9` re-dispatched. **That rolls back code only.** The
schema stays where it is. Which brings us to the rule.

## The rule: expand / contract

Prisma has no down-migrations. There is no `prisma migrate down`, and there
never will be. So a schema change that the *previous* release cannot tolerate
turns a 30-second rollback into a restore-from-backup outage.

**Every schema change must leave the previous release working.** Split anything
destructive across two releases:

| | Release N (expand) | Release N+1 (contract) |
| --- | --- | --- |
| Rename a column | add the new column, write to both, read the old | drop the old column |
| Drop a column | stop reading and writing it in code | drop it |
| Add a required column | add it nullable (or with a default), backfill | make it `NOT NULL` |
| Split a table | create the new table, dual-write | stop writing the old, drop it |
| Add an enum value | add the value; old code must not choke on unknowns | start emitting it |

Ship N, let it soak, *then* ship N+1. If N has to be rolled back you just
re-dispatch the old tag — the expanded schema still satisfies it.

Restoring from the pre-migrate dump is the **emergency lever, not the routine
path**, and on galactus it is worse than it sounds: galactus is the replication
master, DDL replicates through the binlog, and restoring the master from a dump
diverges every replica. GTIDs will not line up and each replica needs a full
re-seed. Assume a restore is a multi-hour, whole-topology event.

## Migration history

`packages/database/prisma/migrations/0000_init/` is a **baseline**. It is the
full schema as it stood on 2026-07-30, generated with:

```bash
prisma migrate diff --from-empty \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script
```

Until then the schema had only ever been applied with `prisma db push`, so no
history existed and the schema state was disconnected from the app version.

### One-time, on every database that already exists

`0000_init` describes tables those databases already have, so `migrate deploy`
would fail with **P3005 "the database schema is not empty"**. Mark it applied
instead of applying it — this writes a `_prisma_migrations` row and changes no
data:

```bash
DATABASE_URL=<the database> npx prisma@5 migrate resolve \
  --applied 0000_init --schema packages/database/prisma/schema.prisma
```

Do this once per database (prod, dev, any local copy). Verify first that the
live schema really does match the baseline — this should print an empty
migration:

```bash
prisma migrate diff --from-url "$DATABASE_URL" \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script
```

If it prints actual statements, the live database has drifted from
`schema.prisma`. Reconcile *before* baselining, or the first real migration
will fail against a schema Prisma believes it already knows.

### From here on

```bash
# edit schema.prisma, then:
pnpm --filter @jorgecuadros/database exec prisma migrate dev --name add_foo
```

Commit the generated `migrations/<timestamp>_add_foo/` directory. `db push` is
now a local-scratch tool only — using it against a database with history
desynchronises it from `_prisma_migrations`.

If you hand-write a migration instead of generating one, check it against what
Prisma would have produced before committing — a hand-written file that drifts
from `schema.prisma` fails on the *next* deploy, not this one:

```bash
prisma migrate diff \
  --from-schema-datamodel <schema.prisma at the previous commit> \
  --to-schema-datamodel packages/database/prisma/schema.prisma --script
```

## Migrations also run at container start

`docker/api-entrypoint.sh` is the api image's `ENTRYPOINT`. It runs
`prisma migrate deploy` and only then `exec`s the API. **If the migration
fails the container exits non-zero and the API never listens.**

That is the point. Serving against a schema that does not match the code is
worse than being down, because the failures are partial and silent — a write
to a column that does not exist yet breaks one feature while the rest of the
app looks healthy.

This does not replace the workflow step, which still runs first and against
the old code. It covers what that step cannot:

- **`skip_migrate: true`.** Previously that left the schema behind with no
  further safety net, and the mismatch surfaced later as a runtime error. Now
  it just moves the migration into the container, so it is a safe choice when
  the runner cannot reach MySQL.
- **Restarts that never touch the workflow** — `restart: unless-stopped`
  bringing the stack back after a host reboot, or a stack re-applied by hand
  in Portainer.

Behaviour worth knowing:

| | |
|---|---|
| `RUN_MIGRATIONS=false` | Skip and start anyway. Plumbed through both app stack files. For when the schema is being moved by hand. |
| `DATABASE_URL` unset | Refuses to start (it would have failed at Nest boot anyway, but this says why). |
| Cannot reach the database (**P1001**) | Retries, default 20 × 3s. Covers a cold `db` container and galactus's MagicDNS lookup right after a reboot. `MIGRATE_MAX_ATTEMPTS` / `MIGRATE_RETRY_SECONDS` tune it. |
| Any other failure | Exits at once. Retrying a broken migration only delays the same error; **P3005** additionally prints the `migrate resolve --applied 0000_init` hint. |

**On replicas.** Both stacks are `replicas: 1` and must stay that way for an
unrelated reason (the servicios email sweep has no DB lock — see the caveats
below). If that ever changes, concurrent `migrate deploy` runs are safe on
their own: Prisma takes a database advisory lock, so the others block and then
find nothing pending. They would each pay the wait at startup, not corrupt
anything.

The prisma CLI has to be present in the runtime layer for any of this. The
image copies `/repo/node_modules` wholesale so it already is, and the
Dockerfile **asserts it at build time** — a missing CLI breaks the image
build rather than a production boot.

## galactus vs cubex

`galactus` is standalone Docker (Portainer endpoint **3**), `cubex` is a 3-node
Swarm (endpoint **2**). They need different compose files because **plain
compose silently ignores Swarm's `deploy:` keys** rather than erroring:

| | Swarm (`deploy/*.stack.yml`) | standalone (`deploy/galactus/*.compose.yml`) |
| --- | --- | --- |
| restart | `deploy.restart_policy` | `restart: unless-stopped` — **without this nothing comes back after a host reboot** |
| placement | `node.labels.jorgecuadros_db == true` | dropped, one host |
| ports | `{mode: ingress}` long syntax | `"3306:3306"` |
| `depends_on` | ignored by Swarm | honoured, with `condition: service_healthy` |
| volumes | named | named (unchanged — the pinning hazard was a Swarm problem) |

Keep the two sets in sync when either changes.

On both hosts, cross-stack traffic goes over the **host address**, not compose
service DNS: db, minio and app are three separate stacks, so three separate
networks. `DATABASE_URL` and `S3_ENDPOINT` name the host and its published
port. Do not "simplify" them to `mysql:3306`.

### galactus is addressed by MagicDNS, and containers need help resolving it

galactus is Tailscale-only once it is installed in the office, so every URL
names `galactus.tail01aa2.ts.net`. Its LAN IP is a DHCP lease and has already
drifted once — never put a `192.168.4.x` address in a secret.

Containers on galactus cannot resolve that name by default. The host runs
systemd-resolved, whose `127.0.0.53` stub is unreachable from inside a
container, so Docker falls back to the upstream resolver listed in
`/run/systemd/resolve/resolv.conf` — the LAN router, which knows nothing about
the tailnet. Routing to `100.x` works fine; only the *lookup* fails, and the
symptom is Prisma **P1001 "can't reach database server"** on a container that
otherwise started cleanly.

`deploy/galactus/jorgecuadros-app.compose.yml` therefore pins the resolver:

```yaml
dns:        [100.100.100.100]        # Tailscale's fixed anycast MagicDNS address
dns_search: [tail01aa2.ts.net]       # this tailnet's suffix
```

Both are overridable (`TAILSCALE_DNS`, `TAILNET_SUFFIX`) if the tailnet changes.
Browser-facing origins need none of this — those names resolve on the client.

## Replication

galactus's MySQL is the **master**; every other MySQL in the estate is a
replica. Consequences that bite:

- `server-id` must be unique across the whole topology (prod `1`, cubex dev
  `11`). A duplicate breaks replication silently.
- GTID is on from first boot, so replicas attach with `SOURCE_AUTO_POSITION=1`.
- `binlog_expire_logs_seconds` is raised to 60 days in the galactus compose file
  (`MYSQL_BINLOG_EXPIRE_SECONDS`). MySQL 8.4 defaults to 30 days; a replica
  offline longer than the retention needs a full re-seed.

Still open, and **not** handled by anything in this repo:

- No replication user with `REPLICATION SLAVE` granted exists yet.
- Nothing sets `read_only` / `super_read_only` on the replicas, so a stray write
  to a replica will diverge it.
- The channel to the VPS crosses the public internet. It needs a tunnel or TLS —
  do not publish raw 3306.

## Seeding the first sign-in account

A freshly migrated database has a schema and **no users**, so nobody can log in.
`prisma migrate deploy` creates tables, never rows; nothing in the deploy path
seeds an account, by design — creating an administrator should be a deliberate
act, not a side effect of shipping code.

`apps/api/scripts/seed-user.mjs` ships inside the API image. On the target host:

```bash
docker exec -e SEED_PASSWORD='<a strong password>' \
  <api-container> node apps/api/scripts/seed-user.mjs
```

Defaults are `admin@jorgecuadros.local` / `ChangeMe!2026` / role `ADMIN`,
overridable with `SEED_EMAIL`, `SEED_PASSWORD`, `SEED_NAME`. **Do not accept the
default password on anything but a dev database** — it is published in this
repo's README. The script upserts by email, so re-running is safe, but it also
**resets the password of an existing account**.

## The session cookie and TLS

`SESSION_COOKIE_SECURE` controls the `Secure` flag on the session cookie. It
defaults to on in production, and it must be explicitly `"false"` for a
deployment served over plain HTTP.

This is not cosmetic. express-session silently declines to emit a `Secure`
cookie over an unencrypted connection: no `Set-Cookie` header is sent at all,
`POST /auth/login` still answers `200` with the user object, no session is
established, every subsequent request gets `403`, and the UI bounces back to
`/login` in a loop. It looks like an auth bug and is really a transport
mismatch.

galactus runs with `SESSION_COOKIE_SECURE=false`, which is acceptable **only**
because it is reachable exclusively over Tailscale — WireGuard already encrypts
the wire, so the cookie never crosses an untrusted network. Turn it back on the
moment the app is served over TLS or reachable off-tailnet. Behind a
TLS-terminating reverse proxy, set `trust proxy` on the Nest app instead of
disabling the flag.

## The MySQL client inside the API image

Alpine's `mysql-client` package is **MariaDB's** client, and it installs an
empty `/usr/lib/mariadb/plugin`. It therefore cannot speak
`caching_sha2_password`, which is MySQL 8.4's default and effectively only auth
method, and every `mysqldump`/`mysql` call from the container fails with:

```
ERROR 1045: Plugin caching_sha2_password could not be loaded:
  ... /usr/lib/mariadb/plugin/caching_sha2_password.so: No such file or directory
```

`mariadb-connector-c` supplies that plugin and is installed in
`docker/api.Dockerfile` for exactly this reason — do not drop it as an unused
dependency. It affects far more than the deploy backup: the entire
**Operaciones** panel (backup, restore, sync, re-import) shells out to these
binaries, so without it none of those work in a container either. The feature
had only ever been exercised with the API running on a developer machine, where
the Oracle client is installed, which is why this went unnoticed until the
first containerised deploy.

## The Operaciones panel needs its own database login

The panel's four jobs all shell out to `mysqldump`/`mysql`, and they cannot do
so as the application user. `mysqldump --single-transaction` issues
`FLUSH TABLES`, which requires the **global** `RELOAD` privilege; the MySQL
image grants the app user only `ALL PRIVILEGES ON jorgecuadros.*` plus
`USAGE ON *.*`. `--skip-lock-tables` does not avoid it. BACKUP therefore failed
outright, and SYNC and RE-IMPORT with it, because both take a safety backup
first.

The API is given an admin login out of band rather than permanently elevating
the user it serves requests as:

```
OPS_DB_ADMIN_USER=root
OPS_DB_ADMIN_PASSWORD=<MYSQL_ROOT_PASSWORD>
```

Both deploy workflows pass these into the app stack from the existing
`MYSQL_ROOT_PASSWORD` secret. Host, port and database still come from
`DATABASE_URL` — the override changes *who logs in*, never *which server*. With
the pair unset the service falls back to the `DATABASE_URL` credentials and logs
a warning, which is what local development wants.

Two more things the panel's dumps now do, for the same reasons the pre-migrate
backup does them (see `deploy/scripts/pre-migrate-backup.mjs`):

- **`--set-gtid-purged=OFF`, but only when the dumper supports it.** galactus is
  the replication *source* with GTID on, so on a MySQL client this flag is what
  keeps every dump from embedding `SET @@GLOBAL.GTID_PURGED` and becoming
  unrestorable onto the server it came from — which is precisely what the
  restore screen exists to do. The panel, however, dumps from *inside the API
  container*, where Alpine's `mysql-client` is MariaDB's: there `mysqldump` is a
  shim over `mariadb-dump`, the flag does not exist, and passing it failed every
  backup with `mysqldump: unknown variable 'set-gtid-purged=OFF'`. So the panel
  probes `mysqldump --help` and passes the flag only if it is advertised,
  invoking `mariadb-dump` directly otherwise (MariaDB writes no GTID state
  unless asked with `--gtid`, so there is nothing to suppress). The pre-migrate
  backup keeps the flag unconditionally — it runs in a real `mysql:8.4` image,
  not in the API container.
- **`set -o pipefail` and a `CREATE TABLE` count.** `mysqldump | gzip` reports
  gzip's exit status, and a `mysqldump` that dies on its first statement still
  produces a ~372-byte perfectly valid archive that passes `gzip -t`. Without
  both checks a failed backup was recorded as a successful one and listed as an
  ordinary restore point. A dump that fails now deletes its own output.

## Known caveats in the deploy path

- The pre-migrate backup step sets `NODE_TLS_REJECT_UNAUTHORIZED=0` because
  Portainer serves a self-signed certificate. It is scoped to that one step,
  which talks to nothing but Portainer. Replacing the certificate and dropping
  the flag is the real fix.
- The runner lives on cubex and must reach the target host's Portainer (9443).
  It should also reach MySQL (3306) for the migrate step, but that is no longer
  load-bearing — dispatch with `skip_migrate: true` and the api container
  applies the migrations itself at start.
- `bootstrap: true` lets the pre-migrate backup be skipped when no API container
  exists yet. Use it for a first-ever deploy only — it is the one switch that
  lets a migration run with no restore point.
- **The API container sends mail on a timer.** Two sweeps run inside it
  (renewal avisos, on by default at 06:00 America/Tijuana; the servicios
  run-all, off by default) — see
  [`MASS_EMAIL_NOTIFICATIONS.md`](MASS_EMAIL_NOTIFICATIONS.md). Two
  consequences for deploys: the cadence lives in `app_settings`, so it
  **survives a redeploy and is not restored by rolling back an image**, and
  running more than one API replica would double-fire the servicios sweep,
  which has no DB lock (the pólizas one does). Keep it single-replica.
- `SES_*` is optional to deploy — the preflight only warns — but the production
  image sets `NODE_ENV=production`, which disables the stdout dev fallback. A
  blank SES config therefore makes every send fail and log `FAILED`. The
  secrets were created 2026-08-02; the preflight warning on the next run is
  what confirms the names are right. Two things it cannot check: that
  `SES_FROM` is a **verified identity** in `SES_REGION`, and that the account
  is **out of the SES sandbox** (in sandbox, delivery is restricted to verified
  recipients, which would fail a real sweep while looking correctly
  configured).