Prod came up with nobody able to log in, in two separate ways.
1. No sign-in account exists. `prisma migrate deploy` creates tables, never
rows, and nothing in the deploy path seeds one — deliberately, since making
an administrator should not be a side effect of shipping code. But
apps/api/scripts was not in the runtime image either, so the only way to
create the first account was to run the script from a developer machine
against a production DATABASE_URL. Ship scripts/ in the image so it can be
run on the host with docker exec. Still never run automatically.
2. Login could not establish a session at all. cookie.secure followed NODE_ENV,
the image sets NODE_ENV=production, and the app is served over plain HTTP —
express-session then silently emits NO Set-Cookie header. POST /auth/login
still answered 200 with the full user object, no session was created, every
later request 403'd, and the UI would have looped back to /login. It reads
as an auth bug and is really a transport mismatch.
The flag is now driven by SESSION_COOKIE_SECURE, still defaulting to
NODE_ENV. An EMPTY value counts as unset rather than false, because compose
turns an absent `${SESSION_COOKIE_SECURE:-}` into the empty string and the
naive check would have quietly dropped Secure on any deployment that merely
passed the variable through.
galactus sets it to "false". That is acceptable ONLY because the host is
reachable exclusively over Tailscale, so WireGuard already encrypts the
wire. It must go back to "true" when the app is served over TLS or exposed
off-tailnet; behind a TLS-terminating proxy, set trust proxy instead.
Verified against live prod: seeded an admin, POST /auth/login returns 200 with
full ADMIN abilities, a wrong password is rejected with 401, and no Set-Cookie
was present before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first successful galactus deploy came up all-green while the web tier was
running a build from two commits earlier. The registry held web:latest from
3ff56e6; the host still had a web:latest cached from 4ee7ec7; the deploy
reported success and served the old one. The API was only current because it
had been pulled by hand during earlier debugging.
Two independent failures, both fixed here.
1. Images are not pulled. The deploy action's `pull: true` does not reliably
refresh an already-cached moving tag on a standalone endpoint. Added a
Pull images step (deploy/scripts/pull-images.mjs) that pulls each image
through Portainer's Docker API with registry credentials and fails the
deploy if a pull fails — note the endpoint answers 200 even when the pull
errored, so the stream body has to be inspected, not just the status.
2. The drift check could not see it. Both the verify step and the web footer
compared APP_VERSION, but on a branch build BOTH tiers report "master", so
equality proved nothing. They now compare gitSha, which is the only field
that differs between two builds of the same branch. api and web come from
one matrix run, so a difference can only mean an image was not replaced.
This needed a /version on the web tier too — previously its build identity
was only readable by scraping window.__APP_BUILD__ out of the HTML.
pull-images.mjs builds the X-Registry-Auth header as URL-safe base64 WITH
padding: Node's "base64url" omits the padding and Portainer's Go decoder
rejects it with "Illegal base64 data at input byte N".
Verified against galactus: pulls both images, and exits non-zero on a
nonexistent tag.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With the image fixed, the API got as far as connecting and then died with
Prisma P1001 "can't reach database server". The cause is DNS, not routing.
galactus runs systemd-resolved, whose 127.0.0.53 stub is unreachable from
inside a container, so Docker falls back to the upstream resolver in
/run/systemd/resolve/resolv.conf — the LAN router, which knows nothing about
the tailnet. Verified from a probe container on galactus: resolving
galactus.tail01aa2.ts.net fails outright, while `nc 100.103.77.46 3306` is
OPEN. Only the lookup was broken.
Pin the api and web services to Tailscale's own resolver (100.100.100.100,
the same anycast address on every tailnet) with this tailnet's search suffix.
Both are overridable via TAILSCALE_DNS / TAILNET_SUFFIX. db and minio need
nothing — they make no outbound calls.
Verified end to end: the published image, unmodified, with only these DNS
settings, boots on galactus against the real database and serves
/health {"status":"ok"}
/version {"service":"api","version":"master","gitSha":"3ff56e6b..."}
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the gap between "what tag did I deploy" and "what is actually running",
and gives the schema a history that can be reasoned about across releases.
Migrations
- Baseline the existing schema as 0000_init (migrate diff --from-empty). The
schema had only ever been applied with `prisma db push`, so no history
existed and schema state was disconnected from app version. Existing
databases must be baselined once with `migrate resolve --applied 0000_init`;
the workflows print this remedy on P3005.
- Run `prisma migrate deploy` as a deploy STEP, not the container CMD — as a
CMD, N replicas would race each other applying the same migration.
Version reporting
- GET /version on the API reports the APP_VERSION / GIT_SHA / BUILD_DATE that
build.yml already baked into both images but nothing ever read.
- The web footer shows the web build and flags an api/web mismatch. The two
cannot drift at build time (one matrix run) but can at deploy time.
- Both deploy workflows now fail if the running API does not report the tag
that was dispatched — a stack naming a tag is not proof of what is running.
- scripts/set-version.mjs stamps every package.json, which had all sat at
0.1.0 while real releases shipped as v1.x.
Pre-migrate backup
- deploy/scripts/pre-migrate-backup.mjs dumps the database from INSIDE the
still-running old API container over Portainer's Docker API, so the file
lands in the volume the Operaciones restore screen reads. A dump taken on
the CI runner would be unreachable by the only restore path we have.
Verifies the artefact with `gzip -t` before letting the migration proceed.
galactus
- deploy/galactus/*.compose.yml: standalone-Docker ports of the Swarm stacks.
Plain compose silently ignores `deploy:`, so restart_policy becomes
`restart: unless-stopped` — without it nothing returns after a host reboot.
- .gitea/workflows/deploy-galactus.yml drives endpoint 3 with its own secrets.
Fixes
- deploy.yml passed `endpoint_id` and `pull_image` to
cssnr/portainer-stack-deploy-action, which has no such inputs (they are
`endpoint` and `pull`). The endpoint was silently never set.
docs/DEPLOY_AND_MIGRATIONS.md documents expand/contract as the rule for schema
changes: Prisma has no down-migrations, so a code rollback never rolls the
schema back, and restoring the replication master from a dump diverges every
replica.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add the missing api/web deployment path on top of the existing image build CI.
- deploy/jorgecuadros-app.stack.yml: PROD app stack (api + web) pulling the
git.mancinas.io registry images. Does not ship mysql/minio (separate stacks);
API reaches them via DATABASE_URL / S3_ENDPOINT. API pinned to the
jorgecuadros_db node for stable ingest/backup volumes; web is stateless.
- deploy/jorgecuadros-app.env.example: documented stack env template.
- .gitea/workflows/deploy.yml: manual (workflow_dispatch) deploy to Portainer
via cssnr/portainer-stack-deploy-action. Inputs: image tag + scope
(app = web+api, full = db+minio+app, applied db->minio->app).
Make the web API origin runtime-configurable instead of build-baked: the root
layout injects window.__API_ORIGIN__ from the API_ORIGIN env (force-dynamic) and
lib/api.ts resolves it at runtime, so one built image serves any deployment.
Also: dev.sh to run both dev servers (frees stale ports first) and move local
dev to ports web 4500 / api 4501.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-minio.stack.yml: S3-compatible object storage (MinIO) for
the platform's document blobs, deployed to the cubex Swarm with the same
statefulness rules as the DB stack (named volume, pinned to the labeled node).
Parametrized for dev/prod as two stacks (dev API 9100/console 9101, prod
9000/9001). Dev deployed + bucket jorgecuadros-documents created.
migration/blob_extract.py: re-reads the LONGBINARY columns via mdb-export
-b hex (staging used -b strip), carves the embedded file out of the Access
OLE wrapper by locating its magic bytes (JPEG/PNG/PDF/GIF/TIFF) and trimming
trailing OLE junk, uploads to MinIO, and writes service_documents /
policy_documents pointer rows. Row->parent alignment uses mdb-export's
deterministic order (== staged _row_num) for policies and numer_id for
properties. Idempotent (truncate doc tables + overwrite by deterministic key);
--limit/--tables for test passes.
Validated on a limited pass: carved blobs are valid JPEGs (ffd8ff..ffd9)
correctly linked to their policies.
requirements.txt: add boto3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-db.stack.yml: canonical internal MySQL for the platform,
targeting the Portainer local endpoint on cubex (3-node Swarm). Parametrized
(MYSQL_PORT / MYSQL_SERVER_ID) so one file deploys both environments as two
stacks with Swarm-namespaced volumes:
dev -> jorgecuadros-dev-db :3307 server-id 11
prod -> jorgecuadros-prod-db :3306 server-id 1 (replication source)
Swarm-correct: named volume (no bind mount), pinned to one node via
node.labels.jorgecuadros_db==true (cubex labeled), binlog+GTID enabled from
the start so prod can be the VPS replication source without reconfigure.
Dev deployed and verified: MySQL 8.4.10 reachable at 192.168.4.212:3307,
gtid_mode ON, database jorgecuadros present. Secrets live in gitignored
deploy/.env.dev, injected via Portainer stack env at deploy time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rmancinasandClaude Opus 5