Five documents asserted the SES_* secrets were unset in Gitea. They now
exist, so all five are corrected rather than leaving the claim to rot in
whichever one a reader opens first.
Replaces the blocker with the two things creating the secrets does NOT
establish, since both fail in ways that look identical to a missing
config: SES_FROM must be a verified identity in SES_REGION, and the
account must be out of the SES sandbox — in sandbox SES only delivers to
verified recipients, so a sweep across 815 policyholders would fail
almost every send while the configuration reads as correct.
Recommends running the first sweep with debug on, which diverts every
recipient and, on the pólizas side, leaves the avisos pending so a failed
test consumes nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The docs still described the state before the last five commits: the
insurance spec called for a `@Cron` literal and a manual mark-as-sent
mutation, PLAN.md had step 12 as "NOT STARTED", and README's module and
route lists predated seven modules.
- MASS_EMAIL_NOTIFICATIONS.md: new "Send flags", "API surface" and
"Scheduled runs" sections; "Cron (future)" removed — it exists. The
flags table says which flags apply where, and why a debug renewal send
must skip both the RenewalNotice row and `lastSuccessfulAt`.
- INSURANCE_FEATURES_SPEC.md: §1 BUILT note listing the three places the
build diverged from the spec; §1.1 and §1.4 marked superseded in place
rather than deleted, so the reasoning stays readable.
- PLAN.md: step 12 renewal emails DONE with the divergences; status
paragraph rewritten.
- README.md: current module/route lists, plus a "Scheduled jobs" section —
a reader cloning this repo had no way to know the API sends mail on a
timer.
- DEPLOY_AND_MIGRATIONS.md: the cadence lives in app_settings and survives
an image rollback, and the servicios sweep has no multi-replica lock.
- RESUME.md: session record for the whole notificaciones arc.
- RENEWAL_NOTICES.md: pointer that this is the legacy record, not what
shipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every backup on galactus died with:
mysqldump: unknown variable 'set-gtid-purged=OFF'
respaldo incompleto eliminado
Alpine's mysql-client is MariaDB's, so `mysqldump` inside the API
container is a shim over `mariadb-dump`, which has no --set-gtid-purged.
That took out BACKUP and, because they take a safety dump first, SYNC
and REIMPORT too.
Probe `mysqldump --help` and pass the flag only when it is advertised,
calling `mariadb-dump` directly otherwise — MariaDB writes no GTID state
unless asked with --gtid, so there is nothing to suppress. Testing
whether mariadb-dump merely exists would be wrong: on a host carrying
both clients it would shadow a perfectly good MySQL mysqldump.
The probe uses a command substitution rather than `--help | grep -q`
because PIPEFAIL is in effect for these commands and grep closing the
pipe early would report a supported flag as unsupported.
pre-migrate-backup.mjs is unaffected — it dumps from a real mysql:8.4
image, not from the API container.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stamps every package.json, commits chore(release): vX.Y.Z, tags and pushes
both refs in one dispatch — patch/minor/major, or an explicit number. Cutting
a release from a laptop is how a manifest bump gets forgotten or a tag lands
on an unpushed commit; the only input here is the number.
Guards: refuses a version that already exists as a tag (releases are
immutable), a no-op bump, a leading `v`, and a malformed number. Checkout is
full-depth because the duplicate-tag check is meaningless against a shallow
clone.
Pushes with a RELEASE_TOKEN PAT rather than the built-in Actions token —
whether a push made with that token re-triggers build.yml depends on the Gitea
version, and a release that quietly publishes no images is worse than one that
fails outright.
Builds and deploys stay separate: the tag push triggers build.yml, and
deploying remains a deliberate dispatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Operaciones panel (backup, restore, sync, re-import) shelled out to
mysqldump as the application user, parsed straight out of DATABASE_URL.
`--single-transaction` issues FLUSH TABLES, which needs the global RELOAD
privilege, and the app user is granted only ALL ON jorgecuadros.* plus
USAGE ON *.*. BACKUP failed outright; SYNC and REIMPORT failed with it,
since both take a safety backup first.
An admin credential is now supplied out of band via OPS_DB_ADMIN_USER /
OPS_DB_ADMIN_PASSWORD, mirroring what deploy/scripts/pre-migrate-backup.mjs
already does, rather than permanently elevating the user the API serves
requests as. Host, port and database still come from DATABASE_URL, so the
override can only change who logs in, never which server. Unset, it falls
back to the DATABASE_URL credentials and warns — local development is
unaffected.
Two defects in the dumps themselves, both shared with the deploy backup
before it was rewritten:
- No --set-gtid-purged=OFF. The production server is the replication source
with GTID on, so every dump embedded SET @@GLOBAL.GTID_PURGED and was
unrestorable onto the server it came from — the one thing the restore
screen is for.
- The pipeline's exit status was gzip's, and gzip succeeded. A mysqldump
that died on its first statement left a small, perfectly valid archive
that the job recorded as SUCCESS and the restore screen listed as an
ordinary restore point. Dumps now run under `set -o pipefail`, assert a
CREATE TABLE count, and delete their own output on failure. Verified with
a stubbed mysqldump: a failing dump exits 1, surfaces the real error,
removes the partial file, and — critically — stops SYNC/REIMPORT before
the ETL touches anything.
Restores gained pipefail too: a corrupt archive made gunzip fail while
mysql, fed a truncated stream, could still exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pre-migrate backup failed with "mysqldump exited 2" and nothing else.
Reproduced on the host with stderr captured:
ERROR 1045: Plugin caching_sha2_password could not be loaded:
/usr/lib/mariadb/plugin/caching_sha2_password.so: No such file or directory
Alpine's `mysql-client` is MariaDB's client and ships an EMPTY plugin
directory, so it cannot perform caching_sha2_password — MySQL 8.4's default and
effectively only auth method. `mariadb-connector-c` provides the plugin.
This was never about the deploy backup alone. Every mysqldump/mysql call from
the API container was broken, which means the whole Operaciones panel — backup,
restore, sync, re-import — could not work in a container. It went unnoticed
because that feature had only ever been run with the API on a developer
machine, where the Oracle client is installed. Verified after the fix: dump
exits 0, gzip valid, 31 CREATE TABLEs.
Also fixed, both found while chasing the above:
- The backup script reported an exit code and nothing else, because a detached
exec captures no output — which is precisely why this needed a manual
reproduction. mysqldump's stderr is now redirected to a file and read back
through a short attached exec on failure, so the deploy log states the cause.
Verified against live prod: the log now carries the 1045 line itself.
- Listing ONLY 100.100.100.100 as the containers' resolver costs them public
DNS, since MagicDNS does not forward upstream unless the tailnet defines
global nameservers. Nothing at runtime needed it, but `apk` inside the
container stopped resolving, and anything outbound would have too. A public
fallback resolver is now listed after MagicDNS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prod came up with nobody able to log in, in two separate ways.
1. No sign-in account exists. `prisma migrate deploy` creates tables, never
rows, and nothing in the deploy path seeds one — deliberately, since making
an administrator should not be a side effect of shipping code. But
apps/api/scripts was not in the runtime image either, so the only way to
create the first account was to run the script from a developer machine
against a production DATABASE_URL. Ship scripts/ in the image so it can be
run on the host with docker exec. Still never run automatically.
2. Login could not establish a session at all. cookie.secure followed NODE_ENV,
the image sets NODE_ENV=production, and the app is served over plain HTTP —
express-session then silently emits NO Set-Cookie header. POST /auth/login
still answered 200 with the full user object, no session was created, every
later request 403'd, and the UI would have looped back to /login. It reads
as an auth bug and is really a transport mismatch.
The flag is now driven by SESSION_COOKIE_SECURE, still defaulting to
NODE_ENV. An EMPTY value counts as unset rather than false, because compose
turns an absent `${SESSION_COOKIE_SECURE:-}` into the empty string and the
naive check would have quietly dropped Secure on any deployment that merely
passed the variable through.
galactus sets it to "false". That is acceptable ONLY because the host is
reachable exclusively over Tailscale, so WireGuard already encrypts the
wire. It must go back to "true" when the app is served over TLS or exposed
off-tailnet; behind a TLS-terminating proxy, set trust proxy instead.
Verified against live prod: seeded an admin, POST /auth/login returns 200 with
full ADMIN abilities, a wrong password is rejected with 401, and no Set-Cookie
was present before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With the image fixed, the API got as far as connecting and then died with
Prisma P1001 "can't reach database server". The cause is DNS, not routing.
galactus runs systemd-resolved, whose 127.0.0.53 stub is unreachable from
inside a container, so Docker falls back to the upstream resolver in
/run/systemd/resolve/resolv.conf — the LAN router, which knows nothing about
the tailnet. Verified from a probe container on galactus: resolving
galactus.tail01aa2.ts.net fails outright, while `nc 100.103.77.46 3306` is
OPEN. Only the lookup was broken.
Pin the api and web services to Tailscale's own resolver (100.100.100.100,
the same anycast address on every tailnet) with this tailnet's search suffix.
Both are overridable via TAILSCALE_DNS / TAILNET_SUFFIX. db and minio need
nothing — they make no outbound calls.
Verified end to end: the published image, unmodified, with only these DNS
settings, boots on galactus against the real database and serves
/health {"status":"ok"}
/version {"service":"api","version":"master","gitSha":"3ff56e6b..."}
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the gap between "what tag did I deploy" and "what is actually running",
and gives the schema a history that can be reasoned about across releases.
Migrations
- Baseline the existing schema as 0000_init (migrate diff --from-empty). The
schema had only ever been applied with `prisma db push`, so no history
existed and schema state was disconnected from app version. Existing
databases must be baselined once with `migrate resolve --applied 0000_init`;
the workflows print this remedy on P3005.
- Run `prisma migrate deploy` as a deploy STEP, not the container CMD — as a
CMD, N replicas would race each other applying the same migration.
Version reporting
- GET /version on the API reports the APP_VERSION / GIT_SHA / BUILD_DATE that
build.yml already baked into both images but nothing ever read.
- The web footer shows the web build and flags an api/web mismatch. The two
cannot drift at build time (one matrix run) but can at deploy time.
- Both deploy workflows now fail if the running API does not report the tag
that was dispatched — a stack naming a tag is not proof of what is running.
- scripts/set-version.mjs stamps every package.json, which had all sat at
0.1.0 while real releases shipped as v1.x.
Pre-migrate backup
- deploy/scripts/pre-migrate-backup.mjs dumps the database from INSIDE the
still-running old API container over Portainer's Docker API, so the file
lands in the volume the Operaciones restore screen reads. A dump taken on
the CI runner would be unreachable by the only restore path we have.
Verifies the artefact with `gzip -t` before letting the migration proceed.
galactus
- deploy/galactus/*.compose.yml: standalone-Docker ports of the Swarm stacks.
Plain compose silently ignores `deploy:`, so restart_policy becomes
`restart: unless-stopped` — without it nothing returns after a host reboot.
- .gitea/workflows/deploy-galactus.yml drives endpoint 3 with its own secrets.
Fixes
- deploy.yml passed `endpoint_id` and `pull_image` to
cssnr/portainer-stack-deploy-action, which has no such inputs (they are
`endpoint` and `pull`). The endpoint was silently never set.
docs/DEPLOY_AND_MIGRATIONS.md documents expand/contract as the rule for schema
changes: Prisma has no down-migrations, so a code rollback never rolls the
schema back, and restoring the replication master from a dump diverges every
replica.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rmancinasandClaude Opus 5