The swarm stack still hard-failed on an unset API_ORIGIN, and both the env
template and the README told the reader to pin it — the exact habit the derived
origin was meant to end. Make it an optional override everywhere, and say that
WEB_ORIGIN is now a list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The browser hard-required API_ORIGIN, so every move of the server — tailnet
today, the 192.168.1.0 office LAN later, a temporary demo domain in between —
meant editing the deploy env and redeploying. Worse, an http:// API origin on a
page served over TLS is blocked outright as mixed active content, which is what
broke the demo on https://jorgecuadros.freakma.com.
The browser now derives the origin from window.location the way a PHP app
would: same host on port 3001 over plain HTTP, or the same-origin /api path
under https (the reverse proxy strips the prefix). API_ORIGIN survives as an
optional override for a deployment that genuinely splits the two hosts, and SSR
still reads process.env because a derived origin is browser-only.
WEB_ORIGIN becomes a comma-separated list to match: one deployment is now
reached under several origins, and a credentialed fetch from an unlisted one
gets no CORS headers and fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every field the replication card showed was self-reported by the replica, and
the two most reassuring ones lie in the same failure. Seconds_Behind_Source
reads 0 when the I/O thread is disconnected — with no incoming event there is
nothing to measure staleness against — and Replica_IO_Running only says the
network thread is alive, not that it is receiving.
Two checks that ask the master instead:
- GTID drift, folded into the polled status. GTID_SUBTRACT(master, replica)
counts transactions the master executed that the replica has not, so a silent
disconnect shows up as a number that climbs instead of a lag that stays 0.
It also isolates transactions carried under the replica's OWN server UUID —
writes that exist nowhere on the master. There are currently 518 of them,
residue of the seed dump load; inert while log_replica_updates is off, and a
real divergence the day anyone promotes that box.
- A full row-by-row comparison behind a button, over the eight tables
my.jorgecuadros.com reads. GTIDs prove the replica applied everything the
master sent; they say nothing about rows changed here by another route, which
is the one failure the rest of the card cannot see.
The comparison hashes CONVERT(col USING binary), not CAST(col AS CHAR). CAST
transcodes into the connection character set, and the two servers do not agree
on it: the client inside the master's container negotiates latin1, the replica's
utf8mb4. Every accented character in a Mexican name, street or note then hashes
differently and the tool reports a permanent mismatch on exactly the tables that
hold free text. Caught by building it and running it — customers.name gave
3344437324815 against 3339150372121 under CAST, and 3339150372121 on both under
CONVERT. All eight tables now match byte for byte.
Verify is POST and audited despite reading nothing: it full-scans both servers,
so a prefetch or a refresh must not be able to start one.
Tests cover the GTID interval arithmetic, which is inclusive at both ends and
easy to get wrong by one in the direction that hides a gap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A full run_all.py pass truncates and rebuilds every table it owns from the
Access extract. That was harmless while the platform was a read-only mirror --
every row came from the extract, so wiping and rebuilding lost nothing. It
stopped being harmless once the platform started minting rows Access has never
heard of: allocated portal NUMids, customers created in the staff UI,
OCR-captured policies, app-booked ledger rows, uploaded documents.
REIMPORT is a button in /operaciones, so that was one click away.
native_guard.py counts what only exists here and exits 3; run_all.py runs it
before the first truncate and stops. Detecting an allocated NUMid needs the
staged Parquet -- the customer holds an ordinary-looking (utilities, DATGRAL,
'1172') ref, so "customer has no refs" cannot see it and only comparing against
the extract can. Missing staging is therefore treated as blocking rather than
as "nothing to protect".
The guard does not teach full mode to preserve anything: --sync already upserts
legacy rows against the existing refs and leaves the rest alone, and rebuilding
that inside full mode would re-implement it. --force-full (checkbox in the
REIMPORT confirm, recorded in the audit log) deletes them deliberately.
Verified against dev: clean before, exit 3 listing utilities/1172 with a
synthetic ref present, clean again after removing it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Customers created in the staff UI had no NUMid and so could not log in to
my.jorgecuadros.com at all: the id is a CustomerLegacyRef row, not a column,
and create() deliberately writes none.
Allocation is a staff action (POST /customers/:id/portal-access, MANAGER)
rather than part of create, because insurance is expected to move to the
platform before utilities and an insurance-only customer has no reason to
spend a utilities id.
The audit that decides which ids are reusable took three passes. "Owns no
rows" matches nobody -- migration gave all 1,171 NUMids a property and a
transaction. "No transaction in N years" also matches nobody -- every customer
carries a synthetic Jan-1 opening-balance row, so everyone looks active this
year. Subtracting that row is what makes dormancy measurable, and it leaves 4
never-used ids and 10 dormant ones on dev. Two further traps are encoded in the
queries: insurance/DATGRAL is a separate id space that reuses the sourceTable
name and runs past 4,000, and ACCOUNT CANCELED is a transaction line type, not
an account state -- all 8 customers carrying it have current-year activity.
Recycling ships switched off (numid.recycleEmpty, default false). Every
reusable id still exists in Access DATGRAL, and a --sync run reassigns refs
with ON DUPLICATE KEY UPDATE customerId, so an id recycled before the utilities
cutover is silently handed back to its Access owner.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BALANCE FORWARD rows are not movements. Access materialized one per
customer per year, dated Jan 1, holding the closing balance of everything
before it — that is what let the portal keep each year in its own table and
still show a correct running balance from one year's rows.
The platform imported those rows AND the real pre-cutover history they
summarize, and every balance aggregate summed the lot. NUMid 501 read
-10,469.29 on the receivables worklist against -14,065.29 on the customer's
own statement and on the legacy portal; the gap was two cash receipts from
2009 and 2012 that the 2026 opening balance had already absorbed.
The scale settles what it is: summed the old way the whole book came to
+20,605,447.86 MXN — the office owing its customers 20.6 million pesos.
Floored, it is -56,855.90. A receivables ledger cannot be 20M in credit.
Adds BALANCE_FLOOR_JOIN + NOT_SUPERSEDED and applies them to balances()
(page and count queries, which must agree), to stats()'s per-currency and
per-domain figures, and to the owing/in-credit split. The four stats()
aggregates moved from Prisma groupBy to raw SQL because groupBy cannot
express a per-customer floor.
statement() takes the same floor as a scalar, which is also what stops
FEE ANUAL and fee15 leaking in. Those are not in
STATEMENT_EXCLUDED_SOURCE_TABLES — that list reproduces legacy's
DATOS2-only `datosfreak` — and they were putting 2,092 pre-cutover fee rows
across 1,062 customers into the statement, skewing it by -5,129,764 against
the number those customers have been quoted for years. Dating rather than
source is the right test: a fee row *after* the opening balance is a real
charge and still counts.
movements() is deliberately left alone. It is a browser over captured rows
— "how much water did we capture in April" — and staff need the historical
rows visible, so it keeps totalling everything, the same asymmetry
NOT_OUTSTANDING already has.
stats() now separates the two questions it was mixing: movements,
ledgerCustomers, crossLineCustomers and the date range stay unfloored
inventory; everything under byCurrency/byDomain is a balance and is floored.
BillingService had no tests. Adds 13 covering the floor's failure modes —
it fails silently, so MIN-vs-MAX, `>` vs `>=`, the NULL branch for customers
with no opening balance, and the join/predicate alias pairing are each
pinned, plus the 501 arithmetic as a regression.
Verified through the real service against the live ledger: balances() and
statement() both return -14,065.29 for 501, matching the portal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
datosfreak's NOPAGO is the legacy "still owed" flag, and the website reads
it directly — account.statement.php splits the statement on NOPAGO = 0 vs
NOPAGO = 1 and renders the latter as "Outstanding Bills Requiring
Attention". transform_transactions.py hardcoded 0, so all 40,421 rows came
across settled and that section renders empty for anyone served off the
platform. Not a missing column: a missing section, with no error.
Only the three DATOS2-shaped tables carry the flag (76 rows set in datos2,
0 in FEE ANUAL and fee15); the EFECTIVO/FM3 cash streams have no such
column and keep the 0 default. Sync mode gets outstanding=VALUES(...) too,
so an additive sync corrects rows already loaded rather than leaving them
settled forever.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deploy was a job inside build.yml gated by
`if: startsWith(github.ref, 'refs/tags/v')`. Gitea draws every job into the
run graph before it evaluates that `if`, so an ordinary push to master showed
a pending "Deploy to galactus" — indistinguishable from prod being about to be
redeployed off an unreleased commit, and the only safe reaction is to cancel
the run, which takes the images down with it.
The gate itself was never wrong (no deploy-galactus run has ever been created
from a branch ref), but a guarantee you cannot see is not much of a guarantee.
`on: push: tags: ["v*"]` in a workflow of its own makes it structural: the
deploy cannot appear on a master build because the workflow does not exist
there.
It replaces `needs: build` by polling the Actions API for the build.yml run at
this tag and requiring it green, so both images are still known to be in the
registry before anything is pulled. AUTO_DEPLOY_GALACTUS still cuts the chain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SYNC ran `run_all.py --sync` without `--stage`, so it depended on staged
Parquet under migration/output. That directory is part of the image, not a
volume, so any redeploy wiped it and the job died on the first transform:
FileNotFoundError: '/repo/migration/output/stg_utilities/datgral.parquet'
Re-staging is also what makes the job's own label true — without it a sync
would replay whatever upload staged last, not the files currently in the
ingest folder.
Staging now counts as a numbered step when it runs, so the Operaciones
progress bar moves during the slowest phase instead of sitting empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cutting a release then had one manual step left: watch build.yml and
dispatch "Deploy to galactus" by hand with the version. Chain it.
build.yml gains a `deploy` job, `needs: build` and gated on
refs/tags/v*, that dispatches deploy-galactus.yml against the tag with
tag=<version> scope=app bootstrap=false skip_migrate=false. `needs`
waits for both matrix legs, so api and web are both in the registry
before prod pulls either — deploy-galactus.yml only pulls, and a
half-pushed pair leaves prod running one new image and one old one.
A dispatch rather than a `workflow_run:` trigger (which Gitea has
supported since 1.24) because deploy-galactus.yml reads
github.event.inputs.* in ten places; under workflow_run all of them are
empty strings, so the deploy would run with no tag. The dispatch keeps
that workflow's contract intact and keeps it hand-runnable, which is
how rollbacks work.
The dispatch is confirmed the same way release.yml confirms the build
started: snapshot the existing deploy-galactus run ids first, then
require a new one to appear. An accepted dispatch that creates no run
is the failure mode that cost v1.0.3 its images, and a plain "is there
a deploy run" check would be satisfied by the previous release.
Kill switch: repo variable AUTO_DEPLOY_GALACTUS=false prints the manual
command instead of deploying. Needs the existing RELEASE_TOKEN secret;
preflight fails loudly and names the manual command if it is unset.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seconds_Behind_Source cannot answer "is it moving?". While the SQL thread
works through one large transaction the lag counter holds still — often at
0 — even though the replica is not caught up. The relay backlog does move,
and it comes out of the SHOW REPLICA STATUS the panel already runs, so this
costs no extra query and no connection to the source.
Adds applyProgress(), which reads Source_Log_File / Read_Source_Log_Pos vs
Relay_Source_Log_File / Exec_Source_Log_Pos and reports the fetched-but-not-
applied byte delta plus a percentage. Both positions are source binlog
coordinates, so they are only comparable while the two threads are on the
same file; across files the delta is meaningless (positions restart at ~4 in
each new file) and is reported as null rather than as a huge negative number.
The percentage deliberately stops at 99.99 while any backlog remains —
binlog positions are large enough that a real backlog of a few KB rounds to
100% and would render a lagging replica as caught up.
Not folded into `healthy`: a non-zero backlog is the normal state of a
working replica between fetch and apply, so alarming on it would cry wolf.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A REIMPORT takes ~110 seconds and, until now, showed only a scrolling log —
there was no way to tell "halfway" from "wedged", which mattered the day one
actually did wedge.
run_all.py emits "[paso i/N] name" before each step and the API derives
progress from the job log. Emitting the marker from the Python rather than
having the UI count STEPS itself means the step count is stated in exactly
one place; adding a step cannot desync the display. Progress is derived, not
stored, for the same reason: the log is already the record of what happened,
and a separate counter could contradict it, which is precisely the confusion
a progress display exists to remove.
While RUNNING, step i is IN PROGRESS rather than finished, so only i-1 count
as done. Counting i would show 100% while the final step was still working —
and the final step (blob_extract) is the slowest, so the bar would sit at
"100%" for the longest stretch of the job.
BACKUP and RESTORE are a single mysqldump with no steps and deliberately
render no bar; a fabricated percentage would be worse than none. The safety
backup that precedes a REIMPORT is likewise named explicitly instead of
showing 0%, which reads as stuck.
Pinned by job-progress.spec.ts, including the literal line run_all.py emits,
so a change to the Python format fails a test rather than silently blanking
the panel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel reported "Error SQL: Replicate_Ignore_Server_Ids:" against a
replica that was healthy — both threads running, zero lag.
`\s` matches newlines in JavaScript, so `^\s*NAME:\s*(.*)$` let the `\s*`
after the colon walk past an EMPTY field's line break and capture the
following line. Last_SQL_Error is blank on a healthy replica and
Replicate_Ignore_Server_Ids happens to be printed immediately after it, so
the blank error field returned the next field's name as its value. Every
empty field was affected; the visible damage was that a healthy replica
rendered as broken, which is the worst direction for a health panel to fail.
Fixed with `[^\S\n]` — horizontal whitespace only — on both sides of the
field name.
Extracted as replicaField() and pinned by replication.spec.ts against the
verbatim output of the live replica, keeping the empty Last_SQL_Error
adjacent to Replicate_Ignore_Server_Ids because that exact adjacency is what
broke. Also covers the literal "NULL" lag surviving as a distinct value from
empty, and a field name that is a suffix of another (Last_Error vs
Last_SQL_Error) not matching the wrong line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ops jobs run as a child of the API process, so no job can outlive it. When a
deploy landed 110 seconds into a REIMPORT, the child died and nothing was
left to finalize the row — it stayed RUNNING forever. Because startJob()
refuses to start while any RUNNING row exists, that one interrupted job
wedged the panel permanently with no way out from the UI; recovering it took
a manual UPDATE against the production database.
A fresh boot is proof that nothing survived, so this is unconditional rather
than filtered on age: "started recently" does not imply "still alive" here.
Rows are updated one at a time rather than with updateMany so the reason can
be APPENDED to the log. A job whose log simply stops mid-step with no
explanation is what made the first occurrence hard to diagnose.
Failure to reconcile is logged and swallowed: a wedged panel is bad, an API
that will not boot is worse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
my.jorgecuadros.com serves customer balances from the Oracle VPS replica.
A replica whose SQL thread has stopped does not error — it keeps answering,
with data frozen at the moment it stopped — so nothing on the customer site
looks wrong and the only signal is a customer complaining about a stale
balance. This puts the failure somewhere a human sees it.
Deliberately does not trust the two fields an operator reaches for first.
Replica_IO_Running reports Yes while the SQL thread is stopped, because the
network thread keeps downloading binlog it will never apply; verified by
stopping SQL_THREAD and watching IO stay Yes. Seconds_Behind_Source reads
NULL whenever EITHER thread is down, so the card renders "sin dato" rather
than "0 s" — showing zero there would report an outage as perfect health.
The problem string is resolved most-specific-first for the same reason.
Shells out to the mysql client because the API has no MySQL driver and the
image already ships one. --ssl is required (the replica sets
require_secure_transport); --ssl-verify-server-cert=0 is deliberate and is
NOT the trade-off the website makes: this hop never leaves Tailscale and the
replica's firewall admits only this host, so WireGuard authenticates the
peer, whereas the DreamHost leg crosses the public internet and pins the CA.
The account behind it holds REPLICATION CLIENT and nothing else — it cannot
read a single row. REPLICA_DB_* unset is a supported state and renders "no
configurada", which is correct in dev and before cutover.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EFECTIVO ledgers have no type column — in Access the transaction type
is implied by which table a row lives in — so unlike DATOS2 there was no
string to map and typeId came out NULL on all 13,496 rows.
That is not just a blank label. handleGetAccountDetails in
my.jorgecuadros.com identifies payments by matching TYPEOFTRX against
('PAYMENT THANK YOU', 'PAYPAL', 'CASH DEPOSIT', 'CHECK DEPOSIT') to reset
the running balance in mode=current. An unlabelled payment is not
recognised, so the balance silently diverges from legacy — 285 rows across
129 customers in the current year alone.
"CASH DEPOSIT" is measured, not chosen: matching the unlabelled rows to the
live site on (NUMid, date, amount) resolves unanimously to that label —
66/66 in the current-year `datosfreak` and 100/100 in the prior-year `2025`
table, the only two periods the site allowlists.
The FM3 fee streams (EFECTIVO FM3 627, CHEQUE FM3 157) have the same
missing-type problem and are deliberately left NULL: every row predates both
exposed periods, so nothing can be matched against a legacy label and none
can reach a customer. Guessing "CHECK DEPOSIT" there would feed the
payment-detection list on no evidence.
type_id_for(None) returns None, so call sites without a label are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
my.jorgecuadros.com reads customer data from the Oracle VPS replica, and a
replica that has silently stopped applying serves stale balances rather
than erroring — so "is it replicating" needed an answer that is not a
human squinting at SHOW REPLICA STATUS.
Runs entirely against the replica over ssh, so it needs no credentials for
the galactus master, and exits non-zero on failure so it can be driven from
cron or a monitor.
It deliberately does not trust the two fields an operator reaches for first.
Replica_IO_Running reports Yes while the SQL thread is stopped, because the
network thread is still downloading binlog it will never apply — verified by
stopping SQL_THREAD and watching IO stay Yes. Seconds_Behind_Source reads 0
both when there is nothing to apply and when nothing is connected. The
trustworthy signal is GTID_SUBTRACT(Retrieved, Executed): binlog fetched but
not applied.
NULL lag means either thread is down, so it is reported as "not applying"
rather than blamed on a specific thread — the thread fields above already
say which, and guessing there produced a wrong diagnosis.
Uses sed rather than `head -n1`; on this machine `head` resolves to LWP's
HTTP head(1), which mangles the pipeline instead of failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two fields the customer-facing site reads were being dropped on the way in
from Access.
transform_transactions.py mapped DATOS2's type string through the Access
`TYPE OF TRX` table and stored NULL on a miss. That table is a stale
pick-list rather than a constraint — staff free-text straight into DATOS2 —
so 78 distinct values covering 3,939 rows never resolved, including
BALANCE FORWARD (1,188) and ANNUAL FEE (1,116). Nothing else on
`transactions` carries the type text, so those rows lost their label
outright and rendered blank. Now mints a type_transactions row from the
literal string when the lookup lacks it; nameEs stays NULL since only the
lookup has translations.
transform_customers.py never carried DATGRAL.TIPO, leaving
customers.minimumBalance empty on every row despite the column existing.
TIPO is the minimum-balance threshold (100/200/300/500; 1,017 of 1,172
customers carry one), not an account type as the name suggests — the
customer app shows it as `minBalance`. Added to the insert list and to the
ON DUPLICATE KEY UPDATE clause, without which --sync would silently skip
it on existing rows.
Both land on the next `run_all.py --sync` reload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five documents asserted the SES_* secrets were unset in Gitea. They now
exist, so all five are corrected rather than leaving the claim to rot in
whichever one a reader opens first.
Replaces the blocker with the two things creating the secrets does NOT
establish, since both fail in ways that look identical to a missing
config: SES_FROM must be a verified identity in SES_REGION, and the
account must be out of the SES sandbox — in sandbox SES only delivers to
verified recipients, so a sweep across 815 policyholders would fail
almost every send while the configuration reads as correct.
Recommends running the first sweep with debug on, which diverts every
recipient and, on the pólizas side, leaves the avisos pending so a failed
test consumes nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Open work was spread across six documents: PLAN's per-step status,
RESUME §6, two specs' collected open questions, and the "Not built"
sections of the two OCR docs. Nothing tracked the two live data defects
except a paragraph inside INSURANCE_FEATURES_SPEC, and nothing at all
recorded that master is 14 commits and 5 migrations past the last tag.
Compiled by reading those six, then checking each claim against the code
and the dev database rather than trusting the prose — which is how the
dead-table finding surfaced and how both insurance defects were confirmed
still open.
Leads with the ship blocker: SES_* is unset in Gitea while the pólizas
sweep defaults to enabled at 06:00, so deploying current master gives a
nightly sweep that fails every run. Set the secrets or disable the
schedule before cutting v1.0.7.
Findings not previously written down anywhere:
- policy_types still holds only AUTO/LICENCIAS/MULT and 5 policies still
have a NULL policyTypeId; policyTypeId is still `String?` with Prisma's
default SetNull, so the spec's recommended Restrict was never applied.
- EmailTemplate / EmailCampaign / EmailLog have zero references in
apps/api/src or apps/web/src. Scaffolded for step 10's "email
campaigns"; notificaciones shipped against email_notification_log
instead. Either wire them or drop them.
- Customer.customerNumber does not exist, so recycling is not merely
unbuilt but unstarted at the schema level.
Linked from PLAN.md and README so it is findable from either entry point.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gives receipt capture the same treatment policy OCR just got: a doc that
records what is in the code, separate from the spec that records what was
designed. RECEIPT_CAPTURE_SPEC.md §2 had accumulated three BUILT notes
totalling ~120 lines of findings, which is the right place for the
evidence but the wrong place to look up how the matcher picks a column.
docs/STATEMENT_OCR.md covers the pipeline, the OCR seam and its
text-layer-first rule, all eight parsers and the ordering constraints
between them, the matcher's two governing rules and the scopedRefField
table, confirm-through-BillingService, the learning write-back, and the
API surface.
Weight goes to the things that are load-bearing and invisible from the
code shape: brand detection must run to completion before layout because
Tijuana bills predial and zona federal off the same treasury header;
scopedRefField is exported because three call sites must agree or a
reference gets learned into a column nothing searches; FEDERAL_ZONE's
accountNumber holds a peso amount, so it fails the null-guards as well
as the lookup; a misread `$` is the dangerous failure, not a missing one.
Also records that CFE/CESPT/Telnor have no unit suite — they predate the
gas/predial extension and were only verified end to end.
Cross-linked from the spec, POLICY_OCR.md, PLAN.md, README and RESUME.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Policy OCR shipped 2026-08-01 (5e9cb12) and was documented nowhere. It is
not in INSURANCE_FEATURES_SPEC.md because it did not come from that
meeting — it came out of building the utility statement OCR pipeline in
RECEIPT_CAPTURE_SPEC.md §2 and noticing the same shape fits carrier
policy PDFs. A reader had no way to find that lineage.
New docs/POLICY_OCR.md covers it end to end, with weight on the three
things that are not obvious from the statement side:
- **One PDF = one policy.** Statements arrive bundled one customer per
page, so there a page is a document. A GMX certificate is one policy
across two pages, so the pages are concatenated and the parser runs
once per file — which is why `pageNumber` is a file ordinal and
`storageKey` is the source PDF, not a page image.
- **The GMX certificate carries no premium at all** — it lives on a
separate recibo PDF. Hence the null-preserving confirm and the
double-gated ledger write.
- **OcrModule was extracted out of StatementsModule to make this
possible**, and that was blocking rather than cosmetic.
Cross-referenced from RECEIPT_CAPTURE_SPEC.md §2 (where it came from),
INSURANCE_FEATURES_SPEC.md (which never proposed it, and whose §4 carrier
API it partly overlaps), PLAN.md step 11, README and RESUME.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The docs still described the state before the last five commits: the
insurance spec called for a `@Cron` literal and a manual mark-as-sent
mutation, PLAN.md had step 12 as "NOT STARTED", and README's module and
route lists predated seven modules.
- MASS_EMAIL_NOTIFICATIONS.md: new "Send flags", "API surface" and
"Scheduled runs" sections; "Cron (future)" removed — it exists. The
flags table says which flags apply where, and why a debug renewal send
must skip both the RenewalNotice row and `lastSuccessfulAt`.
- INSURANCE_FEATURES_SPEC.md: §1 BUILT note listing the three places the
build diverged from the spec; §1.1 and §1.4 marked superseded in place
rather than deleted, so the reasoning stays readable.
- PLAN.md: step 12 renewal emails DONE with the divergences; status
paragraph rewritten.
- README.md: current module/route lists, plus a "Scheduled jobs" section —
a reader cloning this repo had no way to know the API sends mail on a
timer.
- DEPLOY_AND_MIGRATIONS.md: the cadence lives in app_settings and survives
an image rollback, and the servicios sweep has no multi-replica lock.
- RESUME.md: session record for the whole notificaciones arc.
- RENEWAL_NOTICES.md: pointer that this is the legacy record, not what
shipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Flags del envío" panel lived inside the Servicios tab and only
governed the four bulk jobs. The pólizas half had no debug at all, so
there was no way to test a renewal notice without mailing a real
customer. The panel now lives in the /notificaciones shell above the
tabs and both halves read it.
`debug` on the renewal path diverts to the same override inbox as the
servicios jobs and deliberately does NOT write the `RenewalNotice` row
or advance the sweep's `lastSuccessfulAt` — the customer was not
notified, so nothing may gate the letter they are still owed.
`ignoreDayRestriction` and `useEmailLimit` stay estado-de-cuenta-only
and are labelled as such.
Both automatic sweeps are now operator-editable. The renewal cadence
was a `@Cron("0 6 * * *")` literal and servicios had no automatic run
at all; both now resolve through `NotificationScheduleService`, which
stores the cadence in `app_settings` and reinstalls the cron job on
save — no redeploy, no restart. Defaults preserve current behaviour:
pólizas 06:00 daily, servicios off. A scheduled run never inherits the
UI flags; it always sends for real.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NOTIFICATION_ADMIN_EMAILS made "add Beto to the summaries" a redeploy —
the wrong unit of work for a list that changes when office staff change.
Adds `app_settings`, a key/value table for the configuration staff must
be able to change without a deploy, and `SettingsService`, which resolves
every key db -> env -> default and reports which of the three a value
came from. That ladder is what makes the move safe: a deployment behaves
exactly as before until somebody saves in the UI, and the screen can say
"this is still coming from the deployment" rather than implying somebody
chose it.
- new ability `setting:manage` (ADMIN) — deliberately above
`notification:send`, since redirecting the audit summaries is how
someone would quietly stop them being read
- GET/PUT /notifications/settings/admin-emails; read is open to any
logged-in user so the UI can display the list, write is gated
- resolved per job, not cached at boot, or we would reintroduce exactly
the restart-to-apply behaviour being removed
- a saved empty list means "nobody" and does NOT fall through to the env,
or clearing the field would keep mailing the people just removed
Credentials stay in env — see the model doc for where the line is drawn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stack env is assembled from Gitea repo secrets by the deploy
workflows' `env_data` block — there is no .env file on the host for the
app stack. SES was in neither, so `MailService` came up unconfigured on
every deployment and, with NODE_ENV=production killing the stdout dev
fallback, every notification and renewal aviso failed.
Wire SES_REGION / SES_FROM / SES_FROM_NAME / SES_ACCESS_KEY /
SES_SECRET_KEY / SES_CONFIGURATION_SET / NOTIFICATION_ADMIN_EMAILS
through both galactus and cubex. No `_GALACTUS` suffix: one SES identity
serves every deployment.
Kept out of the required-secrets preflight — mail is not needed to boot,
and failing a deploy over it would be wrong. Preflight warns instead,
since the failure is otherwise invisible until someone clicks "Ejecutar".
Also corrects the comments added in the previous commit, which claimed
these belonged in a host env file rather than in CI secrets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renewal avisos left behind only a `RenewalNotice` row, whose sole job is
gating: a row with `sentAt` drops the policy off the pending list. It
cannot represent a failed send or a customer with no address, so the
Pólizas tab had no "Registro de envíos" to show and a sent notice simply
vanished from the list.
Renewals now write `email_notification_log` — the same table the four
bulk jobs write — as `RENEWAL_NOTICE` / `POLICIES`, with rows for
failures and no-email skips too. `RenewalNotice` keeps its gating role
unchanged; the two are complementary, not redundant.
- extend `EmailNotificationType` (+RENEWAL_NOTICE) and
`EmailNotificationServicio` (+POLICIES); `level` now carries the aviso
generation on renewal rows, so every reader must branch on the type
first (`notificationLevelLabel()` is the one place that lives)
- backfill emailed notices (`channel = 'EMAIL'`) into the log; MAIL-channel
rows are legacy printed letters and are deliberately left out
- extract `NotificationLogService`/`NotificationLogModule` as the single
writer, so a feature that sends mail records it without pulling the
bulk-job pipelines into its module
- `GET /notifications/log` and `/stats` take a comma-separated `servicio`
list; each tab reads its own slice. This also fixes the "Omitidos"
view, which mapped to no filter at all and showed every row
- share one `NotificationLogPanel` between both tabs
- pass SES_* / NOTIFICATION_ADMIN_EMAILS through the galactus compose,
which was missing them entirely — mail is runtime config, not a CI
secret, and the prod image sets NODE_ENV=production so a blank config
fails loudly instead of falling back to stdout
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Pólizas tab now sends. Each pending row gets an "Enviar aviso" button
backed by POST /renewals/send, which renders, mails and records the notice
through the same path the daily sweep uses — so a hand-sent letter is
marked exactly like a swept one and drops off the pending list.
Sending is now the only way a notice gets marked as sent. Remove the
manual "Marcar impreso" / "Marcar EMAIL" buttons and the endpoint behind
them (POST /policies/:id/renewal-notices, PoliciesService.markRenewalNotice,
MarkRenewalNoticeDto): they wrote a sentAt with no mail behind it, which
let the list claim a customer was notified when nothing was sent.
sendOne refuses a generation that already has a sentAt (409) so a double
click cannot mail the customer twice, and 400s when the customer has no
email on file. Sweep and single send share the new deliver() helper.
Add POST /notifications/run-all: runs the four notification jobs
(outstanding, payment confirmation, account status, trust confirmation)
sequentially with one shared set of flags from "Flags del envío".
Sequential rather than parallel — the jobs share the SES transport and
account status can self-throttle via useEmailLimit. A job that throws is
captured and the sweep continues, so one bad query cannot swallow the
other three envíos; the aggregate response carries per-job results plus
summed sent/skipped/failed and an errors count.
Audited as a single notification.run-all.run entry so one staff click is
one audit row. UI adds the button to the flags card, with a confirm when
debug is off, and a per-job summary in "Última respuesta".
MailModule's provider used a `useFactory` with no `inject`, so the factory
received `undefined` and `new MailService(config)` threw on `config.get`,
taking the whole API down at boot. The module also wasn't actually
`@Global()` even though both NotificationsModule and RenewalsModule inject
MailService without importing it — that would have failed next. Replaced the
factory with a plain provider (ConfigModule is already `isGlobal`) and marked
the module global.
On the web side, mass email and renewal notices were two menu entries doing
the same job — telling a customer something by email. They are now two tabs
of `/notificaciones` (Servicios and Pólizas), following the Captura pattern:
`/renovaciones` still resolves, opening the same screen on its Pólizas tab so
existing bookmarks keep working.
The notifications page was also the last screen written in raw inline styles,
with blue buttons and filter pills that appear nowhere else in the app. It now
uses the shared design system: btn-primary/btn-outline, the seg segmented
control, card, tx-table, pager, and the servicios/fideicomiso badges.
Two supporting fixes found on the way: NOTIFICATION_STATUS_COLORS hardcoded
hex instead of the theme's positive/negative/muted vars, and `.small` was
referenced in 19 places across the app but never defined in globals.css.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the four legacy PHP scripts under email.notifications/send*.php
with a single NestJS module. Four jobs (outstanding payments, payment
confirmations, account-status alerts with day-of-week gates, trust
payment confirmations) share one MailService modelled on StorageService:
env-driven SES client, null fallback in dev with console logging, refuses
to send in production when unconfigured.
Schema adds email_notification_log (every attempt, sent/failed/skipped)
and account_status_history (one row per threshold hit, Job 3). Enums
encode the legacy wire shape so external log scrapers keep parsing
notificationType keys verbatim.
Web adds /notificaciones with four trigger cards, a flags panel, and a
paginated log browser. New notification:send ability gates all four
endpoints at MANAGER, matching the renewal:send trust tier.
INSURANCE_FEATURES_SPEC §1. The office printed and mailed renewal letters
from the legacy CONTROL <ramo> RENEW[2/3] paper log; 91% of policyholders
have an email on file, so send the notice instead and keep the paper log
as the fallback.
A daily cron (06:00 America/Tijuana) sweeps three generations off
policyTo — 30 and 15 days before expiry, 7 days after — sends each
through SES, and upserts RenewalNotice by [policyId, generation] so a
policy is never notified twice for the same milestone. RenewalNotice now
records providerMessageId, so a later bounce or complaint webhook can be
traced back to the row that sent it.
- customers.emailOptOut excludes a customer from every sweep; editable
from the customer form
- scheduled_job_states holds the sweep's lock and last successful run;
the window is widened to cover days the job did not run, so a weekend
outage does not silently drop a generation
- SES unconfigured is not an error outside production — messages are
logged and skipped, so dev and CI never send
- /renovaciones (renewal:send, MANAGER+) lists what is pending per
generation, runs the sweep by hand, and marks a notice sent by mail
for the customers with no email
- POST /policies/:id/renewal-notices records that manual mark
- the aviso-renovacion report and the emails now share one projection
(reports/renewal-letter.ts) instead of two copies of the mapping
A bad scan, the wrong PDFs or a duplicate upload used to leave a batch
sitting in READY_FOR_REVIEW forever, because the only exits were confirm
(posts to the books) or rejecting every page one at a time. Add a
DISCARDED terminal status to both OCR domains and a single endpoint per
domain that rejects every page still pending in one shot.
Discarding is refused once anything has landed: statements once a page is
POSTED, policies once a page is APPLIED. Those batches did real work and
have to be settled page by page.
- POST /statements/batches/:id/discard
- POST /policy-ocr/batches/:id/discard
- shared DiscardBatchCard on both review screens, gated the same way
Mirrors the utility statement intake on the insurance side: a policy_ocr
batch/document pair of tables, a GMX parser, a matcher keyed on
Policy.policyNumber, and a "Captura" screen under /polizas that proposes
policy -> customer for staff to confirm.
Lifts the OCR seam out of StatementsModule into its own OcrModule so
PolicyOcrModule can inject OCR_PROVIDER without taking on the rest of
the statement pipeline; StatementsModule now imports it and binds
nothing itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the ZONA FEDERAL TIJUANA parser to the statement intake, measured
against 8 pages of real "Zona Federal Marítimo Terrestre" receipts — the
federal maritime-zone occupancy fee the municipality bills on beachfront
lots. Provider read on 8/8, amount on 8/8 (each verified against the
paper), concession clave on 6/8, period on 8/8, deadline on 2/8.
Four things the corpus forced:
- Tijuana bills predial and zona federal from the same treasury: same
header, same Paseo del Centenario address, same ATB-541201 RFC. Every
predial discriminator matches a zona federal page too, so whichever
rule is asked first wins it. The only words exclusive to this layout
are "Marítimo Terrestre", so its brand rule is asked ahead of all
three predial ones — and its structural rule, anchored on the stub's
"Derechos de ocupación", ahead of theirs.
- FEDERAL_ZONE.accountNumber is an amount, not a reference. It holds
DATMEX.zfed, whose 77 values include 246.06, 2369.09, 22653.94 and a
negative -1679, while the concession claves these receipts are keyed
by appear nowhere in the database. Matching on that column could never
hit — and because every row already has a value, the `[field]: null`
guards on learnAccountRefs and on the review blank-service fill would
never fire either, so every page would return to the queue every
bimester forever. The clave moves to meterNumber, joining gas and
Tijuana predial, and the first confirm teaches the match.
- The payable figure is not the printed subtotal. The municipality
rounds to whole pesos and prints the difference on its own "Ajuste Ley
Hacienda Mpal" line (-$0.05 against a 591.05 subtotal, $0.21 against
2,872.79). The "Total a pagar" box carrying the rounded figure sits on
a grey fill and OCR'd on 1 of 8 pages; the SubTotal row read on 8 of
8. So the amount is the rounded subtotal, cross-checked against the
printed box wherever it survives — where it did, it agreed.
- The clave is 2 digits, a letter and 3 digits (12-T -012), not the
cadastral shape, and the letter is kept as printed: toDigits maps D to
0, which turns a real 14-D -014 into 140014. It is printed twice,
which rescued a page whose heading was struck through by the office's
own highlighter — the failure mode behind both missing claves.
Deriving the deadline from the bimester is deliberately not attempted:
it is the 17th of the month after the bimester closes on a current bill,
but four of these eight are late (a $1,000 Multa) and print a
recalculated date, so a derived date would be wrong on exactly the pages
a human most wants to see.
Re-ran the earlier corpora (25 pages: predial Tijuana/Rosarito/Ensenada,
CFE, CESPT, Telnor) through detection to confirm the new rules steal
nothing — all 25 still read as their original provider, including the
five Tijuana predial pages that share the RFC.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds four parsers to the statement intake — GAS TIJUANA plus one per
municipality, because Tijuana, Rosarito and Ensenada issue three
completely different predial documents — and a text-layer fast path for
the born-digital invoices the gas company sends.
Measured against a new corpus of 14 documents / 29 pages: provider read
on 29/29, amount on 26/29, and 21/29 auto-matched against the dev
database (22/29 identified). The eight review cases are all legitimate.
Five things the corpus forced:
- Not every statement is a scan. The gas invoices are born-digital CFDIs
whose text layer is exact; rasterising them only loses information (one
sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new
`OcrProvider.textPages` reads the embedded layer via `pdftotext
-bbox-layout` — same poppler package as `pdftoppm`, so no new
dependency — and OCR stays the fallback for real scans. Poppler's own
`<line>` grouping follows text flow rather than the page, so words are
regrouped by vertical position; without that, a two-column header
leaves every label separated from the value printed beside it.
- The clave catastral is not two letters and six digits. Position three
is a letter in 15 of the 932 stored claves, and digitising the whole
tail mapped a real `MMB01041` to a nonexistent `MM801041`.
- Tijuana predial prints no clave at all. Its only identifier is an
8-digit municipal account carried in a 32-digit payment barcode, which
the legacy database never held, so it goes in `meterNumber` alongside
gas — `accountNumber` holds `DATMEX.predial`, which is not a
per-property key and must not be overwritten. Those pages start cold
and are taught by the first confirm.
- On Rosarito and Ensenada the clave is the primary key, not a fallback:
those receipts print nothing else, so a unique hit auto-matches. On a
utility bill that merely happens to print one it stays a review hint.
- A misread `$` is the dangerous failure. An Ensenada receipt for
$2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large
and look ordinary in the ledger. Predial amounts now require a literal
`$` and a page that cannot produce one goes to review.
The scoped match field is now one exported function rather than three
copies of `kind === "GAS" ? ... : ...`, since the lookup, the
blank-service fill and the confirm write-back have to agree or a
reference gets learned into a column nothing searches.
First tests in this package: 23 specs over the parsers and the text-layer
reader, every fixture a verbatim OCR excerpt from a real receipt. Adds
the jest config they need and a build tsconfig so they stay out of dist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Backend already returns per-status counts via byStatus; render a real
progress bar (X% / N de M / en cola) while PENDING_OCR pages remain,
using the existing progress-track CSS. Falls back to indeterminate
when no docs have been reported yet.
release.yml pushes the release commit and its tag in a single `git push`,
so Gitea created two build.yml runs for the same commit. Only the tag run
matters: it emits the X.Y.Z and X.Y image tags, and since it is the same
commit it publishes `latest` and `sha-<short>` as well. The master run was
pure duplicate work that had to be waited out or cancelled by hand.
Guard the build job with an `if` that skips a branch push whose head commit
message starts with `chore(release):`. Ordinary pushes to master are
unaffected, and tag pushes and manual dispatches always build.
The skipped master run keeps the release commit's sha, which would have let
release.yml's "Verify build.yml started" check go green on it alone even if
the tag run were never created — the exact failure that check exists to
catch. It now also requires the run's ref to be the tag, falling back to the
sha match only when the API reports no ref.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rmancinasandClaude Opus 5
gitea-actions