Legacy ran a year-end corte: it summed the closing year, wrote that total back
as each customer's Jan-1 BALANCE FORWARD, and started the next year clean. The
platform inherited those opening rows but never the years behind them — only
the current year's charges were ever staged, so every prior year held receipts
and no bills. Rendering one would have shown a customer credits with nothing
owed against them, which is worse than showing nothing.
The archives are whole-database Access snapshots named for the period they
hold, so `2025.accdb` is discovered by filename, staged, and loaded through the
existing DATOS2 branch — a snapshot's `datos2` is the identical shape, one year
older. Only the ledger and DATGRAL come out of a snapshot; everything else in
it is a year-stale copy of a live table. The cash side is deliberately left
behind: EFECTIVO is a lifetime journal, so the snapshot's copy is a subset of
the live one and importing it would double-book every prior-year receipt.
Period travels with the row, in legacySourceTable as `datos2@2025`. That tag,
not the date, is what a year view should filter on: the archives are not
cleanly bounded (2025 carries ten undated rows and two dated into 2026) and
legacy never filtered by date either — its reader is `SELECT ... FROM \`2025\``.
The tag also keeps legacyId safe, since it is a positional ordinal that
restarts at 0 in every archive and would otherwise collide row-for-row.
Two guards, because attaching a prior year by NUMid is the one thing here that
can go quietly wrong:
- Reissued numbers are skipped, not imported. Comparing each archive's
DATGRAL against the live one, 13 names moved since 2025 and 40 since 2024;
most are the same customer re-described, but a few are a different
household holding a recycled number, and filing their ledger under the new
owner would show a stranger's charges. Sharing any word of three or more
characters separates a rename from a reissue. Names are compared
legacy-to-legacy: `customers.name` has been through blank-name recovery,
and comparing to it reported 121 drifts where there are 13.
- Every period is checked against the corte identity it must satisfy —
SUM(year N) == BALANCE FORWARD(N+1) — and the result is reported per year.
A truncated export, a file dropped under the wrong year, or a botched
customer match all fail loudly here. 2025 reconciles 1,159/1,167 (99.3%)
and 2024 1,144/1,156 (99.0%); the recycle guard raised 2024 from 98.2%.
Balances are untouched: BALANCE_FLOOR_JOIN floors on the newest BALANCE FORWARD
per customer, so rows behind it are already excluded from every balance read.
Uploads go through the existing ingest endpoint, allowlisted by an anchored
`AAAA.accdb` pattern that also keeps a caller-supplied name inside the ingest
directory. The Operaciones page grows an entry point for an archive that has no
row yet, reading the period off the chosen file's own name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The capture form only ever had prima neta, derecho de póliza and comisión.
The Access form it replaces has seven figures, and the four that were missing
are the ones that make a policy paid in installments add up.
Adds recargo, IVA, prima total and forma de pago to the policy header, the
same breakdown per installment, and a per-line-of-business IVA rate.
IVA and prima total are the only derived figures:
base = prima neta + recargo + derecho de póliza
IVA = round(base * tasa)
total = base + IVA
The recargo is inside the taxable base. That is not a guess — policy 7006785
prints IVA 52.03 on 610.86 + 8.55 + 31.00, and leaving the recargo out gives
51.35, which matches nothing on the page. Both of its money rows are asserted
in premium.spec.ts. The recargo itself is never derived: the carrier quotes it,
so staff key it in, and the field is disabled on ANNUAL/SINGLE. Both derived
figures are stored rather than recomputed on read, and stay editable, because
the printed policy is the record of truth and a later rate change must not
silently restate what was issued.
The rate lives on PolicyType (seeded to 0.08, editable in Catálogos), which is
the legacy one-row IMPUESTOS / IMPUESTOS_AUTOS tables made configurable. The
rate applied is stamped on the policy so an old one reads back at its original
rate.
Per-installment, not two fixed slots on the header: a policy split into several
exhibiciones prices each payment separately — that is why the Access form drew
the money row twice — and a trimestral policy needs four, which the Access
layout could not hold.
Also fixes two losses in the ETL, which is how these went missing:
- `forma_pago` was marked consumed by the coverage sweep and then never
written to any column, so FORMA PAGO existed nowhere in the platform.
- `recargo` and the whole second money row fell into `coveragesJson` as
loose strings, mislabeled as coverage amounts.
transform_policies.py now writes all of it directly;
backfill_policy_premium_breakdown.py recovers it on a database that must not be
re-imported, and strips the migrated keys back out of coveragesJson. Both are
COALESCE-only, so a figure a human has corrected in the app wins.
IVA and TOTAL are NOT backfilled: they were unbound calculated controls on the
Access form, never columns, so there is nothing to recover and every migrated
policy reads null until it is edited.
The backfill warns on 5 annual policies that carry a non-zero recargo — a
contradiction that predates this change and is left for a human, not silently
corrected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A full run_all.py pass truncates and rebuilds every table it owns from the
Access extract. That was harmless while the platform was a read-only mirror --
every row came from the extract, so wiping and rebuilding lost nothing. It
stopped being harmless once the platform started minting rows Access has never
heard of: allocated portal NUMids, customers created in the staff UI,
OCR-captured policies, app-booked ledger rows, uploaded documents.
REIMPORT is a button in /operaciones, so that was one click away.
native_guard.py counts what only exists here and exits 3; run_all.py runs it
before the first truncate and stops. Detecting an allocated NUMid needs the
staged Parquet -- the customer holds an ordinary-looking (utilities, DATGRAL,
'1172') ref, so "customer has no refs" cannot see it and only comparing against
the extract can. Missing staging is therefore treated as blocking rather than
as "nothing to protect".
The guard does not teach full mode to preserve anything: --sync already upserts
legacy rows against the existing refs and leaves the rest alone, and rebuilding
that inside full mode would re-implement it. --force-full (checkbox in the
REIMPORT confirm, recorded in the audit log) deletes them deliberately.
Verified against dev: clean before, exit 3 listing utilities/1172 with a
synthetic ref present, clean again after removing it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
datosfreak's NOPAGO is the legacy "still owed" flag, and the website reads
it directly — account.statement.php splits the statement on NOPAGO = 0 vs
NOPAGO = 1 and renders the latter as "Outstanding Bills Requiring
Attention". transform_transactions.py hardcoded 0, so all 40,421 rows came
across settled and that section renders empty for anyone served off the
platform. Not a missing column: a missing section, with no error.
Only the three DATOS2-shaped tables carry the flag (76 rows set in datos2,
0 in FEE ANUAL and fee15); the EFECTIVO/FM3 cash streams have no such
column and keep the 0 default. Sync mode gets outstanding=VALUES(...) too,
so an additive sync corrects rows already loaded rather than leaving them
settled forever.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SYNC ran `run_all.py --sync` without `--stage`, so it depended on staged
Parquet under migration/output. That directory is part of the image, not a
volume, so any redeploy wiped it and the job died on the first transform:
FileNotFoundError: '/repo/migration/output/stg_utilities/datgral.parquet'
Re-staging is also what makes the job's own label true — without it a sync
would replay whatever upload staged last, not the files currently in the
ingest folder.
Staging now counts as a numbered step when it runs, so the Operaciones
progress bar moves during the slowest phase instead of sitting empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A REIMPORT takes ~110 seconds and, until now, showed only a scrolling log —
there was no way to tell "halfway" from "wedged", which mattered the day one
actually did wedge.
run_all.py emits "[paso i/N] name" before each step and the API derives
progress from the job log. Emitting the marker from the Python rather than
having the UI count STEPS itself means the step count is stated in exactly
one place; adding a step cannot desync the display. Progress is derived, not
stored, for the same reason: the log is already the record of what happened,
and a separate counter could contradict it, which is precisely the confusion
a progress display exists to remove.
While RUNNING, step i is IN PROGRESS rather than finished, so only i-1 count
as done. Counting i would show 100% while the final step was still working —
and the final step (blob_extract) is the slowest, so the bar would sit at
"100%" for the longest stretch of the job.
BACKUP and RESTORE are a single mysqldump with no steps and deliberately
render no bar; a fabricated percentage would be worse than none. The safety
backup that precedes a REIMPORT is likewise named explicitly instead of
showing 0%, which reads as stuck.
Pinned by job-progress.spec.ts, including the literal line run_all.py emits,
so a change to the Python format fails a test rather than silently blanking
the panel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EFECTIVO ledgers have no type column — in Access the transaction type
is implied by which table a row lives in — so unlike DATOS2 there was no
string to map and typeId came out NULL on all 13,496 rows.
That is not just a blank label. handleGetAccountDetails in
my.jorgecuadros.com identifies payments by matching TYPEOFTRX against
('PAYMENT THANK YOU', 'PAYPAL', 'CASH DEPOSIT', 'CHECK DEPOSIT') to reset
the running balance in mode=current. An unlabelled payment is not
recognised, so the balance silently diverges from legacy — 285 rows across
129 customers in the current year alone.
"CASH DEPOSIT" is measured, not chosen: matching the unlabelled rows to the
live site on (NUMid, date, amount) resolves unanimously to that label —
66/66 in the current-year `datosfreak` and 100/100 in the prior-year `2025`
table, the only two periods the site allowlists.
The FM3 fee streams (EFECTIVO FM3 627, CHEQUE FM3 157) have the same
missing-type problem and are deliberately left NULL: every row predates both
exposed periods, so nothing can be matched against a legacy label and none
can reach a customer. Guessing "CHECK DEPOSIT" there would feed the
payment-detection list on no evidence.
type_id_for(None) returns None, so call sites without a label are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two fields the customer-facing site reads were being dropped on the way in
from Access.
transform_transactions.py mapped DATOS2's type string through the Access
`TYPE OF TRX` table and stored NULL on a miss. That table is a stale
pick-list rather than a constraint — staff free-text straight into DATOS2 —
so 78 distinct values covering 3,939 rows never resolved, including
BALANCE FORWARD (1,188) and ANNUAL FEE (1,116). Nothing else on
`transactions` carries the type text, so those rows lost their label
outright and rendered blank. Now mints a type_transactions row from the
literal string when the lookup lacks it; nameEs stays NULL since only the
lookup has translations.
transform_customers.py never carried DATGRAL.TIPO, leaving
customers.minimumBalance empty on every row despite the column existing.
TIPO is the minimum-balance threshold (100/200/300/500; 1,017 of 1,172
customers carry one), not an account type as the name suggests — the
customer app shows it as `minBalance`. Added to the insert list and to the
ON DUPLICATE KEY UPDATE clause, without which --sync would silently skip
it on existing rows.
Both land on the next `run_all.py --sync` reload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every transform resolves its target through dbenv.database_url(), which lets a
DATABASE_URL in the process environment win — that is how the API container
drives a re-import against its own database with no deploy/ directory present.
blob_extract.py was the one step that bypassed it and called load_env()
directly for the MinIO credentials, so the "Operaciones" re-import loaded all
the data and then exited 1 on:
missing /repo/deploy/.env.prod — deploy the 'prod' DB stack and write its
.env first
Give the S3 settings the same resolution as the DB URL: load_env() now returns
{} for an absent file, and setting()/require() layer the process environment on
top of it. blob_extract reads S3_ENDPOINT / S3_BUCKET and accepts either
S3_ACCESS_KEY/S3_SECRET_KEY or MINIO_ROOT_USER/MINIO_ROOT_PASSWORD, matching
the fallback order in storage.service.ts and the vars the api service already
sets in deploy/galactus/jorgecuadros-app.compose.yml. A genuinely missing
setting still fails fast, now naming the variable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Staff key 300+ utility statements per company per month by hand. This adds
the ingest -> split -> OCR -> match -> review pipeline that proposes customer
and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the
existing BillingService.createBatch seam with source=OCR and a per-document
captureRef so machine and hand capture share one write path and audit trail.
Everything was designed against 10 real scanned statements (46 pages of CFE,
CESPT and Telnor bills) rather than from the sample-free spec. The scans have
no text layer at all — they are camera images — so OCR is mandatory, and they
arrive bundled one customer per page. Measured on those pages the parser
identifies the provider 46/46 and reads an account reference 43/46; against
the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with
the rest genuine review cases. That closes the OCR-provider question in favour
of self-hosted Tesseract: it clears the bar for a queue where a human confirms
every row, and OcrProvider keeps a managed API a one-line swap.
The samples corrected three things the spec had wrong or unknown:
- Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and
predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds,
has 663 distinct values across 1135 rows and appears on no statement. The
clave now lives on Property.cadastralKey as the matcher's secondary key;
predial is left untouched. This had been blocking predial matching.
- Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers
(the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber.
- Phone is one billed line per property (534/18/1 across phone1/2/3), so the
new TELEPHONE ServiceKind backfills from phone1 only, not three rows.
Matching is scoped to one column per service kind and never reads the customer
name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this
office holds under CATT, RANDY, because the printed name is the registrant,
not the current owner. Where a provider prints a payment barcode it beats the
printed label (one CFE label OCR'd a digit too many while its barcode was
correct) and the two cross-check, with disagreement forcing review.
Confirming a document whose service had no reference writes it back, so gas
and any other cold start is a one-time cost rather than a permanent queue.
Verified end to end against the live dev API and MinIO: real scans uploaded
over HTTP, matched, confirmed against a check, and the resulting rows checked
in MySQL (negative amounts, captureSource=OCR, concept derived from the batch
kind, captureRef linking back to each page). Re-confirming a posted batch is
refused. Test data was removed afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The office keeps more than one operating account (Utilities banks in MXN,
Seguros in USD), but bank_transactions was a single implicit MXN register by
design. Adds Bank/BankAccount and makes every read and write in the module
scoped to exactly one account.
Schema:
- Bank / BankAccount. Currency is fixed per account and BankTransaction has
no currency column of its own — a movement inherits its account's, the way
a real bank account doesn't mix currencies.
- BankTransaction.bankAccountId, required. A movement with no known account
isn't reconcilable against a statement.
- @@index([bankAccountId, transactionDate]): every read now filters by
account and orders/groups by date.
Migration:
- backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)"
and backfills all 22,669 existing rows onto it, then promotes the column to
NOT NULL and attaches the FK. Standalone because prisma db push cannot add
a required column to a populated table. Idempotent; re-running once a second
account exists does not re-point rows.
- run_all.py runs it (both modes) before transform_bank.py, which now resolves
the account by label and fails fast if it is missing.
API:
- ?bankAccountId= required on list/stats/facets/summary — not optional with an
"all accounts" default, since summing an MXN and a USD register repeats the
currency-collapsing mistake the billing module exists to prevent. Missing is
400, unknown is 404.
- facets() had no account clause at all and summary() has two raw-SQL rollups;
all three are now parameterised. Scoping only one of summary's queries would
leave the year list and its drill-down describing different books.
- New bank/accounts + bank/banks sub-resource under a MANAGER
bank:manage-accounts ability. currency is absent from the update DTO: booked
movements are denominated in it, so editing would re-denominate history.
Capture into a closed account is rejected.
Web:
- /banco gains an account picker (remembered per browser) and reads every
figure in the selected account's currency; the "single currency (MXN)"
doc-comment and the hardcoded MXN formatting are gone.
- New /banco/cuentas for banks and accounts. Accounts are closed, never
deleted — the FK is required, so deleting one would destroy its register.
- /inicio's chequera card names the account it is reading instead of implying
a single register.
Verified against dev + browser: a second USD account showed full read/write
isolation from the MXN register, whose totals were unchanged (22,669
movements, net 1,014,266.97).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces ~40 legacy Access renewal-notice report clones (one per carrier
per coverage tier, e.g. AMPL/RC/LIC RENEW X MES/VENCE ATLAS 13/2013) with
one parameterized aviso-renovacion report driven by real Policy/Vehicle/
coveragesJson data instead of hand-typed label text per clone.
- schema.prisma: add RenewalNotice, replacing the legacy CONTROL <ramo>
RENEW[2/3] X MES paper log of which notice generation was sent
- reports: new "letter" ReportFormat + aviso-renovacion registry entry +
LetterLayout renderer in ReportRunner.tsx
- docs/RENEWAL_NOTICES.md + migration/legacy_report_defs/: extracted (via
Application.SaveAsText, since the VBA project wouldn't load) and
documented the legacy report/query chain this replaces
Coveragesjson key names and a mark-as-sent mutation are still unverified/
unbuilt — see caveats in docs/RENEWAL_NOTICES.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The --sync path had never been run and was broken in several ways. Fixed and
verified against the dev DB (two consecutive syncs, both exit 0, 32/32
assertions: stable PKs, manual-row preservation, changed-row updates,
legacy-delete, no child duplication, zero FK orphans; idempotent).
- policies/properties: reuse each legacy row's existing id (by provenance)
BEFORE building child rows, so children no longer point at a discarded fresh
uuid; rebuild legacy-owned children via scoped delete + reinsert.
- customers: replace zip(customers, refs) (mispaired almost every row) with a
ref-grouped id remap; names now restore and no spurious customers appear.
- drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy
policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert.
- upsert lookup tables (policy_types, insurance_providers, type_transactions,
adjusters) by natural name and remap child FKs instead of inserting fresh
uuids that nothing points at.
- transactions: drop updatedAt=NOW() (no such column); guard report formatting
on NULL legacySourceTable (manual rows). Same report guard in bank.
- add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune
only legacy-owned empties, never manually-added customers.
web: customer-detail mini tx list now strikes voided rows with an "(anulado)"
tag (was the last void-UI rendering gap; /estado-cuenta already handled it).
docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI
browser pass recorded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Working-tree checkpoint of in-progress work carried across prior
sessions on the feat/crud-rbac branch, committed so it lands on the
remote alongside the CI changes.
- Operaciones admin panel: apps/api/src/ops (ingest upload, backup /
restore / re-import jobs) wired into app.module + RBAC abilities, and
the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR
/ BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups.
- migration/sync.py plus transform_*.py / run_all / config / dbenv /
blob_extract adjustments for the additive sync path.
- crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals.
- schema.prisma + PLAN/RESUME doc updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.
reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.
Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.
- reconcile.py: add canon(), which parses a key column to a number when
nearly every populated cell parses and re-emits it at fixed precision.
Applied in keyset() and in the folio-conflict comparison. Rewrite the
group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
two efectivo_like() calls. EFECTIVO loads first and wins collisions.
De-dup on the business key, never on folio — folio is per-table
sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
their verdicts are unchanged.
Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
144 customers owned zero properties, zero policies and zero transactions —
the legacy DATGRAL row exists but nothing in either business line ever
attached to it. They padded the staff customer list with rows that can't be
acted on. 27 were also nameless (dead ID slots); the other 117 have real
names and sometimes contact details, and read as never-activated prospects
or lapsed clients rather than junk. Removing both sets is a deliberate call.
Implemented as a separate step rather than a filter inside
transform_customers.py: emptiness is only knowable after properties, policies
and transactions have loaded, and deciding it there would mean re-deriving
each downstream transform's source-matching logic against the staged Parquet.
Runs after transform_transactions.py in run_all.py.
Safe by construction — a customer with no rows in any of the three tables has
nothing pointing at it, so the delete cannot orphan anything; only its own
customer_legacy_refs go with it. The step asserts zero orphans afterwards.
Every pruned customer is written to output/pruned_customers.csv with its
legacy provenance before the delete, and --dry-run reports without touching
anything. Nothing is unrecoverable: the Access sources are untouched and a
pipeline run without this step brings them all back.
Verified: full run_all.py pass ends at 1538 customers (from 1682), with
1519 properties / 2378 policies / 45861 transactions all intact and zero
orphans. 17 nameless customers remain, all of which carry real records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 44 customers with no recoverable name render as "(SIN NOMBRE)", and
ordering the list by name alone floated all of them to the top — "(" sorts
before every letter — so the first two screens of the customer browser were
nothing but placeholders. Small number, worst possible position.
Adds customers.nameMissing, set by the transform and used as the primary sort
key so those records land at the end of the list. Denormalized rather than
computed in the query because the list is paginated in SQL, so the ordering
has to be expressible as a column.
Applied to the dev DB as an ALTER + UPDATE in place (no truncate), so the
existing loaded data and its FKs were left alone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance),
which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)".
The blank is real — those cells are empty in the Access files, not lost in
extraction — but the rows mostly are not junk: 176 of the 257 carry a
property, a policy, or transactions.
The old PHP importer handled this by skipping blank-name rows outright
(jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was
worse than it looks: every other adapter resolved its customer FK through
the customer_mapping table those skipped rows never entered, so their
properties and policies were silently dropped (customerServiceAdapter.php:45)
and their transactions were written against customer_id 0
(customerBalanceAdapter.php:52). So: recover the name instead of skipping.
Names come from the secondary tables that still carry them, most trustworthy
first — UTILSEG (the office's own hand-maintained name <-> id cross-reference
spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy
rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow
the name its insurance record resolved to. Result: 213 of 257 recovered, 44
still genuinely nameless anywhere in the source.
customers.nameSource records which table each recovered name came from, so a
reconstructed name is never mistaken for one that was really on the record —
the list tags it "nombre recuperado", the detail header names the source, and
a still-unnamed customer renders muted italic instead of as a normal name.
Also fixes run_all.py: transform_properties and transform_policies truncate
service_documents/policy_documents, but blob_extract.py was not in the step
list, so a full re-run left the uploaded MinIO objects with no rows pointing
at them. Hit exactly that while reloading for this change.
Verified end-to-end: full pipeline re-run against dev reproduces every prior
count (1682 customers, 1519 properties, 2378 policies, 45861 transactions,
22354 bank rows, 70 documents) with zero orphans, and both apps build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.
Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.
Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-minio.stack.yml: S3-compatible object storage (MinIO) for
the platform's document blobs, deployed to the cubex Swarm with the same
statefulness rules as the DB stack (named volume, pinned to the labeled node).
Parametrized for dev/prod as two stacks (dev API 9100/console 9101, prod
9000/9001). Dev deployed + bucket jorgecuadros-documents created.
migration/blob_extract.py: re-reads the LONGBINARY columns via mdb-export
-b hex (staging used -b strip), carves the embedded file out of the Access
OLE wrapper by locating its magic bytes (JPEG/PNG/PDF/GIF/TIFF) and trimming
trailing OLE junk, uploads to MinIO, and writes service_documents /
policy_documents pointer rows. Row->parent alignment uses mdb-export's
deterministic order (== staged _row_num) for policies and numer_id for
properties. Idempotent (truncate doc tables + overwrite by deterministic key);
--limit/--tables for test passes.
Validated on a limited pass: carved blobs are valid JPEGs (ffd8ff..ffd9)
correctly linked to their policies.
requirements.txt: add boto3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.
migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).
run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.
Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.
Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).
run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_properties.py loads properties, property_services and
trust_accounts from staged DATMEX/PROFILE, resolving each property's customer
FK through customer_legacy_refs. Services are derived from DATMEX's own
account/route/meter fields (the authoritative data); PROFILE flags — merged
best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only
refine each service's `active`. Trust accounts are 1:1 from DATMEX trust
fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4.
Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped),
3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 /
OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts —
counts track the PROFILE enrollment flags.
Reproducibility (asked: dev must be redoable in prod):
- migration/dbenv.py: single DB-target source = deploy/.env.<env>'s
DATABASE_URL; connect(env) + env_arg() (--env, default dev).
- transform_customers.py / transform_properties.py now take --env instead of
hardcoding .env.dev.
- migration/run_all.py: runs every step in dependency order against --env
(optional --stage re-extracts from Access first). Reproducing dev->prod is
`run_all.py --env prod` after deploying the prod stack + prisma db push.
All steps are idempotent (truncate+rebuild); re-run yields identical counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).
Loaded and validated against the dev DB (192.168.4.212:3307):
1682 customers (1172 utilities master + 510 insurance-only)
2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
560 insurance rows linked via num_util, 0 broken cross-refs
542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.
RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macOS has no Access ODBC driver, so the pyodbc-based extract.py could not
run. Rewrite it to shell out to mdbtools (mdb-tables/mdb-export) while
keeping the public interface (connect/list_tables/read_table/
sanitize_column_name) unchanged, so load_staging.py and config.py are
untouched. connect() now returns the file path as the handle (mdbtools has
no persistent connection).
Behavior details:
- -b strip drops LONGBINARY/OLE bytes so blobs never corrupt the CSV
(documents are extracted separately in migration step 4).
- ISO date/datetime output (-D/-T); staging read as text (dtype=str),
only empty fields treated as null (keep_default_na=False) so literal
"NA"/"NULL" data strings survive.
- mdbtools reads deleted/corrupted records mdbtools omits rather than
aborting, so the old per-row skip loop is no longer needed.
Verified end-to-end: full staging load reproduces the original Windows
run (82 tables, 0 errors) with exact row counts (datos2 16000, EFECTIVO
13697, DATGRAL 1172/1070, DATMEX 1520) and recovers all 764 MULT rows
(the pyodbc path lost 1 to HY109 corruption). Accented-column tables
(PROPANO) read cleanly.
requirements.txt: drop pyodbc/pywin32 (Windows-only), keep pandas/pyarrow/
sqlalchemy/pymysql; document the Windows-only DAO catalog as historical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
docs/LEGACY_DATABASES.md documents all three source Access databases
(every table, column, type, and known data-quality quirk) generated
from a live read of the real files, so no Windows/Access driver is
needed to understand their structure going forward.
New migration/ tooling: catalog_schema.py connects to the real files
and walks every table (including excluded scratch tables);
render_catalog_md.py renders that into the doc's appendix. Raw output
checked in at migration/catalog.json so the doc can be regenerated
without touching Access again.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 5