Migration: recover blank customer names from secondary legacy tables

DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance),
which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)".
The blank is real — those cells are empty in the Access files, not lost in
extraction — but the rows mostly are not junk: 176 of the 257 carry a
property, a policy, or transactions.

The old PHP importer handled this by skipping blank-name rows outright
(jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was
worse than it looks: every other adapter resolved its customer FK through
the customer_mapping table those skipped rows never entered, so their
properties and policies were silently dropped (customerServiceAdapter.php:45)
and their transactions were written against customer_id 0
(customerBalanceAdapter.php:52). So: recover the name instead of skipping.

Names come from the secondary tables that still carry them, most trustworthy
first — UTILSEG (the office's own hand-maintained name <-> id cross-reference
spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy
rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow
the name its insurance record resolved to. Result: 213 of 257 recovered, 44
still genuinely nameless anywhere in the source.

customers.nameSource records which table each recovered name came from, so a
reconstructed name is never mistaken for one that was really on the record —
the list tags it "nombre recuperado", the detail header names the source, and
a still-unnamed customer renders muted italic instead of as a normal name.

Also fixes run_all.py: transform_properties and transform_policies truncate
service_documents/policy_documents, but blob_extract.py was not in the step
list, so a full re-run left the uploaded MinIO objects with no rows pointing
at them. Hit exactly that while reloading for this change.

Verified end-to-end: full pipeline re-run against dev reproduces every prior
count (1682 customers, 1519 properties, 2378 policies, 45861 transactions,
22354 bank rows, 70 documents) with zero orphans, and both apps build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 20:49:43 -07:00
co-authored by Claude Opus 4.8
parent da0fa3cb47
commit 594ee7cfca
9 changed files with 179 additions and 14 deletions
+10 -1
View File
@@ -17,6 +17,10 @@ Then:
Reproducing dev -> prod is exactly `--env prod` (plus --stage if the staged
Parquet isn't present on the machine running it).
Note: the blob_extract step re-reads the original Access files directly (the
blobs are not in the staged Parquet), so the machine running this needs
SOURCE_ROOT + mdbtools + MinIO credentials even without --stage.
"""
from __future__ import annotations
@@ -29,13 +33,18 @@ from pathlib import Path
HERE = Path(__file__).parent
PY = sys.executable # the venv python running this orchestrator
# Dependency order — extend as later modules land (policies, transactions, bank).
# Dependency order. Every step truncates what it owns, so anything downstream
# of a truncated table has to be rebuilt in the same pass — blob_extract is in
# this list because transform_properties and transform_policies truncate
# service_documents / policy_documents, which would otherwise leave the
# uploaded MinIO objects with no rows pointing at them.
STEPS = [
"transform_customers.py", # customers + customer_legacy_refs (everything FKs to these)
"transform_properties.py", # properties + services + trust accounts
"transform_policies.py", # policies + installments/vehicles/drivers/benef/claims/adjusters
"transform_transactions.py", # shared ledger + type_transactions + exchange_rates
"transform_bank.py", # SCOTHIA bank register (no customer FK; independent)
"blob_extract.py", # document pointers; must follow properties + policies
]