Migration: recover blank customer names from secondary legacy tables
DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance), which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)". The blank is real — those cells are empty in the Access files, not lost in extraction — but the rows mostly are not junk: 176 of the 257 carry a property, a policy, or transactions. The old PHP importer handled this by skipping blank-name rows outright (jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was worse than it looks: every other adapter resolved its customer FK through the customer_mapping table those skipped rows never entered, so their properties and policies were silently dropped (customerServiceAdapter.php:45) and their transactions were written against customer_id 0 (customerBalanceAdapter.php:52). So: recover the name instead of skipping. Names come from the secondary tables that still carry them, most trustworthy first — UTILSEG (the office's own hand-maintained name <-> id cross-reference spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow the name its insurance record resolved to. Result: 213 of 257 recovered, 44 still genuinely nameless anywhere in the source. customers.nameSource records which table each recovered name came from, so a reconstructed name is never mistaken for one that was really on the record — the list tags it "nombre recuperado", the detail header names the source, and a still-unnamed customer renders muted italic instead of as a normal name. Also fixes run_all.py: transform_properties and transform_policies truncate service_documents/policy_documents, but blob_extract.py was not in the step list, so a full re-run left the uploaded MinIO objects with no rows pointing at them. Hit exactly that while reloading for this change. Verified end-to-end: full pipeline re-run against dev reproduces every prior count (1682 customers, 1519 properties, 2378 policies, 45861 transactions, 22354 bank rows, 70 documents) with zero orphans, and both apps build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+10
-1
@@ -17,6 +17,10 @@ Then:
|
||||
|
||||
Reproducing dev -> prod is exactly `--env prod` (plus --stage if the staged
|
||||
Parquet isn't present on the machine running it).
|
||||
|
||||
Note: the blob_extract step re-reads the original Access files directly (the
|
||||
blobs are not in the staged Parquet), so the machine running this needs
|
||||
SOURCE_ROOT + mdbtools + MinIO credentials even without --stage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -29,13 +33,18 @@ from pathlib import Path
|
||||
HERE = Path(__file__).parent
|
||||
PY = sys.executable # the venv python running this orchestrator
|
||||
|
||||
# Dependency order — extend as later modules land (policies, transactions, bank).
|
||||
# Dependency order. Every step truncates what it owns, so anything downstream
|
||||
# of a truncated table has to be rebuilt in the same pass — blob_extract is in
|
||||
# this list because transform_properties and transform_policies truncate
|
||||
# service_documents / policy_documents, which would otherwise leave the
|
||||
# uploaded MinIO objects with no rows pointing at them.
|
||||
STEPS = [
|
||||
"transform_customers.py", # customers + customer_legacy_refs (everything FKs to these)
|
||||
"transform_properties.py", # properties + services + trust accounts
|
||||
"transform_policies.py", # policies + installments/vehicles/drivers/benef/claims/adjusters
|
||||
"transform_transactions.py", # shared ledger + type_transactions + exchange_rates
|
||||
"transform_bank.py", # SCOTHIA bank register (no customer FK; independent)
|
||||
"blob_extract.py", # document pointers; must follow properties + policies
|
||||
]
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user