Transform+load: consolidate all insurance lines into policies (step 3)

migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.

Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.

Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).

run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 18:36:29 -07:00
co-authored by Claude Opus 4.8
parent a680ad2bb0
commit 21899e99bb
3 changed files with 406 additions and 4 deletions
+14 -4
View File
@@ -176,10 +176,20 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
spanning both business lines; linked customers enriched with insurance-only ID-doc fields.
COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present
(`load_staging.py --output-dir ./output` first).
- **NEXT:** properties+services (DATMEX/PROFILE), policies (+installments/vehicles/drivers/
beneficiaries/claims), then the ledger union per the reconciliation rules (both EFECTIVO
tables, all three billing tables, provenance-keyed; normalize `monedas` currency variants),
SCOTHIA bank register. Each resolves its customer FK through `customer_legacy_refs`.
- **Properties — DONE** (`migration/transform_properties.py`): 1519 properties (0 orphans),
3486 services, 553 trust accounts from DATMEX/PROFILE; PROFILE flags matched 1519/1519.
- **Policies — DONE** (`migration/transform_policies.py`): config-driven consolidation of all
insurance lines into `policies` (2378: AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5;
10 skipped for unresolved customer, 0 orphans) + 4678 installments, 1110 vehicles, 513
insured_drivers, 126 beneficiaries, 1 claim, 5 policy_types, 15 insurance_providers, 17
adjusters. Unmodeled coverage columns preserved verbatim in `coveragesJson`. Verified a
unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT
policies — the cross-line customer view works at the data layer.
- **NEXT:** the shared ledger union per the reconciliation rules (both EFECTIVO tables, all
three billing tables, provenance-keyed; normalize `monedas` variants), then SCOTHIA bank
register, then document extraction (step 4, LONGBINARY blobs -> object storage).
- Migration is env-parameterized + reproducible: `run_all.py --env <env>` runs customers ->
properties -> policies in order; add `--stage` to re-extract from Access first.
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file: