Transform+load: consolidate all insurance lines into policies (step 3)
migration/transform_policies.py folds every insurance Access table into one `policies` table (policy_types discriminator) plus child tables, via a per-table declarative mapping that absorbs the column-name variance (num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source column not explicitly modeled — the type-specific coverage amounts — is preserved verbatim in coveragesJson, so consolidation loses nothing. Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago + pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries (by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and builds policy_types + insurance_providers lookups. Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678 installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15 providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a customer carrying both a utility property and MULT policies (the unified cross-line view). run_all.py: add policies to the ordered pipeline. Customer FK resolves through insurance customer_legacy_refs, so this runs after customers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -176,10 +176,20 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
|
||||
spanning both business lines; linked customers enriched with insurance-only ID-doc fields.
|
||||
COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present
|
||||
(`load_staging.py --output-dir ./output` first).
|
||||
- **NEXT:** properties+services (DATMEX/PROFILE), policies (+installments/vehicles/drivers/
|
||||
beneficiaries/claims), then the ledger union per the reconciliation rules (both EFECTIVO
|
||||
tables, all three billing tables, provenance-keyed; normalize `monedas` currency variants),
|
||||
SCOTHIA bank register. Each resolves its customer FK through `customer_legacy_refs`.
|
||||
- **Properties — DONE** (`migration/transform_properties.py`): 1519 properties (0 orphans),
|
||||
3486 services, 553 trust accounts from DATMEX/PROFILE; PROFILE flags matched 1519/1519.
|
||||
- **Policies — DONE** (`migration/transform_policies.py`): config-driven consolidation of all
|
||||
insurance lines into `policies` (2378: AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5;
|
||||
10 skipped for unresolved customer, 0 orphans) + 4678 installments, 1110 vehicles, 513
|
||||
insured_drivers, 126 beneficiaries, 1 claim, 5 policy_types, 15 insurance_providers, 17
|
||||
adjusters. Unmodeled coverage columns preserved verbatim in `coveragesJson`. Verified a
|
||||
unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT
|
||||
policies — the cross-line customer view works at the data layer.
|
||||
- **NEXT:** the shared ledger union per the reconciliation rules (both EFECTIVO tables, all
|
||||
three billing tables, provenance-keyed; normalize `monedas` variants), then SCOTHIA bank
|
||||
register, then document extraction (step 4, LONGBINARY blobs -> object storage).
|
||||
- Migration is env-parameterized + reproducible: `run_all.py --env <env>` runs customers ->
|
||||
properties -> policies in order; add `--stage` to re-extract from Access first.
|
||||
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
|
||||
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
|
||||
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file:
|
||||
|
||||
Reference in New Issue
Block a user