Transform+load: unified customer master (migration step 3, customers)

migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).

Loaded and validated against the dev DB (192.168.4.212:3307):
  1682 customers (1172 utilities master + 510 insurance-only)
  2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
  560 insurance rows linked via num_util, 0 broken cross-refs
  542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.

RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 18:18:10 -07:00
co-authored by Claude Opus 4.8
parent b5983ba687
commit ec499ca5f5
3 changed files with 299 additions and 5 deletions
+18 -5
View File
@@ -169,11 +169,24 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
(folio collides; migrate both), the billing tables are disjoint period runs (union all, no
de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The
decided union/de-dup rules are in `PLAN.md` migration step 2.
4. **Transform + load** (plan step 3) — NEXT. Start with `Customer`/`CustomerLegacyRef` (every
other module depends on it): DATGRAL (utilities) is the master; join insurance `DATGRAL`
via its `NUM UTIL` cross-ref + name/address matching; COBRO3 excluded from customers. Then
the ledger union per the reconciliation rules (both EFECTIVO tables, all three billing tables,
provenance-keyed; normalize `monedas` currency variants).
4. **Transform + load** (plan step 3) — IN PROGRESS.
- **Customers — DONE** (`migration/transform_customers.py`). Loaded into the dev DB: 1682
customers (1172 utilities master + 510 insurance-only), 2242 legacy refs (all traceable),
560 insurance rows linked via `num_util` with 0 broken refs, **542 merged identities**
spanning both business lines; linked customers enriched with insurance-only ID-doc fields.
COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present
(`load_staging.py --output-dir ./output` first).
- **NEXT:** properties+services (DATMEX/PROFILE), policies (+installments/vehicles/drivers/
beneficiaries/claims), then the ledger union per the reconciliation rules (both EFECTIVO
tables, all three billing tables, provenance-keyed; normalize `monedas` currency variants),
SCOTHIA bank register. Each resolves its customer FK through `customer_legacy_refs`.
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file:
`deploy/jorgecuadros-db.stack.yml` (same file deploys prod as `jorgecuadros-prod-db` :3306).
Creds in gitignored `deploy/.env.dev`. NOTE: machine `npm` is pnpm-aliased and pnpm ignores
the `workspaces` field — full workspace install needs `pnpm-workspace.yaml` or real npm; for
now Prisma CLI is run via `npx prisma@5`.
5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the