Transform+load: unified customer master (migration step 3, customers)
migration/transform_customers.py builds `customers` + `customer_legacy_refs` from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL is the customer master; insurance DATGRAL folds in via its num_util cross-reference (matches enrich the master with the ID-document fields utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a provenance ref, so the load is auditable and idempotent (truncate+rebuild). Loaded and validated against the dev DB (192.168.4.212:3307): 1682 customers (1172 utilities master + 510 insurance-only) 2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans 560 insurance rows linked via num_util, 0 broken cross-refs 542 merged identities spanning both business lines Spot-checked a merged customer: single record carrying utilities fee + insurance passport/ID enriched in, both provenance refs present. RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -169,11 +169,24 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
|
||||
(folio collides; migrate both), the billing tables are disjoint period runs (union all, no
|
||||
de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The
|
||||
decided union/de-dup rules are in `PLAN.md` migration step 2.
|
||||
4. **Transform + load** (plan step 3) — NEXT. Start with `Customer`/`CustomerLegacyRef` (every
|
||||
other module depends on it): DATGRAL (utilities) is the master; join insurance `DATGRAL`
|
||||
via its `NUM UTIL` cross-ref + name/address matching; COBRO3 excluded from customers. Then
|
||||
the ledger union per the reconciliation rules (both EFECTIVO tables, all three billing tables,
|
||||
provenance-keyed; normalize `monedas` currency variants).
|
||||
4. **Transform + load** (plan step 3) — IN PROGRESS.
|
||||
- **Customers — DONE** (`migration/transform_customers.py`). Loaded into the dev DB: 1682
|
||||
customers (1172 utilities master + 510 insurance-only), 2242 legacy refs (all traceable),
|
||||
560 insurance rows linked via `num_util` with 0 broken refs, **542 merged identities**
|
||||
spanning both business lines; linked customers enriched with insurance-only ID-doc fields.
|
||||
COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present
|
||||
(`load_staging.py --output-dir ./output` first).
|
||||
- **NEXT:** properties+services (DATMEX/PROFILE), policies (+installments/vehicles/drivers/
|
||||
beneficiaries/claims), then the ledger union per the reconciliation rules (both EFECTIVO
|
||||
tables, all three billing tables, provenance-keyed; normalize `monedas` currency variants),
|
||||
SCOTHIA bank register. Each resolves its customer FK through `customer_legacy_refs`.
|
||||
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
|
||||
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
|
||||
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file:
|
||||
`deploy/jorgecuadros-db.stack.yml` (same file deploys prod as `jorgecuadros-prod-db` :3306).
|
||||
Creds in gitignored `deploy/.env.dev`. NOTE: machine `npm` is pnpm-aliased and pnpm ignores
|
||||
the `workspaces` field — full workspace install needs `pnpm-workspace.yaml` or real npm; for
|
||||
now Prisma CLI is run via `npx prisma@5`.
|
||||
5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
|
||||
Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
|
||||
6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the
|
||||
|
||||
Reference in New Issue
Block a user