migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.
Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.
Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).
run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).
Loaded and validated against the dev DB (192.168.4.212:3307):
1682 customers (1172 utilities master + 510 insurance-only)
2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
560 insurance rows linked via num_util, 0 broken cross-refs
542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.
RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
docs/LEGACY_DATABASES.md documents all three source Access databases
(every table, column, type, and known data-quality quirk) generated
from a live read of the real files, so no Windows/Access driver is
needed to understand their structure going forward.
New migration/ tooling: catalog_schema.py connects to the real files
and walks every table (including excluded scratch tables);
render_catalog_md.py renders that into the doc's appendix. Raw output
checked in at migration/catalog.json so the doc can be regenerated
without touching Access again.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 4.8