jorgecuadros-platform/PLAN.md

Unified Customer / Insurance / Utilities Platform — Migration & Rebuild Plan

Context

Jorge Cuadros & Assoc. runs two lines of business — property/utility management (UTILITIES.accdb) and insurance brokerage (SEGUROS 16.mdb + its linked backend SEGUROS 16_be.mdb) — out of separate, decades-old MS Access databases, plus a third file (SCOTHIA.mdb) that's the office's own Scotiabank checking-account register ("chequera"). The same people are customers of both business lines, but today there's no shared customer record: a person's utility account and their insurance policies live in unrelated systems with independent, inconsistent copies of their name/address/contact info. The bank register is a fourth, disconnected source of truth for the money actually moving through the office's own account.

An earlier attempt to modernize this (jorgecuadros-intra-webapp, PHP) got partway there — it correctly recognized that customers should be unified via a bridge table and began normalizing the flat Access tables into real relational tables (properties, policies, home_policies). But every query in its data layer builds SQL by string-concatenating $_POST directly (src/core/db.php), so it's SQL-injectable end to end, and src/core/auth.php compares passwords in plaintext with no hashing. Per your decision, this isn't worth patching — the app layer will be rebuilt from scratch, in a different stack, in a new repo. The Access data and the schema ideas from the old MySQL dump remain the reference for what the business actually needs.

Goal: one system with a single customer record, from which staff can see and manage that customer's utility services and insurance policies and shared billing/transaction history, replacing both Access files.

Confirmed decisions:

• Stack: Next.js (React + TypeScript) frontend, TypeScript backend (NestJS), MySQL, Prisma ORM.
• New repo, not built on top of jorgecuadros-intra-webapp (that repo is reference-only for business logic/field mappings).
• Full historical data migrates — no cutoff. Source tables are small (largest is ~16k rows), so completeness costs little.

Database engine — revised from PostgreSQL to MySQL. The internal platform doesn't stand alone: a separate, pre-existing customer-facing PHP/MySQL portal (mobile-app-backed — see Infrastructure & Sync below) needs to keep reading live account data from it, and that portal's PHP code uses mysqli and is staying as-is (not being rewritten). Running the internal platform on Postgres would mean maintaining a cross-engine sync into MySQL just to feed the portal; using MySQL everywhere removes that translation layer entirely. Prisma supports MySQL natively, so this only changes the datasource provider in the schema — NestJS/Next.js/Prisma all stay.

Infrastructure & sync architecture

The internal platform and the customer-facing portal are two genuinely separate deployments that need to stay in sync, not one app:

• Internal server — hosted on-prem, private IP (192.168.1.xx), not exposed to the internet. Runs jorgecuadros-platform (this rebuild) and the canonical MySQL database — source of truth for everything staff manage (customers, policies, properties, the full ledger).
• Customer-facing portal — an existing PHP app (with a companion mobile app — its usage-tracking schema, jorgecuadros_app at mysql.freakma.com, logs sections like LOGIN, STATEMENT_ALL_NOW, MAKE_PAYMENT, ORDER_PROPANE, and a devices table of push-notification tokens) running on shared hosting. Out of scope to rebuild — stays exactly as it is. It reads/writes a separate live database, utility_dbo at mysql.freakma.com, which the old internal app already connects to externally (src/core/dbConnection.php: getExternalDBConnection()) — though only for email_alert_log in that codebase; the mobile app almost certainly talks to utility_dbo directly for statements/payments/propane orders. Open item: no schema dump of utility_dbo itself exists yet — only this reference to it — so the sync worker's exact target tables/columns can't be finalized until that's available (export or read-only credentials), the same way the Access files and the jorgecuadros_app dumps were provided.
• New VPS (Hetzner or DigitalOcean, to be provisioned) — runs a MySQL instance and becomes what mysql.freakma.com resolves to. Because it's infrastructure you control (unlike the shared-hosting account), it can run real native MySQL replication — the shared-hosting limitation only ever applied to shared hosting itself, not to a VPS. The shared-hosting portal's PHP code needs no changes beyond the DNS target it already points at.
• Internal server ↔ VPS network path: Tailscale (WireGuard mesh) — both machines join the same tailnet, giving the internal server a way to reach the VPS (and vice versa) without opening any inbound port on either the internal LAN or needing the internal server to be internet-facing.

Sync mechanism:

• Internal → VPS (statements, balances, customer profile changes): one-way native MySQL replication (binlog/GTID-based) from the internal MySQL (source) to the VPS MySQL (replica), over the Tailscale link. Standard, well-supported, and only carries the subset of tables/columns the portal actually needs to read — internal-only data (staff notes, adjuster info, activity logs) should live in tables intentionally excluded from what's replicated, so the more internet-exposed side never receives more than the portal requires.
• VPS → Internal (payment submissions, propane order requests): one-way replication can't carry writes back, and multi-master MySQL replication is fragile enough to avoid for a system this size. Instead: a small set of unreplicated inbox tables on the VPS (payment_submissions, propane_order_requests, matching whatever utility_dbo actually uses once its schema is available) that the portal writes to directly, polled every 1–5 minutes by a worker on the internal server (over Tailscale) that turns new rows into real Transaction/propane-request records and marks them processed.

Source data inventory

Extracted live via pyodbc + the Windows Access ODBC driver (dump_schema.py, tables/columns/row counts/PKs for all four files). Access has no real primary or foreign keys defined anywhere — every relationship below is inferred from column-name conventions (NUM id, NUMid, NUMER ID, IDISEG, IDIUT), not enforced constraints, so the migration has to independently validate every join.

UTILITIES.accdb (52 tables, ~538MB, dominated by embedded document blobs):

• DATMEX (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 LONGBINARY columns holding scanned utility bills/IDs.
• DATGRAL / COBRO3 (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). COBRO3 looks like a filtered snapshot of DATGRAL, not a distinct entity.
• PROFILE (1,520) — per-property service enrollment flags, joins 1:1 with DATMEX by NUMERID.
• EFECTIVO (13,697), EFECTIVO FM3 (627), EFECTIVO_BACKUP (12,387), CHEQUE FM3 (157) — cash/check transaction ledgers, near-identical shape, apparent year/program snapshots rather than distinct data.
• FEE ANUAL (1,082), datos2 (16,000), fee15 (1,030), billing (0) — recurring billing/fee transaction logs, again near-duplicate shapes across different periods/exports — needs de-duplication logic, not a straight union.
• TRUSTVENCE (549) — bank trust account fee due-dates.
• TIPO HIST (2,301) — exchange-rate history (date/hour/rate), referenced by the MONEDAS (currency) field used throughout.
• TYPE OF TRX (79) — ES/EN transaction-type lookup (already mirrored as type_transactions in the old MySQL schema — reuse that mapping).
• Scratch/working tables to exclude from migration: BANCO EDITOR, Errores de pegado, TABLE1, PARA BILLING*, FALTANTES *, TELEFONOS FECHAS, TIT, BORRA-equivalents. A few tables (PROPANO, datosfreak, datosfreak2, LUZ TODOS, etc.) have column names with characters pyodbc's UTF-16 path can't decode — need re-extraction with a Latin-1/CP1252 fallback before they can even be classified as real data vs. scratch.

SEGUROS 16_be.mdb (64 tables, ~882MB; SEGUROS 16.mdb is an empty Access "frontend" shell — all real data lives in _be):

• DATGRAL (1,070 rows) — customer master, same shape as the utilities DATGRAL, and critically has a NUM UTIL column — a literal cross-reference to the utilities customer's NUM id. This is the join key for building the unified customer table; it's also what the old app's customer_mapping table was clearly reverse-engineered from.
• One flat table per insurance line of business, all sharing the same repeated-column pattern (policy #, coverage dates, 4 hardcoded payment installments each with its own date/amount/check#/currency, premium breakdown, liquidation status, address, observations, embedded document blobs):
  • INCENDIO (fire), MULT (multi-risk/home — the most complete, matches old app's home_policies), M EMPR (commercial property)
  • Auto: TABLA AUTOS, TABLA AUTOS AMPL, TABLA AUTOS AMPL R, TABLA AUTOS LIMIT, TABLA AUTOS LIMIT R, TABLA AUTOS RC R, MCA2 — seven variants of essentially one "auto policy" concept, differentiated by coverage tier/program. MCA2 also hardcodes 3 driver+vehicle slots as repeated columns (MARCA1/2/3, PLACA1/2/3, NAME1/2/3, LIC1/2/3...) that need to unpivot into child rows.
  • LICENCIAS (driver's-license insurance, MX-specific product) — also hardcodes 3 named insureds per policy.
• BENEF (768) — policy beneficiaries, already a clean child table (policy#, name, address, phone, email).
• DATOS (2 rows now, but structurally real) — claims/siniestros: claim date, description, adjuster, settlement amounts, checks, documents.
• AJUSTADORES / AJUSTADORESATLAS — adjuster contact lists.
• EDOSEG — per-policy premium/account statement summary.
• EFECTIVO (295) — cash ledger, same shape as the utilities one.
• UTILSEG (1,582) — appears to be the utilities↔insurance customer cross-reference table.
• TABLA LIQUIDA * / TABLE DATOS LIQUIDA * — settlement/liquidation batches per policy line (matches liquidation_number already in the old MySQL policies table).
• Exclude from data migration: the * MENS / *MENSAJE tables (AMPL MENS, IN MENS, LIC MENS, MCA2 MENS, ME MENS, MF MENS, RC MENSAJE, RC R MENS) are mail-merge document templates (letters, certificates) stored as blobs, not customer data — these get reimplemented as PDF templates in the new app, not migrated as rows. TODOSJC, TODOS, vigenta casa y auto unicos look like materialized Access query results (saved reports), not source-of-truth data — exclude, and if the report itself is still needed, rebuild it as a real query against the new schema.

SCOTHIA.mdb (7 tables, ~3MB — the office's own Scotiabank checking-account register, "chequera"). Much simpler than the other two files: this is the company's operating bank account, not customer-facing data.

• DATOS E (15,406 rows) — expenses/outgoing (EGRESO): date, transaction type, check/reference NUM, CONCEPTO (payee/description), amount, OPERADO (cleared flag), notes, and a spelled-out amount-in-words field (CANTIDAD EN LETRA, standard Mexican check-writing convention).
• DATOS I (6,948 rows) — income/incoming (INGRESO): same shape, plus a TRANSFERIDO (transferred) flag instead of the amount-in-words field.
• TABLA RAMODOS (66 rows) — a category/business-line lookup ("ramo" = line of business in Mexican insurance terminology) — this is almost certainly what CONCEPTO entries get classified against, i.e. the link between a bank transaction and which part of the business (insurance line, utility service, trust, etc.) it belongs to.
• ban (1 row) — just holds the bank's name; a config singleton, not data.
• INFORME / INFORME BA (0 rows each) and FECHAIF (1 row) — Access report/query scratch tables (date-range parameters and a report shell), same pattern as the scratch tables in the other two files — exclude from migration.

Target architecture

• Frontend: Next.js (App Router) + TypeScript + React. Server components for data-heavy list/detail views (customers, policies, statements); client components for interactive forms.
• Backend: NestJS (TypeScript) REST API — modular by domain (customers, insurance, utilities, billing, auth, admin), matching the module boundaries below. Gives you DI, guards for auth/authorization, and a validation pipeline (class-validator) for free, which directly replaces the old app's biggest weakness (no input validation, no parameterization).
• Database: MySQL, accessed via Prisma (schema-as-code, migrations, generates a typed client — eliminates the raw-SQL-injection class of bug entirely since Prisma parameterizes everything). See Infrastructure & Sync above for why MySQL rather than Postgres.
• Auth: NestJS + Passport, sessions or JWT (pick one during build), bcrypt/argon2 password hashing, role-based guards replacing the old $_SESSION['level']/['role'] checks.
• File/document storage: object storage (S3-compatible) for the scanned documents currently trapped as Access LONGBINARY blobs — extract once during migration, store as files, keep only the pointer + metadata in MySQL. The old repo already anticipated this (src/objects/s3.php exists but appears unused) — same idea, implemented for real this time.
• Deployment: keep Docker as the packaging mechanism (already proven for this project) with a fresh Dockerfile/docker-compose.yml for the Node services + MySQL, running on the internal server described above; CI can stay on Jenkins if that's still the team's pipeline, or move to GitHub Actions if the new repo lives somewhere other than git.freakma.com — flagged as an open decision below since it wasn't part of the stack questions asked.

Target data model (by domain)

All tables get a surrogate id (uuid or serial) plus, where the row came from a legacy table, provenance columns (legacy_source_db, legacy_source_table, legacy_id) so every migrated row can be traced back to its Access original for spot-checking and reconciliation — and so the ETL can be re-run idempotently (upsert on provenance key) as migration bugs get found and fixed.

Identity (the actual point of this project):

• customers — one row per real person/entity, merged from utilities DATGRAL/COBRO3 and insurance DATGRAL, matched via the NUM UTIL cross-reference plus name/address fuzzy-matching for anyone missing that link. Holds name, addresses (MX + US), phones, email, ID document info, status, currency preference.
• customer_legacy_refs — generalizes the old customer_mapping table: (customer_id, source_system, source_table, legacy_numeric_id), one row per legacy record folded into this customer. This is what makes "unified customer base" actually queryable and keeps the merge auditable.

Insurance domain:

• insurance_providers, policy_types (carry over from old schema, already reasonable)
• policies — generic header (policy #, type, provider, customer, dates, premiums, agent, liquidation status), consolidating INCENDIO/MULT/M EMPR/all six auto-table variants/LICENCIAS into one table with a policy_type discriminator, instead of one Access table per line of business.
• policy_payment_installments — unpivots the 4 hardcoded payment-installment columns (1ER PAGO/FECHA PAGO/NO CHEQUE, ... 2, ... 3, ... 4) into rows: due sequence, amount, currency, paid date, check/reference number.
• vehicles — unpivots MCA2's 3 hardcoded vehicle slots (and the single-vehicle auto tables) into one row per vehicle, FK'd to policy + customer.
• insured_drivers — same unpivot for the repeated named-insured/license columns in MCA2/LICENCIAS.
• properties (shared with utilities domain — see below), policy_beneficiaries (from BENEF, already clean), claims (from DATOS), adjusters (from AJUSTADORES*), policy_documents (extracted blobs, typed: ID, prior policy copy, damage photo, etc.).

Utilities domain:

• properties — one row per property (from DATMEX), FK'd to customers, shared with insurance so a property can carry both a home-insurance policy and utility service enrollments — this is the second half of "unified."
• property_services — one row per enrolled service per property (water/electric/gas/cable/trust/property-tax/federal-zone), unpivoting DATMEX's inline service columns and PROFILE's enrollment flags into real rows with account #, route, meter #, due day.
• service_documents (extracted blobs), trust_accounts (from TRUSTVENCE).

Shared financial ledger (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):

• transactions — unifies utilities' EFECTIVO/EFECTIVO FM3/EFECTIVO_BACKUP/FEE ANUAL/datos2/fee15/billing/CHEQUE FM3/IVA 2015 and insurance's EFECTIVO, tagged by domain (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive.
• exchange_rates (from TIPO HIST), type_transactions (carry over ES/EN lookup as-is).
• bank_transactions — the company's own operating bank register, from SCOTHIA's DATOS E/DATOS I unified into one signed-amount table (income positive, expense negative) with a category FK to business_line_categories (from TABLA RAMODOS) and a cleared/operado flag. This is deliberately separate from customer-facing transactions — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the business_line_categories lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
• business_line_categories (from TABLA RAMODOS).

Admin/shared: users (hashed passwords, roles), activity_logs, email_templates/email_campaigns/email_log (carry over the old schema's intent, rebuilt on the new stack).

Migration strategy

Given the amount of near-duplicate/overlapping data across snapshot tables (multiple EFECTIVO* variants, multiple year-stamped billing tables, COBRO3 vs DATGRAL), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.

1. Raw staging load: dump every non-scratch Access table 1:1 into a MySQL staging (per-source schema/database, e.g. stg_utilities/stg_seguros/stg_scothia) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as migration/load_staging.py in the new repo — see Status below. This is the audit trail — nothing is transformed yet. Extraction toolchain note: the original build used pyodbc + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (migration/extract.py) is being reworked to use mdbtools (mdb-tables/mdb-export, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. PROPANO), and per-table exports cleanly. mdbtools does not extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in migration/objects.json + docs/LEGACY_DATABASES_OBJECTS.md, so nothing is lost. The only piece needing extra handling under mdbtools is LONGBINARY blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
2. Reconciliation pass: for each set of overlapping tables (the EFECTIVO variants, the billing-period tables, DATGRAL vs COBRO3), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
3. Transform + load: SQL/TypeScript scripts (versioned in the new repo under migration/) that read staging, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing legacy_* provenance on every row.
4. Document extraction: separate one-off script pulls every LONGBINARY column out to files (named by provenance key), uploads to object storage, and inserts the corresponding *_documents metadata row.
5. Validation: row-count and spot-check reconciliation between staging and final tables (e.g., every legacy customer has exactly one customers row via customer_legacy_refs; sum of migrated transaction amounts per customer matches sum in staging).

Build sequencing

1. Repo scaffold (Next.js + NestJS + Prisma + MySQL, Docker Compose for local dev), CI pipeline, auth module with hashed passwords and role guards.
2. Prisma schema for the full data model above; run migration steps 1–2 (staging load + reconciliation reports) against real data early, since that's where the biggest unknowns are (do NUM UTIL and name-matching actually cover everyone? how bad is the snapshot-table duplication?).
3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.
4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
7. Bank register module (bank_transactions/business_line_categories from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
8. VPS provisioning + Tailscale + MySQL replication setup. utility_dbo's schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (my-jorgecuadros-web) that reads/writes it.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in utility_dbo: peticion_gas (propane requests), PayPal payment writes, notifications_settings, verification_codes — these define the VPS→internal inbox set.
10. Reports/email campaigns/admin — parity with old app's reports.php/emailCampaigns.php intent, rebuilt properly.

Status

Repo scaffolded at jorgecuadros-platform/: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to git.mancinas.io/rmancinas/jorgecuadros-platform (branch master, 3 commits, working tree clean). migration/load_staging.py (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a cursor.columns() UTF-16 decode bug on several tables (worked around by reading metadata from cursor.description instead) and one Jet/ACE-level corrupted record in MULT (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at jorgecuadros-platform/RESUME.md — read both together.

Environment moved Windows → macOS. Source Access files now live at ~/Downloads/JorgeCuadros-Legacy/ (all four: UTILITIES.accdb, SEGUROS 16.mdb, SEGUROS 16_be.mdb, SCOTHIA.mdb). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but no Access ODBC driver and no node_modules installed yet. Consequence: the pyodbc-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), node_modules needs npm install, and the staging output (gitignored Parquet) must be regenerated from scratch.

Portal live DB now in hand. utility_dbo.sql (1.3 GB, 55 tables) and the portal codebase my-jorgecuadros-web (PHP/mysqli, Gitea repo, themed classic/modern, ~397 PHP files, core in scripts/functions.php) are both on disk — resolving the long-standing "utility_dbo schema unknown" blocker. Sync-relevant tables identified: statements/money (utility_bills, accounting, email_alert_log), customer/property (home_owners, home_index, condominium, management, hoa_management, trust_assist), portal-facing policy views (fm2/fm3/fmt, full_coverage, mx_liability, usa_liability), and portal write points (peticion_gas, PayPal payments, notifications_settings, verification_codes). A second dump, jorgecuadros.sql (38 MB, 11 tables — pagos/pagosemail/PROPANO/TRUSTVENCE/etc.), appears to be an older/partial export, not the portal live DB.

Decisions (locked)

• Stack: Next.js + NestJS + Prisma + MySQL (locked earlier — see engine rationale above).
• Extraction toolchain (macOS): mdbtools, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in objects.json/docs/; only LONGBINARY blob extraction (step 4) needs extra handling under mdbtools.
• i18n: Spanish-first UI — matches the source data and staff usage.
• CI/CD: Gitea Actions on git.mancinas.io — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the portainer-gitea-deploy pattern already used on this LAN). Jenkins/git.freakma.com dropped.
• utility_dbo: resolved — full dump + portal code available (see Status).

Open items (ops, not design)

• VPS provisioning: provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
• Old external-DB credential (hardcoded plaintext MySQL password in the old repo's dbConnection.php, in git history) — rotate it regardless, since it's already exposed.

Verification

• Migration: automated row-count/sum reconciliation between staging and final schema per table group (see step 5 above), run as part of the migration script, not a manual spot-check.
• App: standard NestJS unit/integration tests per module (auth guards, Prisma queries), Playwright/Cypress e2e for the core "look up a customer, see their unified policies + services + statement" flow — the thing the whole project exists to deliver.
• Sync: once the VPS replica and inbox tables exist, verify replication lag stays low (a few seconds to low minutes) and that a payment/propane submission on the portal reliably shows up in the internal app within one polling interval, before relying on it operationally.
• Before cutover: run the new app against migrated data side-by-side with the live Access files for a period, comparing balances/statements for a sample of active customers to catch migration logic errors before the Access files are retired.