jorgecuadros-platform/PLAN.md

# Unified Customer / Insurance / Utilities Platform — Migration & Rebuild Plan

## Context

Jorge Cuadros & Assoc. runs two lines of business — property/utility management (`UTILITIES.accdb`) and insurance brokerage (`SEGUROS 16.mdb` + its linked backend `SEGUROS 16_be.mdb`) — out of separate, decades-old MS Access databases, plus a third file (`SCOTHIA.mdb`) that's the office's own Scotiabank checking-account register ("chequera"). The same people are customers of both business lines, but today there's no shared customer record: a person's utility account and their insurance policies live in unrelated systems with independent, inconsistent copies of their name/address/contact info. The bank register is a fourth, disconnected source of truth for the money actually moving through the office's own account.

An earlier attempt to modernize this (`jorgecuadros-intra-webapp`, PHP) got partway there — it correctly recognized that customers should be unified via a bridge table and began normalizing the flat Access tables into real relational tables (`properties`, `policies`, `home_policies`). But every query in its data layer builds SQL by string-concatenating `$_POST` directly (`src/core/db.php`), so it's SQL-injectable end to end, and `src/core/auth.php` compares passwords in plaintext with no hashing. Per your decision, this isn't worth patching — the app layer will be rebuilt from scratch, in a different stack, in a new repo. The Access data and the schema ideas from the old MySQL dump remain the reference for what the business actually needs.

Goal: one system with a single customer record, from which staff can see and manage that customer's utility services *and* insurance policies *and* shared billing/transaction history, replacing both Access files.

**Confirmed decisions:**
- Stack: Next.js (React + TypeScript) frontend, TypeScript backend (NestJS), **MySQL**, Prisma ORM.
- New repo, not built on top of `jorgecuadros-intra-webapp` (that repo is reference-only for business logic/field mappings).
- Full historical data migrates — no cutoff. Source tables are small (largest is ~16k rows), so completeness costs little.

**Database engine — revised from PostgreSQL to MySQL.** The internal platform doesn't stand alone: a separate, pre-existing customer-facing PHP/MySQL portal (mobile-app-backed — see Infrastructure & Sync below) needs to keep reading live account data from it, and that portal's PHP code uses `mysqli` and is staying as-is (not being rewritten). Running the internal platform on Postgres would mean maintaining a cross-engine sync into MySQL just to feed the portal; using MySQL everywhere removes that translation layer entirely. Prisma supports MySQL natively, so this only changes the `datasource` provider in the schema — NestJS/Next.js/Prisma all stay.

## Infrastructure & sync architecture

The internal platform and the customer-facing portal are two genuinely separate deployments that need to stay in sync, not one app:

- **Internal server** — hosted on-prem, private IP (`192.168.1.xx`), **not exposed to the internet**. Runs `jorgecuadros-platform` (this rebuild) and the canonical MySQL database — source of truth for everything staff manage (customers, policies, properties, the full ledger).
- **Customer-facing portal** — an existing PHP app (with a companion mobile app — its usage-tracking schema, `jorgecuadros_app` at `mysql.freakma.com`, logs sections like `LOGIN`, `STATEMENT_ALL_NOW`, `MAKE_PAYMENT`, `ORDER_PROPANE`, and a `devices` table of push-notification tokens) running on shared hosting. **Out of scope to rebuild** — stays exactly as it is. It reads/writes a separate live database, `utility_dbo` at `mysql.freakma.com`, which the old internal app already connects to externally (`src/core/dbConnection.php: getExternalDBConnection()`) — though only for `email_alert_log` in that codebase; the mobile app almost certainly talks to `utility_dbo` directly for statements/payments/propane orders. **Open item:** no schema dump of `utility_dbo` itself exists yet — only this reference to it — so the sync worker's exact target tables/columns can't be finalized until that's available (export or read-only credentials), the same way the Access files and the `jorgecuadros_app` dumps were provided.
- **New VPS** (Hetzner or DigitalOcean, to be provisioned) — runs a MySQL instance and becomes what `mysql.freakma.com` resolves to. Because it's infrastructure you control (unlike the shared-hosting account), it can run **real native MySQL replication** — the shared-hosting limitation only ever applied to shared hosting itself, not to a VPS. The shared-hosting portal's PHP code needs no changes beyond the DNS target it already points at.
- **Internal server ↔ VPS network path:** Tailscale (WireGuard mesh) — both machines join the same tailnet, giving the internal server a way to reach the VPS (and vice versa) without opening any inbound port on either the internal LAN or needing the internal server to be internet-facing.

**Sync mechanism:**
- **Internal → VPS (statements, balances, customer profile changes):** one-way native MySQL replication (binlog/GTID-based) from the internal MySQL (source) to the VPS MySQL (replica), over the Tailscale link. Standard, well-supported, and only carries the subset of tables/columns the portal actually needs to read — internal-only data (staff notes, adjuster info, activity logs) should live in tables intentionally excluded from what's replicated, so the more internet-exposed side never receives more than the portal requires.
- **VPS → Internal (payment submissions, propane order requests):** one-way replication can't carry writes back, and multi-master MySQL replication is fragile enough to avoid for a system this size. Instead: a small set of unreplicated **inbox tables** on the VPS (`payment_submissions`, `propane_order_requests`, matching whatever `utility_dbo` actually uses once its schema is available) that the portal writes to directly, polled every 1–5 minutes by a worker on the internal server (over Tailscale) that turns new rows into real `Transaction`/propane-request records and marks them processed.

## Source data inventory

Extracted live via `pyodbc` + the Windows Access ODBC driver (`dump_schema.py`, tables/columns/row counts/PKs for all four files). Access has **no real primary or foreign keys defined anywhere** — every relationship below is inferred from column-name conventions (`NUM id`, `NUMid`, `NUMER ID`, `IDISEG`, `IDIUT`), not enforced constraints, so the migration has to independently validate every join.

**UTILITIES.accdb** (52 tables, ~538MB, dominated by embedded document blobs):
- `DATMEX` (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 `LONGBINARY` columns holding scanned utility bills/IDs.
- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). ~~`COBRO3` looks like a filtered snapshot of `DATGRAL`~~ **Reconciliation (step 2) corrected this: `COBRO3` is a charge batch (constant fee 75), not a customer snapshot — `DATGRAL` is the sole customer master.**
- `PROFILE` (1,520) — per-property service enrollment flags, joins 1:1 with `DATMEX` by `NUMERID`.
- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers. ~~apparent year/program snapshots rather than distinct data~~ **Reconciliation (step 2) corrected this: EFECTIVO and EFECTIVO_BACKUP are near-disjoint ledgers (folio collides; only 2 business-key matches) — migrate both, no folio de-dup. FM3 tables are a separate fee stream.**
- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs. ~~near-duplicate shapes ... needs de-duplication~~ **Reconciliation (step 2) corrected this: disjoint billing runs from different periods (2017 / 2018 / 2025–26), zero real-identity overlap — straight union, no de-dup.**
- `TRUSTVENCE` (549) — bank trust account fee due-dates.
- `TIPO HIST` (2,301) — exchange-rate history (date/hour/rate), referenced by the `MONEDAS` (currency) field used throughout.
- `TYPE OF TRX` (79) — ES/EN transaction-type lookup (already mirrored as `type_transactions` in the old MySQL schema — reuse that mapping).
- Scratch/working tables to **exclude** from migration: `BANCO EDITOR`, `Errores de pegado`, `TABLE1`, `PARA BILLING*`, `FALTANTES *`, `TELEFONOS FECHAS`, `TIT`, `BORRA`-equivalents. A few tables (`PROPANO`, `datosfreak`, `datosfreak2`, `LUZ TODOS`, etc.) have column names with characters pyodbc's UTF-16 path can't decode — need re-extraction with a Latin-1/CP1252 fallback before they can even be classified as real data vs. scratch.

**SEGUROS 16_be.mdb** (64 tables, ~882MB; `SEGUROS 16.mdb` is an empty Access "frontend" shell — all real data lives in `_be`):
- `DATGRAL` (1,070 rows) — customer master, **same shape as the utilities `DATGRAL`**, and critically has a `NUM UTIL` column — a literal cross-reference to the utilities customer's `NUM id`. This is the join key for building the unified customer table; it's also what the old app's `customer_mapping` table was clearly reverse-engineered from.
- One flat table per insurance line of business, all sharing the same repeated-column pattern (policy #, coverage dates, 4 hardcoded payment installments each with its own date/amount/check#/currency, premium breakdown, liquidation status, address, observations, embedded document blobs):
  - `INCENDIO` (fire), `MULT` (multi-risk/home — the most complete, matches old app's `home_policies`), `M EMPR` (commercial property)
  - Auto: `TABLA AUTOS`, `TABLA AUTOS AMPL`, `TABLA AUTOS AMPL R`, `TABLA AUTOS LIMIT`, `TABLA AUTOS LIMIT R`, `TABLA AUTOS RC R`, `MCA2` — seven variants of essentially one "auto policy" concept, differentiated by coverage tier/program. `MCA2` also hardcodes **3 driver+vehicle slots as repeated columns** (`MARCA1/2/3`, `PLACA1/2/3`, `NAME1/2/3`, `LIC1/2/3`...) that need to unpivot into child rows.
  - `LICENCIAS` (driver's-license insurance, MX-specific product) — also hardcodes 3 named insureds per policy.
- `BENEF` (768) — policy beneficiaries, already a clean child table (policy#, name, address, phone, email).
- `DATOS` (2 rows now, but structurally real) — claims/siniestros: claim date, description, adjuster, settlement amounts, checks, documents.
- `AJUSTADORES` / `AJUSTADORESATLAS` — adjuster contact lists.
- `EDOSEG` — per-policy premium/account statement summary.
- `EFECTIVO` (295) — cash ledger, same shape as the utilities one.
- `UTILSEG` (1,582) — appears to be the utilities↔insurance customer cross-reference table.
- `TABLA LIQUIDA *` / `TABLE DATOS LIQUIDA *` — settlement/liquidation batches per policy line (matches `liquidation_number` already in the old MySQL `policies` table).
- **Exclude from data migration:** the `* MENS` / `*MENSAJE` tables (`AMPL MENS`, `IN MENS`, `LIC MENS`, `MCA2 MENS`, `ME MENS`, `MF MENS`, `RC MENSAJE`, `RC R MENS`) are mail-merge document *templates* (letters, certificates) stored as blobs, not customer data — these get reimplemented as PDF templates in the new app, not migrated as rows. `TODOSJC`, `TODOS`, `vigenta casa y auto unicos` look like materialized Access query results (saved reports), not source-of-truth data — exclude, and if the report itself is still needed, rebuild it as a real query against the new schema.

**SCOTHIA.mdb** (7 tables, ~3MB — the office's own Scotiabank checking-account register, "chequera"). Much simpler than the other two files: this is the company's operating bank account, not customer-facing data.
- `DATOS E` (15,406 rows) — expenses/outgoing (`EGRESO`): date, transaction type, check/reference `NUM`, `CONCEPTO` (payee/description), amount, `OPERADO` (cleared flag), notes, and a spelled-out amount-in-words field (`CANTIDAD EN LETRA`, standard Mexican check-writing convention).
- `DATOS I` (6,948 rows) — income/incoming (`INGRESO`): same shape, plus a `TRANSFERIDO` (transferred) flag instead of the amount-in-words field.
- `TABLA RAMODOS` (66 rows) — a category/business-line lookup ("ramo" = line of business in Mexican insurance terminology) — this is almost certainly what `CONCEPTO` entries get classified against, i.e. the link between a bank transaction and which part of the business (insurance line, utility service, trust, etc.) it belongs to.
- `ban` (1 row) — just holds the bank's name; a config singleton, not data.
- `INFORME` / `INFORME BA` (0 rows each) and `FECHAIF` (1 row) — Access report/query scratch tables (date-range parameters and a report shell), same pattern as the scratch tables in the other two files — **exclude** from migration.

## Target architecture

- **Frontend:** Next.js (App Router) + TypeScript + React. Server components for data-heavy list/detail views (customers, policies, statements); client components for interactive forms.
- **Backend:** NestJS (TypeScript) REST API — modular by domain (customers, insurance, utilities, billing, auth, admin), matching the module boundaries below. Gives you DI, guards for auth/authorization, and a validation pipeline (`class-validator`) for free, which directly replaces the old app's biggest weakness (no input validation, no parameterization).
- **Database:** MySQL, accessed via Prisma (schema-as-code, migrations, generates a typed client — eliminates the raw-SQL-injection class of bug entirely since Prisma parameterizes everything). See Infrastructure & Sync above for why MySQL rather than Postgres.
- **Auth:** NestJS + Passport, sessions or JWT (pick one during build), `bcrypt`/`argon2` password hashing, role-based guards replacing the old `$_SESSION['level']/['role']` checks.
- **File/document storage:** object storage (S3-compatible) for the scanned documents currently trapped as Access `LONGBINARY` blobs — extract once during migration, store as files, keep only the pointer + metadata in MySQL. The old repo already anticipated this (`src/objects/s3.php` exists but appears unused) — same idea, implemented for real this time.
- **Deployment:** keep Docker as the packaging mechanism (already proven for this project) with a fresh `Dockerfile`/`docker-compose.yml` for the Node services + MySQL, running on the internal server described above; CI can stay on Jenkins if that's still the team's pipeline, or move to GitHub Actions if the new repo lives somewhere other than `git.freakma.com` — flagged as an open decision below since it wasn't part of the stack questions asked.

## Target data model (by domain)

All tables get a surrogate `id` (uuid or serial) plus, where the row came from a legacy table, provenance columns (`legacy_source_db`, `legacy_source_table`, `legacy_id`) so every migrated row can be traced back to its Access original for spot-checking and reconciliation — and so the ETL can be re-run idempotently (upsert on provenance key) as migration bugs get found and fixed.

**Identity (the actual point of this project):**
- `customers` — one row per real person/entity, merged from utilities `DATGRAL`/`COBRO3` and insurance `DATGRAL`, matched via the `NUM UTIL` cross-reference plus name/address fuzzy-matching for anyone missing that link. Holds name, addresses (MX + US), phones, email, ID document info, status, currency preference.
- `customer_legacy_refs` — generalizes the old `customer_mapping` table: `(customer_id, source_system, source_table, legacy_numeric_id)`, one row per legacy record folded into this customer. This is what makes "unified customer base" actually queryable and keeps the merge auditable.

**Insurance domain:**
- `insurance_providers`, `policy_types` (carry over from old schema, already reasonable)
- `policies` — generic header (policy #, type, provider, customer, dates, premiums, agent, liquidation status), consolidating `INCENDIO`/`MULT`/`M EMPR`/all six auto-table variants/`LICENCIAS` into one table with a `policy_type` discriminator, instead of one Access table per line of business.
- `policy_payment_installments` — unpivots the 4 hardcoded payment-installment columns (`1ER PAGO`/`FECHA PAGO`/`NO CHEQUE`, `... 2`, `... 3`, `... 4`) into rows: due sequence, amount, currency, paid date, check/reference number.
- `vehicles` — unpivots `MCA2`'s 3 hardcoded vehicle slots (and the single-vehicle auto tables) into one row per vehicle, FK'd to policy + customer.
- `insured_drivers` — same unpivot for the repeated named-insured/license columns in `MCA2`/`LICENCIAS`.
- `properties` (shared with utilities domain — see below), `policy_beneficiaries` (from `BENEF`, already clean), `claims` (from `DATOS`), `adjusters` (from `AJUSTADORES*`), `policy_documents` (extracted blobs, typed: ID, prior policy copy, damage photo, etc.).

**Utilities domain:**
- `properties` — one row per property (from `DATMEX`), FK'd to `customers`, shared with insurance so a property can carry both a home-insurance policy and utility service enrollments — this is the second half of "unified."
- `property_services` — one row per enrolled service per property (water/electric/gas/cable/trust/property-tax/federal-zone), unpivoting `DATMEX`'s inline service columns and `PROFILE`'s enrollment flags into real rows with account #, route, meter #, due day.
- `service_documents` (extracted blobs), `trust_accounts` (from `TRUSTVENCE`).

**Shared financial ledger** (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive. **`amount` is signed:** negative = charge (cargo), positive = credit (abono), so `SUM(amount)` per customer per currency *is* the balance — negative means the customer owes the office. The two currencies are never summed together (see the billing module note in Build sequencing step 6).
- `exchange_rates` (from `TIPO HIST`), `type_transactions` (carry over ES/EN lookup as-is).
- `bank_transactions` — the company's own operating bank register, from SCOTHIA's `DATOS E`/`DATOS I` unified into one signed-amount table (income positive, expense negative) with a `category` FK to `business_line_categories` (from `TABLA RAMODOS`) and a `cleared`/`operado` flag. This is deliberately **separate** from customer-facing `transactions` — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the `business_line_categories` lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
- `business_line_categories` (from `TABLA RAMODOS`).

**Admin/shared:** `users` (hashed passwords, roles), `activity_logs`, `email_templates`/`email_campaigns`/`email_log` (carry over the old schema's intent, rebuilt on the new stack).

## Migration strategy

Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.

1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. The union/de-dup rules below are decided by the data:
   - **`EFECTIVO` vs `EFECTIVO_BACKUP`: `EFECTIVO_BACKUP` is a stale backup copy — de-dup it. (Corrected 2026-07-22; see the box below.)** On the canonicalized business key `(cl,fecha,monto,conepto)`, **12,386 of BACKUP's 12,387 rows already exist verbatim in `EFECTIVO`** — same customer, same timestamp to the second, same amount, same concept text — leaving exactly **1** genuinely new row. `folio` is a per-table sequential number that **collides** (12,363 shared numbers, 12,204 of them on different payments), so it can never be the de-dup key. **Rule: load `EFECTIVO` in full; from `EFECTIVO_BACKUP` load only business-key-new rows.** `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)

     > **Why this was wrong the first time.** The original pass reported only **2** overlapping rows and concluded the two tables were "near-disjoint ledgers, migrate both". That verdict came from a bug in `reconcile.py`, which compared business-key columns as raw strings on the premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: `mdb-export` formats a numeric column from its *Access column type*, so the same amount is emitted as `5000` from one table and `27000.0000` from the other, and no two rows could ever match on `monto`. `reconcile.py` now canonicalizes numeric key columns before comparing. The bad rule had already been loaded: the ledger carried 45,861 rows with **12,386 duplicated payments**, roughly doubling every customer's historical receipt total — which would have made every balance and statement in step 6 wrong. Re-running `run_all.py` brings the ledger to **33,475** rows. Groups 2 and 3 below were re-checked under the fix and their verdicts are unchanged.
   - **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈2025–26, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
   - **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
5. **Validation**: row-count and spot-check reconciliation between `staging` and final tables (e.g., every legacy customer has exactly one `customers` row via `customer_legacy_refs`; sum of migrated transaction amounts per customer matches sum in `staging`).

## Build sequencing

1. Repo scaffold (Next.js + NestJS + Prisma + MySQL, Docker Compose for local dev), CI pipeline, auth module with hashed passwords and role guards.
2. Prisma schema for the full data model above; run migration steps 1–2 (staging load + reconciliation reports) against real data early, since that's where the biggest unknowns are (do `NUM UTIL` and name-matching actually cover everyone? how bad is the snapshot-table duplication?).
3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.
4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions) — **DONE**. `apps/api/src/billing/` + web `/estado-cuenta` and `/estado-cuenta/[id]`. Two questions, two views: a per-customer **balances worklist** (who owes what) and a cross-customer **movement browser** (every charge and credit, filterable by line, concept, origin table and date range, with totals for the whole filtered set). The detail page is the actual statement: balance per currency, the same balance split by business line, charges broken out by concept, and the full movement list with a running balance. **Design constraint that shapes the whole module: balances are reported per currency and never collapsed into one number.** 912 of the 1,269 customers with a ledger move in both MXN and USD, the charge side is MXN-only while receipts arrive in both, and the legacy data never stored the exchange rate applied to a movement — so a single "total balance" would be a figure that never existed in the books.
7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.
10. Reports/email campaigns/admin — parity with old app's `reports.php`/`emailCampaigns.php` intent, rebuilt properly.

## Status

Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to `git.mancinas.io/rmancinas/jorgecuadros-platform` (branch `master`, 3 commits, working tree clean). `migration/load_staging.py` (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at `jorgecuadros-platform/RESUME.md` — read both together.

**Environment moved Windows → macOS.** Source Access files now live at `~/Downloads/JorgeCuadros-Legacy/` (all four: `UTILITIES.accdb`, `SEGUROS 16.mdb`, `SEGUROS 16_be.mdb`, `SCOTHIA.mdb`). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but **no Access ODBC driver and no `node_modules` installed yet**. Consequence: the `pyodbc`-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), `node_modules` needs `npm install`, and the staging output (gitignored Parquet) must be regenerated from scratch.

**Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB.

## Decisions (locked)

- **Stack:** Next.js + NestJS + Prisma + **MySQL** (locked earlier — see engine rationale above).
- **Extraction toolchain (macOS):** **mdbtools**, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in `objects.json`/`docs/`; only `LONGBINARY` blob extraction (step 4) needs extra handling under mdbtools.
- **i18n:** **Spanish-first** UI — matches the source data and staff usage.
- **CI/CD:** **Gitea Actions** on `git.mancinas.io` — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the `portainer-gitea-deploy` pattern already used on this LAN). Jenkins/`git.freakma.com` dropped.
- **`utility_dbo`:** resolved — full dump + portal code available (see Status).

## Open items (ops, not design)

- **VPS provisioning:** provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
- **Old external-DB credential** (hardcoded plaintext MySQL password in the old repo's `dbConnection.php`, in git history) — rotate it regardless, since it's already exposed.

## Verification

- Migration: automated row-count/sum reconciliation between `staging` and final schema per table group (see step 5 above), run as part of the migration script, not a manual spot-check.
- App: standard NestJS unit/integration tests per module (auth guards, Prisma queries), Playwright/Cypress e2e for the core "look up a customer, see their unified policies + services + statement" flow — the thing the whole project exists to deliver.
- Sync: once the VPS replica and inbox tables exist, verify replication lag stays low (a few seconds to low minutes) and that a payment/propane submission on the portal reliably shows up in the internal app within one polling interval, before relying on it operationally.
- Before cutover: run the new app against migrated data side-by-side with the live Access files for a period, comparing balances/statements for a sample of active customers to catch migration logic errors before the Access files are retired.