jorgecuadros-platform/PLAN.md

# Unified Customer / Insurance / Utilities Platform — Migration & Rebuild Plan

## Context

Jorge Cuadros & Assoc. runs two lines of business — property/utility management (`UTILITIES.accdb`) and insurance brokerage (`SEGUROS 16.mdb` + its linked backend `SEGUROS 16_be.mdb`) — out of separate, decades-old MS Access databases, plus a third file (`SCOTHIA.mdb`) that's the office's own Scotiabank checking-account register ("chequera"). The same people are customers of both business lines, but today there's no shared customer record: a person's utility account and their insurance policies live in unrelated systems with independent, inconsistent copies of their name/address/contact info. The bank register is a fourth, disconnected source of truth for the money actually moving through the office's own account.

An earlier attempt to modernize this (`jorgecuadros-intra-webapp`, PHP) got partway there — it correctly recognized that customers should be unified via a bridge table and began normalizing the flat Access tables into real relational tables (`properties`, `policies`, `home_policies`). But every query in its data layer builds SQL by string-concatenating `$_POST` directly (`src/core/db.php`), so it's SQL-injectable end to end, and `src/core/auth.php` compares passwords in plaintext with no hashing. Per your decision, this isn't worth patching — the app layer will be rebuilt from scratch, in a different stack, in a new repo. The Access data and the schema ideas from the old MySQL dump remain the reference for what the business actually needs.

Goal: one system with a single customer record, from which staff can see and manage that customer's utility services *and* insurance policies *and* shared billing/transaction history, replacing both Access files.

**Confirmed decisions:**
- Stack: Next.js (React + TypeScript) frontend, TypeScript backend (NestJS), **MySQL**, Prisma ORM.
- New repo, not built on top of `jorgecuadros-intra-webapp` (that repo is reference-only for business logic/field mappings).
- Full historical data migrates — no cutoff. Source tables are small (largest is ~16k rows), so completeness costs little.

**Database engine — revised from PostgreSQL to MySQL.** The internal platform doesn't stand alone: a separate, pre-existing customer-facing PHP/MySQL portal (mobile-app-backed — see Infrastructure & Sync below) needs to keep reading live account data from it, and that portal's PHP code uses `mysqli` and is staying as-is (not being rewritten). Running the internal platform on Postgres would mean maintaining a cross-engine sync into MySQL just to feed the portal; using MySQL everywhere removes that translation layer entirely. Prisma supports MySQL natively, so this only changes the `datasource` provider in the schema — NestJS/Next.js/Prisma all stay.

## Infrastructure & sync architecture

The internal platform and the customer-facing portal are two genuinely separate deployments that need to stay in sync, not one app:

- **Internal server** — hosted on-prem, private IP (`192.168.1.xx`), **not exposed to the internet**. Runs `jorgecuadros-platform` (this rebuild) and the canonical MySQL database — source of truth for everything staff manage (customers, policies, properties, the full ledger).
- **Customer-facing portal** — an existing PHP app (with a companion mobile app — its usage-tracking schema, `jorgecuadros_app` at `mysql.freakma.com`, logs sections like `LOGIN`, `STATEMENT_ALL_NOW`, `MAKE_PAYMENT`, `ORDER_PROPANE`, and a `devices` table of push-notification tokens) running on shared hosting. **Out of scope to rebuild** — stays exactly as it is. It reads/writes a separate live database, `utility_dbo` at `mysql.freakma.com`, which the old internal app already connects to externally (`src/core/dbConnection.php: getExternalDBConnection()`) — though only for `email_alert_log` in that codebase; the mobile app almost certainly talks to `utility_dbo` directly for statements/payments/propane orders. **Open item:** no schema dump of `utility_dbo` itself exists yet — only this reference to it — so the sync worker's exact target tables/columns can't be finalized until that's available (export or read-only credentials), the same way the Access files and the `jorgecuadros_app` dumps were provided.
- **New VPS** (Hetzner or DigitalOcean, to be provisioned) — runs a MySQL instance and becomes what `mysql.freakma.com` resolves to. Because it's infrastructure you control (unlike the shared-hosting account), it can run **real native MySQL replication** — the shared-hosting limitation only ever applied to shared hosting itself, not to a VPS. The shared-hosting portal's PHP code needs no changes beyond the DNS target it already points at.
- **Internal server ↔ VPS network path:** Tailscale (WireGuard mesh) — both machines join the same tailnet, giving the internal server a way to reach the VPS (and vice versa) without opening any inbound port on either the internal LAN or needing the internal server to be internet-facing.

**Sync mechanism:**
- **Internal → VPS (statements, balances, customer profile changes):** one-way native MySQL replication (binlog/GTID-based) from the internal MySQL (source) to the VPS MySQL (replica), over the Tailscale link. Standard, well-supported, and only carries the subset of tables/columns the portal actually needs to read — internal-only data (staff notes, adjuster info, activity logs) should live in tables intentionally excluded from what's replicated, so the more internet-exposed side never receives more than the portal requires.
- **VPS → Internal (payment submissions, propane order requests):** one-way replication can't carry writes back, and multi-master MySQL replication is fragile enough to avoid for a system this size. Instead: a small set of unreplicated **inbox tables** on the VPS (`payment_submissions`, `propane_order_requests`, matching whatever `utility_dbo` actually uses once its schema is available) that the portal writes to directly, polled every 1–5 minutes by a worker on the internal server (over Tailscale) that turns new rows into real `Transaction`/propane-request records and marks them processed.

## Source data inventory

Extracted live via `pyodbc` + the Windows Access ODBC driver (`dump_schema.py`, tables/columns/row counts/PKs for all four files). Access has **no real primary or foreign keys defined anywhere** — every relationship below is inferred from column-name conventions (`NUM id`, `NUMid`, `NUMER ID`, `IDISEG`, `IDIUT`), not enforced constraints, so the migration has to independently validate every join.

**UTILITIES.accdb** (52 tables, ~538MB, dominated by embedded document blobs):
- `DATMEX` (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 `LONGBINARY` columns holding scanned utility bills/IDs.
- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). ~~`COBRO3` looks like a filtered snapshot of `DATGRAL`~~ **Reconciliation (step 2) corrected this: `COBRO3` is a charge batch (constant fee 75), not a customer snapshot — `DATGRAL` is the sole customer master.**
- `PROFILE` (1,520) — per-property service enrollment flags, joins 1:1 with `DATMEX` by `NUMERID`.
- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers. ~~apparent year/program snapshots rather than distinct data~~ **Reconciliation (step 2) corrected this: EFECTIVO and EFECTIVO_BACKUP are near-disjoint ledgers (folio collides; only 2 business-key matches) — migrate both, no folio de-dup. FM3 tables are a separate fee stream.**
- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs. ~~near-duplicate shapes ... needs de-duplication~~ **Reconciliation (step 2) corrected this: disjoint billing runs from different periods (2017 / 2018 / 2025–26), zero real-identity overlap — straight union, no de-dup.**
- `TRUSTVENCE` (549) — bank trust account fee due-dates.
- `TIPO HIST` (2,301) — exchange-rate history (date/hour/rate), referenced by the `MONEDAS` (currency) field used throughout.
- `TYPE OF TRX` (79) — ES/EN transaction-type lookup (already mirrored as `type_transactions` in the old MySQL schema — reuse that mapping).
- Scratch/working tables to **exclude** from migration: `BANCO EDITOR`, `Errores de pegado`, `TABLE1`, `PARA BILLING*`, `FALTANTES *`, `TELEFONOS FECHAS`, `TIT`, `BORRA`-equivalents. A few tables (`PROPANO`, `datosfreak`, `datosfreak2`, `LUZ TODOS`, etc.) have column names with characters pyodbc's UTF-16 path can't decode — need re-extraction with a Latin-1/CP1252 fallback before they can even be classified as real data vs. scratch.

**SEGUROS 16_be.mdb** (64 tables, ~882MB; `SEGUROS 16.mdb` is an empty Access "frontend" shell — all real data lives in `_be`):
- `DATGRAL` (1,070 rows) — customer master, **same shape as the utilities `DATGRAL`**, and critically has a `NUM UTIL` column — a literal cross-reference to the utilities customer's `NUM id`. This is the join key for building the unified customer table; it's also what the old app's `customer_mapping` table was clearly reverse-engineered from.
- One flat table per insurance line of business, all sharing the same repeated-column pattern (policy #, coverage dates, 4 hardcoded payment installments each with its own date/amount/check#/currency, premium breakdown, liquidation status, address, observations, embedded document blobs):
  - `INCENDIO` (fire), `MULT` (multi-risk/home — the most complete, matches old app's `home_policies`), `M EMPR` (commercial property)
  - Auto: `TABLA AUTOS`, `TABLA AUTOS AMPL`, `TABLA AUTOS AMPL R`, `TABLA AUTOS LIMIT`, `TABLA AUTOS LIMIT R`, `TABLA AUTOS RC R`, `MCA2` — seven variants of essentially one "auto policy" concept, differentiated by coverage tier/program. `MCA2` also hardcodes **3 driver+vehicle slots as repeated columns** (`MARCA1/2/3`, `PLACA1/2/3`, `NAME1/2/3`, `LIC1/2/3`...) that need to unpivot into child rows.
  - `LICENCIAS` (driver's-license insurance, MX-specific product) — also hardcodes 3 named insureds per policy.
- `BENEF` (768) — policy beneficiaries, already a clean child table (policy#, name, address, phone, email).
- `DATOS` (2 rows now, but structurally real) — claims/siniestros: claim date, description, adjuster, settlement amounts, checks, documents.
- `AJUSTADORES` / `AJUSTADORESATLAS` — adjuster contact lists.
- `EDOSEG` — per-policy premium/account statement summary.
- `EFECTIVO` (295) — cash ledger, same shape as the utilities one.
- `UTILSEG` (1,582) — appears to be the utilities↔insurance customer cross-reference table.
- `TABLA LIQUIDA *` / `TABLE DATOS LIQUIDA *` — settlement/liquidation batches per policy line (matches `liquidation_number` already in the old MySQL `policies` table).
- **Exclude from data migration:** the `* MENS` / `*MENSAJE` tables (`AMPL MENS`, `IN MENS`, `LIC MENS`, `MCA2 MENS`, `ME MENS`, `MF MENS`, `RC MENSAJE`, `RC R MENS`) are mail-merge document *templates* (letters, certificates) stored as blobs, not customer data — these get reimplemented as PDF templates in the new app, not migrated as rows. `TODOSJC`, `TODOS`, `vigenta casa y auto unicos` look like materialized Access query results (saved reports), not source-of-truth data — exclude, and if the report itself is still needed, rebuild it as a real query against the new schema.

**SCOTHIA.mdb** (7 tables, ~3MB — the office's own Scotiabank checking-account register, "chequera"). Much simpler than the other two files: this is the company's operating bank account, not customer-facing data.
- `DATOS E` (15,406 rows) — expenses/outgoing (`EGRESO`): date, transaction type, check/reference `NUM`, `CONCEPTO` (payee/description), amount, `OPERADO` (cleared flag), notes, and a spelled-out amount-in-words field (`CANTIDAD EN LETRA`, standard Mexican check-writing convention).
- `DATOS I` (6,948 rows) — income/incoming (`INGRESO`): same shape, plus a `TRANSFERIDO` (transferred) flag instead of the amount-in-words field.
- `TABLA RAMODOS` (66 rows) — a category/business-line lookup ("ramo" = line of business in Mexican insurance terminology) — this is almost certainly what `CONCEPTO` entries get classified against, i.e. the link between a bank transaction and which part of the business (insurance line, utility service, trust, etc.) it belongs to.
- `ban` (1 row) — just holds the bank's name; a config singleton, not data.
- `INFORME` / `INFORME BA` (0 rows each) and `FECHAIF` (1 row) — Access report/query scratch tables (date-range parameters and a report shell), same pattern as the scratch tables in the other two files — **exclude** from migration.

## Target architecture

- **Frontend:** Next.js (App Router) + TypeScript + React. Server components for data-heavy list/detail views (customers, policies, statements); client components for interactive forms.
- **Backend:** NestJS (TypeScript) REST API — modular by domain (customers, insurance, utilities, billing, auth, admin), matching the module boundaries below. Gives you DI, guards for auth/authorization, and a validation pipeline (`class-validator`) for free, which directly replaces the old app's biggest weakness (no input validation, no parameterization).
- **Database:** MySQL, accessed via Prisma (schema-as-code, migrations, generates a typed client — eliminates the raw-SQL-injection class of bug entirely since Prisma parameterizes everything). See Infrastructure & Sync above for why MySQL rather than Postgres.
- **Auth:** NestJS + Passport, sessions or JWT (pick one during build), `bcrypt`/`argon2` password hashing, role-based guards replacing the old `$_SESSION['level']/['role']` checks.
- **File/document storage:** object storage (S3-compatible) for the scanned documents currently trapped as Access `LONGBINARY` blobs — extract once during migration, store as files, keep only the pointer + metadata in MySQL. The old repo already anticipated this (`src/objects/s3.php` exists but appears unused) — same idea, implemented for real this time.
- **Deployment:** keep Docker as the packaging mechanism (already proven for this project) with a fresh `Dockerfile`/`docker-compose.yml` for the Node services + MySQL, running on the internal server described above; CI can stay on Jenkins if that's still the team's pipeline, or move to GitHub Actions if the new repo lives somewhere other than `git.freakma.com` — flagged as an open decision below since it wasn't part of the stack questions asked.

## Target data model (by domain)

All tables get a surrogate `id` (uuid or serial) plus, where the row came from a legacy table, provenance columns (`legacy_source_db`, `legacy_source_table`, `legacy_id`) so every migrated row can be traced back to its Access original for spot-checking and reconciliation — and so the ETL can be re-run idempotently (upsert on provenance key) as migration bugs get found and fixed.

**Identity (the actual point of this project):**
- `customers` — one row per real person/entity, merged from utilities `DATGRAL`/`COBRO3` and insurance `DATGRAL`, matched via the `NUM UTIL` cross-reference plus name/address fuzzy-matching for anyone missing that link. Holds name, addresses (MX + US), phones, email, ID document info, status, currency preference.
- `customer_legacy_refs` — generalizes the old `customer_mapping` table: `(customer_id, source_system, source_table, legacy_numeric_id)`, one row per legacy record folded into this customer. This is what makes "unified customer base" actually queryable and keeps the merge auditable.

**Insurance domain:**
- `insurance_providers`, `policy_types` (carry over from old schema, already reasonable)
- `policies` — generic header (policy #, type, provider, customer, dates, premiums, agent, liquidation status), consolidating `INCENDIO`/`MULT`/`M EMPR`/all six auto-table variants/`LICENCIAS` into one table with a `policy_type` discriminator, instead of one Access table per line of business.
- `policy_payment_installments` — unpivots the 4 hardcoded payment-installment columns (`1ER PAGO`/`FECHA PAGO`/`NO CHEQUE`, `... 2`, `... 3`, `... 4`) into rows: due sequence, amount, currency, paid date, check/reference number.
- `vehicles` — unpivots `MCA2`'s 3 hardcoded vehicle slots (and the single-vehicle auto tables) into one row per vehicle, FK'd to policy + customer.
- `insured_drivers` — same unpivot for the repeated named-insured/license columns in `MCA2`/`LICENCIAS`.
- `properties` (shared with utilities domain — see below), `policy_beneficiaries` (from `BENEF`, already clean), `claims` (from `DATOS`), `adjusters` (from `AJUSTADORES*`), `policy_documents` (extracted blobs, typed: ID, prior policy copy, damage photo, etc.).

**Utilities domain:**
- `properties` — one row per property (from `DATMEX`), FK'd to `customers`, shared with insurance so a property can carry both a home-insurance policy and utility service enrollments — this is the second half of "unified."
- `property_services` — one row per enrolled service per property (water/electric/gas/cable/trust/property-tax/federal-zone), unpivoting `DATMEX`'s inline service columns and `PROFILE`'s enrollment flags into real rows with account #, route, meter #, due day.
- `service_documents` (extracted blobs), `trust_accounts` (from `TRUSTVENCE`).

**Shared financial ledger** (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive. **`amount` is signed:** negative = charge (cargo), positive = credit (abono), so `SUM(amount)` per customer per currency *is* the balance — negative means the customer owes the office. The two currencies are never summed together (see the billing module note in Build sequencing step 6).
- `exchange_rates` (from `TIPO HIST`), `type_transactions` (carry over ES/EN lookup as-is).
- `bank_transactions` — the company's own operating bank register, from SCOTHIA's `DATOS E`/`DATOS I` unified into one signed-amount table (income positive, expense negative) with a `category` FK to `business_line_categories` (from `TABLA RAMODOS`) and a `cleared`/`operado` flag. This is deliberately **separate** from customer-facing `transactions` — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the `business_line_categories` lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
- `business_line_categories` (from `TABLA RAMODOS`).

**Admin/shared:** `users` (hashed passwords, roles), `activity_logs`, `email_templates`/`email_campaigns`/`email_log` (carry over the old schema's intent, rebuilt on the new stack).

## Migration strategy

Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.

1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. The union/de-dup rules below are decided by the data:
   - **`EFECTIVO` vs `EFECTIVO_BACKUP`: `EFECTIVO_BACKUP` is a stale backup copy — de-dup it. (Corrected 2026-07-22; see the box below.)** On the canonicalized business key `(cl,fecha,monto,conepto)`, **12,386 of BACKUP's 12,387 rows already exist verbatim in `EFECTIVO`** — same customer, same timestamp to the second, same amount, same concept text — leaving exactly **1** genuinely new row. `folio` is a per-table sequential number that **collides** (12,363 shared numbers, 12,204 of them on different payments), so it can never be the de-dup key. **Rule: load `EFECTIVO` in full; from `EFECTIVO_BACKUP` load only business-key-new rows.** `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)

     > **Why this was wrong the first time.** The original pass reported only **2** overlapping rows and concluded the two tables were "near-disjoint ledgers, migrate both". That verdict came from a bug in `reconcile.py`, which compared business-key columns as raw strings on the premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: `mdb-export` formats a numeric column from its *Access column type*, so the same amount is emitted as `5000` from one table and `27000.0000` from the other, and no two rows could ever match on `monto`. `reconcile.py` now canonicalizes numeric key columns before comparing. The bad rule had already been loaded: the ledger carried 45,861 rows with **12,386 duplicated payments**, roughly doubling every customer's historical receipt total — which would have made every balance and statement in step 6 wrong. Re-running `run_all.py` brings the ledger to **33,475** rows. Groups 2 and 3 below were re-checked under the fix and their verdicts are unchanged.
   - **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈2025–26, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
   - **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
5. **Validation**: row-count and spot-check reconciliation between `staging` and final tables (e.g., every legacy customer has exactly one `customers` row via `customer_legacy_refs`; sum of migrated transaction amounts per customer matches sum in `staging`).

## Build sequencing

1. Repo scaffold (Next.js + NestJS + Prisma + MySQL, Docker Compose for local dev), CI pipeline, auth module with hashed passwords and role guards.
2. Prisma schema for the full data model above; run migration steps 1–2 (staging load + reconciliation reports) against real data early, since that's where the biggest unknowns are (do `NUM UTIL` and name-matching actually cover everyone? how bad is the snapshot-table duplication?).
3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.
4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions) — **DONE**. `apps/api/src/billing/` + web `/estado-cuenta` and `/estado-cuenta/[id]`. Two questions, two views: a per-customer **balances worklist** (who owes what) and a cross-customer **movement browser** (every charge and credit, filterable by line, concept, origin table and date range, with totals for the whole filtered set). The detail page is the actual statement: balance per currency, the same balance split by business line, charges broken out by concept, and the full movement list with a running balance. **Design constraint that shapes the whole module: balances are reported per currency and never collapsed into one number.** 912 of the 1,269 customers with a ledger move in both MXN and USD, the charge side is MXN-only while receipts arrive in both, and the legacy data never stored the exchange rate applied to a movement — so a single "total balance" would be a figure that never existed in the books.
7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. **The separate Phase B Access additive sync is implemented:** `migration/run_all.py --sync` and the admin `SYNC` job upsert legacy-owned rows without truncating the database or touching manual rows. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.
10. Reports/email campaigns/admin — parity with old app's `reports.php`/`emailCampaigns.php` intent, rebuilt properly.
11. **Receipt capture ("Editor") completion + three net-new ops features — NOT STARTED, spec written.** Full design in [`docs/RECEIPT_CAPTURE_SPEC.md`](docs/RECEIPT_CAPTURE_SPEC.md), from the 2026-07-25/26 meeting with Jorge:
    - **Receipt capture module — DONE** (2026-07-27). The legacy "Editor" replacement, built on the single-movement capture from step 6. Wires up the previously-unused `Transaction.outstanding` (NOPAGO): capture flag on `POST /billing`, `?outstanding=` list filter, `POST /billing/:id/resolve-outstanding` (gated `ledger:create`, not `ledger:void` — resolving *completes* a capture), and exclusion from every balance aggregate exactly as the legacy `SALDOS ULTIMO 0`'s `HAVING NOPAGO = 0` did. Adds `POST /billing/batch` (one `$transaction`, check-level fields shared, per-line customer/amount) and `GET /billing/by-check`, plus the `cheque-count` report replacing `REPORTE CHEQUE COUNT` / `REPORTE POR CHEQUE` / `EDITA CHEQUE ALF|COUNT|NUM` — print/PDF/CSV/XLSX come free from the existing `/reportes/:slug` machinery. Web: `/estado-cuenta/lote` (the actual "Editor" screen, with live reconciliation against the physical check amount), plus an "Estado de pago" filter, a "sin fondos" row tag and a Resolver dialog on `/estado-cuenta`. No new abilities. Verified end-to-end against dev, API + browser.
      **Two pre-existing bugs found and fixed while building it:** (a) `statement()` filtered `legacySourceTable: { notIn: [...] }`, which compiles to SQL `NOT IN` — and `NULL NOT IN (…)` is NULL, so **every app-captured movement was invisible on the customer statement** (438 rows in the movement browser vs 392 on the statement) while still appearing everywhere else. This would have made the whole receipt-capture feature look broken to staff. Now NULL-safe. (b) The balances *count* query omitted the void filter its own page query applied, so the row count disagreed with the rows.
      **OCR seam:** `BillingService.createBatch(dto, opts)` is the single multi-row write path and carries three contract guarantees for the step-11 OCR module to post through — `items[i]` maps to `lines[i]` (so `StatementDocument.postedTransactionId` can be zipped back on), `opts.refs[i]` stamps `captureRef` with a duplicate-post guard that a *voided* row deliberately does not block, and `opts.source` is service-level only so an HTTP client cannot label hand-keyed rows as machine-captured. Backed by a new `TransactionCaptureSource` enum (MANUAL/BATCH/OCR) + `captureRef`, both nullable so the 40,136 migrated rows stay NULL rather than being mislabelled.
    - **PDF/OCR auto-capture** — ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff currently key in by hand. Posts through the capture module above. Matching logic was checked field-by-field against `migration/transform_properties.py`'s actual output and found three real gaps to close first: no `TELEPHONE` service kind exists yet, `PROPERTY_TAX.accountNumber` was migrated from `PREDIAL` not `CLAVE` (needs verification against a real predial statement), and `GAS.meterNumber` was never populated by the migration at all.
    - **Multi-bank chequera** — `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register; today's `bank_transactions` is hardcoded single-account/MXN-only by design (see step 7 above) and needs a required `bankAccountId` plus scoping added to every read path in `bank.service.ts`, including two raw-SQL queries in `summary()`.
    - **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s).
    
    Several open questions block parts of this (OCR provider/budget, the Seguros bank's identity, the clave-catastral-vs-predial mismatch, exact recycling triggers, and whether "recycling" should ever mean true data purge vs. archive-and-reuse-the-number) — see the spec's collected open-questions section.
12. **Insurance features — NOT STARTED, spec written.** Full design in [`docs/INSURANCE_FEATURES_SPEC.md`](docs/INSURANCE_FEATURES_SPEC.md), the insurance half of the same 2026-07-25/26 meeting with Jorge that produced step 11:
    - **Renewal notification emails** — a daily `@nestjs/schedule` sweep that mails the customer 30 days before expiry, 15 days before, and 7 days after, mapping onto `RenewalNotice.generation` 1/2/3 with **no schema change**. Sending is **Amazon SES** (`@aws-sdk/client-sesv2`, mirroring `StorageService`'s optional-client/degrade-don't-crash pattern) — the office already runs SES, so provider and budget are settled, not open. The letter body is the *existing* `aviso-renovacion` report (`reports.registry.ts:623-799`); `@@unique([policyId, generation])` is already-in-place idempotency, so a re-run cannot double-send. Volume ≈260 mails/month, and **815 of the 893 policyholders (91%) have an email**. Also adds the manual mark-as-sent mutation the report's own comment anticipates, so the report's permanently-zero `enviadas` total becomes real. Smallest useful piece — do first.
    - **Liquidación batch workflow** — ~70% already built (`liquidated`/`liquidationNumber`/`liquidationDate` are wired through DTOs, list filter, stats, form and detail page); only the *batch* print-and-mark step is missing, against a live pending set of 226 policies. Adds a ramo-parameterized pending report plus `POST /policies/liquidate-batch` under a new MANAGER `policy:liquidate` ability. Parameterized by ramo, not MULT-only — legacy `TABLA LIQUIDA MF` served `MULT`, `INCENDIO` and `M EMPR` alike.
    - **Certificate / "Solicitud Atlas"** — renders from the same `format: "letter"` machinery `aviso-renovacion` uses, then reaches customers as an extension of the step-8/9 replication (PDF generated here, pushed to MinIO, pointer replicated), **not** as a new public surface in this repo. Half-blocked: "Solicitud" has zero referent in the legacy system and normally means an *application form*, a different artifact from a certificate.
    - **Carrier API integration (ANA Seguros + GMX)** — shape only (`CarrierConnector` + an import-review queue rather than direct `Policy` writes, matching how step 11's OCR results are routed). Carrier research done 2026-07-27: **the two carriers are one company** — both belong to **Grupo Valore** (ANA writes autos, GMX writes daños, which is exactly this database's `AUTO`/`LICENCIAS` vs `MULT`/`INCENDIO`/`M_EMPR` split), so it is one commercial relationship, not two. **ANA has a real live SOAP service** (`server.anaseguros.com.mx/ananetws/service.asmx`, ASP.NET `.asmx`) with a published operation list — catalogs, `CalculaValor`/`CalculaMSI`, `ValidaSerie`, `RecuperaCotizacion`, `Transaccion`. **GMX publishes no machine interface at all**, only human agent portals. ⚠️ **Critical mismatch:** every ANA operation serves *new-business quoting/issuance*, not "list the policies where I am agent of record" — so if the ask is inbound portfolio sync, no evidence exists that either carrier sells it. Blocked on one phone call to Grupo Valore ((55) 5480-4000) for credentials + a direction answer, not on further research. ("GDMX" in the meeting notes was a typo for `GMX` — confirmed 2026-07-27.)

    **Two pre-existing defects were found while verifying this spec and should be fixed as part of the liquidación work:** (a) `policy_types` is missing its `INCENDIO` and `M_EMPR` rows and, because `policies_policyTypeId_fkey` is `ON DELETE SET NULL`, 5 `m_empr` policies silently lost their ramo — 4 of them are pending liquidación and are invisible to every ramo-filtered query; (b) the legacy settlement slots don't match what the target model assumed — `MULT`/`INCENDIO` carry two and `M EMPR` carries four, while `Policy` collapses to one, so ≤41 MULT second settlements were dropped in migration. Spec recommends moving settlement onto `PolicyPaymentInstallment` rather than adding a second slot.

    **One long-standing open question is closed by this spec:** `DATGRAL.[NUM UTIL]` is authoritative for Utilities↔Seguros reconciliation and **`UTILSEG` must not be used** — its numbers resolve to unrelated people under every reading tested (name match 58/1,024 vs. 298/563 for `NUM UTIL`), and where the two sources overlap they contradict each other on 170 of 218 shared ids. This matters to step 11's customer-number recycling, which touches the same identity space.

## Status

Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to `git.mancinas.io/rmancinas/jorgecuadros-platform` (branch `master`, 3 commits, working tree clean). `migration/load_staging.py` (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at `jorgecuadros-platform/RESUME.md` — read both together.

**Environment moved Windows → macOS.** Source Access files now live at `~/Downloads/JorgeCuadros-Legacy/` (all four: `UTILITIES.accdb`, `SEGUROS 16.mdb`, `SEGUROS 16_be.mdb`, `SCOTHIA.mdb`). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but **no Access ODBC driver and no `node_modules` installed yet**. Consequence: the `pyodbc`-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), `node_modules` needs `npm install`, and the staging output (gitignored Parquet) must be regenerated from scratch.

**Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB.

**Step 11 spec written, not built.** `docs/RECEIPT_CAPTURE_SPEC.md` covers the receipt-capture ("Editor") completion plus the three net-new ops features (OCR auto-capture, multi-bank chequera, customer-number recycling) — see Build sequencing step 11 above for the summary. Written from the 2026-07-25/26 meeting notes and verified against the real migration scripts and current API code, not just designed from the meeting notes alone.

**Step 12 spec written, not built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. Two of the four features are much smaller than they sound: the renewal-notice table, its idempotency key and the letter body already exist, and the per-policy liquidación fields are already wired end to end.

## Decisions (locked)

- **Stack:** Next.js + NestJS + Prisma + **MySQL** (locked earlier — see engine rationale above).
- **Extraction toolchain (macOS):** **mdbtools**, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in `objects.json`/`docs/`; only `LONGBINARY` blob extraction (step 4) needs extra handling under mdbtools.
- **i18n:** **Spanish-first** UI — matches the source data and staff usage.
- **CI/CD:** **Gitea Actions** on `git.mancinas.io` — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the `portainer-gitea-deploy` pattern already used on this LAN). Jenkins/`git.freakma.com` dropped.
- **`utility_dbo`:** resolved — full dump + portal code available (see Status).
- **Phase B Access additive sync:** implemented in `migration/run_all.py --sync` and the admin
  `SYNC` job. It preserves manual rows and stable legacy-owned primary keys; end-to-end database
  validation remains before production use.

## Open items (ops, not design)

- **VPS provisioning:** provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
- **Old external-DB credential** (hardcoded plaintext MySQL password in the old repo's `dbConnection.php`, in git history) — rotate it regardless, since it's already exposed.

## Open design questions (steps 11 & 12 — need Jorge before/while building)

Unlike the ops items above, these block design decisions, not just infrastructure. Full detail in each section of `docs/RECEIPT_CAPTURE_SPEC.md` (step 11) and `docs/INSURANCE_FEATURES_SPEC.md` (step 12):

**Step 11 — utilities/ops side:**

- OCR provider/budget for the statement auto-capture pipeline (self-hosted vs. a paid per-page API, given 300+ statements/month/service provider).
- Whether `PROPERTY_TAX.accountNumber` (migrated from `DATMEX.PREDIAL`) is actually the same number as "Clave Catastral" (`DATMEX.CLAVE`) — blocks OCR matching for predial statements until confirmed against a real bill.
- The actual bank name/currency/details for the Seguros USD account, and whether any historical Seguros bank register exists to migrate.
- The exact "1 year inactivity" / "cancelled" triggers for customer-number recycling eligibility.
- Whether customer-number recycling should ever include true PII purge (matching the office's paper-world habit) or archive-and-reuse-the-number is sufficient — recommended default is archive-only, consistent with this project's existing never-hard-delete convention.

**Step 12 — insurance side:**

- Which SES region + verified sending identity/configuration set the renewal mail goes out under, and whether it reuses the existing IAM credentials or gets its own scoped `ses:SendEmail` user. (Provider and budget are *not* open — SES is settled.)
- What to do with the 78 policyholders who have no email on file: skip silently, or produce a print worklist? Recommended: the worklist, since `aviso-renovacion` already renders exactly those letters.
- Whether renewal notices go out in Spanish or English — `Customer` carries no language preference.
- What "garantías" refers to — it has zero referent in the legacy data, and it blocks the liquidación batch's exclusion filter.
- Whether policy settlement should move onto `PolicyPaymentInstallment` (recommended) or gain a second slot on `Policy`, and whether to backfill the ≤41 MULT second settlements lost in migration.
- Whether batch liquidación warrants a new MANAGER-level `policy:liquidate` ability (recommended) or should reuse the existing STAFF-level `policy:update`.
- **What "Solicitud Atlas" actually is** — an application form or a certificate. These are different artifacts with different data and timing; this blocks the whole certificate feature.
- **Carrier integration direction** — outbound quote/issue (which ANA's SOAP service supports today) or inbound sync of the office's existing book (which nothing found suggests either carrier offers)? This decides whether the feature is buildable at all. Bundle with the other three carrier questions into one call to Grupo Valore ((55) 5480-4000): WSDL + credentials for the ANA service, whether a cartera/portfolio download exists for an agent's own book, whether GMX daños has any machine interface, and whether one credential spans both carriers. ("GDMX" is resolved — it was a typo for `GMX`.)

## Verification

- Migration: automated row-count/sum reconciliation between `staging` and final schema per table group (see step 5 above), run as part of the migration script, not a manual spot-check.
- App: standard NestJS unit/integration tests per module (auth guards, Prisma queries), Playwright/Cypress e2e for the core "look up a customer, see their unified policies + services + statement" flow — the thing the whole project exists to deliver.
- **Sync:** Phase B Access additive sync now has automated CLI/admin wiring, but must be verified
  against a disposable DB with stable-PK, manual-row, update, and source-delete cases. The
  separate VPS/portal sync still requires VPS provisioning and inbox-table implementation.
- Before cutover: run the new app against migrated data side-by-side with the live Access files for a period, comparing balances/statements for a sample of active customers to catch migration logic errors before the Access files are retired.