Initial scaffold: unified customer/insurance/utilities platform

Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.

See PLAN.md and RESUME.md for the full architecture and session history.

PLAN.md

@@ -0,0 +1,145 @@
+# Unified Customer / Insurance / Utilities Platform — Migration & Rebuild Plan
+
+## Context
+
+Jorge Cuadros & Assoc. runs two lines of business — property/utility management (`UTILITIES.accdb`) and insurance brokerage (`SEGUROS 16.mdb` + its linked backend `SEGUROS 16_be.mdb`) — out of separate, decades-old MS Access databases, plus a third file (`SCOTHIA.mdb`) that's the office's own Scotiabank checking-account register ("chequera"). The same people are customers of both business lines, but today there's no shared customer record: a person's utility account and their insurance policies live in unrelated systems with independent, inconsistent copies of their name/address/contact info. The bank register is a fourth, disconnected source of truth for the money actually moving through the office's own account.
+
+An earlier attempt to modernize this (`jorgecuadros-intra-webapp`, PHP) got partway there — it correctly recognized that customers should be unified via a bridge table and began normalizing the flat Access tables into real relational tables (`properties`, `policies`, `home_policies`). But every query in its data layer builds SQL by string-concatenating `$_POST` directly (`src/core/db.php`), so it's SQL-injectable end to end, and `src/core/auth.php` compares passwords in plaintext with no hashing. Per your decision, this isn't worth patching — the app layer will be rebuilt from scratch, in a different stack, in a new repo. The Access data and the schema ideas from the old MySQL dump remain the reference for what the business actually needs.
+
+Goal: one system with a single customer record, from which staff can see and manage that customer's utility services *and* insurance policies *and* shared billing/transaction history, replacing both Access files.
+
+**Confirmed decisions:**
+- Stack: Next.js (React + TypeScript) frontend, TypeScript backend (NestJS), **MySQL**, Prisma ORM.
+- New repo, not built on top of `jorgecuadros-intra-webapp` (that repo is reference-only for business logic/field mappings).
+- Full historical data migrates — no cutoff. Source tables are small (largest is ~16k rows), so completeness costs little.
+
+**Database engine — revised from PostgreSQL to MySQL.** The internal platform doesn't stand alone: a separate, pre-existing customer-facing PHP/MySQL portal (mobile-app-backed — see Infrastructure & Sync below) needs to keep reading live account data from it, and that portal's PHP code uses `mysqli` and is staying as-is (not being rewritten). Running the internal platform on Postgres would mean maintaining a cross-engine sync into MySQL just to feed the portal; using MySQL everywhere removes that translation layer entirely. Prisma supports MySQL natively, so this only changes the `datasource` provider in the schema — NestJS/Next.js/Prisma all stay.
+
+## Infrastructure & sync architecture
+
+The internal platform and the customer-facing portal are two genuinely separate deployments that need to stay in sync, not one app:
+
+- **Internal server** — hosted on-prem, private IP (`192.168.1.xx`), **not exposed to the internet**. Runs `jorgecuadros-platform` (this rebuild) and the canonical MySQL database — source of truth for everything staff manage (customers, policies, properties, the full ledger).
+- **Customer-facing portal** — an existing PHP app (with a companion mobile app — its usage-tracking schema, `jorgecuadros_app` at `mysql.freakma.com`, logs sections like `LOGIN`, `STATEMENT_ALL_NOW`, `MAKE_PAYMENT`, `ORDER_PROPANE`, and a `devices` table of push-notification tokens) running on shared hosting. **Out of scope to rebuild** — stays exactly as it is. It reads/writes a separate live database, `utility_dbo` at `mysql.freakma.com`, which the old internal app already connects to externally (`src/core/dbConnection.php: getExternalDBConnection()`) — though only for `email_alert_log` in that codebase; the mobile app almost certainly talks to `utility_dbo` directly for statements/payments/propane orders. **Open item:** no schema dump of `utility_dbo` itself exists yet — only this reference to it — so the sync worker's exact target tables/columns can't be finalized until that's available (export or read-only credentials), the same way the Access files and the `jorgecuadros_app` dumps were provided.
+- **New VPS** (Hetzner or DigitalOcean, to be provisioned) — runs a MySQL instance and becomes what `mysql.freakma.com` resolves to. Because it's infrastructure you control (unlike the shared-hosting account), it can run **real native MySQL replication** — the shared-hosting limitation only ever applied to shared hosting itself, not to a VPS. The shared-hosting portal's PHP code needs no changes beyond the DNS target it already points at.
+- **Internal server ↔ VPS network path:** Tailscale (WireGuard mesh) — both machines join the same tailnet, giving the internal server a way to reach the VPS (and vice versa) without opening any inbound port on either the internal LAN or needing the internal server to be internet-facing.
+
+**Sync mechanism:**
+- **Internal → VPS (statements, balances, customer profile changes):** one-way native MySQL replication (binlog/GTID-based) from the internal MySQL (source) to the VPS MySQL (replica), over the Tailscale link. Standard, well-supported, and only carries the subset of tables/columns the portal actually needs to read — internal-only data (staff notes, adjuster info, activity logs) should live in tables intentionally excluded from what's replicated, so the more internet-exposed side never receives more than the portal requires.
+- **VPS → Internal (payment submissions, propane order requests):** one-way replication can't carry writes back, and multi-master MySQL replication is fragile enough to avoid for a system this size. Instead: a small set of unreplicated **inbox tables** on the VPS (`payment_submissions`, `propane_order_requests`, matching whatever `utility_dbo` actually uses once its schema is available) that the portal writes to directly, polled every 1–5 minutes by a worker on the internal server (over Tailscale) that turns new rows into real `Transaction`/propane-request records and marks them processed.
+
+## Source data inventory
+
+Extracted live via `pyodbc` + the Windows Access ODBC driver (`dump_schema.py`, tables/columns/row counts/PKs for all four files). Access has **no real primary or foreign keys defined anywhere** — every relationship below is inferred from column-name conventions (`NUM id`, `NUMid`, `NUMER ID`, `IDISEG`, `IDIUT`), not enforced constraints, so the migration has to independently validate every join.
+
+**UTILITIES.accdb** (52 tables, ~538MB, dominated by embedded document blobs):
+- `DATMEX` (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 `LONGBINARY` columns holding scanned utility bills/IDs.
+- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). `COBRO3` looks like a filtered snapshot of `DATGRAL`, not a distinct entity.
+- `PROFILE` (1,520) — per-property service enrollment flags, joins 1:1 with `DATMEX` by `NUMERID`.
+- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers, near-identical shape, apparent year/program snapshots rather than distinct data.
+- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs, again near-duplicate shapes across different periods/exports — needs de-duplication logic, not a straight union.
+- `TRUSTVENCE` (549) — bank trust account fee due-dates.
+- `TIPO HIST` (2,301) — exchange-rate history (date/hour/rate), referenced by the `MONEDAS` (currency) field used throughout.
+- `TYPE OF TRX` (79) — ES/EN transaction-type lookup (already mirrored as `type_transactions` in the old MySQL schema — reuse that mapping).
+- Scratch/working tables to **exclude** from migration: `BANCO EDITOR`, `Errores de pegado`, `TABLE1`, `PARA BILLING*`, `FALTANTES *`, `TELEFONOS FECHAS`, `TIT`, `BORRA`-equivalents. A few tables (`PROPANO`, `datosfreak`, `datosfreak2`, `LUZ TODOS`, etc.) have column names with characters pyodbc's UTF-16 path can't decode — need re-extraction with a Latin-1/CP1252 fallback before they can even be classified as real data vs. scratch.
+
+**SEGUROS 16_be.mdb** (64 tables, ~882MB; `SEGUROS 16.mdb` is an empty Access "frontend" shell — all real data lives in `_be`):
+- `DATGRAL` (1,070 rows) — customer master, **same shape as the utilities `DATGRAL`**, and critically has a `NUM UTIL` column — a literal cross-reference to the utilities customer's `NUM id`. This is the join key for building the unified customer table; it's also what the old app's `customer_mapping` table was clearly reverse-engineered from.
+- One flat table per insurance line of business, all sharing the same repeated-column pattern (policy #, coverage dates, 4 hardcoded payment installments each with its own date/amount/check#/currency, premium breakdown, liquidation status, address, observations, embedded document blobs):
+  - `INCENDIO` (fire), `MULT` (multi-risk/home — the most complete, matches old app's `home_policies`), `M EMPR` (commercial property)
+  - Auto: `TABLA AUTOS`, `TABLA AUTOS AMPL`, `TABLA AUTOS AMPL R`, `TABLA AUTOS LIMIT`, `TABLA AUTOS LIMIT R`, `TABLA AUTOS RC R`, `MCA2` — seven variants of essentially one "auto policy" concept, differentiated by coverage tier/program. `MCA2` also hardcodes **3 driver+vehicle slots as repeated columns** (`MARCA1/2/3`, `PLACA1/2/3`, `NAME1/2/3`, `LIC1/2/3`...) that need to unpivot into child rows.
+  - `LICENCIAS` (driver's-license insurance, MX-specific product) — also hardcodes 3 named insureds per policy.
+- `BENEF` (768) — policy beneficiaries, already a clean child table (policy#, name, address, phone, email).
+- `DATOS` (2 rows now, but structurally real) — claims/siniestros: claim date, description, adjuster, settlement amounts, checks, documents.
+- `AJUSTADORES` / `AJUSTADORESATLAS` — adjuster contact lists.
+- `EDOSEG` — per-policy premium/account statement summary.
+- `EFECTIVO` (295) — cash ledger, same shape as the utilities one.
+- `UTILSEG` (1,582) — appears to be the utilities↔insurance customer cross-reference table.
+- `TABLA LIQUIDA *` / `TABLE DATOS LIQUIDA *` — settlement/liquidation batches per policy line (matches `liquidation_number` already in the old MySQL `policies` table).
+- **Exclude from data migration:** the `* MENS` / `*MENSAJE` tables (`AMPL MENS`, `IN MENS`, `LIC MENS`, `MCA2 MENS`, `ME MENS`, `MF MENS`, `RC MENSAJE`, `RC R MENS`) are mail-merge document *templates* (letters, certificates) stored as blobs, not customer data — these get reimplemented as PDF templates in the new app, not migrated as rows. `TODOSJC`, `TODOS`, `vigenta casa y auto unicos` look like materialized Access query results (saved reports), not source-of-truth data — exclude, and if the report itself is still needed, rebuild it as a real query against the new schema.
+
+**SCOTHIA.mdb** (7 tables, ~3MB — the office's own Scotiabank checking-account register, "chequera"). Much simpler than the other two files: this is the company's operating bank account, not customer-facing data.
+- `DATOS E` (15,406 rows) — expenses/outgoing (`EGRESO`): date, transaction type, check/reference `NUM`, `CONCEPTO` (payee/description), amount, `OPERADO` (cleared flag), notes, and a spelled-out amount-in-words field (`CANTIDAD EN LETRA`, standard Mexican check-writing convention).
+- `DATOS I` (6,948 rows) — income/incoming (`INGRESO`): same shape, plus a `TRANSFERIDO` (transferred) flag instead of the amount-in-words field.
+- `TABLA RAMODOS` (66 rows) — a category/business-line lookup ("ramo" = line of business in Mexican insurance terminology) — this is almost certainly what `CONCEPTO` entries get classified against, i.e. the link between a bank transaction and which part of the business (insurance line, utility service, trust, etc.) it belongs to.
+- `ban` (1 row) — just holds the bank's name; a config singleton, not data.
+- `INFORME` / `INFORME BA` (0 rows each) and `FECHAIF` (1 row) — Access report/query scratch tables (date-range parameters and a report shell), same pattern as the scratch tables in the other two files — **exclude** from migration.
+
+## Target architecture
+
+- **Frontend:** Next.js (App Router) + TypeScript + React. Server components for data-heavy list/detail views (customers, policies, statements); client components for interactive forms.
+- **Backend:** NestJS (TypeScript) REST API — modular by domain (customers, insurance, utilities, billing, auth, admin), matching the module boundaries below. Gives you DI, guards for auth/authorization, and a validation pipeline (`class-validator`) for free, which directly replaces the old app's biggest weakness (no input validation, no parameterization).
+- **Database:** MySQL, accessed via Prisma (schema-as-code, migrations, generates a typed client — eliminates the raw-SQL-injection class of bug entirely since Prisma parameterizes everything). See Infrastructure & Sync above for why MySQL rather than Postgres.
+- **Auth:** NestJS + Passport, sessions or JWT (pick one during build), `bcrypt`/`argon2` password hashing, role-based guards replacing the old `$_SESSION['level']/['role']` checks.
+- **File/document storage:** object storage (S3-compatible) for the scanned documents currently trapped as Access `LONGBINARY` blobs — extract once during migration, store as files, keep only the pointer + metadata in MySQL. The old repo already anticipated this (`src/objects/s3.php` exists but appears unused) — same idea, implemented for real this time.
+- **Deployment:** keep Docker as the packaging mechanism (already proven for this project) with a fresh `Dockerfile`/`docker-compose.yml` for the Node services + MySQL, running on the internal server described above; CI can stay on Jenkins if that's still the team's pipeline, or move to GitHub Actions if the new repo lives somewhere other than `git.freakma.com` — flagged as an open decision below since it wasn't part of the stack questions asked.
+
+## Target data model (by domain)
+
+All tables get a surrogate `id` (uuid or serial) plus, where the row came from a legacy table, provenance columns (`legacy_source_db`, `legacy_source_table`, `legacy_id`) so every migrated row can be traced back to its Access original for spot-checking and reconciliation — and so the ETL can be re-run idempotently (upsert on provenance key) as migration bugs get found and fixed.
+
+**Identity (the actual point of this project):**
+- `customers` — one row per real person/entity, merged from utilities `DATGRAL`/`COBRO3` and insurance `DATGRAL`, matched via the `NUM UTIL` cross-reference plus name/address fuzzy-matching for anyone missing that link. Holds name, addresses (MX + US), phones, email, ID document info, status, currency preference.
+- `customer_legacy_refs` — generalizes the old `customer_mapping` table: `(customer_id, source_system, source_table, legacy_numeric_id)`, one row per legacy record folded into this customer. This is what makes "unified customer base" actually queryable and keeps the merge auditable.
+
+**Insurance domain:**
+- `insurance_providers`, `policy_types` (carry over from old schema, already reasonable)
+- `policies` — generic header (policy #, type, provider, customer, dates, premiums, agent, liquidation status), consolidating `INCENDIO`/`MULT`/`M EMPR`/all six auto-table variants/`LICENCIAS` into one table with a `policy_type` discriminator, instead of one Access table per line of business.
+- `policy_payment_installments` — unpivots the 4 hardcoded payment-installment columns (`1ER PAGO`/`FECHA PAGO`/`NO CHEQUE`, `... 2`, `... 3`, `... 4`) into rows: due sequence, amount, currency, paid date, check/reference number.
+- `vehicles` — unpivots `MCA2`'s 3 hardcoded vehicle slots (and the single-vehicle auto tables) into one row per vehicle, FK'd to policy + customer.
+- `insured_drivers` — same unpivot for the repeated named-insured/license columns in `MCA2`/`LICENCIAS`.
+- `properties` (shared with utilities domain — see below), `policy_beneficiaries` (from `BENEF`, already clean), `claims` (from `DATOS`), `adjusters` (from `AJUSTADORES*`), `policy_documents` (extracted blobs, typed: ID, prior policy copy, damage photo, etc.).
+
+**Utilities domain:**
+- `properties` — one row per property (from `DATMEX`), FK'd to `customers`, shared with insurance so a property can carry both a home-insurance policy and utility service enrollments — this is the second half of "unified."
+- `property_services` — one row per enrolled service per property (water/electric/gas/cable/trust/property-tax/federal-zone), unpivoting `DATMEX`'s inline service columns and `PROFILE`'s enrollment flags into real rows with account #, route, meter #, due day.
+- `service_documents` (extracted blobs), `trust_accounts` (from `TRUSTVENCE`).
+
+**Shared financial ledger** (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):
+- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive.
+- `exchange_rates` (from `TIPO HIST`), `type_transactions` (carry over ES/EN lookup as-is).
+- `bank_transactions` — the company's own operating bank register, from SCOTHIA's `DATOS E`/`DATOS I` unified into one signed-amount table (income positive, expense negative) with a `category` FK to `business_line_categories` (from `TABLA RAMODOS`) and a `cleared`/`operado` flag. This is deliberately **separate** from customer-facing `transactions` — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the `business_line_categories` lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
+- `business_line_categories` (from `TABLA RAMODOS`).
+
+**Admin/shared:** `users` (hashed passwords, roles), `activity_logs`, `email_templates`/`email_campaigns`/`email_log` (carry over the old schema's intent, rebuilt on the new stack).
+
+## Migration strategy
+
+Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
+
+1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script (`pyodbc` → `pandas`/SQLAlchemy, same connection approach already validated in this session), across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet.
+2. **Reconciliation pass**: for each set of overlapping tables (the `EFECTIVO` variants, the billing-period tables, `DATGRAL` vs `COBRO3`), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
+3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
+4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
+5. **Validation**: row-count and spot-check reconciliation between `staging` and final tables (e.g., every legacy customer has exactly one `customers` row via `customer_legacy_refs`; sum of migrated transaction amounts per customer matches sum in `staging`).
+
+## Build sequencing
+
+1. Repo scaffold (Next.js + NestJS + Prisma + MySQL, Docker Compose for local dev), CI pipeline, auth module with hashed passwords and role guards.
+2. Prisma schema for the full data model above; run migration steps 1–2 (staging load + reconciliation reports) against real data early, since that's where the biggest unknowns are (do `NUM UTIL` and name-matching actually cover everyone? how bad is the snapshot-table duplication?).
+3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.
+4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
+5. Utilities module (properties, services, trust accounts) on top of the same customer records.
+6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
+7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
+8. VPS provisioning + Tailscale + MySQL replication setup, once `utility_dbo`'s schema is available to finalize exactly which tables/columns get replicated and what the inbox tables need to look like.
+9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8.
+10. Reports/email campaigns/admin — parity with old app's `reports.php`/`emailCampaigns.php` intent, rebuilt properly.
+
+## Status (as of this session)
+
+Repo scaffolded at `jorgecuadros-platform/` (sibling to the Access files): npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps build clean under strict TypeScript. `migration/load_staging.py` (step 1 of Migration strategy) has been run end-to-end against all four real Access files, staging 82 tables to Parquet; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap (Prisma schema, Docker Compose, `.env.example`, migration script's MySQL sink) has since been applied and re-verified (schema validates, client regenerates, API rebuilds clean against MySQL). A companion resume doc lives at `jorgecuadros-platform/RESUME.md` with exact file paths, environment notes, and a session-state summary — read both together.
+
+## Open decisions (not yet locked down)
+
+- **`utility_dbo` schema**: the customer-facing portal's live database — referenced from the old app (`getExternalDBConnection()`) but no dump/access provided yet. Blocks finalizing exactly which tables the sync replicates and what the inbox tables (`payment_submissions`, `propane_order_requests`) need to match.
+- **VPS provisioning**: which provider (Hetzner vs DigitalOcean), size, and who sets up Tailscale + MySQL on it — an ops task outside this session's ability to do directly.
+- **CI/hosting**: keep Jenkins + `git.freakma.com`, or move CI to GitHub Actions if the new repo goes elsewhere?
+- **i18n**: nearly all source data and, presumably, staff usage is in Spanish, while the old app's code/UI was English-labeled internally. Confirm whether the new UI should be Spanish-first, bilingual, or English (matching old app) before frontend work starts.
+
+## Verification
+
+- Migration: automated row-count/sum reconciliation between `staging` and final schema per table group (see step 5 above), run as part of the migration script, not a manual spot-check.
+- App: standard NestJS unit/integration tests per module (auth guards, Prisma queries), Playwright/Cypress e2e for the core "look up a customer, see their unified policies + services + statement" flow — the thing the whole project exists to deliver.
+- Sync: once the VPS replica and inbox tables exist, verify replication lag stays low (a few seconds to low minutes) and that a payment/propane submission on the portal reliably shows up in the internal app within one polling interval, before relying on it operationally.
+- Before cutover: run the new app against migrated data side-by-side with the live Access files for a period, comparing balances/statements for a sample of active customers to catch migration logic errors before the Access files are retired.