Initial scaffold: unified customer/insurance/utilities platform

Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.

See PLAN.md and RESUME.md for the full architecture and session history.

RESUME.md

@@ -0,0 +1,147 @@
+# Resume Notes — Jorge Cuadros & Assoc. Unified Platform
+
+Comprehensive state-of-the-world doc for picking this project back up. Read this
+before doing anything else in a fresh session — it front-loads everything that
+took multiple rounds of investigation to establish.
+
+**Companion doc:** the full architecture/migration plan is copied into this
+repo as [`PLAN.md`](PLAN.md) (source of truth is
+`C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — copy here if that
+one gets updated further). This file is the "what happened and what's next"
+companion to that plan, not a replacement for it. Read both.
+
+---
+
+## 1. The goal
+
+Jorge Cuadros & Assoc. runs two lines of business — property/utility
+management and insurance brokerage — out of separate, decades-old MS Access
+databases, plus a third Access file that's the office's own bank checking
+register. The same people are customers of both lines but there's no shared
+customer record between systems. Goal: **one platform with a single unified
+customer record**, from which staff see and manage that customer's utility
+services *and* insurance policies *and* shared billing/transaction history —
+replacing the Access files and the old, insecure PHP internal app.
+
+There is also a **separate, pre-existing customer-facing portal** (PHP +
+MySQL, with a companion mobile app) that customers use to view statements,
+make payments, and order propane. That portal is **out of scope to rebuild**
+— it stays exactly as-is — but the new platform has to keep it supplied with
+live data. That constraint is what drove the database-engine and
+infrastructure decisions below.
+
+## 2. Where everything lives (file paths)
+
+**Source data (do not modify — read-only references):**
+- `C:\Users\ricar\Downloads\Jorge\UTILITIES.accdb` — utilities business, 52 tables, ~538MB
+- `C:\Users\ricar\Downloads\Jorge\SEGUROS 16.mdb` — insurance frontend shell, **empty**, all data is in `_be`
+- `C:\Users\ricar\Downloads\Jorge\SEGUROS 16_be.mdb` — insurance backend, 64 tables, ~882MB
+- `C:\Users\ricar\Downloads\Jorge\SCOTHIA.mdb` — office's own Scotiabank checking register ("chequera"), 7 tables, ~3MB
+- `C:\Users\ricar\Downloads\jorgecuadros_app.sql` and `jorgecuadros_app (1).sql` — MySQL dumps of the customer-portal's **tracking/analytics** sidecar DB (`browse_tracking`, `devices` push-tokens, `task_tracking`) from `mysql.freakma.com`. **Not** the portal's real data DB — see open item #1 below.
+
+**Old internal app (reference-only, not being built on):**
+- `C:\Users\ricar\Downloads\Jorge\jorgecuadros-intra-webapp` — PHP, MySQL (`webapp_jorgecuadros`). Schema at `db\webapp_jorgecuadros.sql` is a useful reference for field mappings/business logic. Code itself is not being reused — see §4.
+
+**New platform (the actual deliverable, in progress):**
+- `C:\Users\ricar\Downloads\Jorge\jorgecuadros-platform` — the new repo. Not yet a git repository (no commits made this session — user hasn't asked for any yet).
+
+**The plan document:**
+- `C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — full architecture, source-data inventory per table, target data model, migration strategy, infrastructure/sync design, open decisions, build sequencing. **This is the source of truth for the design** — this RESUME.md summarizes it plus session/environment state that isn't in the plan itself.
+
+**Ephemeral / will NOT persist across sessions** (session-scoped temp directory):
+- `C:\Users\ricar\AppData\Local\Temp\claude\...\scratchpad\` — contained exploratory helper scripts (`dump_schema.py`, `summarize_schema.py`, `test_decode_fix.py`) and the schema JSON/txt dumps used during initial analysis, plus a Parquet staging output from one run of the migration script. **None of this needs to be recovered** — the real, permanent versions of the useful scripts are in `jorgecuadros-platform/migration/`, and the Parquet output can be regenerated in under 2 minutes by rerunning `load_staging.py` (see §6).
+
+## 3. Key decisions made this session
+
+| Decision | Answer | Why |
+|---|---|---|
+| Stack | Next.js (React/TS) + NestJS (TS) + Prisma | Type safety, parameterized queries by default (kills the SQL-injection class of bug the old app had everywhere) |
+| Database engine | **MySQL** (not Postgres — reversed mid-session) | The customer-facing portal's PHP code (`mysqli`) isn't being rewritten, and shared hosting can't run Postgres. Using MySQL everywhere avoids a cross-engine sync layer. |
+| Repo | New repo, not built on `jorgecuadros-intra-webapp` | That repo has SQL injection in every query (`src/core/db.php` string-concatenates `$_POST`) and plaintext password comparison (`src/core/auth.php`) — not worth patching |
+| Historical data | Migrate everything, no cutoff | Source tables are small (largest ~16k rows); completeness is cheap |
+| Customer portal | Stays as-is, not rebuilt | Explicit user decision |
+| Infrastructure | Internal server (private) + new VPS (Tailscale-linked) running a MySQL replica | Internal server has no inbound internet exposure; shared hosting can't be a replication target; a VPS you control can be both a real replication node and internet-reachable for the portal |
+| Auth mechanism | Session-based (Passport + `express-session`), Argon2 password hashing | Implemented already — see §5 |
+
+## 4. What was actually built and verified this session
+
+Everything below was **run and confirmed working**, not just written:
+
+### 4.1 Repo scaffold
+- `jorgecuadros-platform/` — npm workspaces (`apps/*`, `packages/*`)
+- `apps/api` — NestJS. **Builds clean** under strict TypeScript (`npx nest build` in `apps/api`, zero errors).
+  - `src/main.ts` — global `ValidationPipe` (whitelist + forbid unknown fields — structural replacement for the old app's total lack of input validation), session middleware, CORS.
+  - `src/auth/` — `AuthService.validateUser()` verifies passwords with `argon2.verify()` (replaces `passwd = '$password'` plaintext SQL comparison in the old app), `LocalStrategy`, `SessionSerializer`, `AuthenticatedGuard` (replaces manually-called `validate_session()`), `AuthController` (`/auth/login`, `/auth/me`, `/auth/logout`).
+  - `src/users/`, `src/prisma/` (global `PrismaModule`/`PrismaService`).
+- `apps/web` — Next.js App Router shell. **Builds clean** (`npx next build`).
+- `packages/database` — Prisma schema + generated client.
+
+### 4.2 Prisma schema (`packages/database/prisma/schema.prisma`)
+Full target data model implementing the plan's design, **validated and generating a working client against MySQL**:
+- **Identity:** `Customer`, `CustomerLegacyRef` (generalizes the old `customer_mapping` bridge table — one row per legacy record folded into a unified customer, with provenance)
+- **Insurance:** `InsuranceProvider`, `PolicyType`, `Policy` (consolidates `INCENDIO`/`MULT`/`M EMPR`/6 auto-table variants/`LICENCIAS` into one table with a type discriminator), `PolicyPaymentInstallment` (unpivots the 4 hardcoded payment-installment columns found on every legacy policy table), `Vehicle` (unpivots `MCA2`'s 3 hardcoded vehicle slots), `InsuredDriver`, `PolicyBeneficiary`, `Claim`, `Adjuster`, `PolicyDocument`
+- **Utilities:** `Property` (shared with insurance — the actual unification point), `PropertyService`, `ServiceDocument`, `TrustAccount`
+- **Shared ledger:** `Transaction` (unifies all the `EFECTIVO*`/billing-period snapshot tables), `ExchangeRate`, `TypeTransaction`
+- **Bank register (SCOTHIA):** `BankTransaction`, `BusinessLineCategory`
+- **Admin:** `User` (hashed passwords, roles), `ActivityLog`, `EmailTemplate`, `EmailCampaign`, `EmailLog`
+
+Every model sourced from a legacy Access table carries `legacySourceDb`/`legacySourceTable`/`legacyId` provenance columns for traceability and idempotent re-runs. Long-text fields (`notes`, `observations`, `description`, etc.) are explicitly `@db.Text` — MySQL's default `String` is `VARCHAR(191)` and would silently truncate them otherwise (this was caught and fixed during the Postgres→MySQL swap).
+
+Regenerate the client any time with:
+```bash
+cd jorgecuadros-platform
+DATABASE_URL="mysql://user:pass@localhost:3306/placeholder" npx prisma generate --schema=packages/database/prisma/schema.prisma
+```
+(A real `DATABASE_URL` isn't needed for `generate`/`validate`, just a syntactically valid one — no live DB was available in this session, see §7.)
+
+### 4.3 Docker Compose / Dockerfiles
+- `docker-compose.yml` — `mysql:8.4` + `api` + `web` services, healthchecked.
+- `docker/api.Dockerfile`, `docker/web.Dockerfile` — multi-stage builds.
+- `.env.example` — `DATABASE_URL`, `SESSION_SECRET`, `WEB_ORIGIN`, `NEXT_PUBLIC_API_ORIGIN`.
+- **Not run** — this environment has no Docker installed (`docker --version` fails). Untested beyond visual review; verify on a machine with Docker before relying on it.
+
+### 4.4 Migration pipeline (`migration/`) — run end-to-end against real data
+- `config.py` — manifest of the 3 Access source files (paths + per-source exclude lists for confirmed-scratch tables, with reasoning in comments)
+- `extract.py` — connects via `pyodbc` + the Windows Access ODBC driver (`Microsoft Access Driver (*.mdb, *.accdb)`, 64-bit — confirmed installed on this machine). Two real bugs found and fixed here:
+  1. `cursor.columns()` hits a UTF-16 decode bug on some tables (confirmed: `PROPANO`, `FALTANTES AGUA`, `TIT`) — fixed by reading column names from `cursor.description` after a `SELECT *` instead.
+  2. `cursor.fetchall()` aborts an entire table on the first corrupted row — confirmed on `MULT` (Jet/ACE-level "Record is deleted" error, HY109). Fixed by fetching row-by-row in a try/except, skipping and logging just the bad row. Recovered 763 of 764 rows in `MULT`. **Verified the cursor advances correctly and doesn't infinite-loop on the bad row** before trusting this for a full run.
+- `load_staging.py` — dumps every non-excluded table into either Parquet (`--output-dir`, no DB needed) or MySQL (`--database-url`, one database per source: `stg_utilities`/`stg_seguros`/`stg_scothia`). **Actually run** in Parquet mode against all three Access files: **82 tables staged successfully, zero unhandled errors**, `MULT`'s corrupted row correctly skipped and logged.
+- `requirements.txt` — `pyodbc`, `pandas`, `pyarrow`, `sqlalchemy`, `pymysql`.
+
+To rerun (from `jorgecuadros-platform/migration`, after `pip install -r requirements.txt`):
+```bash
+python load_staging.py --output-dir ./output              # Parquet, no DB needed — always works
+python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/   # loads into real MySQL once available
+```
+
+## 5. Infrastructure & sync architecture (designed, not yet built)
+
+- **Internal server** — on-prem, private IP `192.168.1.xx`, no inbound internet exposure. Runs the platform + canonical MySQL (source of truth).
+- **New VPS** (Hetzner or DigitalOcean — not yet provisioned) — becomes what `mysql.freakma.com` resolves to. Runs a MySQL replica. Because it's infrastructure the user controls (unlike shared hosting), it can be a real MySQL replication target.
+- **Tailscale** — mesh VPN joining the internal server and the VPS, so they can reach each other without opening inbound ports anywhere.
+- **Internal → VPS:** one-way native MySQL replication (binlog/GTID) for the subset of data the portal needs to read (statements, balances, customer profile). Internal-only tables (staff notes, adjuster info, activity logs) are deliberately excluded from what replicates.
+- **VPS → Internal:** the portal also *writes* (payment submissions, propane orders) — one-way replication can't carry that back, and multi-master MySQL replication was deliberately ruled out as too fragile for this system's size. Instead: unreplicated "inbox" tables on the VPS (`payment_submissions`, `propane_order_requests`) that the portal writes to directly, polled every 1–5 min by a worker on the internal server (over Tailscale) that turns new rows into real records.
+
+## 6. Open items — need input/access before certain next steps can proceed
+
+1. **`utility_dbo` schema is still unknown.** This is the customer portal's actual live data database (referenced in the old app via `getExternalDBConnection()` at `mysql.freakma.com`, used there only for `email_alert_log`, but the mobile app almost certainly reads/writes statements, payments, and propane orders directly against it). The two SQL dumps provided (`jorgecuadros_app.sql`, `jorgecuadros_app (1).sql`) turned out to be a **separate** analytics/tracking database, not this one. When asked, the user pointed back to `SEGUROS 16_be.mdb` — worth revisiting; it's possible the intent was "the source data ultimately comes from the Access files" rather than "here is utility_dbo's schema." **Without the real `utility_dbo` schema, the sync worker's exact target tables/columns for the inbox pattern can't be finalized.** Ask for an export or read-only credentials, same as how the Access files and tracking-DB dumps were provided.
+2. **VPS not yet provisioned** — provider (Hetzner vs DigitalOcean), size, and Tailscale/MySQL setup on it are pending. Ops task, not something done in this session.
+3. **CI/hosting** — keep Jenkins + `git.freakma.com`, or move to GitHub Actions if the new repo lives elsewhere?
+4. **i18n** — nearly all source data and, presumably, staff usage is in Spanish; old app's code/UI was English-labeled. Confirm Spanish-first / bilingual / English before frontend work goes deep.
+5. **Reconciliation pass not started** (migration plan step 2) — the near-duplicate snapshot tables (`EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`, `FEE ANUAL`/`datos2`/`fee15`/`billing`, `DATGRAL` vs `COBRO3`) need a diff/dedupe pass against the staged data before any transform-and-load into the real schema. This is explicitly *not* a "guess the rule up front" thing — the plan calls for writing SQL against the staged Parquet/MySQL data to see what's actually duplicate vs. distinct.
+
+## 7. Environment notes (this machine, in case it matters for reproducing)
+
+- Windows, PowerShell primary, Git Bash also available.
+- Node v25.2.0, npm 11.6.2 — no pnpm, yarn is present but npm workspaces were used throughout.
+- Python 3.9 (`C:\Python39`), `pip install`'d this session: `pyodbc`, `pandas`, `pyarrow`, `psycopg2` (installed but no longer used after the MySQL switch — harmless to leave or remove).
+- MS Access ODBC driver confirmed installed: **64-bit** `Microsoft Access Driver (*.mdb, *.accdb)` (matches 64-bit Python — this pairing matters, a 32/64-bit mismatch would break `pyodbc.connect`).
+- **No Docker, no local MySQL, no local Postgres** on this machine — `docker-compose.yml` and the MySQL-target mode of `load_staging.py` are written but unexecuted here. Test both on whatever machine ends up running this for real.
+- Old repo's `dbConnection.php` has a **hardcoded plaintext MySQL password** for the external DB connection, committed to git history. Not carried forward into the new platform, but worth rotating that credential regardless since it's already exposed in the old repo's history.
+
+## 8. Suggested next session starting point
+
+1. Chase down open item #1 (`utility_dbo` schema) — it blocks finalizing the sync design concretely.
+2. Start the reconciliation pass (plan step 2, open item #5) against the staged Parquet data — rerun `load_staging.py --output-dir` first since the original output didn't persist.
+3. Once reconciliation rules are known, write the transform-and-load scripts (plan step 3) that populate the real Prisma-managed MySQL tables from staging, starting with the `Customer`/`CustomerLegacyRef` module since every other module depends on it.
+4. In parallel or after: build out the Customer module in `apps/api`/`apps/web` (list/search/detail) — the first real feature, per Build Sequencing step 3 in the plan.