# Resume Notes — Jorge Cuadros & Assoc. Unified Platform Comprehensive state-of-the-world doc for picking this project back up. Read this before doing anything else in a fresh session — it front-loads everything that took multiple rounds of investigation to establish. **Companion doc:** the full architecture/migration plan is copied into this repo as [`PLAN.md`](PLAN.md) (source of truth is `C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — copy here if that one gets updated further). This file is the "what happened and what's next" companion to that plan, not a replacement for it. Read both. --- ## 1. The goal Jorge Cuadros & Assoc. runs two lines of business — property/utility management and insurance brokerage — out of separate, decades-old MS Access databases, plus a third Access file that's the office's own bank checking register. The same people are customers of both lines but there's no shared customer record between systems. Goal: **one platform with a single unified customer record**, from which staff see and manage that customer's utility services *and* insurance policies *and* shared billing/transaction history — replacing the Access files and the old, insecure PHP internal app. There is also a **separate, pre-existing customer-facing portal** (PHP + MySQL, with a companion mobile app) that customers use to view statements, make payments, and order propane. That portal is **out of scope to rebuild** — it stays exactly as-is — but the new platform has to keep it supplied with live data. That constraint is what drove the database-engine and infrastructure decisions below. ## 2. Where everything lives (file paths) **Source data (do not modify — read-only references):** - `C:\Users\ricar\Downloads\Jorge\UTILITIES.accdb` — utilities business, 52 tables, ~538MB - `C:\Users\ricar\Downloads\Jorge\SEGUROS 16.mdb` — insurance frontend shell, **empty**, all data is in `_be` - `C:\Users\ricar\Downloads\Jorge\SEGUROS 16_be.mdb` — insurance backend, 64 tables, ~882MB - `C:\Users\ricar\Downloads\Jorge\SCOTHIA.mdb` — office's own Scotiabank checking register ("chequera"), 7 tables, ~3MB - **Full structural reference for all three, usable without Windows or the original files:** [`docs/LEGACY_DATABASES.md`](docs/LEGACY_DATABASES.md) — every table, every column with type/nullability, the cross-reference keys between the three databases, and every known data-quality quirk (the UTF-16 decode bug, the corrupted `MULT` row, near-duplicate snapshot tables, etc.), all generated from a live read of the real files via `migration/catalog_schema.py`. Regenerate it if the source files change; the raw JSON it's built from is checked in at `migration/catalog.json`. - **Queries/Forms/Reports reference:** [`docs/LEGACY_DATABASES_OBJECTS.md`](docs/LEGACY_DATABASES_OBJECTS.md) — none of this is visible via ODBC/`pyodbc`; it required DAO COM automation (`migration/catalog_objects.py`, needs `pywin32`) instead. Found 311 Reports, 271 Forms, and 1,274 Queries (751 "real," the rest Access-internal hidden subquery caches) across the three populated files — importantly, `SEGUROS 16.mdb` (which has zero data tables) turned out to hold *all* of the insurance line's Reports/Forms/Queries; `SEGUROS 16_be.mdb` is confirmed pure data storage. The real queries' full SQL text is the best available record of actual business logic (billing math, renewal batching) — worth reading before reimplementing any given feature from scratch. Raw JSON checked in at `migration/objects.json`. - `C:\Users\ricar\Downloads\jorgecuadros_app.sql` and `jorgecuadros_app (1).sql` — MySQL dumps of the customer-portal's **tracking/analytics** sidecar DB (`browse_tracking`, `devices` push-tokens, `task_tracking`) from `mysql.freakma.com`. **Not** the portal's real data DB — see open item #1 below. **Old internal app (reference-only, not being built on):** - `C:\Users\ricar\Downloads\Jorge\jorgecuadros-intra-webapp` — PHP, MySQL (`webapp_jorgecuadros`). Schema at `db\webapp_jorgecuadros.sql` is a useful reference for field mappings/business logic. Code itself is not being reused — see §4. **New platform (the actual deliverable, in progress):** - `C:\Users\ricar\Downloads\Jorge\jorgecuadros-platform` — the new repo. Not yet a git repository (no commits made this session — user hasn't asked for any yet). **The plan document:** - `C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — full architecture, source-data inventory per table, target data model, migration strategy, infrastructure/sync design, open decisions, build sequencing. **This is the source of truth for the design** — this RESUME.md summarizes it plus session/environment state that isn't in the plan itself. **Ephemeral / will NOT persist across sessions** (session-scoped temp directory): - `C:\Users\ricar\AppData\Local\Temp\claude\...\scratchpad\` — contained exploratory helper scripts (`dump_schema.py`, `summarize_schema.py`, `test_decode_fix.py`) and the schema JSON/txt dumps used during initial analysis, plus a Parquet staging output from one run of the migration script. **None of this needs to be recovered** — the real, permanent versions of the useful scripts are in `jorgecuadros-platform/migration/`, and the Parquet output can be regenerated in under 2 minutes by rerunning `load_staging.py` (see §6). ## 3. Key decisions made this session | Decision | Answer | Why | |---|---|---| | Stack | Next.js (React/TS) + NestJS (TS) + Prisma | Type safety, parameterized queries by default (kills the SQL-injection class of bug the old app had everywhere) | | Database engine | **MySQL** (not Postgres — reversed mid-session) | The customer-facing portal's PHP code (`mysqli`) isn't being rewritten, and shared hosting can't run Postgres. Using MySQL everywhere avoids a cross-engine sync layer. | | Repo | New repo, not built on `jorgecuadros-intra-webapp` | That repo has SQL injection in every query (`src/core/db.php` string-concatenates `$_POST`) and plaintext password comparison (`src/core/auth.php`) — not worth patching | | Historical data | Migrate everything, no cutoff | Source tables are small (largest ~16k rows); completeness is cheap | | Customer portal | Stays as-is, not rebuilt | Explicit user decision | | Infrastructure | Internal server (private) + new VPS (Tailscale-linked) running a MySQL replica | Internal server has no inbound internet exposure; shared hosting can't be a replication target; a VPS you control can be both a real replication node and internet-reachable for the portal | | Auth mechanism | Session-based (Passport + `express-session`), Argon2 password hashing | Implemented already — see §5 | ## 4. What was actually built and verified this session Everything below was **run and confirmed working**, not just written: ### 4.1 Repo scaffold - `jorgecuadros-platform/` — npm workspaces (`apps/*`, `packages/*`) - `apps/api` — NestJS. **Builds clean** under strict TypeScript (`npx nest build` in `apps/api`, zero errors). - `src/main.ts` — global `ValidationPipe` (whitelist + forbid unknown fields — structural replacement for the old app's total lack of input validation), session middleware, CORS. - `src/auth/` — `AuthService.validateUser()` verifies passwords with `argon2.verify()` (replaces `passwd = '$password'` plaintext SQL comparison in the old app), `LocalStrategy`, `SessionSerializer`, `AuthenticatedGuard` (replaces manually-called `validate_session()`), `AuthController` (`/auth/login`, `/auth/me`, `/auth/logout`). - `src/users/`, `src/prisma/` (global `PrismaModule`/`PrismaService`). - `apps/web` — Next.js App Router shell. **Builds clean** (`npx next build`). - `packages/database` — Prisma schema + generated client. ### 4.2 Prisma schema (`packages/database/prisma/schema.prisma`) Full target data model implementing the plan's design, **validated and generating a working client against MySQL**: - **Identity:** `Customer`, `CustomerLegacyRef` (generalizes the old `customer_mapping` bridge table — one row per legacy record folded into a unified customer, with provenance) - **Insurance:** `InsuranceProvider`, `PolicyType`, `Policy` (consolidates `INCENDIO`/`MULT`/`M EMPR`/6 auto-table variants/`LICENCIAS` into one table with a type discriminator), `PolicyPaymentInstallment` (unpivots the 4 hardcoded payment-installment columns found on every legacy policy table), `Vehicle` (unpivots `MCA2`'s 3 hardcoded vehicle slots), `InsuredDriver`, `PolicyBeneficiary`, `Claim`, `Adjuster`, `PolicyDocument` - **Utilities:** `Property` (shared with insurance — the actual unification point), `PropertyService`, `ServiceDocument`, `TrustAccount` - **Shared ledger:** `Transaction` (unifies all the `EFECTIVO*`/billing-period snapshot tables), `ExchangeRate`, `TypeTransaction` - **Bank register (SCOTHIA):** `BankTransaction`, `BusinessLineCategory` - **Admin:** `User` (hashed passwords, roles), `ActivityLog`, `EmailTemplate`, `EmailCampaign`, `EmailLog` Every model sourced from a legacy Access table carries `legacySourceDb`/`legacySourceTable`/`legacyId` provenance columns for traceability and idempotent re-runs. Long-text fields (`notes`, `observations`, `description`, etc.) are explicitly `@db.Text` — MySQL's default `String` is `VARCHAR(191)` and would silently truncate them otherwise (this was caught and fixed during the Postgres→MySQL swap). Regenerate the client any time with: ```bash cd jorgecuadros-platform DATABASE_URL="mysql://user:pass@localhost:3306/placeholder" npx prisma generate --schema=packages/database/prisma/schema.prisma ``` (A real `DATABASE_URL` isn't needed for `generate`/`validate`, just a syntactically valid one — no live DB was available in this session, see §7.) ### 4.3 Docker Compose / Dockerfiles - `docker-compose.yml` — `mysql:8.4` + `api` + `web` services, healthchecked. - `docker/api.Dockerfile`, `docker/web.Dockerfile` — multi-stage builds. - `.env.example` — `DATABASE_URL`, `SESSION_SECRET`, `WEB_ORIGIN`, `NEXT_PUBLIC_API_ORIGIN`. - **Not run** — this environment has no Docker installed (`docker --version` fails). Untested beyond visual review; verify on a machine with Docker before relying on it. ### 4.4 Migration pipeline (`migration/`) — run end-to-end against real data - `config.py` — manifest of the 3 Access source files (paths + per-source exclude lists for confirmed-scratch tables, with reasoning in comments) - `extract.py` — connects via `pyodbc` + the Windows Access ODBC driver (`Microsoft Access Driver (*.mdb, *.accdb)`, 64-bit — confirmed installed on this machine). Two real bugs found and fixed here: 1. `cursor.columns()` hits a UTF-16 decode bug on some tables (confirmed: `PROPANO`, `FALTANTES AGUA`, `TIT`) — fixed by reading column names from `cursor.description` after a `SELECT *` instead. 2. `cursor.fetchall()` aborts an entire table on the first corrupted row — confirmed on `MULT` (Jet/ACE-level "Record is deleted" error, HY109). Fixed by fetching row-by-row in a try/except, skipping and logging just the bad row. Recovered 763 of 764 rows in `MULT`. **Verified the cursor advances correctly and doesn't infinite-loop on the bad row** before trusting this for a full run. - `load_staging.py` — dumps every non-excluded table into either Parquet (`--output-dir`, no DB needed) or MySQL (`--database-url`, one database per source: `stg_utilities`/`stg_seguros`/`stg_scothia`). **Actually run** in Parquet mode against all three Access files: **82 tables staged successfully, zero unhandled errors**, `MULT`'s corrupted row correctly skipped and logged. - `requirements.txt` — `pyodbc`, `pandas`, `pyarrow`, `sqlalchemy`, `pymysql`. To rerun (from `jorgecuadros-platform/migration`, after `pip install -r requirements.txt`): ```bash python load_staging.py --output-dir ./output # Parquet, no DB needed — always works python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/ # loads into real MySQL once available ``` ## 5. Infrastructure & sync architecture (designed, not yet built) - **Internal server** — on-prem, private IP `192.168.1.xx`, no inbound internet exposure. Runs the platform + canonical MySQL (source of truth). - **New VPS** (Hetzner or DigitalOcean — not yet provisioned) — becomes what `mysql.freakma.com` resolves to. Runs a MySQL replica. Because it's infrastructure the user controls (unlike shared hosting), it can be a real MySQL replication target. - **Tailscale** — mesh VPN joining the internal server and the VPS, so they can reach each other without opening inbound ports anywhere. - **Internal → VPS:** one-way native MySQL replication (binlog/GTID) for the subset of data the portal needs to read (statements, balances, customer profile). Internal-only tables (staff notes, adjuster info, activity logs) are deliberately excluded from what replicates. - **VPS → Internal:** the portal also *writes* (payment submissions, propane orders) — one-way replication can't carry that back, and multi-master MySQL replication was deliberately ruled out as too fragile for this system's size. Instead: unreplicated "inbox" tables on the VPS (`payment_submissions`, `propane_order_requests`) that the portal writes to directly, polled every 1–5 min by a worker on the internal server (over Tailscale) that turns new rows into real records. ## 6. Open items — need input/access before certain next steps can proceed 1. **`utility_dbo` schema is still unknown.** This is the customer portal's actual live data database (referenced in the old app via `getExternalDBConnection()` at `mysql.freakma.com`, used there only for `email_alert_log`, but the mobile app almost certainly reads/writes statements, payments, and propane orders directly against it). The two SQL dumps provided (`jorgecuadros_app.sql`, `jorgecuadros_app (1).sql`) turned out to be a **separate** analytics/tracking database, not this one. When asked, the user pointed back to `SEGUROS 16_be.mdb` — worth revisiting; it's possible the intent was "the source data ultimately comes from the Access files" rather than "here is utility_dbo's schema." **Without the real `utility_dbo` schema, the sync worker's exact target tables/columns for the inbox pattern can't be finalized.** Ask for an export or read-only credentials, same as how the Access files and tracking-DB dumps were provided. 2. **VPS not yet provisioned** — provider (Hetzner vs DigitalOcean), size, and Tailscale/MySQL setup on it are pending. Ops task, not something done in this session. 3. **CI/hosting** — keep Jenkins + `git.freakma.com`, or move to GitHub Actions if the new repo lives elsewhere? 4. **i18n** — nearly all source data and, presumably, staff usage is in Spanish; old app's code/UI was English-labeled. Confirm Spanish-first / bilingual / English before frontend work goes deep. 5. **Reconciliation pass not started** (migration plan step 2) — the near-duplicate snapshot tables (`EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`, `FEE ANUAL`/`datos2`/`fee15`/`billing`, `DATGRAL` vs `COBRO3`) need a diff/dedupe pass against the staged data before any transform-and-load into the real schema. This is explicitly *not* a "guess the rule up front" thing — the plan calls for writing SQL against the staged Parquet/MySQL data to see what's actually duplicate vs. distinct. ## 7. Environment notes (this machine, in case it matters for reproducing) - Windows, PowerShell primary, Git Bash also available. - Node v25.2.0, npm 11.6.2 — no pnpm, yarn is present but npm workspaces were used throughout. - Python 3.9 (`C:\Python39`), `pip install`'d this session: `pyodbc`, `pandas`, `pyarrow`, `psycopg2` (installed but no longer used after the MySQL switch — harmless to leave or remove). - MS Access ODBC driver confirmed installed: **64-bit** `Microsoft Access Driver (*.mdb, *.accdb)` (matches 64-bit Python — this pairing matters, a 32/64-bit mismatch would break `pyodbc.connect`). - **No Docker, no local MySQL, no local Postgres** on this machine — `docker-compose.yml` and the MySQL-target mode of `load_staging.py` are written but unexecuted here. Test both on whatever machine ends up running this for real. - Old repo's `dbConnection.php` has a **hardcoded plaintext MySQL password** for the external DB connection, committed to git history. Not carried forward into the new platform, but worth rotating that credential regardless since it's already exposed in the old repo's history. ## 8. Plan locked — next actions **All four previously-open decisions are now locked (2026-07-22).** See `PLAN.md` → "Decisions (locked)" for the authoritative record: - Extraction toolchain (macOS): **mdbtools** (installed + verified against the real files). - i18n: **Spanish-first**. - CI/CD: **Gitea Actions** on `git.mancinas.io` → registry → Portainer. - `utility_dbo`: **resolved** — full dump (`utility_dbo.sql`, 1.3 GB, 55 tables) and the portal codebase (`~/PhpstormProjects/my-jorgecuadros-web`) are both on disk. **Environment: moved Windows → macOS.** Sources now at `~/Downloads/JorgeCuadros-Legacy/` (all four files). This machine has Docker, MySQL/MariaDB client, Node 22, Python 3.14, Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet not present. **Execution queue (in order):** 1. **Port the extraction layer to mdbtools.** Rewrite `migration/extract.py` to shell out to `mdb-tables`/`mdb-export` instead of `pyodbc`. Keep the same public interface (`connect`/`list_tables`/`read_table`) so `load_staging.py` and `config.py` are unchanged beyond the already-fixed `SOURCE_ROOT`. Carry over the two hard-won fixes conceptually: accented-column tables (mdbtools reads `PROPANO` cleanly — verified) and the corrupted `MULT` row (mdb-export's `-b` / error handling; confirm the bad row is skipped, not fatal). 2. **Re-run staging** (`python load_staging.py --output-dir ./output`) to regenerate the staged data on this machine, then load into a local MySQL (`docker compose up mysql`) for SQL reconciliation. 3. **Reconciliation pass** (plan step 2) — **DONE** (`migration/reconcile.py` → `RECONCILIATION.md`). Overturned all three "duplicate" assumptions: EFECTIVO/BACKUP are near-disjoint ledgers (folio collides; migrate both), the billing tables are disjoint period runs (union all, no de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The decided union/de-dup rules are in `PLAN.md` migration step 2. 4. **Transform + load** (plan step 3) — IN PROGRESS. - **Customers — DONE** (`migration/transform_customers.py`). Loaded into the dev DB: 1682 customers (1172 utilities master + 510 insurance-only), 2242 legacy refs (all traceable), 560 insurance rows linked via `num_util` with 0 broken refs, **542 merged identities** spanning both business lines; linked customers enriched with insurance-only ID-doc fields. COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present (`load_staging.py --output-dir ./output` first). - **Properties — DONE** (`migration/transform_properties.py`): 1519 properties (0 orphans), 3486 services, 553 trust accounts from DATMEX/PROFILE; PROFILE flags matched 1519/1519. - **Policies — DONE** (`migration/transform_policies.py`): config-driven consolidation of all insurance lines into `policies` (2378: AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5; 10 skipped for unresolved customer, 0 orphans) + 4678 installments, 1110 vehicles, 513 insured_drivers, 126 beneficiaries, 1 claim, 5 policy_types, 15 insurance_providers, 17 adjusters. Unmodeled coverage columns preserved verbatim in `coveragesJson`. Verified a unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT policies — the cross-line customer view works at the data layer. - **Shared ledger — DONE** (`migration/transform_transactions.py`): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans) unioning both EFECTIVO tables (13696+12386, no folio de-dup), all three billing tables (datos2/FEE ANUAL/fee15), the FM3 fee stream (amount=fee+tax+multa), IVA 2015 (nominal date), and insurance EFECTIVO — per the reconciliation rules; plus 79 `type_transactions` (EN/ES) and 2301 `exchange_rates`. Skipped 22 no-customer + 303 no-date (mostly datos2 blanks). - **Bank register — DONE** (`migration/transform_bank.py`): 22354 `bank_transactions` from SCOTHIA DATOS I/E as signed amounts (income +, expense -; net +899,375.77), 66 `business_line_categories`. No customer FK; categoryId left null (concept->ramo classifier is a later enhancement). - **Documents — DONE** (`migration/blob_extract.py`, migration step 4): carves embedded files out of the Access OLE wrapper (magic-byte detection) and uploads to MinIO on cubex (stack `jorgecuadros-dev-minio`, S3 at 192.168.4.212:9100, bucket jorgecuadros-documents), writing service_documents/policy_documents pointer rows. Loaded 70 documents (3 service bills + 67 policy foto/docs, 0 orphans, ~290 MB). **Data finding:** the LONGBINARY columns are almost entirely empty — DATMEX's real bill-scan columns are ILUZ/IAGUA/IPREDIAL/ITEL (not doc_1/doc_2), but only 3 cells populated across 1520 rows; the *_MENS tables are mail-merge templates (correctly excluded). The 538MB/882MB source files are mostly Access bloat, not documents. **Migration steps 1-4 COMPLETE.** - **Customer module (plan step 3) — DONE**: `apps/api/src/customers/` (list/search/detail/stats) + `apps/web` `/clientes` and `/clientes/[id]`, Spanish-first, verified against real data. - **Insurance module (plan step 4) — DONE**: `apps/api/src/policies/` (`GET /policies` with search over policy number / customer / agent / plate / driver name, vigencia buckets active|expiring|expired|undated, ramo + aseguradora + liquidada filters, 5 sorts; `/policies/stats`, `/policies/facets`, `/policies/:id`) + web `/polizas` (renewals-first browser, clickable stat cells) and `/polizas/[id]`. Cross-links both ways with the customer view. **Data finding:** the `policies.total` column is dead — only 2 of 2378 rows are non-zero (1585 are literally 0, 791 null) and one of those two is *lower* than its own net premium, so every premium headline and the premium sort use `netPremium` (populated on 2377/2378). This also fixed a live bug on the customer detail page, which was showing "$0.00 Total" for 1585 policies. - **Utilities module (plan step 5) — DONE**: `apps/api/src/properties/` (`GET /properties` with search over address / customer / service account number / meter / trust number / phones, filters for service kind, municipality, trust bank, trust bucket (with|without|active|expiring|expired|undated) and `hasServices`, 5 sorts; `/properties/stats`, `/properties/facets`, `/properties/:id`) + web `/servicios` (renewals-first property browser with clickable stat cells and a clickable service-mix strip) and `/servicios/[id]` (services, fideicomiso, linked policy, owner + sibling properties, owner-level utility ledger, documents). Cross-links both ways with the customer and policy views. **Data findings:** (a) the trust renewal date staff chase is `trust_accounts.dueDate2` — DATMEX's `vence2`, one year after `vence1` on 531 of 541 dated trusts (18 due within 30 days, 119 already overdue); (b) `properties.zone` is effectively dead (1444 of 1519 null, the rest near-unique), so it is not a facet; (c) the municipality that bills a property lives in the *predial* service's `notes` (ROSARITO 566 / TIJUANA 221 / ENSENADA 152, 939/939 populated) — that is the real geographic filter. `PropertyService.notes` means something different per kind (municipality / CFE PAR-IMPAR cycle / gas supply type), so the UI labels it per kind. 240 of 1519 properties have no service rows at all — surfaced as its own bucket. - **NEXT:** plan step 6 — the shared billing/statements view across both business lines. - Full pipeline reproducible in one command: `run_all.py --env ` runs customers -> properties -> policies -> transactions -> bank in order (all idempotent); add `--stage` to re-extract from the Access files first. Verified end-to-end against dev. 5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack `jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled `jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file: `deploy/jorgecuadros-db.stack.yml` (same file deploys prod as `jorgecuadros-prod-db` :3306). Creds in gitignored `deploy/.env.dev`. NOTE: machine `npm` is pnpm-aliased and pnpm ignores the `workspaces` field — full workspace install needs `pnpm-workspace.yaml` or real npm; for now Prisma CLI is run via `npx prisma@5`. 5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature, Spanish-first UI. Run `npm install` at repo root first (node_modules absent here). 6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the VPS→internal inbox tables against the real `utility_dbo` schema and the portal's read/write points in `my-jorgecuadros-web` (`peticion_gas`, PayPal payments, `notifications_settings`). Only genuinely-pending item is **VPS provisioning** (ops task — provider/size/Tailscale+MySQL).