jorgecuadros-platform/RESUME.md

Resume Notes — Jorge Cuadros & Assoc. Unified Platform

Comprehensive state-of-the-world doc for picking this project back up. Read this before doing anything else in a fresh session — it front-loads everything that took multiple rounds of investigation to establish.

Companion doc: the full architecture/migration plan is copied into this repo as PLAN.md (source of truth is C:\Users\ricar\.claude\plans\logical-yawning-tome.md — copy here if that one gets updated further). This file is the "what happened and what's next" companion to that plan, not a replacement for it. Read both.

---

1. The goal

Jorge Cuadros & Assoc. runs two lines of business — property/utility management and insurance brokerage — out of separate, decades-old MS Access databases, plus a third Access file that's the office's own bank checking register. The same people are customers of both lines but there's no shared customer record between systems. Goal: one platform with a single unified customer record, from which staff see and manage that customer's utility services and insurance policies and shared billing/transaction history — replacing the Access files and the old, insecure PHP internal app.

There is also a separate, pre-existing customer-facing portal (PHP + MySQL, with a companion mobile app) that customers use to view statements, make payments, and order propane. That portal is out of scope to rebuild — it stays exactly as-is — but the new platform has to keep it supplied with live data. That constraint is what drove the database-engine and infrastructure decisions below.

2. Where everything lives (file paths)

Source data (do not modify — read-only references):

• C:\Users\ricar\Downloads\Jorge\UTILITIES.accdb — utilities business, 52 tables, ~538MB
• C:\Users\ricar\Downloads\Jorge\SEGUROS 16.mdb — insurance frontend shell, empty, all data is in _be
• C:\Users\ricar\Downloads\Jorge\SEGUROS 16_be.mdb — insurance backend, 64 tables, ~882MB
• C:\Users\ricar\Downloads\Jorge\SCOTHIA.mdb — office's own Scotiabank checking register ("chequera"), 7 tables, ~3MB
• Full structural reference for all three, usable without Windows or the original files: docs/LEGACY_DATABASES.md — every table, every column with type/nullability, the cross-reference keys between the three databases, and every known data-quality quirk (the UTF-16 decode bug, the corrupted MULT row, near-duplicate snapshot tables, etc.), all generated from a live read of the real files via migration/catalog_schema.py. Regenerate it if the source files change; the raw JSON it's built from is checked in at migration/catalog.json.
• Queries/Forms/Reports reference: docs/LEGACY_DATABASES_OBJECTS.md — none of this is visible via ODBC/pyodbc; it required DAO COM automation (migration/catalog_objects.py, needs pywin32) instead. Found 311 Reports, 271 Forms, and 1,274 Queries (751 "real," the rest Access-internal hidden subquery caches) across the three populated files — importantly, SEGUROS 16.mdb (which has zero data tables) turned out to hold all of the insurance line's Reports/Forms/Queries; SEGUROS 16_be.mdb is confirmed pure data storage. The real queries' full SQL text is the best available record of actual business logic (billing math, renewal batching) — worth reading before reimplementing any given feature from scratch. Raw JSON checked in at migration/objects.json.
• C:\Users\ricar\Downloads\jorgecuadros_app.sql and jorgecuadros_app (1).sql — MySQL dumps of the customer-portal's tracking/analytics sidecar DB (browse_tracking, devices push-tokens, task_tracking) from mysql.freakma.com. Not the portal's real data DB — see open item #1 below.

Old internal app (reference-only, not being built on):

• C:\Users\ricar\Downloads\Jorge\jorgecuadros-intra-webapp — PHP, MySQL (webapp_jorgecuadros). Schema at db\webapp_jorgecuadros.sql is a useful reference for field mappings/business logic. Code itself is not being reused — see §4.

New platform (the actual deliverable, in progress):

• C:\Users\ricar\Downloads\Jorge\jorgecuadros-platform — the new repo. Not yet a git repository (no commits made this session — user hasn't asked for any yet).

The plan document:

• C:\Users\ricar\.claude\plans\logical-yawning-tome.md — full architecture, source-data inventory per table, target data model, migration strategy, infrastructure/sync design, open decisions, build sequencing. This is the source of truth for the design — this RESUME.md summarizes it plus session/environment state that isn't in the plan itself.

Ephemeral / will NOT persist across sessions (session-scoped temp directory):

• C:\Users\ricar\AppData\Local\Temp\claude\...\scratchpad\ — contained exploratory helper scripts (dump_schema.py, summarize_schema.py, test_decode_fix.py) and the schema JSON/txt dumps used during initial analysis, plus a Parquet staging output from one run of the migration script. None of this needs to be recovered — the real, permanent versions of the useful scripts are in jorgecuadros-platform/migration/, and the Parquet output can be regenerated in under 2 minutes by rerunning load_staging.py (see §6).

3. Key decisions made this session

Decision	Answer	Why
Stack	Next.js (React/TS) + NestJS (TS) + Prisma	Type safety, parameterized queries by default (kills the SQL-injection class of bug the old app had everywhere)
Database engine	MySQL (not Postgres — reversed mid-session)	The customer-facing portal's PHP code (mysqli) isn't being rewritten, and shared hosting can't run Postgres. Using MySQL everywhere avoids a cross-engine sync layer.
Repo	New repo, not built on jorgecuadros-intra-webapp	That repo has SQL injection in every query (src/core/db.php string-concatenates $_POST) and plaintext password comparison (src/core/auth.php) — not worth patching
Historical data	Migrate everything, no cutoff	Source tables are small (largest ~16k rows); completeness is cheap
Customer portal	Stays as-is, not rebuilt	Explicit user decision
Infrastructure	Internal server (private) + new VPS (Tailscale-linked) running a MySQL replica	Internal server has no inbound internet exposure; shared hosting can't be a replication target; a VPS you control can be both a real replication node and internet-reachable for the portal
Auth mechanism	Session-based (Passport + express-session), Argon2 password hashing	Implemented already — see §5

4. What was actually built and verified this session

Everything below was run and confirmed working, not just written:

4.1 Repo scaffold

• jorgecuadros-platform/ — npm workspaces (apps/*, packages/*)
• apps/api — NestJS. Builds clean under strict TypeScript (npx nest build in apps/api, zero errors).
  • src/main.ts — global ValidationPipe (whitelist + forbid unknown fields — structural replacement for the old app's total lack of input validation), session middleware, CORS.
  • src/auth/ — AuthService.validateUser() verifies passwords with argon2.verify() (replaces passwd = '$password' plaintext SQL comparison in the old app), LocalStrategy, SessionSerializer, AuthenticatedGuard (replaces manually-called validate_session()), AuthController (/auth/login, /auth/me, /auth/logout).
  • src/users/, src/prisma/ (global PrismaModule/PrismaService).
• apps/web — Next.js App Router shell. Builds clean (npx next build).
• packages/database — Prisma schema + generated client.

4.2 Prisma schema (packages/database/prisma/schema.prisma)

Full target data model implementing the plan's design, validated and generating a working client against MySQL:

• Identity: Customer, CustomerLegacyRef (generalizes the old customer_mapping bridge table — one row per legacy record folded into a unified customer, with provenance)
• Insurance: InsuranceProvider, PolicyType, Policy (consolidates INCENDIO/MULT/M EMPR/6 auto-table variants/LICENCIAS into one table with a type discriminator), PolicyPaymentInstallment (unpivots the 4 hardcoded payment-installment columns found on every legacy policy table), Vehicle (unpivots MCA2's 3 hardcoded vehicle slots), InsuredDriver, PolicyBeneficiary, Claim, Adjuster, PolicyDocument
• Utilities: Property (shared with insurance — the actual unification point), PropertyService, ServiceDocument, TrustAccount
• Shared ledger: Transaction (unifies all the EFECTIVO*/billing-period snapshot tables), ExchangeRate, TypeTransaction
• Bank register (SCOTHIA): BankTransaction, BusinessLineCategory
• Admin: User (hashed passwords, roles), ActivityLog, EmailTemplate, EmailCampaign, EmailLog

Every model sourced from a legacy Access table carries legacySourceDb/legacySourceTable/legacyId provenance columns for traceability and idempotent re-runs. Long-text fields (notes, observations, description, etc.) are explicitly @db.Text — MySQL's default String is VARCHAR(191) and would silently truncate them otherwise (this was caught and fixed during the Postgres→MySQL swap).

Regenerate the client any time with:

cd jorgecuadros-platform
DATABASE_URL="mysql://user:pass@localhost:3306/placeholder" npx prisma generate --schema=packages/database/prisma/schema.prisma

(A real DATABASE_URL isn't needed for generate/validate, just a syntactically valid one — no live DB was available in this session, see §7.)

4.3 Docker Compose / Dockerfiles

• docker-compose.yml — mysql:8.4 + api + web services, healthchecked.
• docker/api.Dockerfile, docker/web.Dockerfile — multi-stage builds.
• .env.example — DATABASE_URL, SESSION_SECRET, WEB_ORIGIN, NEXT_PUBLIC_API_ORIGIN.
• Not run — this environment has no Docker installed (docker --version fails). Untested beyond visual review; verify on a machine with Docker before relying on it.

4.4 Migration pipeline (migration/) — run end-to-end against real data

• config.py — manifest of the 3 Access source files (paths + per-source exclude lists for confirmed-scratch tables, with reasoning in comments)
• extract.py — connects via pyodbc + the Windows Access ODBC driver (Microsoft Access Driver (*.mdb, *.accdb), 64-bit — confirmed installed on this machine). Two real bugs found and fixed here:
  1. cursor.columns() hits a UTF-16 decode bug on some tables (confirmed: PROPANO, FALTANTES AGUA, TIT) — fixed by reading column names from cursor.description after a SELECT * instead.
  2. cursor.fetchall() aborts an entire table on the first corrupted row — confirmed on MULT (Jet/ACE-level "Record is deleted" error, HY109). Fixed by fetching row-by-row in a try/except, skipping and logging just the bad row. Recovered 763 of 764 rows in MULT. Verified the cursor advances correctly and doesn't infinite-loop on the bad row before trusting this for a full run.
• load_staging.py — dumps every non-excluded table into either Parquet (--output-dir, no DB needed) or MySQL (--database-url, one database per source: stg_utilities/stg_seguros/stg_scothia). Actually run in Parquet mode against all three Access files: 82 tables staged successfully, zero unhandled errors, MULT's corrupted row correctly skipped and logged.
• requirements.txt — pyodbc, pandas, pyarrow, sqlalchemy, pymysql.

To rerun (from jorgecuadros-platform/migration, after pip install -r requirements.txt):

python load_staging.py --output-dir ./output              # Parquet, no DB needed — always works
python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/   # loads into real MySQL once available

5. Infrastructure & sync architecture (designed, not yet built)

• Internal server — on-prem, private IP 192.168.1.xx, no inbound internet exposure. Runs the platform + canonical MySQL (source of truth).
• New VPS (Hetzner or DigitalOcean — not yet provisioned) — becomes what mysql.freakma.com resolves to. Runs a MySQL replica. Because it's infrastructure the user controls (unlike shared hosting), it can be a real MySQL replication target.
• Tailscale — mesh VPN joining the internal server and the VPS, so they can reach each other without opening inbound ports anywhere.
• Internal → VPS: one-way native MySQL replication (binlog/GTID) for the subset of data the portal needs to read (statements, balances, customer profile). Internal-only tables (staff notes, adjuster info, activity logs) are deliberately excluded from what replicates.
• VPS → Internal: the portal also writes (payment submissions, propane orders) — one-way replication can't carry that back, and multi-master MySQL replication was deliberately ruled out as too fragile for this system's size. Instead: unreplicated "inbox" tables on the VPS (payment_submissions, propane_order_requests) that the portal writes to directly, polled every 1–5 min by a worker on the internal server (over Tailscale) that turns new rows into real records.

6. Open items — need input/access before certain next steps can proceed

1. utility_dbo schema is still unknown. This is the customer portal's actual live data database (referenced in the old app via getExternalDBConnection() at mysql.freakma.com, used there only for email_alert_log, but the mobile app almost certainly reads/writes statements, payments, and propane orders directly against it). The two SQL dumps provided (jorgecuadros_app.sql, jorgecuadros_app (1).sql) turned out to be a separate analytics/tracking database, not this one. When asked, the user pointed back to SEGUROS 16_be.mdb — worth revisiting; it's possible the intent was "the source data ultimately comes from the Access files" rather than "here is utility_dbo's schema." Without the real utility_dbo schema, the sync worker's exact target tables/columns for the inbox pattern can't be finalized. Ask for an export or read-only credentials, same as how the Access files and tracking-DB dumps were provided.
2. VPS not yet provisioned — provider (Hetzner vs DigitalOcean), size, and Tailscale/MySQL setup on it are pending. Ops task, not something done in this session.
3. CI/hosting — keep Jenkins + git.freakma.com, or move to GitHub Actions if the new repo lives elsewhere?
4. i18n — nearly all source data and, presumably, staff usage is in Spanish; old app's code/UI was English-labeled. Confirm Spanish-first / bilingual / English before frontend work goes deep.
5. Reconciliation pass not started (migration plan step 2) — the near-duplicate snapshot tables (EFECTIVO/EFECTIVO FM3/EFECTIVO_BACKUP, FEE ANUAL/datos2/fee15/billing, DATGRAL vs COBRO3) need a diff/dedupe pass against the staged data before any transform-and-load into the real schema. This is explicitly not a "guess the rule up front" thing — the plan calls for writing SQL against the staged Parquet/MySQL data to see what's actually duplicate vs. distinct.

7. Environment notes (this machine, in case it matters for reproducing)

• Windows, PowerShell primary, Git Bash also available.
• Node v25.2.0, npm 11.6.2 — no pnpm, yarn is present but npm workspaces were used throughout.
• Python 3.9 (C:\Python39), pip install'd this session: pyodbc, pandas, pyarrow, psycopg2 (installed but no longer used after the MySQL switch — harmless to leave or remove).
• MS Access ODBC driver confirmed installed: 64-bit Microsoft Access Driver (*.mdb, *.accdb) (matches 64-bit Python — this pairing matters, a 32/64-bit mismatch would break pyodbc.connect).
• No Docker, no local MySQL, no local Postgres on this machine — docker-compose.yml and the MySQL-target mode of load_staging.py are written but unexecuted here. Test both on whatever machine ends up running this for real.
• Old repo's dbConnection.php has a hardcoded plaintext MySQL password for the external DB connection, committed to git history. Not carried forward into the new platform, but worth rotating that credential regardless since it's already exposed in the old repo's history.

8. Plan locked — next actions

All four previously-open decisions are now locked (2026-07-22). See PLAN.md → "Decisions (locked)" for the authoritative record:

• Extraction toolchain (macOS): mdbtools (installed + verified against the real files).
• i18n: Spanish-first.
• CI/CD: Gitea Actions on git.mancinas.io → registry → Portainer.
• utility_dbo: resolved — full dump (utility_dbo.sql, 1.3 GB, 55 tables) and the portal codebase (~/PhpstormProjects/my-jorgecuadros-web) are both on disk.

Environment: moved Windows → macOS. Sources now at ~/Downloads/JorgeCuadros-Legacy/ (all four files). This machine has Docker, MySQL/MariaDB client, Node 22, Python 3.14, Homebrew. No Access ODBC driver, node_modules not installed, staging Parquet not present.

Execution queue (in order):

1. Port the extraction layer to mdbtools. Rewrite migration/extract.py to shell out to mdb-tables/mdb-export instead of pyodbc. Keep the same public interface (connect/list_tables/read_table) so load_staging.py and config.py are unchanged beyond the already-fixed SOURCE_ROOT. Carry over the two hard-won fixes conceptually: accented-column tables (mdbtools reads PROPANO cleanly — verified) and the corrupted MULT row (mdb-export's -b / error handling; confirm the bad row is skipped, not fatal).
2. Re-run staging (python load_staging.py --output-dir ./output) to regenerate the staged data on this machine, then load into a local MySQL (docker compose up mysql) for SQL reconciliation.
3. Reconciliation pass (plan step 2) — DONE (migration/reconcile.py → RECONCILIATION.md). Overturned all three "duplicate" assumptions: EFECTIVO/BACKUP are near-disjoint ledgers (folio collides; migrate both), the billing tables are disjoint period runs (union all, no de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The decided union/de-dup rules are in PLAN.md migration step 2.
4. Transform + load (plan step 3) — IN PROGRESS.
   • Customers — DONE (migration/transform_customers.py). Loaded into the dev DB: 1682 customers (1172 utilities master + 510 insurance-only), 2242 legacy refs (all traceable), 560 insurance rows linked via num_util with 0 broken refs, 542 merged identities spanning both business lines; linked customers enriched with insurance-only ID-doc fields. COBRO3 excluded. Re-runnable (truncate+rebuild); needs staged Parquet present (load_staging.py --output-dir ./output first).
   • Properties — DONE (migration/transform_properties.py): 1519 properties (0 orphans), 3486 services, 553 trust accounts from DATMEX/PROFILE; PROFILE flags matched 1519/1519.
   • Policies — DONE (migration/transform_policies.py): config-driven consolidation of all insurance lines into policies (2378: AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5; 10 skipped for unresolved customer, 0 orphans) + 4678 installments, 1110 vehicles, 513 insured_drivers, 126 beneficiaries, 1 claim, 5 policy_types, 15 insurance_providers, 17 adjusters. Unmodeled coverage columns preserved verbatim in coveragesJson. Verified a unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT policies — the cross-line customer view works at the data layer.
   • NEXT: the shared ledger union per the reconciliation rules (both EFECTIVO tables, all three billing tables, provenance-keyed; normalize monedas variants), then SCOTHIA bank register, then document extraction (step 4, LONGBINARY blobs -> object storage).
   • Migration is env-parameterized + reproducible: run_all.py --env <env> runs customers -> properties -> policies in order; add --stage to re-extract from Access first. 5b. Infra done: dev MySQL deployed to the cubex Swarm via Portainer API as stack jorgecuadros-dev-db (MySQL 8.4, 192.168.4.212:3307, node cubex labeled jorgecuadros_db=true); Prisma schema pushed (26 tables). Stack file: deploy/jorgecuadros-db.stack.yml (same file deploys prod as jorgecuadros-prod-db :3306). Creds in gitignored deploy/.env.dev. NOTE: machine npm is pnpm-aliased and pnpm ignores the workspaces field — full workspace install needs pnpm-workspace.yaml or real npm; for now Prisma CLI is run via npx prisma@5.
5. Customer module in apps/api/apps/web (list/search/detail) — first real feature, Spanish-first UI. Run npm install at repo root first (node_modules absent here).
6. Sync design finalization — now unblocked: map the internal→VPS replicated subset and the VPS→internal inbox tables against the real utility_dbo schema and the portal's read/write points in my-jorgecuadros-web (peticion_gas, PayPal payments, notifications_settings).

Only genuinely-pending item is VPS provisioning (ops task — provider/size/Tailscale+MySQL).