jorgecuadros-platform/PLAN.md

Unified Customer / Insurance / Utilities Platform — Migration & Rebuild Plan

Context

Jorge Cuadros & Assoc. runs two lines of business — property/utility management (UTILITIES.accdb) and insurance brokerage (SEGUROS 16.mdb + its linked backend SEGUROS 16_be.mdb) — out of separate, decades-old MS Access databases, plus a third file (SCOTHIA.mdb) that's the office's own Scotiabank checking-account register ("chequera"). The same people are customers of both business lines, but today there's no shared customer record: a person's utility account and their insurance policies live in unrelated systems with independent, inconsistent copies of their name/address/contact info. The bank register is a fourth, disconnected source of truth for the money actually moving through the office's own account.

An earlier attempt to modernize this (jorgecuadros-intra-webapp, PHP) got partway there — it correctly recognized that customers should be unified via a bridge table and began normalizing the flat Access tables into real relational tables (properties, policies, home_policies). But every query in its data layer builds SQL by string-concatenating $_POST directly (src/core/db.php), so it's SQL-injectable end to end, and src/core/auth.php compares passwords in plaintext with no hashing. Per your decision, this isn't worth patching — the app layer will be rebuilt from scratch, in a different stack, in a new repo. The Access data and the schema ideas from the old MySQL dump remain the reference for what the business actually needs.

Goal: one system with a single customer record, from which staff can see and manage that customer's utility services and insurance policies and shared billing/transaction history, replacing both Access files.

Confirmed decisions:

• Stack: Next.js (React + TypeScript) frontend, TypeScript backend (NestJS), MySQL, Prisma ORM.
• New repo, not built on top of jorgecuadros-intra-webapp (that repo is reference-only for business logic/field mappings).
• Full historical data migrates — no cutoff. Source tables are small (largest is ~16k rows), so completeness costs little.

Database engine — revised from PostgreSQL to MySQL. The internal platform doesn't stand alone: a separate, pre-existing customer-facing PHP/MySQL portal (mobile-app-backed — see Infrastructure & Sync below) needs to keep reading live account data from it, and that portal's PHP code uses mysqli and is staying as-is (not being rewritten). Running the internal platform on Postgres would mean maintaining a cross-engine sync into MySQL just to feed the portal; using MySQL everywhere removes that translation layer entirely. Prisma supports MySQL natively, so this only changes the datasource provider in the schema — NestJS/Next.js/Prisma all stay.

Infrastructure & sync architecture

The internal platform and the customer-facing portal are two genuinely separate deployments that need to stay in sync, not one app:

• Internal server — hosted on-prem, private IP (192.168.1.xx), not exposed to the internet. Runs jorgecuadros-platform (this rebuild) and the canonical MySQL database — source of truth for everything staff manage (customers, policies, properties, the full ledger).
• Customer-facing portal — an existing PHP app (with a companion mobile app — its usage-tracking schema, jorgecuadros_app at mysql.freakma.com, logs sections like LOGIN, STATEMENT_ALL_NOW, MAKE_PAYMENT, ORDER_PROPANE, and a devices table of push-notification tokens) running on shared hosting. Out of scope to rebuild — stays exactly as it is. It reads/writes a separate live database, utility_dbo at mysql.freakma.com, which the old internal app already connects to externally (src/core/dbConnection.php: getExternalDBConnection()) — though only for email_alert_log in that codebase; the mobile app almost certainly talks to utility_dbo directly for statements/payments/propane orders. Open item: no schema dump of utility_dbo itself exists yet — only this reference to it — so the sync worker's exact target tables/columns can't be finalized until that's available (export or read-only credentials), the same way the Access files and the jorgecuadros_app dumps were provided.
• New VPS (Hetzner or DigitalOcean, to be provisioned) — runs a MySQL instance and becomes what mysql.freakma.com resolves to. Because it's infrastructure you control (unlike the shared-hosting account), it can run real native MySQL replication — the shared-hosting limitation only ever applied to shared hosting itself, not to a VPS. The shared-hosting portal's PHP code needs no changes beyond the DNS target it already points at.
• Internal server ↔ VPS network path: Tailscale (WireGuard mesh) — both machines join the same tailnet, giving the internal server a way to reach the VPS (and vice versa) without opening any inbound port on either the internal LAN or needing the internal server to be internet-facing.

Sync mechanism:

• Internal → VPS (statements, balances, customer profile changes): one-way native MySQL replication (binlog/GTID-based) from the internal MySQL (source) to the VPS MySQL (replica), over the Tailscale link. Standard, well-supported, and only carries the subset of tables/columns the portal actually needs to read — internal-only data (staff notes, adjuster info, activity logs) should live in tables intentionally excluded from what's replicated, so the more internet-exposed side never receives more than the portal requires.
• VPS → Internal (payment submissions, propane order requests): one-way replication can't carry writes back, and multi-master MySQL replication is fragile enough to avoid for a system this size. Instead: a small set of unreplicated inbox tables on the VPS (payment_submissions, propane_order_requests, matching whatever utility_dbo actually uses once its schema is available) that the portal writes to directly, polled every 1–5 minutes by a worker on the internal server (over Tailscale) that turns new rows into real Transaction/propane-request records and marks them processed.

Source data inventory

Extracted live via pyodbc + the Windows Access ODBC driver (dump_schema.py, tables/columns/row counts/PKs for all four files). Access has no real primary or foreign keys defined anywhere — every relationship below is inferred from column-name conventions (NUM id, NUMid, NUMER ID, IDISEG, IDIUT), not enforced constraints, so the migration has to independently validate every join.

UTILITIES.accdb (52 tables, ~538MB, dominated by embedded document blobs):

• DATMEX (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 LONGBINARY columns holding scanned utility bills/IDs.
• DATGRAL / COBRO3 (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). COBRO3 looks like a filtered snapshot of DATGRAL Reconciliation (step 2) corrected this: COBRO3 is a charge batch (constant fee 75), not a customer snapshot — DATGRAL is the sole customer master.
• PROFILE (1,520) — per-property service enrollment flags, joins 1:1 with DATMEX by NUMERID.
• EFECTIVO (13,697), EFECTIVO FM3 (627), EFECTIVO_BACKUP (12,387), CHEQUE FM3 (157) — cash/check transaction ledgers. apparent year/program snapshots rather than distinct data Reconciliation (step 2) corrected this: EFECTIVO and EFECTIVO_BACKUP are near-disjoint ledgers (folio collides; only 2 business-key matches) — migrate both, no folio de-dup. FM3 tables are a separate fee stream.
• FEE ANUAL (1,082), datos2 (16,000), fee15 (1,030), billing (0) — recurring billing/fee transaction logs. near-duplicate shapes ... needs de-duplication Reconciliation (step 2) corrected this: disjoint billing runs from different periods (2017 / 2018 / 2025–26), zero real-identity overlap — straight union, no de-dup.
• TRUSTVENCE (549) — bank trust account fee due-dates.
• TIPO HIST (2,301) — exchange-rate history (date/hour/rate), referenced by the MONEDAS (currency) field used throughout.
• TYPE OF TRX (79) — ES/EN transaction-type lookup (already mirrored as type_transactions in the old MySQL schema — reuse that mapping).
• Scratch/working tables to exclude from migration: BANCO EDITOR, Errores de pegado, TABLE1, PARA BILLING*, FALTANTES *, TELEFONOS FECHAS, TIT, BORRA-equivalents. A few tables (PROPANO, datosfreak, datosfreak2, LUZ TODOS, etc.) have column names with characters pyodbc's UTF-16 path can't decode — need re-extraction with a Latin-1/CP1252 fallback before they can even be classified as real data vs. scratch.

SEGUROS 16_be.mdb (64 tables, ~882MB; SEGUROS 16.mdb is an empty Access "frontend" shell — all real data lives in _be):

• DATGRAL (1,070 rows) — customer master, same shape as the utilities DATGRAL, and critically has a NUM UTIL column — a literal cross-reference to the utilities customer's NUM id. This is the join key for building the unified customer table; it's also what the old app's customer_mapping table was clearly reverse-engineered from.
• One flat table per insurance line of business, all sharing the same repeated-column pattern (policy #, coverage dates, 4 hardcoded payment installments each with its own date/amount/check#/currency, premium breakdown, liquidation status, address, observations, embedded document blobs):
  • INCENDIO (fire), MULT (multi-risk/home — the most complete, matches old app's home_policies), M EMPR (commercial property)
  • Auto: TABLA AUTOS, TABLA AUTOS AMPL, TABLA AUTOS AMPL R, TABLA AUTOS LIMIT, TABLA AUTOS LIMIT R, TABLA AUTOS RC R, MCA2 — seven variants of essentially one "auto policy" concept, differentiated by coverage tier/program. MCA2 also hardcodes 3 driver+vehicle slots as repeated columns (MARCA1/2/3, PLACA1/2/3, NAME1/2/3, LIC1/2/3...) that need to unpivot into child rows.
  • LICENCIAS (driver's-license insurance, MX-specific product) — also hardcodes 3 named insureds per policy.
• BENEF (768) — policy beneficiaries, already a clean child table (policy#, name, address, phone, email).
• DATOS (2 rows now, but structurally real) — claims/siniestros: claim date, description, adjuster, settlement amounts, checks, documents.
• AJUSTADORES / AJUSTADORESATLAS — adjuster contact lists.
• EDOSEG — per-policy premium/account statement summary.
• EFECTIVO (295) — cash ledger, same shape as the utilities one.
• UTILSEG (1,582) — appears to be the utilities↔insurance customer cross-reference table.
• TABLA LIQUIDA * / TABLE DATOS LIQUIDA * — settlement/liquidation batches per policy line (matches liquidation_number already in the old MySQL policies table).
• Exclude from data migration: the * MENS / *MENSAJE tables (AMPL MENS, IN MENS, LIC MENS, MCA2 MENS, ME MENS, MF MENS, RC MENSAJE, RC R MENS) are mail-merge document templates (letters, certificates) stored as blobs, not customer data — these get reimplemented as PDF templates in the new app, not migrated as rows. TODOSJC, TODOS, vigenta casa y auto unicos look like materialized Access query results (saved reports), not source-of-truth data — exclude, and if the report itself is still needed, rebuild it as a real query against the new schema.

SCOTHIA.mdb (7 tables, ~3MB — the office's own Scotiabank checking-account register, "chequera"). Much simpler than the other two files: this is the company's operating bank account, not customer-facing data.

• DATOS E (15,406 rows) — expenses/outgoing (EGRESO): date, transaction type, check/reference NUM, CONCEPTO (payee/description), amount, OPERADO (cleared flag), notes, and a spelled-out amount-in-words field (CANTIDAD EN LETRA, standard Mexican check-writing convention).
• DATOS I (6,948 rows) — income/incoming (INGRESO): same shape, plus a TRANSFERIDO (transferred) flag instead of the amount-in-words field.
• TABLA RAMODOS (66 rows) — a category/business-line lookup ("ramo" = line of business in Mexican insurance terminology) — this is almost certainly what CONCEPTO entries get classified against, i.e. the link between a bank transaction and which part of the business (insurance line, utility service, trust, etc.) it belongs to.
• ban (1 row) — just holds the bank's name; a config singleton, not data.
• INFORME / INFORME BA (0 rows each) and FECHAIF (1 row) — Access report/query scratch tables (date-range parameters and a report shell), same pattern as the scratch tables in the other two files — exclude from migration.

Target architecture

• Frontend: Next.js (App Router) + TypeScript + React. Server components for data-heavy list/detail views (customers, policies, statements); client components for interactive forms.
• Backend: NestJS (TypeScript) REST API — modular by domain (customers, insurance, utilities, billing, auth, admin), matching the module boundaries below. Gives you DI, guards for auth/authorization, and a validation pipeline (class-validator) for free, which directly replaces the old app's biggest weakness (no input validation, no parameterization).
• Database: MySQL, accessed via Prisma (schema-as-code, migrations, generates a typed client — eliminates the raw-SQL-injection class of bug entirely since Prisma parameterizes everything). See Infrastructure & Sync above for why MySQL rather than Postgres.
• Auth: NestJS + Passport, sessions or JWT (pick one during build), bcrypt/argon2 password hashing, role-based guards replacing the old $_SESSION['level']/['role'] checks.
• File/document storage: object storage (S3-compatible) for the scanned documents currently trapped as Access LONGBINARY blobs — extract once during migration, store as files, keep only the pointer + metadata in MySQL. The old repo already anticipated this (src/objects/s3.php exists but appears unused) — same idea, implemented for real this time.
• Deployment: keep Docker as the packaging mechanism (already proven for this project) with a fresh Dockerfile/docker-compose.yml for the Node services + MySQL, running on the internal server described above; CI can stay on Jenkins if that's still the team's pipeline, or move to GitHub Actions if the new repo lives somewhere other than git.freakma.com — flagged as an open decision below since it wasn't part of the stack questions asked.

Target data model (by domain)

All tables get a surrogate id (uuid or serial) plus, where the row came from a legacy table, provenance columns (legacy_source_db, legacy_source_table, legacy_id) so every migrated row can be traced back to its Access original for spot-checking and reconciliation — and so the ETL can be re-run idempotently (upsert on provenance key) as migration bugs get found and fixed.

Identity (the actual point of this project):

• customers — one row per real person/entity, merged from utilities DATGRAL/COBRO3 and insurance DATGRAL, matched via the NUM UTIL cross-reference plus name/address fuzzy-matching for anyone missing that link. Holds name, addresses (MX + US), phones, email, ID document info, status, currency preference.
• customer_legacy_refs — generalizes the old customer_mapping table: (customer_id, source_system, source_table, legacy_numeric_id), one row per legacy record folded into this customer. This is what makes "unified customer base" actually queryable and keeps the merge auditable.

Insurance domain:

• insurance_providers, policy_types (carry over from old schema, already reasonable)
• policies — generic header (policy #, type, provider, customer, dates, premiums, agent, liquidation status), consolidating INCENDIO/MULT/M EMPR/all six auto-table variants/LICENCIAS into one table with a policy_type discriminator, instead of one Access table per line of business.
• policy_payment_installments — unpivots the 4 hardcoded payment-installment columns (1ER PAGO/FECHA PAGO/NO CHEQUE, ... 2, ... 3, ... 4) into rows: due sequence, amount, currency, paid date, check/reference number.
• vehicles — unpivots MCA2's 3 hardcoded vehicle slots (and the single-vehicle auto tables) into one row per vehicle, FK'd to policy + customer.
• insured_drivers — same unpivot for the repeated named-insured/license columns in MCA2/LICENCIAS.
• properties (shared with utilities domain — see below), policy_beneficiaries (from BENEF, already clean), claims (from DATOS), adjusters (from AJUSTADORES*), policy_documents (extracted blobs, typed: ID, prior policy copy, damage photo, etc.).

Utilities domain:

• properties — one row per property (from DATMEX), FK'd to customers, shared with insurance so a property can carry both a home-insurance policy and utility service enrollments — this is the second half of "unified."
• property_services — one row per enrolled service per property (water/electric/gas/cable/trust/property-tax/federal-zone), unpivoting DATMEX's inline service columns and PROFILE's enrollment flags into real rows with account #, route, meter #, due day.
• service_documents (extracted blobs), trust_accounts (from TRUSTVENCE).

Shared financial ledger (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):

• transactions — unifies utilities' EFECTIVO/EFECTIVO FM3/EFECTIVO_BACKUP/FEE ANUAL/datos2/fee15/billing/CHEQUE FM3/IVA 2015 and insurance's EFECTIVO, tagged by domain (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive. amount is signed: negative = charge (cargo), positive = credit (abono), so SUM(amount) per customer per currency is the balance — negative means the customer owes the office. The two currencies are never summed together (see the billing module note in Build sequencing step 6).
• exchange_rates (from TIPO HIST), type_transactions (carry over ES/EN lookup as-is).
• bank_transactions — the company's own operating bank register, from SCOTHIA's DATOS E/DATOS I unified into one signed-amount table (income positive, expense negative) with a category FK to business_line_categories (from TABLA RAMODOS) and a cleared/operado flag. This is deliberately separate from customer-facing transactions — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the business_line_categories lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
• business_line_categories (from TABLA RAMODOS).

Admin/shared: users (hashed passwords, roles), activity_logs, email_templates/email_campaigns/email_log (carry over the old schema's intent, rebuilt on the new stack).

Migration strategy

Given the amount of near-duplicate/overlapping data across snapshot tables (multiple EFECTIVO* variants, multiple year-stamped billing tables, COBRO3 vs DATGRAL), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.

1. Raw staging load: dump every non-scratch Access table 1:1 into a MySQL staging (per-source schema/database, e.g. stg_utilities/stg_seguros/stg_scothia) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as migration/load_staging.py in the new repo — see Status below. This is the audit trail — nothing is transformed yet. Extraction toolchain note: the original build used pyodbc + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (migration/extract.py) is being reworked to use mdbtools (mdb-tables/mdb-export, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. PROPANO), and per-table exports cleanly. mdbtools does not extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in migration/objects.json + docs/LEGACY_DATABASES_OBJECTS.md, so nothing is lost. The only piece needing extra handling under mdbtools is LONGBINARY blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
2. Reconciliation pass — DONE (migration/reconcile.py → migration/RECONCILIATION.md, run against the staged data). For each set of overlapping tables, it probes a deliberate business key (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. The union/de-dup rules below are decided by the data:

   • EFECTIVO vs EFECTIVO_BACKUP: EFECTIVO_BACKUP is a stale backup copy — de-dup it. (Corrected 2026-07-22; see the box below.) On the canonicalized business key (cl,fecha,monto,conepto), 12,386 of BACKUP's 12,387 rows already exist verbatim in EFECTIVO — same customer, same timestamp to the second, same amount, same concept text — leaving exactly 1 genuinely new row. folio is a per-table sequential number that collides (12,363 shared numbers, 12,204 of them on different payments), so it can never be the de-dup key. Rule: load EFECTIVO in full; from EFECTIVO_BACKUP load only business-key-new rows. EFECTIVO FM3/CHEQUE FM3 are a separate fee/tax/multa stream, migrated distinctly. (monedas needs currency normalization — PESOS/Pesos/DOLLARS variants.)

     Why this was wrong the first time. The original pass reported only 2 overlapping rows and concluded the two tables were "near-disjoint ledgers, migrate both". That verdict came from a bug in reconcile.py, which compared business-key columns as raw strings on the premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: mdb-export formats a numeric column from its Access column type, so the same amount is emitted as 5000 from one table and 27000.0000 from the other, and no two rows could ever match on monto. reconcile.py now canonicalizes numeric key columns before comparing. The bad rule had already been loaded: the ledger carried 45,861 rows with 12,386 duplicated payments, roughly doubling every customer's historical receipt total — which would have made every balance and statement in step 6 wrong. Re-running run_all.py brings the ledger to 33,475 rows. Groups 2 and 3 below were re-checked under the fix and their verdicts are unchanged.

   • datos2 vs FEE ANUAL vs fee15: not near-duplicate exports. They are disjoint billing runs from different periods (datos2 ≈2025–26, FEE ANUAL 2018-01-03, fee15 2017-01-10 — each period-table refer is a single constant); zero real-identity overlap. Rule: migrate all three, no de-dup; keep datos2.due_date (null for the others).

   • DATGRAL vs COBRO3: COBRO3 is not a filtered snapshot of the customer master — its fee is a constant 75 for all 181 rows (a saved charge worklist / "cobro" = collection), and every num_id already exists in DATGRAL. Rule: DATGRAL is the sole utilities customer master; COBRO3 contributes zero customers — model its 181 rows as charge transactions if worth keeping, else exclude.

3. Transform + load: SQL/TypeScript scripts (versioned in the new repo under migration/) that read staging, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing legacy_* provenance on every row.
4. Document extraction: separate one-off script pulls every LONGBINARY column out to files (named by provenance key), uploads to object storage, and inserts the corresponding *_documents metadata row.
5. Validation: row-count and spot-check reconciliation between staging and final tables (e.g., every legacy customer has exactly one customers row via customer_legacy_refs; sum of migrated transaction amounts per customer matches sum in staging).

Build sequencing

1. Repo scaffold (Next.js + NestJS + Prisma + MySQL, Docker Compose for local dev), CI pipeline, auth module with hashed passwords and role guards.

2. Prisma schema for the full data model above; run migration steps 1–2 (staging load + reconciliation reports) against real data early, since that's where the biggest unknowns are (do NUM UTIL and name-matching actually cover everyone? how bad is the snapshot-table duplication?).

3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.

4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.

5. Utilities module (properties, services, trust accounts) on top of the same customer records.

6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions) — DONE. apps/api/src/billing/ + web /estado-cuenta and /estado-cuenta/[id]. Two questions, two views: a per-customer balances worklist (who owes what) and a cross-customer movement browser (every charge and credit, filterable by line, concept, origin table and date range, with totals for the whole filtered set). The detail page is the actual statement: balance per currency, the same balance split by business line, charges broken out by concept, and the full movement list with a running balance. Design constraint that shapes the whole module: balances are reported per currency and never collapsed into one number. 912 of the 1,269 customers with a ledger move in both MXN and USD, the charge side is MXN-only while receipts arrive in both, and the legacy data never stored the exchange rate applied to a movement — so a single "total balance" would be a figure that never existed in the books.

7. Bank register module (bank_transactions/business_line_categories from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.

8. VPS provisioning + Tailscale + MySQL replication setup. utility_dbo's schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (my-jorgecuadros-web) that reads/writes it.

9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. The separate Phase B Access additive sync is implemented: migration/run_all.py --sync and the admin SYNC job upsert legacy-owned rows without truncating the database or touching manual rows. Portal write points confirmed present in utility_dbo: peticion_gas (propane requests), PayPal payment writes, notifications_settings, verification_codes — these define the VPS→internal inbox set.

10. Reports/email campaigns/admin — parity with old app's reports.php/emailCampaigns.php intent, rebuilt properly.

11. Receipt capture ("Editor") completion + three net-new ops features — NOT STARTED, spec written. Full design in docs/RECEIPT_CAPTURE_SPEC.md, from the 2026-07-25/26 meeting with Jorge:

    • Receipt capture module — DONE (2026-07-27). The legacy "Editor" replacement, built on the single-movement capture from step 6. Wires up the previously-unused Transaction.outstanding (NOPAGO): capture flag on POST /billing, ?outstanding= list filter, POST /billing/:id/resolve-outstanding (gated ledger:create, not ledger:void — resolving completes a capture), and exclusion from every balance aggregate exactly as the legacy SALDOS ULTIMO 0's HAVING NOPAGO = 0 did. Adds POST /billing/batch (one $transaction, check-level fields shared, per-line customer/amount) and GET /billing/by-check, plus the cheque-count report replacing REPORTE CHEQUE COUNT / REPORTE POR CHEQUE / EDITA CHEQUE ALF|COUNT|NUM — print/PDF/CSV/XLSX come free from the existing /reportes/:slug machinery. Web: /estado-cuenta/lote (the actual "Editor" screen, with live reconciliation against the physical check amount), plus an "Estado de pago" filter, a "sin fondos" row tag and a Resolver dialog on /estado-cuenta. No new abilities. Verified end-to-end against dev, API + browser. Two pre-existing bugs found and fixed while building it: (a) statement() filtered legacySourceTable: { notIn: [...] }, which compiles to SQL NOT IN — and NULL NOT IN (…) is NULL, so every app-captured movement was invisible on the customer statement (438 rows in the movement browser vs 392 on the statement) while still appearing everywhere else. This would have made the whole receipt-capture feature look broken to staff. Now NULL-safe. (b) The balances count query omitted the void filter its own page query applied, so the row count disagreed with the rows. OCR seam: BillingService.createBatch(dto, opts) is the single multi-row write path and carries three contract guarantees for the step-11 OCR module to post through — items[i] maps to lines[i] (so StatementDocument.postedTransactionId can be zipped back on), opts.refs[i] stamps captureRef with a duplicate-post guard that a voided row deliberately does not block, and opts.source is service-level only so an HTTP client cannot label hand-keyed rows as machine-captured. Backed by a new TransactionCaptureSource enum (MANUAL/BATCH/OCR) + captureRef, both nullable so the 40,136 migrated rows stay NULL rather than being mislabelled.
    • PDF/OCR auto-capture — DONE (2026-08-01). The ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff key in by hand, built in apps/api/src/statements/ and posting through §1.2's createBatch seam with source: "OCR" and a per-document captureRef. Web: /recibos + /recibos/:id. Abilities statement:ingest/statement:review (STAFF — the review step is what makes machine capture safe at that tier). OCR is self-hosted Tesseract behind a swappable OcrProvider interface; tesseract-ocr, tesseract-ocr-data-spa and poppler-utils were added to the API image. Every decision was driven by 10 real scans (46 pages). Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database 39/46 (85%) exact auto-match, 40/46 (87%) identified, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive bundled one customer per page. The three gaps are closed, and two of them were mis-stated in the spec. (a) TELEPHONE now exists and is backfilled from Property.phone1 only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) Clave catastral ≠ predial: DATMEX.clave (934 rows, KA903009) is what CESPT and predial bills actually print, while predial — what PROPERTY_TAX.accountNumber holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on Property.cadastralKey as the matcher's secondary key and predial is left untouched. (c) Gas was not a dead end: 160 of the 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), all recovered into GAS.meterNumber. Matching is scoped per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off NO. DE SERVICIO, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost.
    • Policy OCR capture — DONE (2026-08-01), unplanned — it came out of building the bullet above. Full write-up in docs/POLICY_OCR.md. Once the receipt pipeline existed it was obvious the same render→OCR→parse→match→review shape fits the other stack of paper this office keys in by hand: the carrier policy PDFs behind every Policy row. Built in apps/api/src/policy-ocr/ with a GMX parser, policy_ocr_batches/policy_ocr_documents, and abilities policy:ingest/policy:ocr-review (STAFF, same trust tier and same reason). Web: /polizas/captura is the "automática" tab of the policy-creation screen (/polizas/nuevo is the manual one, both render PolicyCaptura.tsx) with the review queue at /polizas/captura/[id]. The OcrProvider seam was extracted out of StatementsModule into its own OcrModule to make this possible — that was blocking, not cosmetic; StatementsModule now imports it and binds nothing. The statement pipeline's core assumption inverts here. Utility statements arrive bundled one customer per page, so there a page is a document; a GMX certificate is one policy across two pages (header on 1, coverage table on 2), so the pipeline concatenates the pages and runs the parser and matcher once per file. PolicyOcrDocument.pageNumber is therefore the file ordinal in the batch, and storageKey points at the source PDF (the review screen embeds the exact artifact the office received) rather than at a page image. Matching is on Policy.policyNumber alone and never the printed insured name — the same registrant-vs-owner drift that rules names out on the utility side. Zero hits means a new policy and confirm creates it; more than one is surfaced, never auto-picked. The GMX certificate carries no premium at all — the figure lives on a separate recibo PDF — so the premium fields stay null with a note saying why, confirm never overwrites an existing premium with null, and the optional ledger write is gated on staff ticking postPremium and a premium actually parsing. 8/8 parser tests against one real document (HC_Folio_000767_Traduccion.pdf). GMX is the only carrier implemented; the dispatcher is a pattern table, so a second one is a parser function and two entries.
    • Multi-bank chequera — DONE (2026-07-27). Bank/BankAccount models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. bank_transactions gained a required bankAccountId (plus an (bankAccountId, transactionDate) index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by migration/backfill_bank_accounts.py — a standalone step because prisma db push cannot add a required column to a populated table. It is idempotent and now runs inside run_all.py (both normal and --sync) ahead of transform_bank.py, which fails fast if the account is missing. Every read path in bank.service.ts is account-scoped, including facets() (which had no filter at all) and both raw-SQL rollups in summary(). API: ?bankAccountId= is required on list/stats/facets/summary — not optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new bank/accounts + bank/banks sub-resource under a MANAGER bank:manage-accounts ability. Web: /banco gained an account picker (remembered per browser) and reads every figure in the selected account's currency, /banco/cuentas manages banks and accounts, and /inicio's chequera card names the account it is showing instead of implying one register. An account's currency is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged.
    • Customer-number recycling — promotes the legacy NUM id (currently only inside customer_legacy_refs) into a first-class, reusable Customer.customerNumber, automates finding candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic rownum_N/insrow_N placeholders in transform_customers.py, not real NUM ids).

    Several open questions block parts of this (OCR provider/budget, the Seguros bank's identity, the clave-catastral-vs-predial mismatch, exact recycling triggers, and whether "recycling" should ever mean true data purge vs. archive-and-reuse-the-number) — see the spec's collected open-questions section.

12. Insurance features — one of four built, rest spec'd. Full design in docs/INSURANCE_FEATURES_SPEC.md, the insurance half of the same 2026-07-25/26 meeting with Jorge that produced step 11:

    • Renewal notification emails — DONE (2026-08-01, extended 08-02). A sweep that mails the customer 30 days before expiry, 15 days before, and 7 days after, mapping onto RenewalNotice.generation 1/2/3 with no schema change. Sending is Amazon SES (@aws-sdk/client-sesv2, mirroring StorageService's optional-client/degrade-don't-crash pattern). The letter body is the existing aviso-renovacion report; @@unique([policyId, generation]) is already-in-place idempotency, so a re-run cannot double-send. Volume ≈260 mails/month, and 815 of the 893 policyholders (91%) have an email. Three things came out differently from the spec. (a) The manual mark-as-sent mutation was dropped on purpose — a button that marks a notice sent without sending anything lets the list claim a customer was told when they were not. POST /renewals/send replaced it: sending from the list is the marking, and the report's enviadas total becomes real the same way. (b) The send history is not renewal-specific — every attempt, including the failures and no-email skips a RenewalNotice row cannot represent, also writes email_notification_log as RENEWAL_NOTICE/POLICIES, shared with the four bulk jobs from docs/MASS_EMAIL_NOTIFICATIONS.md. RenewalNotice stays gating state; the log is history. (c) The @Cron("0 6 * * *") literal the spec called for lasted one day: both this sweep and the servicios jobs now take their cadence from NotificationScheduleService, stored in app_settings and reinstalled on save — no redeploy. Defaults preserve the old behaviour (pólizas 06:00 daily, servicios off). Both halves live on one screen. /notificaciones has Servicios and Pólizas tabs over the one log; /renovaciones is an alias onto the Pólizas tab. The send flags (debug in particular) sit in the shell above the tabs and govern both — before that there was no way to test a renewal aviso without mailing a real customer. A debug send diverts the mail, skips the RenewalNotice upsert and does not advance the sweep's lastSuccessfulAt; all three are needed together, or a test run silently narrows tomorrow's window and drops the letters it only pretended to send. Still blocked in production: the SES_* variables are wired through the deploy workflow but unset in Gitea, so a production send fails loudly rather than going out.
    • Liquidación batch workflow — ~70% already built (liquidated/liquidationNumber/liquidationDate are wired through DTOs, list filter, stats, form and detail page); only the batch print-and-mark step is missing, against a live pending set of 226 policies. Adds a ramo-parameterized pending report plus POST /policies/liquidate-batch under a new MANAGER policy:liquidate ability. Parameterized by ramo, not MULT-only — legacy TABLA LIQUIDA MF served MULT, INCENDIO and M EMPR alike.
    • Certificate / "Solicitud Atlas" — renders from the same format: "letter" machinery aviso-renovacion uses, then reaches customers as an extension of the step-8/9 replication (PDF generated here, pushed to MinIO, pointer replicated), not as a new public surface in this repo. Half-blocked: "Solicitud" has zero referent in the legacy system and normally means an application form, a different artifact from a certificate.
    • Carrier API integration (ANA Seguros + GMX) — shape only (CarrierConnector + an import-review queue rather than direct Policy writes, matching how step 11's OCR results are routed). Carrier research done 2026-07-27: the two carriers are one company — both belong to Grupo Valore (ANA writes autos, GMX writes daños, which is exactly this database's AUTO/LICENCIAS vs MULT/INCENDIO/M_EMPR split), so it is one commercial relationship, not two. ANA has a real live SOAP service (server.anaseguros.com.mx/ananetws/service.asmx, ASP.NET .asmx) with a published operation list — catalogs, CalculaValor/CalculaMSI, ValidaSerie, RecuperaCotizacion, Transaccion. GMX publishes no machine interface at all, only human agent portals. ⚠️ Critical mismatch: every ANA operation serves new-business quoting/issuance, not "list the policies where I am agent of record" — so if the ask is inbound portfolio sync, no evidence exists that either carrier sells it. Blocked on one phone call to Grupo Valore ((55) 5480-4000) for credentials + a direction answer, not on further research. ("GDMX" in the meeting notes was a typo for GMX — confirmed 2026-07-27.)

    Two pre-existing defects were found while verifying this spec and should be fixed as part of the liquidación work: (a) policy_types is missing its INCENDIO and M_EMPR rows and, because policies_policyTypeId_fkey is ON DELETE SET NULL, 5 m_empr policies silently lost their ramo — 4 of them are pending liquidación and are invisible to every ramo-filtered query; (b) the legacy settlement slots don't match what the target model assumed — MULT/INCENDIO carry two and M EMPR carries four, while Policy collapses to one, so ≤41 MULT second settlements were dropped in migration. Spec recommends moving settlement onto PolicyPaymentInstallment rather than adding a second slot.

    One long-standing open question is closed by this spec: DATGRAL.[NUM UTIL] is authoritative for Utilities↔Seguros reconciliation and UTILSEG must not be used — its numbers resolve to unrelated people under every reading tested (name match 58/1,024 vs. 298/563 for NUM UTIL), and where the two sources overlap they contradict each other on 170 of 218 shared ids. This matters to step 11's customer-number recycling, which touches the same identity space.

Status

Repo scaffolded at jorgecuadros-platform/: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to git.mancinas.io/rmancinas/jorgecuadros-platform (branch master, 3 commits, working tree clean). migration/load_staging.py (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a cursor.columns() UTF-16 decode bug on several tables (worked around by reading metadata from cursor.description instead) and one Jet/ACE-level corrupted record in MULT (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at jorgecuadros-platform/RESUME.md — read both together.

Environment moved Windows → macOS. Source Access files now live at ~/Downloads/JorgeCuadros-Legacy/ (all four: UTILITIES.accdb, SEGUROS 16.mdb, SEGUROS 16_be.mdb, SCOTHIA.mdb). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but no Access ODBC driver and no node_modules installed yet. Consequence: the pyodbc-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), node_modules needs npm install, and the staging output (gitignored Parquet) must be regenerated from scratch.

Portal live DB now in hand. utility_dbo.sql (1.3 GB, 55 tables) and the portal codebase my-jorgecuadros-web (PHP/mysqli, Gitea repo, themed classic/modern, ~397 PHP files, core in scripts/functions.php) are both on disk — resolving the long-standing "utility_dbo schema unknown" blocker. Sync-relevant tables identified: statements/money (utility_bills, accounting, email_alert_log), customer/property (home_owners, home_index, condominium, management, hoa_management, trust_assist), portal-facing policy views (fm2/fm3/fmt, full_coverage, mx_liability, usa_liability), and portal write points (peticion_gas, PayPal payments, notifications_settings, verification_codes). A second dump, jorgecuadros.sql (38 MB, 11 tables — pagos/pagosemail/PROPANO/TRUSTVENCE/etc.), appears to be an older/partial export, not the portal live DB.

Step 11 is now three-quarters built. Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. docs/RECEIPT_CAPTURE_SPEC.md carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown.

It also produced a feature nobody planned. The statement OCR pipeline generalised: the same render→OCR→parse→match→review shape reads carrier policy PDFs into Policy rows, which is docs/POLICY_OCR.md (built 2026-08-01, GMX only so far). It belongs to step 12's subject matter but to step 11's lineage, and it is in no spec — worth knowing before reading INSURANCE_FEATURES_SPEC.md, which does not mention it. It also partly overlaps what §4's carrier API was wanted for, and unlike that section it is not blocked on a phone call.

Step 12 is one-quarter built. docs/INSURANCE_FEATURES_SPEC.md covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. §1 renewal emails is done (2026-08-01/02) and carries a BUILT note recording the three places the build diverged from the spec; §2 liquidación is still the smallest remaining piece, since the per-policy fields are already wired end to end.

Notifications are one screen, not two features. The four legacy mass-email jobs (docs/MASS_EMAIL_NOTIFICATIONS.md) and the insurance renewal avisos both mean "tell a customer something by email", so they are tabs of /notificaciones over one email_notification_log, with one shared flags panel and one schedule editor. app_settings + SettingsService (db → env → default) is the operator-config seam they introduced: summary recipients and both sweep cadences live there, so changing any of them is a save, not a redeploy. Credentials stay in the environment.

Decisions (locked)

• Stack: Next.js + NestJS + Prisma + MySQL (locked earlier — see engine rationale above).
• Extraction toolchain (macOS): mdbtools, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in objects.json/docs/; only LONGBINARY blob extraction (step 4) needs extra handling under mdbtools.
• i18n: Spanish-first UI — matches the source data and staff usage.
• CI/CD: Gitea Actions on git.mancinas.io — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the portainer-gitea-deploy pattern already used on this LAN). Jenkins/git.freakma.com dropped.
• utility_dbo: resolved — full dump + portal code available (see Status).
• Phase B Access additive sync: implemented in migration/run_all.py --sync and the admin SYNC job. It preserves manual rows and stable legacy-owned primary keys; end-to-end database validation remains before production use.

Open items (ops, not design)

• VPS provisioning: provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
• Old external-DB credential (hardcoded plaintext MySQL password in the old repo's dbConnection.php, in git history) — rotate it regardless, since it's already exposed.

Open design questions (steps 11 & 12 — need Jorge before/while building)

Unlike the ops items above, these block design decisions, not just infrastructure. Full detail in each section of docs/RECEIPT_CAPTURE_SPEC.md (step 11) and docs/INSURANCE_FEATURES_SPEC.md (step 12):

Step 11 — utilities/ops side:

• OCR provider/budget — CLOSED: self-hosted Tesseract, chosen on measured accuracy against real scans, so there is no per-page cost to approve.
• Whether PROPERTY_TAX.accountNumber (from DATMEX.PREDIAL) is the same number as "Clave Catastral" (DATMEX.CLAVE) — CLOSED: they are different numbers. Answered from real CESPT bills plus the staged data; the clave is now migrated separately and predial was left alone.
• Whether the CFE figure to charge is the rounded headline/barcode amount ($268 — what is actually paid at the window) or the exact breakdown Total ($268.88). The parser takes the barcode amount; one confirmation from Jorge would settle it.
• The actual bank name/currency/details for the Seguros USD account, and whether any historical Seguros bank register exists to migrate. (Multi-bank support itself is built — this is now only the missing content: staff can open the account in /banco/cuentas the moment the answer arrives, and it starts empty unless a historical register turns up.)
• The exact "1 year inactivity" / "cancelled" triggers for customer-number recycling eligibility.
• Whether customer-number recycling should ever include true PII purge (matching the office's paper-world habit) or archive-and-reuse-the-number is sufficient — recommended default is archive-only, consistent with this project's existing never-hard-delete convention.

Step 12 — insurance side:

• Which SES region + verified sending identity/configuration set the renewal mail goes out under, and whether it reuses the existing IAM credentials or gets its own scoped ses:SendEmail user. (Provider and budget are not open — SES is settled.)
• What to do with the 78 policyholders who have no email on file: skip silently, or produce a print worklist? Recommended: the worklist, since aviso-renovacion already renders exactly those letters.
• Whether renewal notices go out in Spanish or English — Customer carries no language preference.
• What "garantías" refers to — it has zero referent in the legacy data, and it blocks the liquidación batch's exclusion filter.
• Whether policy settlement should move onto PolicyPaymentInstallment (recommended) or gain a second slot on Policy, and whether to backfill the ≤41 MULT second settlements lost in migration.
• Whether batch liquidación warrants a new MANAGER-level policy:liquidate ability (recommended) or should reuse the existing STAFF-level policy:update.
• What "Solicitud Atlas" actually is — an application form or a certificate. These are different artifacts with different data and timing; this blocks the whole certificate feature.
• Carrier integration direction — outbound quote/issue (which ANA's SOAP service supports today) or inbound sync of the office's existing book (which nothing found suggests either carrier offers)? This decides whether the feature is buildable at all. Bundle with the other three carrier questions into one call to Grupo Valore ((55) 5480-4000): WSDL + credentials for the ANA service, whether a cartera/portfolio download exists for an agent's own book, whether GMX daños has any machine interface, and whether one credential spans both carriers. ("GDMX" is resolved — it was a typo for GMX.)

Verification

• Migration: automated row-count/sum reconciliation between staging and final schema per table group (see step 5 above), run as part of the migration script, not a manual spot-check.
• App: standard NestJS unit/integration tests per module (auth guards, Prisma queries), Playwright/Cypress e2e for the core "look up a customer, see their unified policies + services + statement" flow — the thing the whole project exists to deliver.
• Sync: Phase B Access additive sync now has automated CLI/admin wiring, but must be verified against a disposable DB with stable-PK, manual-row, update, and source-delete cases. The separate VPS/portal sync still requires VPS provisioning and inbox-table implementation.
• Before cutover: run the new app against migrated data side-by-side with the live Access files for a period, comparing balances/statements for a sample of active customers to catch migration logic errors before the Access files are retired.