Lock plan: macOS move, mdbtools extraction, resolve open decisions

Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.

- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
  with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
  CI=Gitea Actions, utility_dbo=available); update Status for the macOS
  move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PLAN.md

@@ -107,7 +107,7 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
 
 Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
 
-1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script (`pyodbc` → `pandas`/SQLAlchemy, same connection approach already validated in this session), across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet.
+1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
 2. **Reconciliation pass**: for each set of overlapping tables (the `EFECTIVO` variants, the billing-period tables, `DATGRAL` vs `COBRO3`), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
 3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
 4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
@@ -122,20 +122,30 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
 5. Utilities module (properties, services, trust accounts) on top of the same customer records.
 6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
 7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
-8. VPS provisioning + Tailscale + MySQL replication setup, once `utility_dbo`'s schema is available to finalize exactly which tables/columns get replicated and what the inbox tables need to look like.
-9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8.
+8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
+9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.
 10. Reports/email campaigns/admin — parity with old app's `reports.php`/`emailCampaigns.php` intent, rebuilt properly.
 
-## Status (as of this session)
+## Status
 
-Repo scaffolded at `jorgecuadros-platform/` (sibling to the Access files): npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps build clean under strict TypeScript. `migration/load_staging.py` (step 1 of Migration strategy) has been run end-to-end against all four real Access files, staging 82 tables to Parquet; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap (Prisma schema, Docker Compose, `.env.example`, migration script's MySQL sink) has since been applied and re-verified (schema validates, client regenerates, API rebuilds clean against MySQL). A companion resume doc lives at `jorgecuadros-platform/RESUME.md` with exact file paths, environment notes, and a session-state summary — read both together.
+Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to `git.mancinas.io/rmancinas/jorgecuadros-platform` (branch `master`, 3 commits, working tree clean). `migration/load_staging.py` (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at `jorgecuadros-platform/RESUME.md` — read both together.
 
-## Open decisions (not yet locked down)
+**Environment moved Windows → macOS.** Source Access files now live at `~/Downloads/JorgeCuadros-Legacy/` (all four: `UTILITIES.accdb`, `SEGUROS 16.mdb`, `SEGUROS 16_be.mdb`, `SCOTHIA.mdb`). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but **no Access ODBC driver and no `node_modules` installed yet**. Consequence: the `pyodbc`-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), `node_modules` needs `npm install`, and the staging output (gitignored Parquet) must be regenerated from scratch.
 
-- **`utility_dbo` schema**: the customer-facing portal's live database — referenced from the old app (`getExternalDBConnection()`) but no dump/access provided yet. Blocks finalizing exactly which tables the sync replicates and what the inbox tables (`payment_submissions`, `propane_order_requests`) need to match.
-- **VPS provisioning**: which provider (Hetzner vs DigitalOcean), size, and who sets up Tailscale + MySQL on it — an ops task outside this session's ability to do directly.
-- **CI/hosting**: keep Jenkins + `git.freakma.com`, or move CI to GitHub Actions if the new repo goes elsewhere?
-- **i18n**: nearly all source data and, presumably, staff usage is in Spanish, while the old app's code/UI was English-labeled internally. Confirm whether the new UI should be Spanish-first, bilingual, or English (matching old app) before frontend work starts.
+**Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB.
+
+## Decisions (locked)
+
+- **Stack:** Next.js + NestJS + Prisma + **MySQL** (locked earlier — see engine rationale above).
+- **Extraction toolchain (macOS):** **mdbtools**, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in `objects.json`/`docs/`; only `LONGBINARY` blob extraction (step 4) needs extra handling under mdbtools.
+- **i18n:** **Spanish-first** UI — matches the source data and staff usage.
+- **CI/CD:** **Gitea Actions** on `git.mancinas.io` — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the `portainer-gitea-deploy` pattern already used on this LAN). Jenkins/`git.freakma.com` dropped.
+- **`utility_dbo`:** resolved — full dump + portal code available (see Status).
+
+## Open items (ops, not design)
+
+- **VPS provisioning:** provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
+- **Old external-DB credential** (hardcoded plaintext MySQL password in the old repo's `dbConnection.php`, in git history) — rotate it regardless, since it's already exposed.
 
 ## Verification
 

RESUME.md

@@ -141,9 +141,37 @@ python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/   # l
 - **No Docker, no local MySQL, no local Postgres** on this machine — `docker-compose.yml` and the MySQL-target mode of `load_staging.py` are written but unexecuted here. Test both on whatever machine ends up running this for real.
 - Old repo's `dbConnection.php` has a **hardcoded plaintext MySQL password** for the external DB connection, committed to git history. Not carried forward into the new platform, but worth rotating that credential regardless since it's already exposed in the old repo's history.
 
-## 8. Suggested next session starting point
+## 8. Plan locked — next actions
 
-1. Chase down open item #1 (`utility_dbo` schema) — it blocks finalizing the sync design concretely.
-2. Start the reconciliation pass (plan step 2, open item #5) against the staged Parquet data — rerun `load_staging.py --output-dir` first since the original output didn't persist.
-3. Once reconciliation rules are known, write the transform-and-load scripts (plan step 3) that populate the real Prisma-managed MySQL tables from staging, starting with the `Customer`/`CustomerLegacyRef` module since every other module depends on it.
-4. In parallel or after: build out the Customer module in `apps/api`/`apps/web` (list/search/detail) — the first real feature, per Build Sequencing step 3 in the plan.
+**All four previously-open decisions are now locked (2026-07-22).** See `PLAN.md`
+→ "Decisions (locked)" for the authoritative record:
+- Extraction toolchain (macOS): **mdbtools** (installed + verified against the real files).
+- i18n: **Spanish-first**.
+- CI/CD: **Gitea Actions** on `git.mancinas.io` → registry → Portainer.
+- `utility_dbo`: **resolved** — full dump (`utility_dbo.sql`, 1.3 GB, 55 tables) and the
+  portal codebase (`~/PhpstormProjects/my-jorgecuadros-web`) are both on disk.
+
+**Environment: moved Windows → macOS.** Sources now at `~/Downloads/JorgeCuadros-Legacy/`
+(all four files). This machine has Docker, MySQL/MariaDB client, Node 22, Python 3.14,
+Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet not present.
+
+**Execution queue (in order):**
+1. **Port the extraction layer to mdbtools.** Rewrite `migration/extract.py` to shell out to
+   `mdb-tables`/`mdb-export` instead of `pyodbc`. Keep the same public interface
+   (`connect`/`list_tables`/`read_table`) so `load_staging.py` and `config.py` are unchanged
+   beyond the already-fixed `SOURCE_ROOT`. Carry over the two hard-won fixes conceptually:
+   accented-column tables (mdbtools reads `PROPANO` cleanly — verified) and the corrupted `MULT`
+   row (mdb-export's `-b` / error handling; confirm the bad row is skipped, not fatal).
+2. **Re-run staging** (`python load_staging.py --output-dir ./output`) to regenerate the staged
+   data on this machine, then load into a local MySQL (`docker compose up mysql`) for SQL reconciliation.
+3. **Reconciliation pass** (plan step 2) against staged data — the `EFECTIVO*` variants, the
+   billing-period tables, `DATGRAL` vs `COBRO3`. Produce diff/dedupe rules from the data.
+4. **Transform + load** (plan step 3), starting with `Customer`/`CustomerLegacyRef` — every
+   other module depends on it. Use the `NUM UTIL` cross-ref + name/address matching.
+5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
+   Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
+6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the
+   VPS→internal inbox tables against the real `utility_dbo` schema and the portal's read/write
+   points in `my-jorgecuadros-web` (`peticion_gas`, PayPal payments, `notifications_settings`).
+
+Only genuinely-pending item is **VPS provisioning** (ops task — provider/size/Tailscale+MySQL).

migration/config.py

@@ -8,13 +8,18 @@ templates, materialized Access query results) are excluded. Anything with
 real rows and an ambiguous purpose (e.g. PROPANO, datosfreak, pagos email)
 is loaded into staging anyway — the reconciliation pass decides what to do
 with it, per the migration plan's "don't guess the rule up front" principle.
-See C:\\Users\\ricar\\.claude\\plans\\logical-yawning-tome.md for the full
-rationale per table group.
+See PLAN.md (repo root) for the full rationale per table group.
+
+Environment note: this project moved from Windows to macOS. The original
+extraction path (pyodbc + the Windows Access ODBC driver in extract.py) does
+not work on macOS; extraction is being reworked to use mdbtools
+(`brew install mdbtools`). This SOURCE_ROOT points at the macOS location of
+the four Access source files.
 """
 
 from pathlib import Path
 
-SOURCE_ROOT = Path(r"C:\Users\ricar\Downloads\Jorge")
+SOURCE_ROOT = Path.home() / "Downloads" / "JorgeCuadros-Legacy"
 
 SOURCES = {
     "utilities": {