rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Lock plan: macOS move, mdbtools extraction, resolve open decisions

Browse Source 
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.

- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
  with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
  CI=Gitea Actions, utility_dbo=available); update Status for the macOS
  move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 17:08:43 -07:00
co-authored by Claude Opus 4.8
parent 0268ed896c
commit 22a53bc562
3 changed files with 61 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+20 -10
View File
@@ -107,7 +107,7 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script (`pyodbc` → `pandas`/SQLAlchemy, same connection approach already validated in this session), across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet.
1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
2. **Reconciliation pass**: for each set of overlapping tables (the `EFECTIVO` variants, the billing-period tables, `DATGRAL` vs `COBRO3`), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
@@ -122,20 +122,30 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
8. VPS provisioning + Tailscale + MySQL replication setup, once `utility_dbo`'s schema is available to finalize exactly which tables/columns get replicated and what the inbox tables need to look like.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8.
8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.
10. Reports/email campaigns/admin — parity with old app's `reports.php`/`emailCampaigns.php` intent, rebuilt properly.
## Status (as of this session)
## Status
Repo scaffolded at `jorgecuadros-platform/` (sibling to the Access files): npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps build clean under strict TypeScript. `migration/load_staging.py` (step 1 of Migration strategy) has been run end-to-end against all four real Access files, staging 82 tables to Parquet; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap (Prisma schema, Docker Compose, `.env.example`, migration script's MySQL sink) has since been applied and re-verified (schema validates, client regenerates, API rebuilds clean against MySQL). A companion resume doc lives at `jorgecuadros-platform/RESUME.md` with exact file paths, environment notes, and a session-state summary — read both together.
Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a real hashed-password (Argon2) session-auth module replacing the old plaintext SQL comparison, Next.js web shell, Prisma schema covering the full data model above — both apps built clean under strict TypeScript on the original machine. Committed and pushed to `git.mancinas.io/rmancinas/jorgecuadros-platform` (branch `master`, 3 commits, working tree clean). `migration/load_staging.py` (step 1 of Migration strategy) was run end-to-end against all four real Access files on the original machine, staging 82 tables; it surfaced and fixed two real data issues: a `cursor.columns()` UTF-16 decode bug on several tables (worked around by reading metadata from `cursor.description` instead) and one Jet/ACE-level corrupted record in `MULT` (now skipped and logged rather than aborting the whole table). Schema/infra were built against Postgres first, then switched to MySQL after the shared-hosting/portal-sync constraint came up — the provider swap has been applied and re-verified. A companion resume doc lives at `jorgecuadros-platform/RESUME.md` — read both together.
## Open decisions (not yet locked down)
**Environment moved Windows → macOS.** Source Access files now live at `~/Downloads/JorgeCuadros-Legacy/` (all four: `UTILITIES.accdb`, `SEGUROS 16.mdb`, `SEGUROS 16_be.mdb`, `SCOTHIA.mdb`). This machine has Docker, a MySQL/MariaDB client, Node 22, Python 3.14, and Homebrew — but **no Access ODBC driver and no `node_modules` installed yet**. Consequence: the `pyodbc`-based extraction layer must be reworked for mdbtools (see Migration strategy step 1), `node_modules` needs `npm install`, and the staging output (gitignored Parquet) must be regenerated from scratch.
- **`utility_dbo` schema**: the customer-facing portal's live database — referenced from the old app (`getExternalDBConnection()`) but no dump/access provided yet. Blocks finalizing exactly which tables the sync replicates and what the inbox tables (`payment_submissions`, `propane_order_requests`) need to match.
- **VPS provisioning**: which provider (Hetzner vs DigitalOcean), size, and who sets up Tailscale + MySQL on it — an ops task outside this session's ability to do directly.
- **CI/hosting**: keep Jenkins + `git.freakma.com`, or move CI to GitHub Actions if the new repo goes elsewhere?
- **i18n**: nearly all source data and, presumably, staff usage is in Spanish, while the old app's code/UI was English-labeled internally. Confirm whether the new UI should be Spanish-first, bilingual, or English (matching old app) before frontend work starts.
**Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB.
## Decisions (locked)
- **Stack:** Next.js + NestJS + Prisma + **MySQL** (locked earlier — see engine rationale above).
- **Extraction toolchain (macOS):** **mdbtools**, verified against the real files. Forms/Reports/Queries were already extracted on Windows and are frozen in `objects.json`/`docs/`; only `LONGBINARY` blob extraction (step 4) needs extra handling under mdbtools.
- **i18n:** **Spanish-first** UI — matches the source data and staff usage.
- **CI/CD:** **Gitea Actions** on `git.mancinas.io` — build image, push to the git.mancinas.io container registry, deploy to Portainer (mirrors the `portainer-gitea-deploy` pattern already used on this LAN). Jenkins/`git.freakma.com` dropped.
- **`utility_dbo`:** resolved — full dump + portal code available (see Status).
## Open items (ops, not design)
- **VPS provisioning:** provider (Hetzner vs DigitalOcean), size, and Tailscale + MySQL replica setup on it — an ops task, still pending. Design is settled; only the box is missing.
- **Old external-DB credential** (hardcoded plaintext MySQL password in the old repo's `dbConnection.php`, in git history) — rotate it regardless, since it's already exposed.
## Verification