rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record step 6, correct the EFECTIVO verdict, refresh stale state

Browse Source 
PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
  with the corrected de-dup rule, plus a box explaining why the original
  verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
  summed.
- Build sequencing step 6 marked done.

RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
  "not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
  worth knowing — run_all.py vs single transforms, `next build` clobbering a
  running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
  many sessions ago. Renumbered, with an explicit "next" block.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 23:29:45 -07:00
co-authored by Claude Opus 4.8
parent 2c6a6bf60b
commit 12a1523073
2 changed files with 182 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESUME.md
+176 -78
View File
@@ -4,11 +4,11 @@ Comprehensive state-of-the-world doc for picking this project back up. Read this
before doing anything else in a fresh session — it front-loads everything that
took multiple rounds of investigation to establish.
**Companion doc:** the full architecture/migration plan is copied into this
repo as [`PLAN.md`](PLAN.md) (source of truth is
`C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — copy here if that
one gets updated further). This file is the "what happened and what's next"
companion to that plan, not a replacement for it. Read both.
**Companion doc:** the full architecture/migration plan is [`PLAN.md`](PLAN.md) in
this repo — **that is the source of truth for the design.** (It began as
`~/.claude/plans/logical-yawning-tome.md` on the old Windows machine; that copy is
gone and no longer authoritative.) This file is the "what happened and what's next"
companion, not a replacement. Read both.
---
@@ -32,28 +32,37 @@ infrastructure decisions below.
## 2. Where everything lives (file paths)
**Source data (do not modify — read-only references):**
- `C:\Users\ricar\Downloads\Jorge\UTILITIES.accdb` — utilities business, 52 tables, ~538MB
- `C:\Users\ricar\Downloads\Jorge\SEGUROS 16.mdb` — insurance frontend shell, **empty**, all data is in `_be`
- `C:\Users\ricar\Downloads\Jorge\SEGUROS 16_be.mdb` — insurance backend, 64 tables, ~882MB
- `C:\Users\ricar\Downloads\Jorge\SCOTHIA.mdb` — office's own Scotiabank checking register ("chequera"), 7 tables, ~3MB
> Paths below are the **current macOS machine**. The project moved Windows → macOS on
> 2026-07-22; anything still written as `C:\Users\ricar\...` in older notes is stale.
**Source data (do not modify — read-only references), all in `~/Downloads/JorgeCuadros-Legacy/`:**
- `UTILITIES.accdb` — utilities business, 52 tables, ~538MB
- `SEGUROS 16.mdb` — insurance frontend shell, **no data tables**, but holds *all* of the
insurance line's Reports/Forms/Queries
- `SEGUROS 16_be.mdb` — insurance backend, 64 tables, ~882MB
- `SCOTHIA.mdb` — office's own Scotiabank checking register ("chequera"), 7 tables, ~3MB
- `utility_dbo.sql` — customer portal's live DB dump (1.3 GB, 55 tables)
- `jorgecuadros.sql` — older/partial export (38 MB, 11 tables), **not** the portal live DB
- **Full structural reference for all three, usable without Windows or the original files:** [`docs/LEGACY_DATABASES.md`](docs/LEGACY_DATABASES.md) — every table, every column with type/nullability, the cross-reference keys between the three databases, and every known data-quality quirk (the UTF-16 decode bug, the corrupted `MULT` row, near-duplicate snapshot tables, etc.), all generated from a live read of the real files via `migration/catalog_schema.py`. Regenerate it if the source files change; the raw JSON it's built from is checked in at `migration/catalog.json`.
- **Queries/Forms/Reports reference:** [`docs/LEGACY_DATABASES_OBJECTS.md`](docs/LEGACY_DATABASES_OBJECTS.md) — none of this is visible via ODBC/`pyodbc`; it required DAO COM automation (`migration/catalog_objects.py`, needs `pywin32`) instead. Found 311 Reports, 271 Forms, and 1,274 Queries (751 "real," the rest Access-internal hidden subquery caches) across the three populated files — importantly, `SEGUROS 16.mdb` (which has zero data tables) turned out to hold *all* of the insurance line's Reports/Forms/Queries; `SEGUROS 16_be.mdb` is confirmed pure data storage. The real queries' full SQL text is the best available record of actual business logic (billing math, renewal batching) — worth reading before reimplementing any given feature from scratch. Raw JSON checked in at `migration/objects.json`.
- `C:\Users\ricar\Downloads\jorgecuadros_app.sql` and `jorgecuadros_app (1).sql` — MySQL dumps of the customer-portal's **tracking/analytics** sidecar DB (`browse_tracking`, `devices` push-tokens, `task_tracking`) from `mysql.freakma.com`. **Not** the portal's real data DB — see open item #1 below.
- `jorgecuadros_app.sql` / `jorgecuadros_app (1).sql` (on the old machine) — MySQL dumps of the portal's **tracking/analytics** sidecar DB (`browse_tracking`, `devices` push-tokens, `task_tracking`). **Not** the portal's real data DB; superseded by `utility_dbo.sql` above.
**Customer-facing portal (out of scope to rebuild, but the sync target):**
- `~/PhpstormProjects/my-jorgecuadros-web` — PHP/`mysqli`, ~397 files, core in `scripts/functions.php`. Reads/writes `utility_dbo`.
**Old internal app (reference-only, not being built on):**
- `C:\Users\ricar\Downloads\Jorge\jorgecuadros-intra-webapp` — PHP, MySQL (`webapp_jorgecuadros`). Schema at `db\webapp_jorgecuadros.sql` is a useful reference for field mappings/business logic. Code itself is not being reused — see §4.
- `jorgecuadros-intra-webapp` (on the old machine) — PHP, MySQL (`webapp_jorgecuadros`). Its `db/webapp_jorgecuadros.sql` is a useful reference for field mappings/business logic. Code itself is not reused — see §4.
**New platform (the actual deliverable, in progress):**
- `C:\Users\ricar\Downloads\Jorge\jorgecuadros-platform` — the new repo. Not yet a git repository (no commits made this session — user hasn't asked for any yet).
**New platform (the actual deliverable):**
- `~/WebstormProjects/jorgecuadros-platform` — the repo. **Is** a git repo, branch `master`, 21 commits, remote `git.mancinas.io/rmancinas/jorgecuadros-platform`.
**The plan document:**
- `C:\Users\ricar\.claude\plans\logical-yawning-tome.md` — full architecture, source-data inventory per table, target data model, migration strategy, infrastructure/sync design, open decisions, build sequencing. **This is the source of truth for the design** — this RESUME.md summarizes it plus session/environment state that isn't in the plan itself.
- [`PLAN.md`](PLAN.md) in this repo — full architecture, source-data inventory per table, target data model, migration strategy, infrastructure/sync design, locked decisions, build sequencing. **This is now the source of truth for the design** (the original `~/.claude/plans/logical-yawning-tome.md` lived on the old Windows machine). This RESUME.md is the "what happened / what's next" companion.
**Ephemeral / will NOT persist across sessions** (session-scoped temp directory):
- `C:\Users\ricar\AppData\Local\Temp\claude\...\scratchpad\` — contained exploratory helper scripts (`dump_schema.py`, `summarize_schema.py`, `test_decode_fix.py`) and the schema JSON/txt dumps used during initial analysis, plus a Parquet staging output from one run of the migration script. **None of this needs to be recovered** — the real, permanent versions of the useful scripts are in `jorgecuadros-platform/migration/`, and the Parquet output can be regenerated in under 2 minutes by rerunning `load_staging.py` (see §6).
**Staged data (gitignored, regenerable):**
- `migration/output/stg_utilities|stg_seguros|stg_scothia/*.parquet` — regenerate in ~2 min with `load_staging.py --output-dir ./output`. Every transform step reads from here.
## 3. Key decisions made this session
## 3. Key decisions (locked — see `PLAN.md` → "Decisions (locked)")
| Decision | Answer | Why |
|---|---|---|
@@ -65,9 +74,10 @@ infrastructure decisions below.
| Infrastructure | Internal server (private) + new VPS (Tailscale-linked) running a MySQL replica | Internal server has no inbound internet exposure; shared hosting can't be a replication target; a VPS you control can be both a real replication node and internet-reachable for the portal |
| Auth mechanism | Session-based (Passport + `express-session`), Argon2 password hashing | Implemented already — see §5 |
## 4. What was actually built and verified this session
## 4. What is built and verified
Everything below was **run and confirmed working**, not just written:
Everything below was **run and confirmed working**, not just written. §8 carries the
per-module detail and the running status; this section is the structural tour.
### 4.1 Repo scaffold
- `jorgecuadros-platform/` — npm workspaces (`apps/*`, `packages/*`)
@@ -94,7 +104,7 @@ Regenerate the client any time with:
cd jorgecuadros-platform
DATABASE_URL="mysql://user:pass@localhost:3306/placeholder" npx prisma generate --schema=packages/database/prisma/schema.prisma
```
(A real `DATABASE_URL` isn't needed for `generate`/`validate`, just a syntactically valid one — no live DB was available in this session, see §7.)
(A real `DATABASE_URL` isn't needed for `generate`/`validate`, just a syntactically valid one. A live dev DB *is* available now — see §7 — so `prisma db push` works too.)
### 4.3 Docker Compose / Dockerfiles
- `docker-compose.yml` — `mysql:8.4` + `api` + `web` services, healthchecked.
@@ -103,17 +113,20 @@ DATABASE_URL="mysql://user:pass@localhost:3306/placeholder" npx prisma generate
- **Not run** — this environment has no Docker installed (`docker --version` fails). Untested beyond visual review; verify on a machine with Docker before relying on it.
### 4.4 Migration pipeline (`migration/`) — run end-to-end against real data
- `config.py` — manifest of the 3 Access source files (paths + per-source exclude lists for confirmed-scratch tables, with reasoning in comments)
- `extract.py` — connects via `pyodbc` + the Windows Access ODBC driver (`Microsoft Access Driver (*.mdb, *.accdb)`, 64-bit — confirmed installed on this machine). Two real bugs found and fixed here:
1. `cursor.columns()` hits a UTF-16 decode bug on some tables (confirmed: `PROPANO`, `FALTANTES AGUA`, `TIT`) — fixed by reading column names from `cursor.description` after a `SELECT *` instead.
2. `cursor.fetchall()` aborts an entire table on the first corrupted row — confirmed on `MULT` (Jet/ACE-level "Record is deleted" error, HY109). Fixed by fetching row-by-row in a try/except, skipping and logging just the bad row. Recovered 763 of 764 rows in `MULT`. **Verified the cursor advances correctly and doesn't infinite-loop on the bad row** before trusting this for a full run.
- `load_staging.py` — dumps every non-excluded table into either Parquet (`--output-dir`, no DB needed) or MySQL (`--database-url`, one database per source: `stg_utilities`/`stg_seguros`/`stg_scothia`). **Actually run** in Parquet mode against all three Access files: **82 tables staged successfully, zero unhandled errors**, `MULT`'s corrupted row correctly skipped and logged.
- `requirements.txt` — `pyodbc`, `pandas`, `pyarrow`, `sqlalchemy`, `pymysql`.
- `config.py` — manifest of the Access source files (`SOURCE_ROOT` + per-source exclude lists for confirmed-scratch tables, with reasoning in comments)
- `extract.py` — shells out to **mdbtools** (`mdb-tables` / `mdb-export`, Homebrew). Rewritten from the original `pyodbc` + Windows Access ODBC version during the macOS move; public interface (`connect`/`list_tables`/`read_table`) unchanged. mdbtools also sidesteps both bugs the pyodbc path needed workarounds for: it reads accented-column tables (`PROPANO`, `FALTANTES AGUA`, `TIT`) cleanly instead of hitting a UTF-16 decode error, and it doesn't abort a whole table on `MULT`'s corrupted row.
- What mdbtools **cannot** do is read Forms/Reports/Queries. Those were already captured on Windows via DAO COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md` — nothing is lost, but they can't be re-extracted on this machine.
- `load_staging.py` — dumps every non-excluded table into either Parquet (`--output-dir`, no DB needed) or MySQL (`--database-url`, one database per source: `stg_utilities`/`stg_seguros`/`stg_scothia`). 82 tables staged, zero unhandled errors.
- `reconcile.py` → `RECONCILIATION.md` — the duplicate/distinct pass (step 2). See §8 step 3.
- `transform_*.py`, `prune_empty_customers.py`, `blob_extract.py` — steps 3–4, all idempotent (truncate + rebuild).
- `run_all.py` — **the entry point.** Runs every step in dependency order. See the ⚠️ in §7 for why you should never run a single transform on its own.
- `dbenv.py` — `--env <name>` reads `deploy/.env.<name>` for the target DB.
- `requirements.txt` — `pandas`, `pyarrow`, `sqlalchemy`, `pymysql`, `boto3` (no `pyodbc` — that was the Windows path).
To rerun (from `jorgecuadros-platform/migration`, after `pip install -r requirements.txt`):
To rerun (from `migration/`, venv at `migration/.venv`):
```bash
python load_staging.py --output-dir ./output # Parquet, no DB needed — always works
python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/ # loads into real MySQL once available
./.venv/bin/python load_staging.py --output-dir ./output # re-extract from Access (needs mdbtools + the source files)
./.venv/bin/python run_all.py --env dev # full transform+load; add --stage to re-extract first
```
## 5. Infrastructure & sync architecture (designed, not yet built)
@@ -124,22 +137,47 @@ python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/ # l
- **Internal → VPS:** one-way native MySQL replication (binlog/GTID) for the subset of data the portal needs to read (statements, balances, customer profile). Internal-only tables (staff notes, adjuster info, activity logs) are deliberately excluded from what replicates.
- **VPS → Internal:** the portal also *writes* (payment submissions, propane orders) — one-way replication can't carry that back, and multi-master MySQL replication was deliberately ruled out as too fragile for this system's size. Instead: unreplicated "inbox" tables on the VPS (`payment_submissions`, `propane_order_requests`) that the portal writes to directly, polled every 1–5 min by a worker on the internal server (over Tailscale) that turns new rows into real records.
## 6. Open items — need input/access before certain next steps can proceed
## 6. Open items
1. **`utility_dbo` schema is still unknown.** This is the customer portal's actual live data database (referenced in the old app via `getExternalDBConnection()` at `mysql.freakma.com`, used there only for `email_alert_log`, but the mobile app almost certainly reads/writes statements, payments, and propane orders directly against it). The two SQL dumps provided (`jorgecuadros_app.sql`, `jorgecuadros_app (1).sql`) turned out to be a **separate** analytics/tracking database, not this one. When asked, the user pointed back to `SEGUROS 16_be.mdb` — worth revisiting; it's possible the intent was "the source data ultimately comes from the Access files" rather than "here is utility_dbo's schema." **Without the real `utility_dbo` schema, the sync worker's exact target tables/columns for the inbox pattern can't be finalized.** Ask for an export or read-only credentials, same as how the Access files and tracking-DB dumps were provided.
2. **VPS not yet provisioned** — provider (Hetzner vs DigitalOcean), size, and Tailscale/MySQL setup on it are pending. Ops task, not something done in this session.
3. **CI/hosting** — keep Jenkins + `git.freakma.com`, or move to GitHub Actions if the new repo lives elsewhere?
4. **i18n** — nearly all source data and, presumably, staff usage is in Spanish; old app's code/UI was English-labeled. Confirm Spanish-first / bilingual / English before frontend work goes deep.
5. **Reconciliation pass not started** (migration plan step 2) — the near-duplicate snapshot tables (`EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`, `FEE ANUAL`/`datos2`/`fee15`/`billing`, `DATGRAL` vs `COBRO3`) need a diff/dedupe pass against the staged data before any transform-and-load into the real schema. This is explicitly *not* a "guess the rule up front" thing — the plan calls for writing SQL against the staged Parquet/MySQL data to see what's actually duplicate vs. distinct.
**Resolved since this section was first written** (kept as a pointer, not a to-do):
`utility_dbo` schema (full dump on disk), CI/CD (Gitea Actions), i18n (Spanish-first), and
the reconciliation pass (done, then corrected) are all closed. See §3 and §8.
## 7. Environment notes (this machine, in case it matters for reproducing)
**Still open:**
1. **VPS not yet provisioned** — provider (Hetzner vs DigitalOcean), size, Tailscale + MySQL
replica setup. Pure ops task; the design is settled (§5). This is the only genuinely
blocking item left on the roadmap.
2. **Sync worker not built** — unblocked now that `utility_dbo` and the portal code are on
disk, but depends on the VPS existing. Portal write points to poll: `peticion_gas`,
PayPal payments, `notifications_settings`, `verification_codes`.
3. **Old external-DB credential** — the old repo's `dbConnection.php` has a hardcoded
plaintext MySQL password committed to git history. Not carried into the new platform,
but rotate it regardless; it is already exposed.
4. **`bank_transactions.categoryId` is null on all 22354 rows** — the concept→ramo
classifier was deferred. Needed before any "insurance vs utilities vs trust" split of
the office's own bank activity.
5. **`TRASPASOS PAYPAL` is a clearing account, not a customer** — carries -7.03M MXN over
309 movements and therefore tops the adeudo worklist. Deliberately not special-cased in
code; needs a business decision on how to model it.
- Windows, PowerShell primary, Git Bash also available.
- Node v25.2.0, npm 11.6.2 — no pnpm, yarn is present but npm workspaces were used throughout.
- Python 3.9 (`C:\Python39`), `pip install`'d this session: `pyodbc`, `pandas`, `pyarrow`, `psycopg2` (installed but no longer used after the MySQL switch — harmless to leave or remove).
- MS Access ODBC driver confirmed installed: **64-bit** `Microsoft Access Driver (*.mdb, *.accdb)` (matches 64-bit Python — this pairing matters, a 32/64-bit mismatch would break `pyodbc.connect`).
- **No Docker, no local MySQL, no local Postgres** on this machine — `docker-compose.yml` and the MySQL-target mode of `load_staging.py` are written but unexecuted here. Test both on whatever machine ends up running this for real.
- Old repo's `dbConnection.php` has a **hardcoded plaintext MySQL password** for the external DB connection, committed to git history. Not carried forward into the new platform, but worth rotating that credential regardless since it's already exposed in the old repo's history.
## 7. Environment notes (current macOS machine)
- macOS (Darwin 25.5.0), zsh. Node v22.23.0. Python 3.14.6 in `migration/.venv`. Homebrew, Docker, MySQL/MariaDB client all present.
- **mdbtools** installed via Homebrew — the extraction toolchain. No Access ODBC driver (and none needed).
- **`npm` is pnpm-aliased**, and pnpm ignores the `workspaces` field. Consequences:
- there is **no root `node_modules/.bin`**. Binaries live per-app: `apps/api/node_modules/.bin/nest`, `apps/web/node_modules/.bin/next`.
- Prisma CLI is run as `npx prisma@5`.
- **Dev servers** (both must be up to use the UI):
- API `cd apps/api && ./node_modules/.bin/nest start --watch` → `:3001`
- Web `cd apps/web && ./node_modules/.bin/next dev` → `:3000`
- Dev login: `admin@jorgecuadros.local`, password from `apps/api/scripts/seed-user.mjs` (`SEED_PASSWORD` env overrides the default).
- **Dev DB**: `192.168.4.212:3307` (cubex Swarm stack `jorgecuadros-dev-db`). Credentials in gitignored `deploy/.env.dev`. **MinIO** for documents: `192.168.4.212:9100`, bucket `jorgecuadros-documents`.
**Traps worth knowing before you lose an hour to one:**
- ⚠️ **Never run a single `transform_*.py` on its own — use `run_all.py`.** Each step truncates what it owns, so a lone run orphans everything downstream. `prune_empty_customers.py` must re-run after any ledger change, and `blob_extract.py` must follow properties + policies or the uploaded MinIO objects end up with no rows pointing at them.
- ⚠️ **Never run `next build` while `next dev` is running** — they share `.next` and the dev server starts serving blank white pages. Recovery: kill the dev server, `rm -rf apps/web/.next`, restart.
- ⚠️ **`mdb-export` formats numerics per Access column type** (`5000` from one table, `27000.0000` from another). Never string-compare staged Parquet numerics across two tables — canonicalize first. This exact trap produced a wrong, *locked* migration decision that shipped 12386 duplicate rows into the ledger (§8 step 4).
- Shell on this machine: `head` is aliased to an HTTP HEAD tool — use `/usr/bin/head`. `grep --include=*.md` trips zsh globbing — quote the pattern.
## 8. Plan locked — next actions
@@ -151,24 +189,36 @@ python load_staging.py --database-url mysql+pymysql://user:pass@host:3306/ # l
- `utility_dbo`: **resolved** — full dump (`utility_dbo.sql`, 1.3 GB, 55 tables) and the
portal codebase (`~/PhpstormProjects/my-jorgecuadros-web`) are both on disk.
**Environment: moved Windows → macOS.** Sources now at `~/Downloads/JorgeCuadros-Legacy/`
(all four files). This machine has Docker, MySQL/MariaDB client, Node 22, Python 3.14,
Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet not present.
**Environment: moved Windows → macOS** (2026-07-22). See §7 for the current machine.
**Execution queue (in order):**
1. **Port the extraction layer to mdbtools.** Rewrite `migration/extract.py` to shell out to
**Execution queue.** Steps 1–6 below are **done**; they are kept because each carries the
data findings and corrections that came out of doing it. Skip to the ⏭ marker at the end
for what's actually next.
1. ~~**Port the extraction layer to mdbtools.**~~ **DONE.** Rewrote `migration/extract.py` to shell out to
`mdb-tables`/`mdb-export` instead of `pyodbc`. Keep the same public interface
(`connect`/`list_tables`/`read_table`) so `load_staging.py` and `config.py` are unchanged
beyond the already-fixed `SOURCE_ROOT`. Carry over the two hard-won fixes conceptually:
accented-column tables (mdbtools reads `PROPANO` cleanly — verified) and the corrupted `MULT`
row (mdb-export's `-b` / error handling; confirm the bad row is skipped, not fatal).
2. **Re-run staging** (`python load_staging.py --output-dir ./output`) to regenerate the staged
data on this machine, then load into a local MySQL (`docker compose up mysql`) for SQL reconciliation.
3. **Reconciliation pass** (plan step 2) — **DONE** (`migration/reconcile.py` → `RECONCILIATION.md`).
Overturned all three "duplicate" assumptions: EFECTIVO/BACKUP are near-disjoint ledgers
(folio collides; migrate both), the billing tables are disjoint period runs (union all, no
de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The
decided union/de-dup rules are in `PLAN.md` migration step 2.
2. ~~**Re-run staging**~~ **DONE** — staged Parquet regenerated on this machine
(`load_staging.py --output-dir ./output`), 82 tables.
3. **Reconciliation pass** (plan step 2) — **DONE** (`migration/reconcile.py` → `RECONCILIATION.md`),
**and corrected 2026-07-22.** Current verdicts:
- `EFECTIVO_BACKUP` is a **stale backup copy of `EFECTIVO`** — 12386 of its 12387 rows are
verbatim duplicates (customer + timestamp-to-the-second + amount + concept text), leaving
1 new row. Load EFECTIVO in full, de-dup BACKUP on the business key. **Never de-dup on
`folio`** — it is per-table sequential and collides (12363 shared numbers, 12204 of them
on different payments).
- The billing tables (`datos2`/`FEE ANUAL`/`fee15`) are disjoint period runs — union all,
no de-dup.
- `COBRO3` is a charge batch, not a customer snapshot — `DATGRAL` is the sole master.
⚠️ **This file and `PLAN.md` previously said the opposite about EFECTIVO** ("near-disjoint
ledgers, migrate both"). That was a bug, not a finding — see the Shared ledger entry in
step 4 below for the root cause and the fix. If you read a doc, comment, or commit message
from before 2026-07-22 that says "migrate both, no folio de-dup", it is stale.
The authoritative rules live in `PLAN.md` migration step 2.
4. **Transform + load** (plan step 3) — IN PROGRESS.
- **Customers — DONE** (`migration/transform_customers.py`). Loaded into the dev DB: 1682
customers (1172 utilities master + 510 insurance-only), 2242 legacy refs (all traceable),
@@ -185,12 +235,23 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
adjusters. Unmodeled coverage columns preserved verbatim in `coveragesJson`. Verified a
unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT
policies — the cross-line customer view works at the data layer.
- **Shared ledger — DONE** (`migration/transform_transactions.py`): 45861 transactions
(UTILITY 45566 / INSURANCE 295, 0 orphans) unioning both EFECTIVO tables (13696+12386,
no folio de-dup), all three billing tables (datos2/FEE ANUAL/fee15), the FM3 fee stream
(amount=fee+tax+multa), IVA 2015 (nominal date), and insurance EFECTIVO — per the
reconciliation rules; plus 79 `type_transactions` (EN/ES) and 2301 `exchange_rates`.
Skipped 22 no-customer + 303 no-date (mostly datos2 blanks).
- **Shared ledger — DONE** (`migration/transform_transactions.py`): **33475** transactions
(UTILITY 33180 / INSURANCE 295, 0 orphans) unioning EFECTIVO (13695) **plus only the 1
business-key-new row from EFECTIVO_BACKUP**, all three billing tables
(datos2/FEE ANUAL/fee15), the FM3 fee stream (amount=fee+tax+multa), IVA 2015 (nominal
date), and insurance EFECTIVO; plus 79 `type_transactions` and 2301 `exchange_rates`.
Skipped 22 no-customer + 272 no-date + **12417 EFECTIVO_BACKUP duplicates**.
**Corrected 2026-07-22 — this used to load 45861 rows.** `reconcile.py` had string-compared
`monto`, which mdb-export serializes at a different precision per Access column type
(`5000` vs `27000.0000`), so it saw 2 overlapping rows instead of 12386 and ruled
EFECTIVO_BACKUP an independent ledger. It is a stale backup copy: 12386 of its 12387 rows
match an EFECTIVO row on customer + timestamp-to-the-second + amount + concept text. The
ledger was double-counting those payments, roughly doubling every customer's historical
receipt total. Both `reconcile.py` (canonicalizes numeric key columns now) and
`transform_transactions.py` (de-dups on the business key, never on `folio` — folio
collides) are fixed, and `run_all.py --env dev` has been re-run end to end.
**Lesson for any future reconciliation: never compare mdb-export output as raw strings
across two tables — canonicalize numerics first.**
- **Bank register — DONE** (`migration/transform_bank.py`): 22354 `bank_transactions` from
SCOTHIA DATOS I/E as signed amounts (income +, expense -; net +899,375.77), 66
`business_line_categories`. No customer FK; categoryId left null (concept->ramo classifier
@@ -233,21 +294,58 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
geographic filter. `PropertyService.notes` means something different per kind
(municipality / CFE PAR-IMPAR cycle / gas supply type), so the UI labels it per kind.
240 of 1519 properties have no service rows at all — surfaced as its own bucket.
- **NEXT:** plan step 6 — the shared billing/statements view across both business lines.
- Full pipeline reproducible in one command: `run_all.py --env <env>` runs customers ->
properties -> policies -> transactions -> bank in order (all idempotent); add `--stage`
to re-extract from the Access files first. Verified end-to-end against dev.
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file:
`deploy/jorgecuadros-db.stack.yml` (same file deploys prod as `jorgecuadros-prod-db` :3306).
Creds in gitignored `deploy/.env.dev`. NOTE: machine `npm` is pnpm-aliased and pnpm ignores
the `workspaces` field — full workspace install needs `pnpm-workspace.yaml` or real npm; for
now Prisma CLI is run via `npx prisma@5`.
5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the
VPS→internal inbox tables against the real `utility_dbo` schema and the portal's read/write
points in `my-jorgecuadros-web` (`peticion_gas`, PayPal payments, `notifications_settings`).
- **Billing / statements module (plan step 6) — DONE**: `apps/api/src/billing/`
(`GET /billing` movement browser with search over customer / referencia / cheque /
concepto / periodo, filters for línea, moneda, cargo-vs-abono, concepto (typeId), origin
table and a from/to date range, 5 sorts, and **totals for the whole filtered set**;
`GET /billing/balances` per-customer balances with owing/credit/settled buckets and 4
sorts; `/billing/stats`, `/billing/facets`, `/billing/customers/:id`) + web
`/estado-cuenta` (two tabs: "Saldos por cliente" worklist and "Movimientos" ledger) and
`/estado-cuenta/[id]` (the statement: balance per currency, the same balance split by
business line, cargos por concepto with proportional bars, and the full movement table
with a running balance). Cross-links from the customer and property detail pages.
**Data findings:**
(a) `transactions.amount` is a *signed* ledger — every charge type is negative without
exception (WATER 3115/3117, ELECTRIC 2191/2191, PROPERTY TAXES 926/926, TRUST FEE
188/188) and every deposit type positive (CHECK/CASH DEPOSIT, PAYPAL, all of EFECTIVO),
so `SUM(amount)` is the balance and negative = the customer owes.
(b) **Currency is not summable.** 912 of the 1269 customers with a ledger move in both
MXN and USD; the charge side (datos2/FEE ANUAL/fee15) is MXN-only while receipts arrive
in both, and no per-movement exchange rate was ever stored. Every figure in the module is
per currency; the balance filter/sort takes a currency argument rather than collapsing.
(c) `type_transactions.nameEs` is **entirely null** — the legacy `TYPE OF TRX` table has
an `ESPAÑOL` column but all 79 rows are empty, so the API can only return English names.
`labels.ts:TX_TYPE_LABELS` supplies Spanish for the real service/payment categories; the
rest of the 79 "types" are payee names (LORETO GONZALEZ, ALBERCAS VALLARTA…) that fall
through untranslated, which is correct.
(d) The biggest debtor by far is **"TRASPASOS PAYPAL"** (-7.03M MXN over 309 movements) —
a house/clearing account, not a person. Left in rather than special-cased, but it will
head the adeudo worklist until someone decides how to model it.
- Full pipeline reproducible in one command: `run_all.py --env <env>` runs customers →
properties → policies → transactions → prune → bank → blobs in order (all idempotent);
add `--stage` to re-extract from the Access files first. Verified end-to-end against dev.
Only genuinely-pending item is **VPS provisioning** (ops task — provider/size/Tailscale+MySQL).
5. **Infra** — **DONE.** Dev MySQL deployed to the cubex Swarm via the Portainer API as stack
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file
`deploy/jorgecuadros-db.stack.yml` deploys prod from the same file as
`jorgecuadros-prod-db` on :3306. MinIO for documents deployed as `jorgecuadros-dev-minio`.
6. **Staff web UI** — **DONE** for all four modules (§8 step 4: clientes, polizas, servicios,
estado-cuenta). Spanish-first, session-cookie auth against the API, verified against real
migrated data.
---
⏭ **NEXT — where to pick up:**
- **Plan step 7: bank register module.** SCOTHIA data is already migrated (22354
`bank_transactions` + 66 `business_line_categories`) and has **no customer FK**, so it is
self-contained and low-risk — API + `/banco` browser only. Blocked on nothing.
Ties into open item §6.4 (`categoryId` is null on every row).
- **Plan step 8–9: VPS + sync worker.** Blocked on VPS provisioning (§6.1) — the only real
external dependency left.
- **Plan step 10: reports / email campaigns / admin.**
- **Uncommitted work:** the billing module and the ledger de-dup fix are written, built and
verified but **not committed** — `git status` is dirty at the time of writing. The browser
visual pass on `/estado-cuenta` was also never completed (it needs an interactive login).