docs: record step 6, correct the EFECTIVO verdict, refresh stale state

PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
  with the corrected de-dup rule, plus a box explaining why the original
  verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
  summed.
- Build sequencing step 6 marked done.

RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
  "not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
  worth knowing — run_all.py vs single transforms, `next build` clobbering a
  running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
  many sessions ago. Renumbered, with an explicit "next" block.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 23:29:45 -07:00
co-authored by Claude Opus 4.8
parent 2c6a6bf60b
commit 12a1523073
2 changed files with 182 additions and 82 deletions
+6 -4
View File
@@ -96,7 +96,7 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
- `service_documents` (extracted blobs), `trust_accounts` (from `TRUSTVENCE`).
**Shared financial ledger** (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive.
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive. **`amount` is signed:** negative = charge (cargo), positive = credit (abono), so `SUM(amount)` per customer per currency *is* the balance — negative means the customer owes the office. The two currencies are never summed together (see the billing module note in Build sequencing step 6).
- `exchange_rates` (from `TIPO HIST`), `type_transactions` (carry over ES/EN lookup as-is).
- `bank_transactions` — the company's own operating bank register, from SCOTHIA's `DATOS E`/`DATOS I` unified into one signed-amount table (income positive, expense negative) with a `category` FK to `business_line_categories` (from `TABLA RAMODOS`) and a `cleared`/`operado` flag. This is deliberately **separate** from customer-facing `transactions` — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the `business_line_categories` lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
- `business_line_categories` (from `TABLA RAMODOS`).
@@ -108,8 +108,10 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 13.
2. **Reconciliation pass****DONE** (`migration/reconcile.py``migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. **Outcome overturned all three of the plan's original "duplicate" assumptions — the union/de-dup rules below are now decided by the data:**
- **`EFECTIVO` vs `EFECTIVO_BACKUP`:** *not* a live/backup duplicate pair. `folio` is a per-table sequential number that **collides** (12,363 shared folio numbers, all carrying different transactions); on the real business key `(cl,fecha,monto,conepto)` only **2 rows** overlap. They are near-disjoint ledgers (BACKUP ≈ 20172022, EFECTIVO recent). **Rule: migrate both**, keyed internally by `(legacy_source_table, folio)` provenance; no folio de-dup, don't drop BACKUP. `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
2. **Reconciliation pass****DONE** (`migration/reconcile.py``migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. The union/de-dup rules below are decided by the data:
- **`EFECTIVO` vs `EFECTIVO_BACKUP`: `EFECTIVO_BACKUP` is a stale backup copy — de-dup it. (Corrected 2026-07-22; see the box below.)** On the canonicalized business key `(cl,fecha,monto,conepto)`, **12,386 of BACKUP's 12,387 rows already exist verbatim in `EFECTIVO`** — same customer, same timestamp to the second, same amount, same concept text — leaving exactly **1** genuinely new row. `folio` is a per-table sequential number that **collides** (12,363 shared numbers, 12,204 of them on different payments), so it can never be the de-dup key. **Rule: load `EFECTIVO` in full; from `EFECTIVO_BACKUP` load only business-key-new rows.** `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
> **Why this was wrong the first time.** The original pass reported only **2** overlapping rows and concluded the two tables were "near-disjoint ledgers, migrate both". That verdict came from a bug in `reconcile.py`, which compared business-key columns as raw strings on the premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: `mdb-export` formats a numeric column from its *Access column type*, so the same amount is emitted as `5000` from one table and `27000.0000` from the other, and no two rows could ever match on `monto`. `reconcile.py` now canonicalizes numeric key columns before comparing. The bad rule had already been loaded: the ledger carried 45,861 rows with **12,386 duplicated payments**, roughly doubling every customer's historical receipt total — which would have made every balance and statement in step 6 wrong. Re-running `run_all.py` brings the ledger to **33,475** rows. Groups 2 and 3 below were re-checked under the fix and their verdicts are unchanged.
- **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈202526, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
- **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
@@ -123,7 +125,7 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 35 for customers only.
4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions)**DONE**. `apps/api/src/billing/` + web `/estado-cuenta` and `/estado-cuenta/[id]`. Two questions, two views: a per-customer **balances worklist** (who owes what) and a cross-customer **movement browser** (every charge and credit, filterable by line, concept, origin table and date range, with totals for the whole filtered set). The detail page is the actual statement: balance per currency, the same balance split by business line, charges broken out by concept, and the full movement list with a running balance. **Design constraint that shapes the whole module: balances are reported per currency and never collapsed into one number.** 912 of the 1,269 customers with a ledger move in both MXN and USD, the charge side is MXN-only while receipts arrive in both, and the legacy data never stored the exchange rate applied to a movement — so a single "total balance" would be a figure that never existed in the books.
7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.