docs: record step 6, correct the EFECTIVO verdict, refresh stale state
PLAN.md: - Migration step 2: replace the "near-disjoint ledgers, migrate both" rule with the corrected de-dup rule, plus a box explaining why the original verdict was wrong so the reversal is auditable rather than silent. - Note that transactions.amount is signed and that currencies are never summed. - Build sequencing step 6 marked done. RESUME.md — the execution queue still stated the reverted EFECTIVO verdict verbatim, so a fresh session reading top-to-bottom would have hit the old rule in step 3 and the correction in step 4 with no way to tell which won. Beyond that fix, several sections still described the pre-macOS-move world: - §2: every source path was C:\Users\ricar\...; the repo was described as "not yet a git repository". - §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools. - §6: four of five "open items" were already resolved. - §7: documented the old Windows box. Now the macOS machine, plus the traps worth knowing — run_all.py vs single transforms, `next build` clobbering a running dev server's .next, and the mdb-export numeric formatting trap. - §8: items were mis-numbered (5b before 5) and item 5 was work finished many sessions ago. Renumbered, with an explicit "next" block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -96,7 +96,7 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
|
||||
- `service_documents` (extracted blobs), `trust_accounts` (from `TRUSTVENCE`).
|
||||
|
||||
**Shared financial ledger** (one office, one set of books — no reason to keep insurance and utility transactions in separate schemas):
|
||||
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive.
|
||||
- `transactions` — unifies utilities' `EFECTIVO`/`EFECTIVO FM3`/`EFECTIVO_BACKUP`/`FEE ANUAL`/`datos2`/`fee15`/`billing`/`CHEQUE FM3`/`IVA 2015` and insurance's `EFECTIVO`, tagged by `domain` (utility/insurance/trust) and carrying the provenance columns so the de-duplication across those overlapping snapshot tables is traceable, not destructive. **`amount` is signed:** negative = charge (cargo), positive = credit (abono), so `SUM(amount)` per customer per currency *is* the balance — negative means the customer owes the office. The two currencies are never summed together (see the billing module note in Build sequencing step 6).
|
||||
- `exchange_rates` (from `TIPO HIST`), `type_transactions` (carry over ES/EN lookup as-is).
|
||||
- `bank_transactions` — the company's own operating bank register, from SCOTHIA's `DATOS E`/`DATOS I` unified into one signed-amount table (income positive, expense negative) with a `category` FK to `business_line_categories` (from `TABLA RAMODOS`) and a `cleared`/`operado` flag. This is deliberately **separate** from customer-facing `transactions` — it's the office's own bank reconciliation book, not money owed by/to a customer — but sharing the `business_line_categories` lookup lets you eventually answer "how much of our actual bank activity ties back to insurance vs. utilities vs. trust," which is a natural reporting win from unifying these three sources.
|
||||
- `business_line_categories` (from `TABLA RAMODOS`).
|
||||
@@ -108,8 +108,10 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
|
||||
Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
|
||||
|
||||
1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
|
||||
2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. **Outcome overturned all three of the plan's original "duplicate" assumptions — the union/de-dup rules below are now decided by the data:**
|
||||
- **`EFECTIVO` vs `EFECTIVO_BACKUP`:** *not* a live/backup duplicate pair. `folio` is a per-table sequential number that **collides** (12,363 shared folio numbers, all carrying different transactions); on the real business key `(cl,fecha,monto,conepto)` only **2 rows** overlap. They are near-disjoint ledgers (BACKUP ≈ 2017–2022, EFECTIVO recent). **Rule: migrate both**, keyed internally by `(legacy_source_table, folio)` provenance; no folio de-dup, don't drop BACKUP. `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
|
||||
2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. The union/de-dup rules below are decided by the data:
|
||||
- **`EFECTIVO` vs `EFECTIVO_BACKUP`: `EFECTIVO_BACKUP` is a stale backup copy — de-dup it. (Corrected 2026-07-22; see the box below.)** On the canonicalized business key `(cl,fecha,monto,conepto)`, **12,386 of BACKUP's 12,387 rows already exist verbatim in `EFECTIVO`** — same customer, same timestamp to the second, same amount, same concept text — leaving exactly **1** genuinely new row. `folio` is a per-table sequential number that **collides** (12,363 shared numbers, 12,204 of them on different payments), so it can never be the de-dup key. **Rule: load `EFECTIVO` in full; from `EFECTIVO_BACKUP` load only business-key-new rows.** `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
|
||||
|
||||
> **Why this was wrong the first time.** The original pass reported only **2** overlapping rows and concluded the two tables were "near-disjoint ledgers, migrate both". That verdict came from a bug in `reconcile.py`, which compared business-key columns as raw strings on the premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: `mdb-export` formats a numeric column from its *Access column type*, so the same amount is emitted as `5000` from one table and `27000.0000` from the other, and no two rows could ever match on `monto`. `reconcile.py` now canonicalizes numeric key columns before comparing. The bad rule had already been loaded: the ledger carried 45,861 rows with **12,386 duplicated payments**, roughly doubling every customer's historical receipt total — which would have made every balance and statement in step 6 wrong. Re-running `run_all.py` brings the ledger to **33,475** rows. Groups 2 and 3 below were re-checked under the fix and their verdicts are unchanged.
|
||||
- **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈2025–26, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
|
||||
- **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
|
||||
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
|
||||
@@ -123,7 +125,7 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
|
||||
3. Customer module (list/search/detail — the unified view is the core deliverable) backed by finished migration steps 3–5 for customers only.
|
||||
4. Insurance module (policies, vehicles, beneficiaries, claims) on top of the same customer records.
|
||||
5. Utilities module (properties, services, trust accounts) on top of the same customer records.
|
||||
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions).
|
||||
6. Shared billing/statements module (the payoff: one statement per customer spanning both utility and insurance transactions) — **DONE**. `apps/api/src/billing/` + web `/estado-cuenta` and `/estado-cuenta/[id]`. Two questions, two views: a per-customer **balances worklist** (who owes what) and a cross-customer **movement browser** (every charge and credit, filterable by line, concept, origin table and date range, with totals for the whole filtered set). The detail page is the actual statement: balance per currency, the same balance split by business line, charges broken out by concept, and the full movement list with a running balance. **Design constraint that shapes the whole module: balances are reported per currency and never collapsed into one number.** 912 of the 1,269 customers with a ledger move in both MXN and USD, the charge side is MXN-only while receipts arrive in both, and the legacy data never stored the exchange rate applied to a movement — so a single "total balance" would be a figure that never existed in the books.
|
||||
7. Bank register module (`bank_transactions`/`business_line_categories` from SCOTHIA) — small, self-contained, and has no customer FK, so it can slot in independently once the core migration pipeline exists; low risk, low priority relative to the customer-facing modules.
|
||||
8. VPS provisioning + Tailscale + MySQL replication setup. `utility_dbo`'s schema is now available (full dump on disk — 55 tables; see Status), so the exact replicated table/column set and inbox-table shape can be finalized against the real portal DB and the portal PHP code (`my-jorgecuadros-web`) that reads/writes it.
|
||||
9. Sync worker (push replicated tables' relevant subset, poll inbox tables for payment/propane submissions) — depends on step 8. Portal write points confirmed present in `utility_dbo`: `peticion_gas` (propane requests), PayPal payment writes, `notifications_settings`, `verification_codes` — these define the VPS→internal inbox set.
|
||||
|
||||
Reference in New Issue
Block a user