rmancinasandClaude Opus 4.8
9de8e4e6c0
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint ledgers" and the transform loaded both in full. That verdict was a bug, not a finding. reconcile.py compared the business key (cl, fecha, monto, conepto) as raw strings, on the stated premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: mdb-export formats a numeric column from its Access column type, so the same amount is emitted as `5000` from one table and `27000.0000` from the other. No two rows could ever match on `monto`, which is why the pass reported 2 overlapping rows. Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's 12387 rows already exist verbatim in EFECTIVO — same customer, same timestamp to the second, same amount, same concept text — leaving exactly one genuinely new row. The ledger was carrying 12386 duplicated payments, roughly doubling every customer's historical receipt total. - reconcile.py: add canon(), which parses a key column to a number when nearly every populated cell parses and re-emits it at fixed precision. Applied in keyset() and in the folio-conflict comparison. Rewrite the group-1 verdict and the module docstring's method note. - transform_transactions.py: share a business-key `seen` set between the two efectivo_like() calls. EFECTIVO loads first and wins collisions. De-dup on the business key, never on folio — folio is per-table sequential and collides on 12204 different payments. - Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix; their verdicts are unchanged. Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
41 lines
4.4 KiB
Markdown
41 lines
4.4 KiB
Markdown
# Reconciliation Report — Overlapping Legacy Tables
|
||
|
||
_Migration plan step 2. Generated by `reconcile.py` from staged Parquet (`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py > RECONCILIATION.md`._
|
||
|
||
**Headline:** `EFECTIVO_BACKUP` *is* a duplicate of `EFECTIVO` and must not be double-loaded; the billing tables are genuinely disjoint period runs; `COBRO3` is a mislabeled charge batch, not a customer master. See each group's decided rule.
|
||
|
||
## 1. Cash ledger — `EFECTIVO` vs `EFECTIVO_BACKUP`
|
||
|
||
- `EFECTIVO`: 13697 rows. `EFECTIVO_BACKUP`: 12387 rows.
|
||
- `folio` is per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio *numbers* appear in both.
|
||
- **But of those 12363 shared folio numbers, 12204 carry a *different* transaction** (differ on ['cl', 'fecha', 'monto', 'conepto']). → `folio` collides; it is NOT a stable cross-table id, and it cannot be the de-dup key.
|
||
- On the real business key `['cl', 'fecha', 'monto', 'conepto']`, canonicalized: **12386 rows in both**, 1279 only in EFECTIVO, 1 only in BACKUP — i.e. all but 1 of BACKUP's 12387 rows already exist verbatim in EFECTIVO, same customer, same timestamp to the second, same amount, same concept text.
|
||
- Yearly row counts track each other almost exactly from 2006 to 2023 (e.g. 2017: 1036 vs 994), which is what a stale copy looks like — not two ledgers covering different eras.
|
||
|
||
**Decided rule (corrected):** `EFECTIVO_BACKUP` is a **stale backup copy of `EFECTIVO`**, not an independent ledger. Load `EFECTIVO` in full, and load from `EFECTIVO_BACKUP` only the rows whose canonicalized business key is absent from `EFECTIVO` (1 row(s)). Loading both in full double-counts 12386 payments and doubles nearly every customer's historical receipt total, which makes any statement or balance view wrong. De-dup on the business key, **not** on `folio` (it collides).
|
||
|
||
> This reverses the original verdict in this report, which read `{b}` as 2. That number came from string-comparing `monto`, which `mdb-export` serializes with a different precision per table (`5000` vs `27000.0000`) — see the module docstring.
|
||
|
||
> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — `fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct transactions, not reconciled against EFECTIVO.
|
||
|
||
> Data-quality note for the transform: `monedas` has case/spelling variants (`PESOS`/`Pesos`, `DOLARES`/`Dolares`/`DOLLARS`) — normalize currency on load.
|
||
|
||
## 2. Billing/fee logs — `datos2` vs `FEE ANUAL` vs `fee15`
|
||
|
||
- Rows: `datos2` 16000, `FEE ANUAL` 1082, `fee15` 1030.
|
||
- `date` spans: `datos2` ('2025-01-08 00:00:00', '2026-05-15 00:00:00'), `FEE ANUAL` ('2018-01-03 00:00:00', '2018-01-03 00:00:00'), `fee15` ('2017-01-10 00:00:00', '2017-01-10 00:00:00') — **three different periods**.
|
||
- `FEE ANUAL.refer` and `fee15.refer` are each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.
|
||
- Real-identity overlap `['numid', 'date', 'chargecredit']`: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.
|
||
|
||
**Decided rule:** `datos2`, `FEE ANUAL`, and `fee15` are **disjoint historical billing runs from different periods** (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate **all three** into the billing side of `transactions`, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve `datos2.due_date` (the other two lack it — leave null for their rows).
|
||
|
||
## 3. Customer master — `DATGRAL` vs `COBRO3`
|
||
|
||
- `DATGRAL` 1172 rows / `COBRO3` 181 rows; both unique on `num_id`.
|
||
- Every `COBRO3.num_id` is in `DATGRAL` (181/181); it adds **no new customer id**.
|
||
- **`COBRO3.fee` is constant** (1 distinct value: ['75']) while `DATGRAL.fee` varies per customer — so COBRO3 is not a snapshot of the master's fee field.
|
||
- Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved *charge worklist*, not the authoritative customer record.
|
||
|
||
**Decided rule:** `COBRO3` is a **billing/charge batch** ('cobro' = collection), not a customer table. `DATGRAL` is the sole utilities customer master. **Do not** merge COBRO3 into `customers` or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge `transactions` (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.
|
||
|