rmancinasandClaude Opus 4.8
9de8e4e6c0
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint ledgers" and the transform loaded both in full. That verdict was a bug, not a finding. reconcile.py compared the business key (cl, fecha, monto, conepto) as raw strings, on the stated premise that "every table went through the same mdb-export path, so identical source values serialize identically". They don't: mdb-export formats a numeric column from its Access column type, so the same amount is emitted as `5000` from one table and `27000.0000` from the other. No two rows could ever match on `monto`, which is why the pass reported 2 overlapping rows. Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's 12387 rows already exist verbatim in EFECTIVO — same customer, same timestamp to the second, same amount, same concept text — leaving exactly one genuinely new row. The ledger was carrying 12386 duplicated payments, roughly doubling every customer's historical receipt total. - reconcile.py: add canon(), which parses a key column to a number when nearly every populated cell parses and re-emits it at fixed precision. Applied in keyset() and in the folio-conflict comparison. Rewrite the group-1 verdict and the module docstring's method note. - transform_transactions.py: share a business-key `seen` set between the two efectivo_like() calls. EFECTIVO loads first and wins collisions. De-dup on the business key, never on folio — folio is per-table sequential and collides on 12204 different payments. - Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix; their verdicts are unchanged. Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4.4 KiB
Reconciliation Report — Overlapping Legacy Tables
Migration plan step 2. Generated by reconcile.py from staged Parquet (output/stg_utilities). Regenerate: ./.venv/bin/python reconcile.py > RECONCILIATION.md.
Headline: EFECTIVO_BACKUP is a duplicate of EFECTIVO and must not be double-loaded; the billing tables are genuinely disjoint period runs; COBRO3 is a mislabeled charge batch, not a customer master. See each group's decided rule.
1. Cash ledger — EFECTIVO vs EFECTIVO_BACKUP
EFECTIVO: 13697 rows.EFECTIVO_BACKUP: 12387 rows.foliois per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio numbers appear in both.- But of those 12363 shared folio numbers, 12204 carry a different transaction (differ on ['cl', 'fecha', 'monto', 'conepto']). →
foliocollides; it is NOT a stable cross-table id, and it cannot be the de-dup key. - On the real business key
['cl', 'fecha', 'monto', 'conepto'], canonicalized: 12386 rows in both, 1279 only in EFECTIVO, 1 only in BACKUP — i.e. all but 1 of BACKUP's 12387 rows already exist verbatim in EFECTIVO, same customer, same timestamp to the second, same amount, same concept text. - Yearly row counts track each other almost exactly from 2006 to 2023 (e.g. 2017: 1036 vs 994), which is what a stale copy looks like — not two ledgers covering different eras.
Decided rule (corrected): EFECTIVO_BACKUP is a stale backup copy of EFECTIVO, not an independent ledger. Load EFECTIVO in full, and load from EFECTIVO_BACKUP only the rows whose canonicalized business key is absent from EFECTIVO (1 row(s)). Loading both in full double-counts 12386 payments and doubles nearly every customer's historical receipt total, which makes any statement or balance view wrong. De-dup on the business key, not on folio (it collides).
This reverses the original verdict in this report, which read
{b}as 2. That number came from string-comparingmonto, whichmdb-exportserializes with a different precision per table (5000vs27000.0000) — see the module docstring.
EFECTIVO FM3(627) andCHEQUE FM3(157) are a separate stream —fee/tax/multacolumns instead ofmonto— and migrate as distinct transactions, not reconciled against EFECTIVO.
Data-quality note for the transform:
monedashas case/spelling variants (PESOS/Pesos,DOLARES/Dolares/DOLLARS) — normalize currency on load.
2. Billing/fee logs — datos2 vs FEE ANUAL vs fee15
- Rows:
datos216000,FEE ANUAL1082,fee151030. datespans:datos2('2025-01-08 00:00:00', '2026-05-15 00:00:00'),FEE ANUAL('2018-01-03 00:00:00', '2018-01-03 00:00:00'),fee15('2017-01-10 00:00:00', '2017-01-10 00:00:00') — three different periods.FEE ANUAL.referandfee15.referare each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.- Real-identity overlap
['numid', 'date', 'chargecredit']: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.
Decided rule: datos2, FEE ANUAL, and fee15 are disjoint historical billing runs from different periods (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate all three into the billing side of transactions, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve datos2.due_date (the other two lack it — leave null for their rows).
3. Customer master — DATGRAL vs COBRO3
DATGRAL1172 rows /COBRO3181 rows; both unique onnum_id.- Every
COBRO3.num_idis inDATGRAL(181/181); it adds no new customer id. COBRO3.feeis constant (1 distinct value: ['75']) whileDATGRAL.feevaries per customer — so COBRO3 is not a snapshot of the master's fee field.- Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved charge worklist, not the authoritative customer record.
Decided rule: COBRO3 is a billing/charge batch ('cobro' = collection), not a customer table. DATGRAL is the sole utilities customer master. Do not merge COBRO3 into customers or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge transactions (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.