rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(migration): de-duplicate EFECTIVO_BACKUP against EFECTIVO

Browse Source 
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.

reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.

Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.

- reconcile.py: add canon(), which parses a key column to a number when
  nearly every populated cell parses and re-emits it at fixed precision.
  Applied in keyset() and in the folio-conflict comparison. Rewrite the
  group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
  two efectivo_like() calls. EFECTIVO loads first and wins collisions.
  De-dup on the business key, never on folio — folio is per-table
  sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
  their verdicts are unchanged.

Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 23:28:18 -07:00
co-authored by Claude Opus 4.8
parent 61193586a5
commit 9de8e4e6c0
3 changed files with 105 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
migration/transform_transactions.py
+41 -6
View File
@@ -4,7 +4,12 @@ Migration plan step 3 (shared ledger): unify every cash/billing ledger into
Runs AFTER transform_customers.py (customer FK is required).
Union rules come from the reconciliation pass (RECONCILIATION.md):
- EFECTIVO + EFECTIVO_BACKUP -> both (near-disjoint ledgers; no folio de-dup)
- EFECTIVO + EFECTIVO_BACKUP -> EFECTIVO in full, plus only the BACKUP rows
whose business key is absent from EFECTIVO
(BACKUP is a stale copy: 12386 of its 12387
rows are verbatim duplicates). De-dup on the
business key, never on folio (folio collides
across the two tables).
- datos2 + FEE ANUAL + fee15 -> all three (disjoint period runs; no de-dup)
- EFECTIVO FM3 / CHEQUE FM3 -> distinct fee stream (amount = fee+tax+multa)
- IVA 2015 -> its own snapshot (no date column -> nominal)
@@ -117,7 +122,7 @@ def main():
# --- transactions ---
tx = []
skip_cust = skip_date = 0
skip_cust = skip_date = skip_dupe = 0
def add(cid, domain, tdate, amount, currency, *, period=None, reference=None,
typeid=None, check=None, message=None, src_db=None, src_tbl=None, legacy=None):
@@ -125,10 +130,33 @@ def main():
amount if amount is not None else Decimal(0), currency, None, check,
message, 0, src_db, src_tbl, legacy))
def efectivo_like(src, name, domain, custmap, src_db, legacy_tbl):
nonlocal skip_cust, skip_date
# Business key of a real cash payment. `folio` is deliberately excluded: it
# is a per-table sequential number that collides between EFECTIVO and
# EFECTIVO_BACKUP (12363 shared numbers, 12204 of them on different
# payments), so it identifies nothing across tables.
def biz_key(r):
return (
norm_id(r["cl"]),
s(r["fecha"]),
dec(r["monto"], Decimal(0)),
s(r["conepto"]),
)
def efectivo_like(src, name, domain, custmap, src_db, legacy_tbl, *, seen=None):
"""Load an EFECTIVO-shaped cash ledger.
`seen` (a set) makes the load de-duplicating: keys are added to it as
rows load, and a row whose key is already present is skipped. That is
how EFECTIVO_BACKUP contributes only its genuinely-new rows.
"""
nonlocal skip_cust, skip_date, skip_dupe
df = load(src, name)
for _, r in df.iterrows():
if seen is not None:
key = biz_key(r)
if key in seen:
skip_dupe += 1; continue
seen.add(key)
cid = custmap.get(norm_id(r["cl"]))
if not cid:
skip_cust += 1; continue
@@ -181,8 +209,14 @@ def main():
reference=s(r["recibo"]), message="IVA 2015",
src_db="UTILITIES", src_tbl="IVA 2015", legacy=str(int(r["_row_num"])))
efectivo_like("stg_utilities", "efectivo", "UTILITY", util_cust, "UTILITIES", "EFECTIVO")
efectivo_like("stg_utilities", "efectivo_backup", "UTILITY", util_cust, "UTILITIES", "EFECTIVO_BACKUP")
# Shared across both calls so BACKUP is de-duplicated against EFECTIVO —
# order matters: EFECTIVO is the live table and loads first, so a collision
# always resolves in its favour.
cash_seen: set = set()
efectivo_like("stg_utilities", "efectivo", "UTILITY", util_cust, "UTILITIES",
"EFECTIVO", seen=cash_seen)
efectivo_like("stg_utilities", "efectivo_backup", "UTILITY", util_cust, "UTILITIES",
"EFECTIVO_BACKUP", seen=cash_seen)
fm3("efectivo_fm3", "EFECTIVO FM3")
fm3("cheque_fm3", "CHEQUE FM3", check_col="num_cheque")
billing("datos2", "datos2")
@@ -217,6 +251,7 @@ def main():
print("=== Transactions load complete ===")
print(f" skipped (unresolved customer): {skip_cust}")
print(f" skipped (unparseable date) : {skip_date}")
print(f" skipped (EFECTIVO_BACKUP dup): {skip_dupe}")
print(f" -> transactions : {count('transactions')}")
print(f" by domain : {dict(by_dom)}")
for src, n in by_src: