rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(migration): de-duplicate EFECTIVO_BACKUP against EFECTIVO

Browse Source 
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.

reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.

Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.

- reconcile.py: add canon(), which parses a key column to a number when
  nearly every populated cell parses and re-emits it at fixed precision.
  Applied in keyset() and in the folio-conflict comparison. Rewrite the
  group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
  two efectivo_like() calls. EFECTIVO loads first and wins collisions.
  De-dup on the business key, never on folio — folio is per-table
  sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
  their verdicts are unchanged.

Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 23:28:18 -07:00
co-authored by Claude Opus 4.8
parent 61193586a5
commit 9de8e4e6c0
3 changed files with 105 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
migration/reconcile.py
+57 -21
View File
@@ -17,9 +17,17 @@ which is a trap — it hides *why*. The interesting question is whether two
tables hold the SAME economic records under cosmetic differences, or
genuinely DIFFERENT records. So each group is probed on a deliberate
*business key* (the columns that identify the real-world thing) and the
volatile/identity columns are examined separately. Every table went through
the same mdb-export path, so identical source values serialize identically —
string comparison on a chosen key is a valid identity test.
volatile/identity columns are examined separately.
Method correction (2026-07-22): this file previously assumed that "every table
went through the same mdb-export path, so identical source values serialize
identically". That is false. `mdb-export` formats a numeric column according to
its *Access column type*, so the same amount is emitted as `5000` from one table
and `27000.0000` from another. String-comparing a numeric key column therefore
reports near-zero overlap between tables holding identical records — which is
exactly what produced the original (wrong) "EFECTIVO / EFECTIVO_BACKUP are
near-disjoint ledgers" verdict. Every key column is now canonicalized (numeric
columns parsed and re-formatted to a fixed precision) before comparison.
"""
from __future__ import annotations
@@ -41,8 +49,26 @@ def load(name: str) -> pd.DataFrame:
return df
def canon(sr: pd.Series) -> pd.Series:
"""Canonicalize one key column so it compares across tables.
A column that parses as a number in (nearly) every populated cell is
re-emitted at fixed precision, which erases the per-table formatting
mdb-export applies from the Access column type. Anything else (dates,
free text) is left as the already-stripped string.
"""
populated = sr.ne(_NULL)
if not populated.any():
return sr
num = pd.to_numeric(sr.where(populated), errors="coerce")
if num.notna().sum() < 0.9 * populated.sum():
return sr
return num.map(lambda v: _NULL if pd.isna(v) else f"{v:.4f}").astype("string")
def keyset(df: pd.DataFrame, cols: list[str]) -> set[str]:
return set(df[cols].agg("\x1f".join, axis=1))
keyed = pd.DataFrame({c: canon(df[c]) for c in cols})
return set(keyed.agg("\x1f".join, axis=1))
def overlap(a: pd.DataFrame, b: pd.DataFrame, cols: list[str]):
@@ -65,9 +91,10 @@ def main() -> None:
"(`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py "
"> RECONCILIATION.md`._")
p("")
p("**Headline:** none of the three suspected \"duplicate\" groups are what "
"the plan assumed. They are disjoint historical/period data or a "
"mislabeled batch — see each group's decided rule.")
p("**Headline:** `EFECTIVO_BACKUP` *is* a duplicate of `EFECTIVO` and must "
"not be double-loaded; the billing tables are genuinely disjoint period "
"runs; `COBRO3` is a mislabeled charge batch, not a customer master. See "
"each group's decided rule.")
p("")
# ------------------------------------------------------------------ #
@@ -88,24 +115,33 @@ def main() -> None:
m = ef[ef['folio'].isin(both_folio)].drop_duplicates('folio').set_index('folio')
n = efb[efb['folio'].isin(both_folio)].drop_duplicates('folio').set_index('folio')
ci = m.index.intersection(n.index)
folio_conflict = int((m.loc[ci, biz] != n.loc[ci, biz]).any(axis=1).sum())
mk = pd.DataFrame({c: canon(m.loc[ci, c]) for c in biz})
nk = pd.DataFrame({c: canon(n.loc[ci, c]) for c in biz})
folio_conflict = int((mk != nk).any(axis=1).sum())
p(f"- **But of those {len(ci)} shared folio numbers, {folio_conflict} carry "
f"a *different* transaction** (differ on {biz}). → `folio` collides; it is "
"NOT a stable cross-table id.")
"NOT a stable cross-table id, and it cannot be the de-dup key.")
b, oa, ob = overlap(ef, efb, biz)
p(f"- On the real business key `{biz}`: **{b} rows in both**, {oa} only in "
f"EFECTIVO, {ob} only in BACKUP.")
p("- Date ranges are different eras: BACKUP is dominated by 2017–2022 "
"records, EFECTIVO by recent ones.")
p(f"- On the real business key `{biz}`, canonicalized: **{b} rows in both**, "
f"{oa} only in EFECTIVO, {ob} only in BACKUP — i.e. all but {ob} of "
f"BACKUP's {len(efb)} rows already exist verbatim in EFECTIVO, same "
"customer, same timestamp to the second, same amount, same concept text.")
p("- Yearly row counts track each other almost exactly from 2006 to 2023 "
"(e.g. 2017: 1036 vs 994), which is what a stale copy looks like — not "
"two ledgers covering different eras.")
p("")
p("**Decided rule:** the two tables are **near-disjoint ledgers**, not a "
"live/backup duplicate pair (only " + str(b) + " shared payments out of "
"~13k+12k). Migrate **both** into `transactions`, each row keyed "
"internally by `(legacy_source_table, folio)` provenance — do **not** "
"de-dup on `folio` (it collides) and do **not** drop BACKUP (it holds "
"~12k older payments absent from EFECTIVO). The " + str(b) + " business-"
"key matches are the only possible double-counts and should be spot-"
"checked, but at that volume they don't threaten balance integrity.")
p("**Decided rule (corrected):** `EFECTIVO_BACKUP` is a **stale backup copy "
"of `EFECTIVO`**, not an independent ledger. Load `EFECTIVO` in full, and "
"load from `EFECTIVO_BACKUP` only the rows whose canonicalized business "
f"key is absent from `EFECTIVO` ({ob} row(s)). Loading both in full "
f"double-counts {b} payments and doubles nearly every customer's "
"historical receipt total, which makes any statement or balance view "
"wrong. De-dup on the business key, **not** on `folio` (it collides).")
p("")
p("> This reverses the original verdict in this report, which put that "
"overlap at 2. That number came from string-comparing `monto`, which `mdb-export` "
"serializes with a different precision per table (`5000` vs "
"`27000.0000`) — see the module docstring.")
p("")
p("> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — "
"`fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct "