rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reconciliation pass (migration step 2): decide union/de-dup rules

Browse Source 
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:

- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
  sequential number that collides (12,363 shared folio numbers, every one a
  different transaction); real business-key (cl,fecha,monto,conepto) overlap
  is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
  migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
  don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
  (2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
  no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
  filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
  is the sole customer master, COBRO3 contributes zero customers.

Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.

Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 17:23:33 -07:00
co-authored by Claude Opus 4.8
parent dc9866c24a
commit 81f62430b3
4 changed files with 238 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
migration/RECONCILIATION.md
+38
View File
@@ -0,0 +1,38 @@
# Reconciliation Report — Overlapping Legacy Tables
_Migration plan step 2. Generated by `reconcile.py` from staged Parquet (`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py > RECONCILIATION.md`._
**Headline:** none of the three suspected "duplicate" groups are what the plan assumed. They are disjoint historical/period data or a mislabeled batch — see each group's decided rule.
## 1. Cash ledger — `EFECTIVO` vs `EFECTIVO_BACKUP`
- `EFECTIVO`: 13697 rows. `EFECTIVO_BACKUP`: 12387 rows.
- `folio` is per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio *numbers* appear in both.
- **But of those 12363 shared folio numbers, 12363 carry a *different* transaction** (differ on ['cl', 'fecha', 'monto', 'conepto']). → `folio` collides; it is NOT a stable cross-table id.
- On the real business key `['cl', 'fecha', 'monto', 'conepto']`: **2 rows in both**, 13663 only in EFECTIVO, 12385 only in BACKUP.
- Date ranges are different eras: BACKUP is dominated by 2017–2022 records, EFECTIVO by recent ones.
**Decided rule:** the two tables are **near-disjoint ledgers**, not a live/backup duplicate pair (only 2 shared payments out of ~13k+12k). Migrate **both** into `transactions`, each row keyed internally by `(legacy_source_table, folio)` provenance — do **not** de-dup on `folio` (it collides) and do **not** drop BACKUP (it holds ~12k older payments absent from EFECTIVO). The 2 business-key matches are the only possible double-counts and should be spot-checked, but at that volume they don't threaten balance integrity.
> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — `fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct transactions, not reconciled against EFECTIVO.
> Data-quality note for the transform: `monedas` has case/spelling variants (`PESOS`/`Pesos`, `DOLARES`/`Dolares`/`DOLLARS`) — normalize currency on load.
## 2. Billing/fee logs — `datos2` vs `FEE ANUAL` vs `fee15`
- Rows: `datos2` 16000, `FEE ANUAL` 1082, `fee15` 1030.
- `date` spans: `datos2` ('2025-01-08 00:00:00', '2026-05-15 00:00:00'), `FEE ANUAL` ('2018-01-03 00:00:00', '2018-01-03 00:00:00'), `fee15` ('2017-01-10 00:00:00', '2017-01-10 00:00:00') — **three different periods**.
- `FEE ANUAL.refer` and `fee15.refer` are each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.
- Real-identity overlap `['numid', 'date', 'chargecredit']`: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.
**Decided rule:** `datos2`, `FEE ANUAL`, and `fee15` are **disjoint historical billing runs from different periods** (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate **all three** into the billing side of `transactions`, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve `datos2.due_date` (the other two lack it — leave null for their rows).
## 3. Customer master — `DATGRAL` vs `COBRO3`
- `DATGRAL` 1172 rows / `COBRO3` 181 rows; both unique on `num_id`.
- Every `COBRO3.num_id` is in `DATGRAL` (181/181); it adds **no new customer id**.
- **`COBRO3.fee` is constant** (1 distinct value: ['75']) while `DATGRAL.fee` varies per customer — so COBRO3 is not a snapshot of the master's fee field.
- Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved *charge worklist*, not the authoritative customer record.
**Decided rule:** `COBRO3` is a **billing/charge batch** ('cobro' = collection), not a customer table. `DATGRAL` is the sole utilities customer master. **Do not** merge COBRO3 into `customers` or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge `transactions` (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.