rmancinasandClaude Opus 4.8
81f62430b3
Add migration/reconcile.py (reads staged Parquet) and the generated migration/RECONCILIATION.md. Naive full-row matching across the suspected "duplicate" groups gives a misleading ~0 overlap, so each group is probed on a deliberate business key instead. The evidence overturns all three of the plan's original assumptions: - EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table sequential number that collides (12,363 shared folio numbers, every one a different transaction); real business-key (cl,fecha,monto,conepto) overlap is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule: migrate both, keyed by (source_table, folio) provenance, no folio de-dup, don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream. - datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods (2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three, no de-dup; keep datos2.due_date. - DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL is the sole customer master, COBRO3 contributes zero customers. Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization at transform time. Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and RESUME.md (queue: reconciliation done, transform+load next). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
39 lines
3.9 KiB
Markdown
39 lines
3.9 KiB
Markdown
# Reconciliation Report — Overlapping Legacy Tables
|
||
|
||
_Migration plan step 2. Generated by `reconcile.py` from staged Parquet (`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py > RECONCILIATION.md`._
|
||
|
||
**Headline:** none of the three suspected "duplicate" groups are what the plan assumed. They are disjoint historical/period data or a mislabeled batch — see each group's decided rule.
|
||
|
||
## 1. Cash ledger — `EFECTIVO` vs `EFECTIVO_BACKUP`
|
||
|
||
- `EFECTIVO`: 13697 rows. `EFECTIVO_BACKUP`: 12387 rows.
|
||
- `folio` is per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio *numbers* appear in both.
|
||
- **But of those 12363 shared folio numbers, 12363 carry a *different* transaction** (differ on ['cl', 'fecha', 'monto', 'conepto']). → `folio` collides; it is NOT a stable cross-table id.
|
||
- On the real business key `['cl', 'fecha', 'monto', 'conepto']`: **2 rows in both**, 13663 only in EFECTIVO, 12385 only in BACKUP.
|
||
- Date ranges are different eras: BACKUP is dominated by 2017–2022 records, EFECTIVO by recent ones.
|
||
|
||
**Decided rule:** the two tables are **near-disjoint ledgers**, not a live/backup duplicate pair (only 2 shared payments out of ~13k+12k). Migrate **both** into `transactions`, each row keyed internally by `(legacy_source_table, folio)` provenance — do **not** de-dup on `folio` (it collides) and do **not** drop BACKUP (it holds ~12k older payments absent from EFECTIVO). The 2 business-key matches are the only possible double-counts and should be spot-checked, but at that volume they don't threaten balance integrity.
|
||
|
||
> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — `fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct transactions, not reconciled against EFECTIVO.
|
||
|
||
> Data-quality note for the transform: `monedas` has case/spelling variants (`PESOS`/`Pesos`, `DOLARES`/`Dolares`/`DOLLARS`) — normalize currency on load.
|
||
|
||
## 2. Billing/fee logs — `datos2` vs `FEE ANUAL` vs `fee15`
|
||
|
||
- Rows: `datos2` 16000, `FEE ANUAL` 1082, `fee15` 1030.
|
||
- `date` spans: `datos2` ('2025-01-08 00:00:00', '2026-05-15 00:00:00'), `FEE ANUAL` ('2018-01-03 00:00:00', '2018-01-03 00:00:00'), `fee15` ('2017-01-10 00:00:00', '2017-01-10 00:00:00') — **three different periods**.
|
||
- `FEE ANUAL.refer` and `fee15.refer` are each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.
|
||
- Real-identity overlap `['numid', 'date', 'chargecredit']`: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.
|
||
|
||
**Decided rule:** `datos2`, `FEE ANUAL`, and `fee15` are **disjoint historical billing runs from different periods** (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate **all three** into the billing side of `transactions`, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve `datos2.due_date` (the other two lack it — leave null for their rows).
|
||
|
||
## 3. Customer master — `DATGRAL` vs `COBRO3`
|
||
|
||
- `DATGRAL` 1172 rows / `COBRO3` 181 rows; both unique on `num_id`.
|
||
- Every `COBRO3.num_id` is in `DATGRAL` (181/181); it adds **no new customer id**.
|
||
- **`COBRO3.fee` is constant** (1 distinct value: ['75']) while `DATGRAL.fee` varies per customer — so COBRO3 is not a snapshot of the master's fee field.
|
||
- Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved *charge worklist*, not the authoritative customer record.
|
||
|
||
**Decided rule:** `COBRO3` is a **billing/charge batch** ('cobro' = collection), not a customer table. `DATGRAL` is the sole utilities customer master. **Do not** merge COBRO3 into `customers` or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge `transactions` (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.
|
||
|