jorgecuadros-platform/migration/RECONCILIATION.md

Reconciliation Report — Overlapping Legacy Tables

Migration plan step 2. Generated by reconcile.py from staged Parquet (output/stg_utilities). Regenerate: ./.venv/bin/python reconcile.py > RECONCILIATION.md.

Headline: none of the three suspected "duplicate" groups are what the plan assumed. They are disjoint historical/period data or a mislabeled batch — see each group's decided rule.

1. Cash ledger — EFECTIVO vs EFECTIVO_BACKUP

• EFECTIVO: 13697 rows. EFECTIVO_BACKUP: 12387 rows.
• folio is per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio numbers appear in both.
• But of those 12363 shared folio numbers, 12363 carry a different transaction (differ on ['cl', 'fecha', 'monto', 'conepto']). → folio collides; it is NOT a stable cross-table id.
• On the real business key ['cl', 'fecha', 'monto', 'conepto']: 2 rows in both, 13663 only in EFECTIVO, 12385 only in BACKUP.
• Date ranges are different eras: BACKUP is dominated by 2017–2022 records, EFECTIVO by recent ones.

Decided rule: the two tables are near-disjoint ledgers, not a live/backup duplicate pair (only 2 shared payments out of ~13k+12k). Migrate both into transactions, each row keyed internally by (legacy_source_table, folio) provenance — do not de-dup on folio (it collides) and do not drop BACKUP (it holds ~12k older payments absent from EFECTIVO). The 2 business-key matches are the only possible double-counts and should be spot-checked, but at that volume they don't threaten balance integrity.

EFECTIVO FM3 (627) and CHEQUE FM3 (157) are a separate stream — fee/tax/multa columns instead of monto — and migrate as distinct transactions, not reconciled against EFECTIVO.

Data-quality note for the transform: monedas has case/spelling variants (PESOS/Pesos, DOLARES/Dolares/DOLLARS) — normalize currency on load.

2. Billing/fee logs — datos2 vs FEE ANUAL vs fee15

• Rows: datos2 16000, FEE ANUAL 1082, fee15 1030.
• date spans: datos2 ('2025-01-08 00:00:00', '2026-05-15 00:00:00'), FEE ANUAL ('2018-01-03 00:00:00', '2018-01-03 00:00:00'), fee15 ('2017-01-10 00:00:00', '2017-01-10 00:00:00') — three different periods.
• FEE ANUAL.refer and fee15.refer are each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.
• Real-identity overlap ['numid', 'date', 'chargecredit']: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.

Decided rule: datos2, FEE ANUAL, and fee15 are disjoint historical billing runs from different periods (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate all three into the billing side of transactions, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve datos2.due_date (the other two lack it — leave null for their rows).

3. Customer master — DATGRAL vs COBRO3

• DATGRAL 1172 rows / COBRO3 181 rows; both unique on num_id.
• Every COBRO3.num_id is in DATGRAL (181/181); it adds no new customer id.
• COBRO3.fee is constant (1 distinct value: ['75']) while DATGRAL.fee varies per customer — so COBRO3 is not a snapshot of the master's fee field.
• Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved charge worklist, not the authoritative customer record.

Decided rule: COBRO3 is a billing/charge batch ('cobro' = collection), not a customer table. DATGRAL is the sole utilities customer master. Do not merge COBRO3 into customers or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge transactions (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.