rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reconciliation pass (migration step 2): decide union/de-dup rules

Browse Source 
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:

- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
  sequential number that collides (12,363 shared folio numbers, every one a
  different transaction); real business-key (cl,fecha,monto,conepto) overlap
  is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
  migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
  don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
  (2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
  no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
  filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
  is the sole customer master, COBRO3 contributes zero customers.

Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.

Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 17:23:33 -07:00
co-authored by Claude Opus 4.8
parent dc9866c24a
commit 81f62430b3
4 changed files with 238 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESUME.md
+10 -4
View File
@@ -164,10 +164,16 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
row (mdb-export's `-b` / error handling; confirm the bad row is skipped, not fatal).
2. **Re-run staging** (`python load_staging.py --output-dir ./output`) to regenerate the staged
data on this machine, then load into a local MySQL (`docker compose up mysql`) for SQL reconciliation.
3. **Reconciliation pass** (plan step 2) against staged data — the `EFECTIVO*` variants, the
billing-period tables, `DATGRAL` vs `COBRO3`. Produce diff/dedupe rules from the data.
4. **Transform + load** (plan step 3), starting with `Customer`/`CustomerLegacyRef` — every
other module depends on it. Use the `NUM UTIL` cross-ref + name/address matching.
3. **Reconciliation pass** (plan step 2) — **DONE** (`migration/reconcile.py` → `RECONCILIATION.md`).
Overturned all three "duplicate" assumptions: EFECTIVO/BACKUP are near-disjoint ledgers
(folio collides; migrate both), the billing tables are disjoint period runs (union all, no
de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The
decided union/de-dup rules are in `PLAN.md` migration step 2.
4. **Transform + load** (plan step 3) — NEXT. Start with `Customer`/`CustomerLegacyRef` (every
other module depends on it): DATGRAL (utilities) is the master; join insurance `DATGRAL`
via its `NUM UTIL` cross-ref + name/address matching; COBRO3 excluded from customers. Then
the ledger union per the reconciliation rules (both EFECTIVO tables, all three billing tables,
provenance-keyed; normalize `monedas` currency variants).
5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the