Reconciliation pass (migration step 2): decide union/de-dup rules

Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:

- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
  sequential number that collides (12,363 shared folio numbers, every one a
  different transaction); real business-key (cl,fecha,monto,conepto) overlap
  is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
  migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
  don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
  (2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
  no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
  filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
  is the sole customer master, COBRO3 contributes zero customers.

Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.

Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PLAN.md

@@ -34,10 +34,10 @@ Extracted live via `pyodbc` + the Windows Access ODBC driver (`dump_schema.py`,
 
 **UTILITIES.accdb** (52 tables, ~538MB, dominated by embedded document blobs):
 - `DATMEX` (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 `LONGBINARY` columns holding scanned utility bills/IDs.
-- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). `COBRO3` looks like a filtered snapshot of `DATGRAL`, not a distinct entity.
+- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). ~~`COBRO3` looks like a filtered snapshot of `DATGRAL`~~ **Reconciliation (step 2) corrected this: `COBRO3` is a charge batch (constant fee 75), not a customer snapshot — `DATGRAL` is the sole customer master.**
 - `PROFILE` (1,520) — per-property service enrollment flags, joins 1:1 with `DATMEX` by `NUMERID`.
-- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers, near-identical shape, apparent year/program snapshots rather than distinct data.
-- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs, again near-duplicate shapes across different periods/exports — needs de-duplication logic, not a straight union.
+- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers. ~~apparent year/program snapshots rather than distinct data~~ **Reconciliation (step 2) corrected this: EFECTIVO and EFECTIVO_BACKUP are near-disjoint ledgers (folio collides; only 2 business-key matches) — migrate both, no folio de-dup. FM3 tables are a separate fee stream.**
+- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs. ~~near-duplicate shapes ... needs de-duplication~~ **Reconciliation (step 2) corrected this: disjoint billing runs from different periods (2017 / 2018 / 2025–26), zero real-identity overlap — straight union, no de-dup.**
 - `TRUSTVENCE` (549) — bank trust account fee due-dates.
 - `TIPO HIST` (2,301) — exchange-rate history (date/hour/rate), referenced by the `MONEDAS` (currency) field used throughout.
 - `TYPE OF TRX` (79) — ES/EN transaction-type lookup (already mirrored as `type_transactions` in the old MySQL schema — reuse that mapping).
@@ -108,7 +108,10 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
 Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
 
 1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
-2. **Reconciliation pass**: for each set of overlapping tables (the `EFECTIVO` variants, the billing-period tables, `DATGRAL` vs `COBRO3`), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
+2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. **Outcome overturned all three of the plan's original "duplicate" assumptions — the union/de-dup rules below are now decided by the data:**
+   - **`EFECTIVO` vs `EFECTIVO_BACKUP`:** *not* a live/backup duplicate pair. `folio` is a per-table sequential number that **collides** (12,363 shared folio numbers, all carrying different transactions); on the real business key `(cl,fecha,monto,conepto)` only **2 rows** overlap. They are near-disjoint ledgers (BACKUP ≈ 2017–2022, EFECTIVO recent). **Rule: migrate both**, keyed internally by `(legacy_source_table, folio)` provenance; no folio de-dup, don't drop BACKUP. `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
+   - **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈2025–26, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
+   - **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
 3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
 4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
 5. **Validation**: row-count and spot-check reconciliation between `staging` and final tables (e.g., every legacy customer has exactly one `customers` row via `customer_legacy_refs`; sum of migrated transaction amounts per customer matches sum in `staging`).