The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.
reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.
Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.
- reconcile.py: add canon(), which parses a key column to a number when
nearly every populated cell parses and re-emits it at fixed precision.
Applied in keyset() and in the folio-conflict comparison. Rewrite the
group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
two efectivo_like() calls. EFECTIVO loads first and wins collisions.
De-dup on the business key, never on folio — folio is per-table
sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
their verdicts are unchanged.
Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.
migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).
run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rmancinasandClaude Opus 4.8