The --sync path had never been run and was broken in several ways. Fixed and
verified against the dev DB (two consecutive syncs, both exit 0, 32/32
assertions: stable PKs, manual-row preservation, changed-row updates,
legacy-delete, no child duplication, zero FK orphans; idempotent).
- policies/properties: reuse each legacy row's existing id (by provenance)
BEFORE building child rows, so children no longer point at a discarded fresh
uuid; rebuild legacy-owned children via scoped delete + reinsert.
- customers: replace zip(customers, refs) (mispaired almost every row) with a
ref-grouped id remap; names now restore and no spurious customers appear.
- drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy
policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert.
- upsert lookup tables (policy_types, insurance_providers, type_transactions,
adjusters) by natural name and remap child FKs instead of inserting fresh
uuids that nothing points at.
- transactions: drop updatedAt=NOW() (no such column); guard report formatting
on NULL legacySourceTable (manual rows). Same report guard in bank.
- add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune
only legacy-owned empties, never manually-added customers.
web: customer-detail mini tx list now strikes voided rows with an "(anulado)"
tag (was the last void-UI rendering gap; /estado-cuenta already handled it).
docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI
browser pass recorded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
144 customers owned zero properties, zero policies and zero transactions —
the legacy DATGRAL row exists but nothing in either business line ever
attached to it. They padded the staff customer list with rows that can't be
acted on. 27 were also nameless (dead ID slots); the other 117 have real
names and sometimes contact details, and read as never-activated prospects
or lapsed clients rather than junk. Removing both sets is a deliberate call.
Implemented as a separate step rather than a filter inside
transform_customers.py: emptiness is only knowable after properties, policies
and transactions have loaded, and deciding it there would mean re-deriving
each downstream transform's source-matching logic against the staged Parquet.
Runs after transform_transactions.py in run_all.py.
Safe by construction — a customer with no rows in any of the three tables has
nothing pointing at it, so the delete cannot orphan anything; only its own
customer_legacy_refs go with it. The step asserts zero orphans afterwards.
Every pruned customer is written to output/pruned_customers.csv with its
legacy provenance before the delete, and --dry-run reports without touching
anything. Nothing is unrecoverable: the Access sources are untouched and a
pipeline run without this step brings them all back.
Verified: full run_all.py pass ends at 1538 customers (from 1682), with
1519 properties / 2378 policies / 45861 transactions all intact and zero
orphans. 17 nameless customers remain, all of which carry real records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rmancinasandClaude Opus 4.8