144 customers owned zero properties, zero policies and zero transactions —
the legacy DATGRAL row exists but nothing in either business line ever
attached to it. They padded the staff customer list with rows that can't be
acted on. 27 were also nameless (dead ID slots); the other 117 have real
names and sometimes contact details, and read as never-activated prospects
or lapsed clients rather than junk. Removing both sets is a deliberate call.
Implemented as a separate step rather than a filter inside
transform_customers.py: emptiness is only knowable after properties, policies
and transactions have loaded, and deciding it there would mean re-deriving
each downstream transform's source-matching logic against the staged Parquet.
Runs after transform_transactions.py in run_all.py.
Safe by construction — a customer with no rows in any of the three tables has
nothing pointing at it, so the delete cannot orphan anything; only its own
customer_legacy_refs go with it. The step asserts zero orphans afterwards.
Every pruned customer is written to output/pruned_customers.csv with its
legacy provenance before the delete, and --dry-run reports without touching
anything. Nothing is unrecoverable: the Access sources are untouched and a
pipeline run without this step brings them all back.
Verified: full run_all.py pass ends at 1538 customers (from 1682), with
1519 properties / 2378 policies / 45861 transactions all intact and zero
orphans. 17 nameless customers remain, all of which carry real records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rmancinasandClaude Opus 4.8
fa9b696752