Transform+load: properties/services/trust + env-parameterize migration

migration/transform_properties.py loads properties, property_services and
trust_accounts from staged DATMEX/PROFILE, resolving each property's customer
FK through customer_legacy_refs. Services are derived from DATMEX's own
account/route/meter fields (the authoritative data); PROFILE flags — merged
best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only
refine each service's `active`. Trust accounts are 1:1 from DATMEX trust
fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4.

Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped),
3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 /
OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts —
counts track the PROFILE enrollment flags.

Reproducibility (asked: dev must be redoable in prod):
- migration/dbenv.py: single DB-target source = deploy/.env.<env>'s
  DATABASE_URL; connect(env) + env_arg() (--env, default dev).
- transform_customers.py / transform_properties.py now take --env instead of
  hardcoding .env.dev.
- migration/run_all.py: runs every step in dependency order against --env
  (optional --stage re-extracts from Access first). Reproducing dev->prod is
  `run_all.py --env prod` after deploying the prod stack + prisma db push.

All steps are idempotent (truncate+rebuild); re-run yields identical counts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 18:29:31 -07:00
co-authored by Claude Opus 4.8
parent ec499ca5f5
commit a680ad2bb0
4 changed files with 383 additions and 11 deletions
+5 -11
View File
@@ -31,9 +31,9 @@ from decimal import Decimal, InvalidOperation
from pathlib import Path
import pandas as pd
import pymysql
REPO = Path(__file__).resolve().parents[1]
from dbenv import connect, env_arg
STG = Path(__file__).parent / "output"
NULL = ""
NOW = datetime.now(timezone.utc).replace(tzinfo=None) # naive UTC for MySQL DATETIME
@@ -161,15 +161,9 @@ _CUST_COLS = [
def main() -> None:
envf = REPO / "deploy" / ".env.dev"
env = dict(
l.strip().split("=", 1) for l in envf.read_text().splitlines()
if "=" in l and not l.startswith("#") and not l.startswith("DATABASE_URL")
)
conn = pymysql.connect(
host="192.168.4.212", port=int(env["MYSQL_PORT"]), user=env["MYSQL_USER"],
password=env["MYSQL_PASSWORD"], database=env["MYSQL_DATABASE"], autocommit=False,
)
env = env_arg()
conn = connect(env)
print(f"[customers] target env: {env}")
cur = conn.cursor()
# Fresh, idempotent rebuild.