Transform+load: properties/services/trust + env-parameterize migration
migration/transform_properties.py loads properties, property_services and trust_accounts from staged DATMEX/PROFILE, resolving each property's customer FK through customer_legacy_refs. Services are derived from DATMEX's own account/route/meter fields (the authoritative data); PROFILE flags — merged best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only refine each service's `active`. Trust accounts are 1:1 from DATMEX trust fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4. Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped), 3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 / OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts — counts track the PROFILE enrollment flags. Reproducibility (asked: dev must be redoable in prod): - migration/dbenv.py: single DB-target source = deploy/.env.<env>'s DATABASE_URL; connect(env) + env_arg() (--env, default dev). - transform_customers.py / transform_properties.py now take --env instead of hardcoding .env.dev. - migration/run_all.py: runs every step in dependency order against --env (optional --stage re-extracts from Access first). Reproducing dev->prod is `run_all.py --env prod` after deploying the prod stack + prisma db push. All steps are idempotent (truncate+rebuild); re-run yields identical counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -31,9 +31,9 @@ from decimal import Decimal, InvalidOperation
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import pymysql
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
from dbenv import connect, env_arg
|
||||
|
||||
STG = Path(__file__).parent / "output"
|
||||
NULL = "∅"
|
||||
NOW = datetime.now(timezone.utc).replace(tzinfo=None) # naive UTC for MySQL DATETIME
|
||||
@@ -161,15 +161,9 @@ _CUST_COLS = [
|
||||
|
||||
|
||||
def main() -> None:
|
||||
envf = REPO / "deploy" / ".env.dev"
|
||||
env = dict(
|
||||
l.strip().split("=", 1) for l in envf.read_text().splitlines()
|
||||
if "=" in l and not l.startswith("#") and not l.startswith("DATABASE_URL")
|
||||
)
|
||||
conn = pymysql.connect(
|
||||
host="192.168.4.212", port=int(env["MYSQL_PORT"]), user=env["MYSQL_USER"],
|
||||
password=env["MYSQL_PASSWORD"], database=env["MYSQL_DATABASE"], autocommit=False,
|
||||
)
|
||||
env = env_arg()
|
||||
conn = connect(env)
|
||||
print(f"[customers] target env: {env}")
|
||||
cur = conn.cursor()
|
||||
|
||||
# Fresh, idempotent rebuild.
|
||||
|
||||
Reference in New Issue
Block a user