rmancinasandClaude Opus 4.8
21899e99bb
migration/transform_policies.py folds every insurance Access table into one `policies` table (policy_types discriminator) plus child tables, via a per-table declarative mapping that absorbs the column-name variance (num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source column not explicitly modeled — the type-specific coverage amounts — is preserved verbatim in coveragesJson, so consolidation loses nothing. Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago + pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries (by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and builds policy_types + insurance_providers lookups. Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678 installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15 providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a customer carrying both a utility property and MULT policies (the unified cross-line view). run_all.py: add policies to the ordered pipeline. Customer FK resolves through insurance customer_legacy_refs, so this runs after customers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
65 lines
2.2 KiB
Python
65 lines
2.2 KiB
Python
"""
|
|
Run the full data migration against one environment, in dependency order.
|
|
|
|
Every step is idempotent (truncate + rebuild), so this is safe to re-run. The
|
|
target DB is chosen with --env (reads deploy/.env.<env>); the same staged
|
|
Parquet feeds every environment.
|
|
|
|
Prerequisites (once per environment, NOT done here):
|
|
1. DB stack deployed (deploy/jorgecuadros-db.stack.yml) and deploy/.env.<env> written.
|
|
2. Prisma schema pushed to it:
|
|
DATABASE_URL="<that env's url>" \
|
|
npx prisma@5 db push --schema=packages/database/prisma/schema.prisma
|
|
|
|
Then:
|
|
./.venv/bin/python run_all.py --env dev # data only (staging already present)
|
|
./.venv/bin/python run_all.py --env prod --stage # re-extract from Access first, then load
|
|
|
|
Reproducing dev -> prod is exactly `--env prod` (plus --stage if the staged
|
|
Parquet isn't present on the machine running it).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).parent
|
|
PY = sys.executable # the venv python running this orchestrator
|
|
|
|
# Dependency order — extend as later modules land (policies, transactions, bank).
|
|
STEPS = [
|
|
"transform_customers.py", # customers + customer_legacy_refs (everything FKs to these)
|
|
"transform_properties.py", # properties + services + trust accounts
|
|
"transform_policies.py", # policies + installments/vehicles/drivers/benef/claims/adjusters
|
|
]
|
|
|
|
|
|
def run(cmd: list[str]) -> None:
|
|
print("+ " + " ".join(cmd), flush=True)
|
|
r = subprocess.run(cmd)
|
|
if r.returncode:
|
|
sys.exit(r.returncode)
|
|
|
|
|
|
def main() -> None:
|
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--env", default="dev", help="target environment (reads deploy/.env.<env>)")
|
|
ap.add_argument("--stage", action="store_true",
|
|
help="re-run the raw staging load first (needs the Access files + mdbtools)")
|
|
args = ap.parse_args()
|
|
|
|
if args.stage:
|
|
run([PY, str(HERE / "load_staging.py"), "--output-dir", str(HERE / "output")])
|
|
|
|
for step in STEPS:
|
|
run([PY, str(HERE / step), "--env", args.env])
|
|
|
|
print(f"\n✓ migration complete for env={args.env}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|