rmancinasandClaude Opus 4.8
83e3cb8f47
migration/transform_transactions.py unions every cash/billing ledger into `transactions` per the reconciliation rules: both EFECTIVO tables (no folio de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3 fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and exchange_rates lookups. Customer FK resolves through customer_legacy_refs; rows with no resolvable customer/date are skipped and counted. Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans), 79 type_transactions, 2301 exchange_rates. migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as signed amounts (income +, expense -) and TABLA RAMODOS into business_line_categories. Deliberately customer-independent (office's own checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77), 66 categories; categoryId left null (concept->ramo classifier is future work). run_all.py: pipeline now customers -> properties -> policies -> transactions -> bank, all idempotent. Verified full end-to-end run against dev. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
67 lines
2.4 KiB
Python
67 lines
2.4 KiB
Python
"""
|
|
Run the full data migration against one environment, in dependency order.
|
|
|
|
Every step is idempotent (truncate + rebuild), so this is safe to re-run. The
|
|
target DB is chosen with --env (reads deploy/.env.<env>); the same staged
|
|
Parquet feeds every environment.
|
|
|
|
Prerequisites (once per environment, NOT done here):
|
|
1. DB stack deployed (deploy/jorgecuadros-db.stack.yml) and deploy/.env.<env> written.
|
|
2. Prisma schema pushed to it:
|
|
DATABASE_URL="<that env's url>" \
|
|
npx prisma@5 db push --schema=packages/database/prisma/schema.prisma
|
|
|
|
Then:
|
|
./.venv/bin/python run_all.py --env dev # data only (staging already present)
|
|
./.venv/bin/python run_all.py --env prod --stage # re-extract from Access first, then load
|
|
|
|
Reproducing dev -> prod is exactly `--env prod` (plus --stage if the staged
|
|
Parquet isn't present on the machine running it).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).parent
|
|
PY = sys.executable # the venv python running this orchestrator
|
|
|
|
# Dependency order — extend as later modules land (policies, transactions, bank).
|
|
STEPS = [
|
|
"transform_customers.py", # customers + customer_legacy_refs (everything FKs to these)
|
|
"transform_properties.py", # properties + services + trust accounts
|
|
"transform_policies.py", # policies + installments/vehicles/drivers/benef/claims/adjusters
|
|
"transform_transactions.py", # shared ledger + type_transactions + exchange_rates
|
|
"transform_bank.py", # SCOTHIA bank register (no customer FK; independent)
|
|
]
|
|
|
|
|
|
def run(cmd: list[str]) -> None:
|
|
print("+ " + " ".join(cmd), flush=True)
|
|
r = subprocess.run(cmd)
|
|
if r.returncode:
|
|
sys.exit(r.returncode)
|
|
|
|
|
|
def main() -> None:
|
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--env", default="dev", help="target environment (reads deploy/.env.<env>)")
|
|
ap.add_argument("--stage", action="store_true",
|
|
help="re-run the raw staging load first (needs the Access files + mdbtools)")
|
|
args = ap.parse_args()
|
|
|
|
if args.stage:
|
|
run([PY, str(HERE / "load_staging.py"), "--output-dir", str(HERE / "output")])
|
|
|
|
for step in STEPS:
|
|
run([PY, str(HERE / step), "--env", args.env])
|
|
|
|
print(f"\n✓ migration complete for env={args.env}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|