rmancinasandClaude Opus 5
9ba5d2d09a
The office keeps more than one operating account (Utilities banks in MXN, Seguros in USD), but bank_transactions was a single implicit MXN register by design. Adds Bank/BankAccount and makes every read and write in the module scoped to exactly one account. Schema: - Bank / BankAccount. Currency is fixed per account and BankTransaction has no currency column of its own — a movement inherits its account's, the way a real bank account doesn't mix currencies. - BankTransaction.bankAccountId, required. A movement with no known account isn't reconcilable against a statement. - @@index([bankAccountId, transactionDate]): every read now filters by account and orders/groups by date. Migration: - backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)" and backfills all 22,669 existing rows onto it, then promotes the column to NOT NULL and attaches the FK. Standalone because prisma db push cannot add a required column to a populated table. Idempotent; re-running once a second account exists does not re-point rows. - run_all.py runs it (both modes) before transform_bank.py, which now resolves the account by label and fails fast if it is missing. API: - ?bankAccountId= required on list/stats/facets/summary — not optional with an "all accounts" default, since summing an MXN and a USD register repeats the currency-collapsing mistake the billing module exists to prevent. Missing is 400, unknown is 404. - facets() had no account clause at all and summary() has two raw-SQL rollups; all three are now parameterised. Scoping only one of summary's queries would leave the year list and its drill-down describing different books. - New bank/accounts + bank/banks sub-resource under a MANAGER bank:manage-accounts ability. currency is absent from the update DTO: booked movements are denominated in it, so editing would re-denominate history. Capture into a closed account is rejected. Web: - /banco gains an account picker (remembered per browser) and reads every figure in the selected account's currency; the "single currency (MXN)" doc-comment and the hardcoded MXN formatting are gone. - New /banco/cuentas for banks and accounts. Accounts are closed, never deleted — the FK is required, so deleting one would destroy its register. - /inicio's chequera card names the account it is reading instead of implying a single register. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged (22,669 movements, net 1,014,266.97). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
99 lines
3.6 KiB
Python
99 lines
3.6 KiB
Python
"""
|
|
Run the full data migration against one environment, in dependency order.
|
|
|
|
Every step is idempotent (truncate + rebuild), so this is safe to re-run. The
|
|
target DB is chosen with --env (reads deploy/.env.<env>); the same staged
|
|
Parquet feeds every environment.
|
|
|
|
Prerequisites (once per environment, NOT done here):
|
|
1. DB stack deployed (deploy/jorgecuadros-db.stack.yml) and deploy/.env.<env> written.
|
|
2. Prisma schema pushed to it:
|
|
DATABASE_URL="<that env's url>" \
|
|
npx prisma@5 db push --schema=packages/database/prisma/schema.prisma
|
|
|
|
Then:
|
|
./.venv/bin/python run_all.py --env dev # data only (staging already present)
|
|
./.venv/bin/python run_all.py --env prod --stage # re-extract from Access first, then load
|
|
|
|
Reproducing dev -> prod is exactly `--env prod` (plus --stage if the staged
|
|
Parquet isn't present on the machine running it).
|
|
|
|
Note: the blob_extract step re-reads the original Access files directly (the
|
|
blobs are not in the staged Parquet), so the machine running this needs
|
|
SOURCE_ROOT + mdbtools + MinIO credentials even without --stage.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).parent
|
|
PY = sys.executable # the venv python running this orchestrator
|
|
|
|
# Dependency order. Every step truncates what it owns, so anything downstream
|
|
# of a truncated table has to be rebuilt in the same pass — blob_extract is in
|
|
# this list because transform_properties and transform_policies truncate
|
|
# service_documents / policy_documents, which would otherwise leave the
|
|
# uploaded MinIO objects with no rows pointing at them.
|
|
STEPS = [
|
|
"transform_customers.py",
|
|
"transform_properties.py",
|
|
"transform_policies.py",
|
|
"transform_transactions.py",
|
|
"prune_empty_customers.py",
|
|
# Seeds the Scotiabank chequera that every SCOTHIA movement is booked into;
|
|
# transform_bank.py fails fast without it.
|
|
"backfill_bank_accounts.py",
|
|
"transform_bank.py",
|
|
"blob_extract.py",
|
|
]
|
|
|
|
SYNC_STEPS = [
|
|
"transform_customers.py",
|
|
"transform_properties.py",
|
|
"transform_policies.py",
|
|
"transform_transactions.py",
|
|
# Manual-safe prune: drops legacy-owned empties that the customer upsert
|
|
# re-creates from Parquet, but leaves manually-added customers alone.
|
|
"prune_empty_customers.py",
|
|
# Seeds the Scotiabank chequera that every SCOTHIA movement is booked into;
|
|
# transform_bank.py fails fast without it.
|
|
"backfill_bank_accounts.py",
|
|
"transform_bank.py",
|
|
]
|
|
|
|
|
|
def run(cmd: list[str]) -> None:
|
|
print("+ " + " ".join(cmd), flush=True)
|
|
r = subprocess.run(cmd)
|
|
if r.returncode:
|
|
sys.exit(r.returncode)
|
|
|
|
|
|
def main() -> None:
|
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--env", default="dev", help="target environment (reads deploy/.env.<env>)")
|
|
ap.add_argument("--stage", action="store_true",
|
|
help="re-run the raw staging load first (needs the Access files + mdbtools)")
|
|
ap.add_argument("--sync", action="store_true",
|
|
help="upsert legacy rows and archive removed legacy rows; preserve manual rows")
|
|
args = ap.parse_args()
|
|
|
|
if args.stage:
|
|
run([PY, str(HERE / "load_staging.py"), "--output-dir", str(HERE / "output")])
|
|
|
|
for step in SYNC_STEPS if args.sync else STEPS:
|
|
cmd = [PY, str(HERE / step), "--env", args.env]
|
|
if args.sync:
|
|
cmd.append("--sync")
|
|
run(cmd)
|
|
|
|
print(f"\n✓ migration complete for env={args.env}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|