feat(bank): multi-bank chequera — required bankAccountId, per-account scoping
The office keeps more than one operating account (Utilities banks in MXN, Seguros in USD), but bank_transactions was a single implicit MXN register by design. Adds Bank/BankAccount and makes every read and write in the module scoped to exactly one account. Schema: - Bank / BankAccount. Currency is fixed per account and BankTransaction has no currency column of its own — a movement inherits its account's, the way a real bank account doesn't mix currencies. - BankTransaction.bankAccountId, required. A movement with no known account isn't reconcilable against a statement. - @@index([bankAccountId, transactionDate]): every read now filters by account and orders/groups by date. Migration: - backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)" and backfills all 22,669 existing rows onto it, then promotes the column to NOT NULL and attaches the FK. Standalone because prisma db push cannot add a required column to a populated table. Idempotent; re-running once a second account exists does not re-point rows. - run_all.py runs it (both modes) before transform_bank.py, which now resolves the account by label and fails fast if it is missing. API: - ?bankAccountId= required on list/stats/facets/summary — not optional with an "all accounts" default, since summing an MXN and a USD register repeats the currency-collapsing mistake the billing module exists to prevent. Missing is 400, unknown is 404. - facets() had no account clause at all and summary() has two raw-SQL rollups; all three are now parameterised. Scoping only one of summary's queries would leave the year list and its drill-down describing different books. - New bank/accounts + bank/banks sub-resource under a MANAGER bank:manage-accounts ability. currency is absent from the update DTO: booked movements are denominated in it, so editing would re-denominate history. Capture into a closed account is rejected. Web: - /banco gains an account picker (remembered per browser) and reads every figure in the selected account's currency; the "single currency (MXN)" doc-comment and the hardcoded MXN formatting are gone. - New /banco/cuentas for banks and accounts. Accounts are closed, never deleted — the FK is required, so deleting one would destroy its register. - /inicio's chequera card names the account it is reading instead of implying a single register. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged (22,669 movements, net 1,014,266.97). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,192 @@
|
||||
"""
|
||||
One-off schema+data step for the multi-bank chequera
|
||||
(docs/RECEIPT_CAPTURE_SPEC.md §3).
|
||||
|
||||
`bank_transactions.bankAccountId` is REQUIRED in the Prisma schema, so
|
||||
`prisma db push` cannot introduce it on a table that already holds 22k rows.
|
||||
This script does the ordered dance that push can't:
|
||||
|
||||
1. create `banks` / `bank_accounts` (same DDL Prisma generates)
|
||||
2. seed the one account every existing row belongs to — Scotiabank MXN,
|
||||
the office's Utilities chequera, which is all `SCOTHIA.mdb` ever was
|
||||
3. add `bankAccountId` NULLable, backfill every row to that account,
|
||||
then promote it to NOT NULL and attach the FK + index
|
||||
|
||||
On a database that predates the feature, run it BEFORE `prisma db push`; push
|
||||
then sees no drift. On a fresh environment push creates the tables itself and
|
||||
this only seeds the rows. Either way `transform_bank.py` needs the account to
|
||||
exist, so `run_all.py` runs it first. Idempotent — safe to re-run, and
|
||||
re-running once a second account exists does NOT re-point rows (the backfill
|
||||
only touches NULLs).
|
||||
|
||||
./.venv/bin/python backfill_bank_accounts.py --env dev
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
from dbenv import connect
|
||||
from sync import parse_mode
|
||||
|
||||
# The account every migrated SCOTHIA row belongs to. Its id is derived, not
|
||||
# random, so a re-run against a half-applied database finds the same row and
|
||||
# `transform_bank.py` can resolve it by label without a lookup table.
|
||||
SCOTIABANK = "Scotiabank"
|
||||
UTILITIES_ACCOUNT = "Utilities — Scotiabank (MXN)"
|
||||
|
||||
|
||||
def table_exists(c, name: str) -> bool:
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM information_schema.tables "
|
||||
"WHERE table_schema = DATABASE() AND table_name = %s",
|
||||
(name,),
|
||||
)
|
||||
return c.fetchone()[0] > 0
|
||||
|
||||
|
||||
def column_exists(c, table: str, column: str) -> bool:
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM information_schema.columns "
|
||||
"WHERE table_schema = DATABASE() AND table_name = %s AND column_name = %s",
|
||||
(table, column),
|
||||
)
|
||||
return c.fetchone()[0] > 0
|
||||
|
||||
|
||||
def constraint_exists(c, table: str, name: str) -> bool:
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM information_schema.table_constraints "
|
||||
"WHERE table_schema = DATABASE() AND table_name = %s AND constraint_name = %s",
|
||||
(table, name),
|
||||
)
|
||||
return c.fetchone()[0] > 0
|
||||
|
||||
|
||||
def index_exists(c, table: str, name: str) -> bool:
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM information_schema.statistics "
|
||||
"WHERE table_schema = DATABASE() AND table_name = %s AND index_name = %s",
|
||||
(table, name),
|
||||
)
|
||||
return c.fetchone()[0] > 0
|
||||
|
||||
|
||||
def main():
|
||||
# `--sync` is accepted and ignored: this step is idempotent by nature, so
|
||||
# it behaves identically in both modes and can sit in run_all's two lists.
|
||||
env, _sync_mode = parse_mode()
|
||||
conn = connect(env)
|
||||
c = conn.cursor()
|
||||
print(f"[bank-accounts] target env: {env}")
|
||||
|
||||
# --- 1. tables ----------------------------------------------------------
|
||||
if not table_exists(c, "banks"):
|
||||
c.execute(
|
||||
"""
|
||||
CREATE TABLE `banks` (
|
||||
`id` VARCHAR(191) NOT NULL,
|
||||
`name` VARCHAR(191) NOT NULL,
|
||||
`country` VARCHAR(191) NULL,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `banks_name_key` (`name`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
|
||||
"""
|
||||
)
|
||||
print(" created banks")
|
||||
|
||||
if not table_exists(c, "bank_accounts"):
|
||||
c.execute(
|
||||
"""
|
||||
CREATE TABLE `bank_accounts` (
|
||||
`id` VARCHAR(191) NOT NULL,
|
||||
`bankId` VARCHAR(191) NOT NULL,
|
||||
`label` VARCHAR(191) NOT NULL,
|
||||
`currency` ENUM('USD','MXN') NOT NULL,
|
||||
`businessLine` ENUM('UTILITY','INSURANCE','TRUST') NULL,
|
||||
`active` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
PRIMARY KEY (`id`),
|
||||
KEY `bank_accounts_bankId_fkey` (`bankId`),
|
||||
CONSTRAINT `bank_accounts_bankId_fkey` FOREIGN KEY (`bankId`)
|
||||
REFERENCES `banks` (`id`) ON DELETE RESTRICT ON UPDATE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
|
||||
"""
|
||||
)
|
||||
print(" created bank_accounts")
|
||||
|
||||
# --- 2. seed the Utilities/Scotiabank chequera --------------------------
|
||||
c.execute("SELECT id FROM banks WHERE name = %s", (SCOTIABANK,))
|
||||
row = c.fetchone()
|
||||
if row:
|
||||
bank_id = row[0]
|
||||
else:
|
||||
bank_id = str(uuid.uuid4())
|
||||
c.execute(
|
||||
"INSERT INTO banks (id, name, country) VALUES (%s, %s, %s)",
|
||||
(bank_id, SCOTIABANK, "MX"),
|
||||
)
|
||||
print(f" seeded bank {SCOTIABANK}")
|
||||
|
||||
c.execute("SELECT id FROM bank_accounts WHERE label = %s", (UTILITIES_ACCOUNT,))
|
||||
row = c.fetchone()
|
||||
if row:
|
||||
account_id = row[0]
|
||||
else:
|
||||
account_id = str(uuid.uuid4())
|
||||
c.execute(
|
||||
"INSERT INTO bank_accounts (id, bankId, label, currency, businessLine, active) "
|
||||
"VALUES (%s, %s, %s, 'MXN', 'UTILITY', 1)",
|
||||
(account_id, bank_id, UTILITIES_ACCOUNT),
|
||||
)
|
||||
print(f" seeded account {UTILITIES_ACCOUNT}")
|
||||
print(f" account id: {account_id}")
|
||||
|
||||
# --- 3. column, backfill, promote to NOT NULL ---------------------------
|
||||
if not column_exists(c, "bank_transactions", "bankAccountId"):
|
||||
c.execute("ALTER TABLE `bank_transactions` ADD COLUMN `bankAccountId` VARCHAR(191) NULL")
|
||||
print(" added bank_transactions.bankAccountId (nullable)")
|
||||
|
||||
c.execute(
|
||||
"UPDATE bank_transactions SET bankAccountId = %s WHERE bankAccountId IS NULL",
|
||||
(account_id,),
|
||||
)
|
||||
print(f" backfilled {c.rowcount} movement(s) to {UTILITIES_ACCOUNT}")
|
||||
|
||||
c.execute("SELECT COUNT(*) FROM bank_transactions WHERE bankAccountId IS NULL")
|
||||
orphans = c.fetchone()[0]
|
||||
if orphans:
|
||||
raise SystemExit(f"abort: {orphans} bank_transactions still have no account")
|
||||
|
||||
c.execute("ALTER TABLE `bank_transactions` MODIFY `bankAccountId` VARCHAR(191) NOT NULL")
|
||||
|
||||
if not index_exists(c, "bank_transactions", "bank_transactions_bankAccountId_transactionDate_idx"):
|
||||
c.execute(
|
||||
"CREATE INDEX `bank_transactions_bankAccountId_transactionDate_idx` "
|
||||
"ON `bank_transactions` (`bankAccountId`, `transactionDate`)"
|
||||
)
|
||||
print(" created (bankAccountId, transactionDate) index")
|
||||
|
||||
if not constraint_exists(c, "bank_transactions", "bank_transactions_bankAccountId_fkey"):
|
||||
c.execute(
|
||||
"ALTER TABLE `bank_transactions` "
|
||||
"ADD CONSTRAINT `bank_transactions_bankAccountId_fkey` FOREIGN KEY (`bankAccountId`) "
|
||||
"REFERENCES `bank_accounts` (`id`) ON DELETE RESTRICT ON UPDATE CASCADE"
|
||||
)
|
||||
print(" attached bankAccountId FK")
|
||||
|
||||
conn.commit()
|
||||
|
||||
c.execute(
|
||||
"SELECT a.label, a.currency, COUNT(t.id), COALESCE(SUM(t.amount), 0) "
|
||||
"FROM bank_accounts a LEFT JOIN bank_transactions t ON t.bankAccountId = a.id "
|
||||
"GROUP BY a.id, a.label, a.currency ORDER BY a.label"
|
||||
)
|
||||
print("=== Multi-bank chequera ready ===")
|
||||
for label, currency, n, total in c.fetchall():
|
||||
print(f" {label:36} {currency} {n:6} movimientos neto {total}")
|
||||
print(" validation: OK")
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -44,6 +44,9 @@ STEPS = [
|
||||
"transform_policies.py",
|
||||
"transform_transactions.py",
|
||||
"prune_empty_customers.py",
|
||||
# Seeds the Scotiabank chequera that every SCOTHIA movement is booked into;
|
||||
# transform_bank.py fails fast without it.
|
||||
"backfill_bank_accounts.py",
|
||||
"transform_bank.py",
|
||||
"blob_extract.py",
|
||||
]
|
||||
@@ -56,6 +59,9 @@ SYNC_STEPS = [
|
||||
# Manual-safe prune: drops legacy-owned empties that the customer upsert
|
||||
# re-creates from Parquet, but leaves manually-added customers alone.
|
||||
"prune_empty_customers.py",
|
||||
# Seeds the Scotiabank chequera that every SCOTHIA movement is booked into;
|
||||
# transform_bank.py fails fast without it.
|
||||
"backfill_bank_accounts.py",
|
||||
"transform_bank.py",
|
||||
]
|
||||
|
||||
|
||||
@@ -10,12 +10,20 @@ Sources:
|
||||
from the spelled-out "cantidad en letra"
|
||||
- TABLA RAMODOS -> business_line_categories (line-of-business lookup)
|
||||
|
||||
Bank account: SCOTHIA is the Utilities MXN chequera and nothing else — DATOS
|
||||
E/I carry no bank or currency column — so every row loads against the single
|
||||
account seeded by `backfill_bank_accounts.py`, which must have run first.
|
||||
`banks` / `bank_accounts` are NOT truncated here; only the movements are. (In
|
||||
full-rebuild mode that still clears app-captured rows on every account, the
|
||||
same whole-database truncate every transform in this pipeline does — use
|
||||
`--sync` to upsert instead.)
|
||||
|
||||
Category link: DATOS E/I have no explicit FK to TABLA RAMODOS — the ramo is
|
||||
inferred from the CONCEPTO text, which is a fuzzy classification, not a stored
|
||||
key. So the categories are loaded but bank_transactions.categoryId is left
|
||||
NULL for now; a concept->ramo classifier is a later enhancement.
|
||||
|
||||
Idempotent (truncate + rebuild). Run:
|
||||
Idempotent (rebuild the legacy rows). Run:
|
||||
./.venv/bin/python transform_bank.py --env dev
|
||||
"""
|
||||
|
||||
@@ -27,6 +35,7 @@ from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from backfill_bank_accounts import UTILITIES_ACCOUNT
|
||||
from dbenv import connect, env_arg
|
||||
from sync import parse_mode
|
||||
|
||||
@@ -78,6 +87,19 @@ def main():
|
||||
print(f"[bank] target env: {env}")
|
||||
c = conn.cursor()
|
||||
|
||||
# Every SCOTHIA row belongs to the one Utilities MXN chequera. Resolved by
|
||||
# label rather than created here, so this script can't silently open a
|
||||
# second copy of the account if the backfill hasn't run.
|
||||
c.execute("SELECT id FROM bank_accounts WHERE label = %s", (UTILITIES_ACCOUNT,))
|
||||
row = c.fetchone()
|
||||
if not row:
|
||||
raise SystemExit(
|
||||
f"missing bank account {UTILITIES_ACCOUNT!r} — run "
|
||||
f"backfill_bank_accounts.py --env {env} first"
|
||||
)
|
||||
account_id = row[0]
|
||||
print(f"[bank] account: {UTILITIES_ACCOUNT} ({account_id})")
|
||||
|
||||
# business_line_categories (dedup TABLA RAMODOS)
|
||||
cats, seen = [], set()
|
||||
for _, r in load("tabla_ramodos").iterrows():
|
||||
@@ -96,7 +118,8 @@ def main():
|
||||
skip_date += 1
|
||||
return
|
||||
rows.append((
|
||||
str(uuid.uuid4()), td, s(r["tipo"]), s(r["num"]), s(r["concepto"]),
|
||||
str(uuid.uuid4()), account_id,
|
||||
td, s(r["tipo"]), s(r["num"]), s(r["concepto"]),
|
||||
amount, None, # categoryId left NULL (see header)
|
||||
1 if truthy(r["operado"]) else 0,
|
||||
1 if (income and truthy(r["transferido"])) else 0,
|
||||
@@ -110,9 +133,17 @@ def main():
|
||||
for _, r in load("datos_e").iterrows():
|
||||
add(r, -(dec(r["egreso"], Decimal(0))), income=False)
|
||||
|
||||
COLS = (
|
||||
"id,bankAccountId,transactionDate,transactionType,reference,concept,amount,"
|
||||
"categoryId,cleared,transferred,notes,amountInWords,legacySourceTable,legacyId"
|
||||
)
|
||||
PLACEHOLDERS = ",".join(["%s"] * 14)
|
||||
|
||||
if sync_mode:
|
||||
# bankAccountId is deliberately absent from the UPDATE clause: an
|
||||
# account moved by hand in the app must not be dragged back.
|
||||
for row in rows:
|
||||
c.execute("INSERT INTO bank_transactions (id,transactionDate,transactionType,reference,concept,amount,categoryId,cleared,transferred,notes,amountInWords,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE transactionDate=VALUES(transactionDate),transactionType=VALUES(transactionType),reference=VALUES(reference),concept=VALUES(concept),amount=VALUES(amount),cleared=VALUES(cleared),transferred=VALUES(transferred),notes=VALUES(notes),amountInWords=VALUES(amountInWords),voidedAt=NULL", row)
|
||||
c.execute(f"INSERT INTO bank_transactions ({COLS}) VALUES ({PLACEHOLDERS}) ON DUPLICATE KEY UPDATE transactionDate=VALUES(transactionDate),transactionType=VALUES(transactionType),reference=VALUES(reference),concept=VALUES(concept),amount=VALUES(amount),cleared=VALUES(cleared),transferred=VALUES(transferred),notes=VALUES(notes),amountInWords=VALUES(amountInWords),voidedAt=NULL", row)
|
||||
else:
|
||||
c.execute("SET FOREIGN_KEY_CHECKS=0")
|
||||
for t in ("bank_transactions", "business_line_categories"):
|
||||
@@ -120,7 +151,7 @@ def main():
|
||||
c.execute("SET FOREIGN_KEY_CHECKS=1")
|
||||
c.executemany("INSERT INTO business_line_categories (id,name) VALUES (%s,%s)", cats)
|
||||
c.executemany(
|
||||
"INSERT INTO bank_transactions (id,transactionDate,transactionType,reference,concept,amount,categoryId,cleared,transferred,notes,amountInWords,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", rows)
|
||||
f"INSERT INTO bank_transactions ({COLS}) VALUES ({PLACEHOLDERS})", rows)
|
||||
conn.commit()
|
||||
|
||||
def count(t):
|
||||
@@ -136,6 +167,15 @@ def main():
|
||||
for src, n, tot in by_src:
|
||||
print(f" {(src or '(manual)'):10} {n:6} sum {tot}")
|
||||
print(f" net balance movement : {net}")
|
||||
# Per account, never a cross-account total: the registers are in different
|
||||
# currencies and summing them produces a figure that never existed.
|
||||
c.execute(
|
||||
"SELECT a.label, a.currency, COUNT(t.id), COALESCE(SUM(t.amount), 0) "
|
||||
"FROM bank_accounts a LEFT JOIN bank_transactions t ON t.bankAccountId = a.id "
|
||||
"GROUP BY a.id, a.label, a.currency ORDER BY a.label"
|
||||
)
|
||||
for label, currency, n, total in c.fetchall():
|
||||
print(f" {label:34} {currency} {n:6} neto {total}")
|
||||
print(f" -> business_line_categories: {count('business_line_categories')}")
|
||||
print(" validation: OK")
|
||||
conn.close()
|
||||
|
||||
Reference in New Issue
Block a user