fix(migration): recover transaction type labels and minimum balance

Two fields the customer-facing site reads were being dropped on the way in
from Access.

transform_transactions.py mapped DATOS2's type string through the Access
`TYPE OF TRX` table and stored NULL on a miss. That table is a stale
pick-list rather than a constraint — staff free-text straight into DATOS2 —
so 78 distinct values covering 3,939 rows never resolved, including
BALANCE FORWARD (1,188) and ANNUAL FEE (1,116). Nothing else on
`transactions` carries the type text, so those rows lost their label
outright and rendered blank. Now mints a type_transactions row from the
literal string when the lookup lacks it; nameEs stays NULL since only the
lookup has translations.

transform_customers.py never carried DATGRAL.TIPO, leaving
customers.minimumBalance empty on every row despite the column existing.
TIPO is the minimum-balance threshold (100/200/300/500; 1,017 of 1,172
customers carry one), not an account type as the name suggests — the
customer app shows it as `minBalance`. Added to the insert list and to the
ON DUPLICATE KEY UPDATE clause, without which --sync would silently skip
it on existing rows.

Both land on the next `run_all.py --sync` reload.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-03 12:30:03 -07:00
co-authored by Claude Opus 5
parent 2fa12890f5
commit 7226772c22
2 changed files with 32 additions and 3 deletions
+8 -2
View File
@@ -189,6 +189,11 @@ def customer_from_utilities(row, name_index) -> dict:
customerSince=as_date(row["cliente_desde"]),
status=as_bool(row["status"]),
feeAmount=as_decimal(row["fee"]),
# DATGRAL.TIPO is the minimum-balance threshold (100/200/300/500 —
# 1,017 of 1,172 customers carry one), NOT an identification or account
# type as the column name suggests. It reaches the website as
# datosfreak.TIPO and is returned to the customer app as `minBalance`.
minimumBalance=as_decimal(row["tipo"]),
updatedAt=NOW,
)
@@ -217,6 +222,7 @@ def customer_from_insurance(row, name_index) -> dict:
customerSince=None,
status=1,
feeAmount=None,
minimumBalance=None,
updatedAt=NOW,
)
@@ -225,7 +231,7 @@ _CUST_COLS = [
"id", "name", "nameSource", "nameMissing", "addressLine1", "addressLine2", "city", "state", "zipCode",
"country", "phone", "mobile", "fax", "email", "notes", "identificationType",
"identificationNumber", "identificationExpiration", "customerSince",
"status", "feeAmount", "updatedAt",
"status", "feeAmount", "minimumBalance", "updatedAt",
]
@@ -316,7 +322,7 @@ def main() -> None:
remap[rec["id"]] = stable or rec["id"]
for rec in customers:
rec["id"] = remap[rec["id"]]
cur.execute(f"INSERT INTO customers ({','.join(f'`{c}`' for c in _CUST_COLS)}) VALUES ({placeholders}) ON DUPLICATE KEY UPDATE name=VALUES(name),nameSource=VALUES(nameSource),nameMissing=VALUES(nameMissing),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),city=VALUES(city),state=VALUES(state),zipCode=VALUES(zipCode),country=VALUES(country),phone=VALUES(phone),mobile=VALUES(mobile),fax=VALUES(fax),email=VALUES(email),notes=VALUES(notes),identificationType=VALUES(identificationType),identificationNumber=VALUES(identificationNumber),identificationExpiration=VALUES(identificationExpiration),customerSince=VALUES(customerSince),status=VALUES(status),feeAmount=VALUES(feeAmount),updatedAt=VALUES(updatedAt)", tuple(rec[c] for c in _CUST_COLS))
cur.execute(f"INSERT INTO customers ({','.join(f'`{c}`' for c in _CUST_COLS)}) VALUES ({placeholders}) ON DUPLICATE KEY UPDATE name=VALUES(name),nameSource=VALUES(nameSource),nameMissing=VALUES(nameMissing),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),city=VALUES(city),state=VALUES(state),zipCode=VALUES(zipCode),country=VALUES(country),phone=VALUES(phone),mobile=VALUES(mobile),fax=VALUES(fax),email=VALUES(email),notes=VALUES(notes),identificationType=VALUES(identificationType),identificationNumber=VALUES(identificationNumber),identificationExpiration=VALUES(identificationExpiration),customerSince=VALUES(customerSince),status=VALUES(status),feeAmount=VALUES(feeAmount),minimumBalance=VALUES(minimumBalance),updatedAt=VALUES(updatedAt)", tuple(rec[c] for c in _CUST_COLS))
for ref in refs:
cur.execute("INSERT INTO customer_legacy_refs (id,customerId,sourceSystem,sourceTable,legacyId) VALUES (%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId)",
(ref[0], remap[ref[1]], ref[2], ref[3], ref[4]))