feat(statements): OCR intake for scanned utility bills
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,155 @@
|
||||
"""
|
||||
Closes the three data gaps the statement-OCR matcher depends on
|
||||
(docs/RECEIPT_CAPTURE_SPEC.md §2, "Matching logic").
|
||||
|
||||
OCR matching is only as good as the field it matches against, and a
|
||||
field-by-field check of real scanned statements against what
|
||||
`transform_properties.py` actually loaded turned up three mismatches. This
|
||||
script fixes them on an existing database; `transform_properties.py` has been
|
||||
updated in the same commit so a full re-migration produces them directly.
|
||||
|
||||
1. CLAVE CATASTRAL — printed on both the CESPT water bill ("Cve. Cat.:
|
||||
KB078025") and the predial statement, and held in `DATMEX.clave` (934
|
||||
rows, format `[A-Z]{2}[0-9]{6}`). It was never migrated. What
|
||||
`PROPERTY_TAX.accountNumber` carries instead is `DATMEX.predial`, a
|
||||
different, purely numeric column that is *not* unique — 663 distinct
|
||||
values across 1135 filled rows — and appears on no statement. So predial
|
||||
is left exactly where it is, and the clave lands on `Property` (it is a
|
||||
property-level key, which is why two different services both print it).
|
||||
|
||||
2. GAS — `GAS.meterNumber` is empty for all 334 rows because the transform
|
||||
put `DATMEX.gas` into `notes`. That column is mixed: 160 rows hold a real
|
||||
numeric account/meter number, the remaining 174 hold a tank descriptor
|
||||
("ESTACIONARIO", "CILINDRO"). The numeric ones are recoverable now; the
|
||||
descriptors legitimately have no number, so those statements still start
|
||||
cold and get their number from the first human confirmation.
|
||||
|
||||
3. TELEPHONE — no such `ServiceKind` existed, so a Telnor bill had nothing to
|
||||
match against. One service row is created per property with a `phone1`.
|
||||
Only phone1: 534 properties have one, 18 have a phone2 and exactly 1 has a
|
||||
phone3, so the secondaries are alternate contacts rather than separately
|
||||
billed lines.
|
||||
|
||||
Idempotent — re-running updates nothing it has already done, and it never
|
||||
overwrites a value a human has since corrected.
|
||||
|
||||
./.venv/bin/python backfill_statement_match_fields.py --env dev
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from dbenv import connect
|
||||
from sync import parse_mode
|
||||
|
||||
STG = Path(__file__).parent / "output" / "stg_utilities"
|
||||
NULL = "∅"
|
||||
|
||||
|
||||
def s(v):
|
||||
if v is None or pd.isna(v):
|
||||
return None
|
||||
v = str(v).strip()
|
||||
return None if v in ("", NULL, "0") else v
|
||||
|
||||
|
||||
def main():
|
||||
# `--sync` is accepted and ignored — the script is idempotent either way.
|
||||
env, _sync_mode = parse_mode()
|
||||
conn = connect(env)
|
||||
c = conn.cursor()
|
||||
print(f"[statement-match-fields] target env: {env}")
|
||||
|
||||
# --- 1. clave catastral -> properties.cadastralKey ----------------------
|
||||
# Joined on provenance, the same key transform_properties.py writes, so a
|
||||
# property that was re-created by a later sync still lines up.
|
||||
dm = pd.read_parquet(STG / "datmex.parquet")
|
||||
claves = []
|
||||
for _, row in dm.iterrows():
|
||||
clave = s(row["clave"])
|
||||
if clave:
|
||||
claves.append((clave, str(int(row["_row_num"]))))
|
||||
|
||||
updated = 0
|
||||
for clave, legacy_id in claves:
|
||||
c.execute(
|
||||
"UPDATE properties SET cadastralKey = %s "
|
||||
"WHERE legacySourceTable = 'DATMEX' AND legacyId = %s AND cadastralKey IS NULL",
|
||||
(clave, legacy_id),
|
||||
)
|
||||
updated += c.rowcount
|
||||
print(f" cadastralKey: set on {updated} propert(ies) ({len(claves)} in source)")
|
||||
|
||||
# --- 2. gas account numbers out of notes -> GAS.meterNumber -------------
|
||||
# REGEXP rather than a Python loop: the value is already sitting in the
|
||||
# notes column, so this is one pass over 334 rows inside the database.
|
||||
c.execute(
|
||||
"UPDATE property_services SET meterNumber = notes "
|
||||
"WHERE kind = 'GAS' AND meterNumber IS NULL "
|
||||
"AND notes REGEXP '^[0-9]{5,}$'"
|
||||
)
|
||||
print(f" GAS.meterNumber: recovered {c.rowcount} account number(s) from notes")
|
||||
|
||||
# --- 3. TELEPHONE service rows ------------------------------------------
|
||||
# Digits only, matching how the transform now writes them: a scanned Telnor
|
||||
# bill prints "664 609 3444" and reduces to the stored local 6093444 once
|
||||
# the LADA is stripped, which is the matcher's job, not this script's.
|
||||
c.execute(
|
||||
"SELECT p.id, p.phone1 FROM properties p "
|
||||
"WHERE p.phone1 IS NOT NULL AND p.phone1 <> '' "
|
||||
"AND NOT EXISTS (SELECT 1 FROM property_services ps "
|
||||
" WHERE ps.propertyId = p.id AND ps.kind = 'TELEPHONE')"
|
||||
)
|
||||
rows = c.fetchall()
|
||||
made = []
|
||||
for pid, phone in rows:
|
||||
digits = "".join(ch for ch in str(phone) if ch.isdigit())
|
||||
if digits:
|
||||
made.append((str(uuid.uuid4()), pid, digits))
|
||||
if made:
|
||||
c.executemany(
|
||||
"INSERT INTO property_services "
|
||||
"(id, propertyId, kind, accountNumber, active, notes) "
|
||||
"VALUES (%s, %s, 'TELEPHONE', %s, 1, 'from DATMEX.telefono')",
|
||||
made,
|
||||
)
|
||||
print(f" TELEPHONE: created {len(made)} service row(s)")
|
||||
|
||||
conn.commit()
|
||||
|
||||
# --- validation ---------------------------------------------------------
|
||||
c.execute("SELECT COUNT(*) FROM properties WHERE cadastralKey IS NOT NULL")
|
||||
n_clave = c.fetchone()[0]
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM property_services WHERE kind='GAS' AND meterNumber IS NOT NULL"
|
||||
)
|
||||
n_gas = c.fetchone()[0]
|
||||
c.execute("SELECT COUNT(*) FROM property_services WHERE kind='TELEPHONE'")
|
||||
n_tel = c.fetchone()[0]
|
||||
|
||||
# A clave that is not unique would silently make the secondary match key
|
||||
# ambiguous, which is worse than not having one — surface it rather than
|
||||
# letting the matcher discover it a statement at a time.
|
||||
c.execute(
|
||||
"SELECT COUNT(*) FROM (SELECT cadastralKey FROM properties "
|
||||
"WHERE cadastralKey IS NOT NULL GROUP BY cadastralKey HAVING COUNT(*) > 1) d"
|
||||
)
|
||||
dupe_claves = c.fetchone()[0]
|
||||
|
||||
print("=== Statement match fields ready ===")
|
||||
print(f" properties with cadastralKey : {n_clave}")
|
||||
print(f" GAS services with meterNumber: {n_gas}")
|
||||
print(f" TELEPHONE services : {n_tel}")
|
||||
print(f" duplicated cadastralKey values: {dupe_claves}"
|
||||
+ (" (matcher treats these as ambiguous)" if dupe_claves else ""))
|
||||
assert n_clave > 0 and n_tel > 0, "backfill produced nothing — check staging output"
|
||||
print(" validation: OK")
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user