feat(statements): OCR intake for scanned utility bills
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m41s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m18s

Staff key 300+ utility statements per company per month by hand. This adds
the ingest -> split -> OCR -> match -> review pipeline that proposes customer
and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the
existing BillingService.createBatch seam with source=OCR and a per-document
captureRef so machine and hand capture share one write path and audit trail.

Everything was designed against 10 real scanned statements (46 pages of CFE,
CESPT and Telnor bills) rather than from the sample-free spec. The scans have
no text layer at all — they are camera images — so OCR is mandatory, and they
arrive bundled one customer per page. Measured on those pages the parser
identifies the provider 46/46 and reads an account reference 43/46; against
the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with
the rest genuine review cases. That closes the OCR-provider question in favour
of self-hosted Tesseract: it clears the bar for a queue where a human confirms
every row, and OcrProvider keeps a managed API a one-line swap.

The samples corrected three things the spec had wrong or unknown:

- Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and
  predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds,
  has 663 distinct values across 1135 rows and appears on no statement. The
  clave now lives on Property.cadastralKey as the matcher's secondary key;
  predial is left untouched. This had been blocking predial matching.
- Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers
  (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber.
- Phone is one billed line per property (534/18/1 across phone1/2/3), so the
  new TELEPHONE ServiceKind backfills from phone1 only, not three rows.

Matching is scoped to one column per service kind and never reads the customer
name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this
office holds under CATT, RANDY, because the printed name is the registrant,
not the current owner. Where a provider prints a payment barcode it beats the
printed label (one CFE label OCR'd a digit too many while its barcode was
correct) and the two cross-check, with disagreement forcing review.

Confirming a document whose service had no reference writes it back, so gas
and any other cold start is a one-time cost rather than a permanent queue.

Verified end to end against the live dev API and MinIO: real scans uploaded
over HTTP, matched, confirmed against a check, and the resulting rows checked
in MySQL (negative amounts, captureSource=OCR, concept derived from the batch
kind, captureRef linking back to each page). Re-confirming a posted batch is
refused. Test data was removed afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-01 00:42:35 -07:00
co-authored by Claude Opus 5
parent 121952fdc1
commit 4d5008b545
26 changed files with 3077 additions and 19 deletions
+25 -5
View File
@@ -147,7 +147,7 @@ def main():
props.append((
pid, cust_id, s(row["direccion"]), ", ".join(addr2_parts) or None,
s(row["telefono"]), s(row["telefono2"]), s(row["telefono3"]),
s(row["zona"]), "DATMEX", legacy_id,
s(row["zona"]), s(row["clave"]), "DATMEX", legacy_id,
))
@@ -178,9 +178,16 @@ def main():
svc("ELECTRIC", account=s(row[rc]),
notes=s(row["luz_tipo"]) if rc == "rpu" else None,
active=flag("electric"))
# GAS
# GAS — the column mixes two things: an account/meter number for 160 of
# the 334 filled rows, and a tank descriptor ("ESTACIONARIO",
# "CILINDRO") for the rest. Only the numeric form can be matched
# against a scanned gas statement, so it is promoted to meterNumber;
# the descriptor stays a note, as before.
if s(row["gas"]) or flag("gas1", False):
svc("GAS", due=s(row["gas_vence"]), notes=s(row["gas"]), active=flag("gas1"))
gas_val = s(row["gas"])
gas_meter = gas_val if gas_val and gas_val.isdigit() and len(gas_val) >= 5 else None
svc("GAS", meter=gas_meter, due=s(row["gas_vence"]),
notes=s(row["gas"]), active=flag("gas1"))
# CABLE
if s(row["cable_num"]) or (s_keep0(row["cable_sky"]) or "0") in _TRUE:
svc("CABLE", account=s(row["cable_num"]), route=s(row["cia_cable"]),
@@ -194,6 +201,19 @@ def main():
if s(row["zfed"]) or flag("federalzone", False):
svc("FEDERAL_ZONE", account=s(row["zfed"]), notes=s(row["zfed_t"]),
active=flag("federalzone"))
# TELEPHONE — DATMEX never had a phone *service*, only the contact
# numbers unpivoted into Property.phone1/2/3 above, even though the
# legacy ledger billed phone as its own transaction type. A Telnor bill
# can only be matched against a service row, so the primary number
# becomes one. Only phone1: of 1518 properties, 534 have phone1, 18
# phone2 and exactly 1 phone3 — the secondaries are alternate contacts,
# not additional billed lines. Stored as the bare local number, which is
# how DATMEX holds it and what a printed bill reduces to once the 664
# Tijuana LADA is stripped.
tel = s(row["telefono"])
if tel:
svc("TELEPHONE", account="".join(ch for ch in tel if ch.isdigit()) or None,
notes="from DATMEX.telefono")
# ALARM
if s(row["alarm_system"]):
svc("ALARM", notes=s(row["alarm_system"]))
@@ -218,7 +238,7 @@ def main():
cur.execute("DELETE ps FROM property_services ps JOIN properties p ON p.id=ps.propertyId WHERE p.legacyId IS NOT NULL")
cur.execute("DELETE ta FROM trust_accounts ta JOIN properties p ON p.id=ta.propertyId WHERE p.legacyId IS NOT NULL")
cur.executemany(
"INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),archivedAt=NULL", props)
"INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),cadastralKey=VALUES(cadastralKey),archivedAt=NULL", props)
delete_missing(cur, "properties", ("legacySourceTable", "legacyId"), prop_keys, "WHERE legacyId IS NOT NULL")
cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services)
cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts)
@@ -227,7 +247,7 @@ def main():
for t in ("property_services", "service_documents", "trust_accounts", "properties"):
cur.execute(f"TRUNCATE TABLE {t}")
cur.execute("SET FOREIGN_KEY_CHECKS=1")
cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props)
cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props)
cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services)
cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts)
conn.commit()