feat(statements): OCR intake for scanned utility bills
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -442,3 +442,87 @@ for what's actually next.
|
||||
verified vs dev: Anular buttons admin-gated, voided rows struck + excluded from totals,
|
||||
clicking Anular voids end-to-end (note: it uses a blocking `window.confirm`). Customer-detail
|
||||
mini tx list now also strikes voided rows ("(anulado)" tag) — was the last void-UI gap.
|
||||
|
||||
---
|
||||
|
||||
## Statement OCR intake (`/recibos`) — DONE 2026-08-01
|
||||
|
||||
Plan step 11 §2 (`docs/RECEIPT_CAPTURE_SPEC.md` §2). The last big utilities
|
||||
feature: staff scan the month's utility bills and the machine proposes customer
|
||||
+ amount per page, instead of keying 300+ statements per company by hand. Built
|
||||
in `apps/api/src/statements/` and `apps/web/src/app/recibos/`, posting through
|
||||
step 11 §1.2's `BillingService.createBatch` seam (`source: "OCR"`, per-document
|
||||
`captureRef`) so machine and hand capture share one write path and one audit
|
||||
trail. Abilities `statement:ingest` / `statement:review`, both STAFF.
|
||||
|
||||
**Verified end to end against the live dev API + MinIO**, not just built: real
|
||||
CFE and Telnor scans uploaded over HTTP, OCR'd, matched, confirmed against a
|
||||
check, and the resulting rows checked in MySQL — negative (charge) amounts,
|
||||
`captureSource = OCR`, concept auto-derived from the batch's service kind,
|
||||
`captureRef` linking each transaction back to its page. Re-confirming a posted
|
||||
batch is refused. All test data was removed afterwards.
|
||||
|
||||
**Everything here was decided from 10 real scanned statements (46 pages), not
|
||||
from the sample-free spec.** Shipped-parser results on them: provider 46/46,
|
||||
account reference 43/46, amount 42/46, due date 44/46; matched against the dev
|
||||
database, **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The rest are
|
||||
real review cases (one shared account number, three phones not on file, one
|
||||
clave not in the book, one page too poor to read).
|
||||
|
||||
Findings that corrected the spec, each of which changed the build:
|
||||
|
||||
- **The scans have no text layer at all** — they are camera images of paper, so
|
||||
OCR is mandatory rather than a convenience, and they arrive **bundled, one
|
||||
customer per page**.
|
||||
- **Clave catastral is not predial.** `DATMEX.clave` (934 rows, `KA903009`) is
|
||||
what CESPT and predial bills print; `DATMEX.predial` — which
|
||||
`PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135
|
||||
rows and appears on no statement. The clave now lives on
|
||||
`Property.cadastralKey` as the matcher's secondary key; predial was left
|
||||
untouched. This is the question that had been blocking predial matching.
|
||||
- **Gas was recoverable after all.** The spec said no legacy gas number existed;
|
||||
in fact 160 of 334 `DATMEX.gas` values are real account numbers (the rest are
|
||||
`ESTACIONARIO`/`CILINDRO` descriptors). Recovered into `GAS.meterNumber`.
|
||||
- **Phone is one billed line per property** (534 / 18 / 1 across phone1/2/3), so
|
||||
`TELEPHONE` — a new `ServiceKind` — backfills from `phone1` only.
|
||||
- **Never match on the printed name.** A CESPT receipt for account `5365218`
|
||||
reads `ARNAIZ ROSAS ELSA AURORA`; the office's book, corroborated by the
|
||||
clave, has `CATT, RANDY`. The name on a utility bill is the registrant, not
|
||||
the current owner.
|
||||
|
||||
`migration/backfill_statement_match_fields.py` closes those three data gaps on
|
||||
an existing database (idempotent, wired into `run_all.py` after
|
||||
`transform_properties.py`, which now produces them directly on a full rebuild).
|
||||
Applied to dev: 934 claves, 160 gas numbers, 534 TELEPHONE rows.
|
||||
|
||||
Implementation notes worth keeping:
|
||||
|
||||
- OCR is self-hosted **Tesseract** behind an `OcrProvider` interface — the
|
||||
provider question is closed on measured accuracy, and a managed API stays a
|
||||
one-line swap in `statements.module.ts`. `tesseract-ocr`,
|
||||
`tesseract-ocr-data-spa` and `poppler-utils` were added to the API image; if
|
||||
they are missing the module reports itself unavailable and only this feature
|
||||
is disabled.
|
||||
- **Payment barcodes beat printed labels.** One CFE label OCR'd a digit too
|
||||
many while its barcode was correct, so the barcode is the source and the label
|
||||
the cross-check; disagreement forces review.
|
||||
- **Detect the provider by brand first, layout only as a fallback** — and never
|
||||
interleave the two passes. A scanned CESPT header came back as `E BAJA ES
|
||||
PAGO / EALIFORNIA`, which is why the layout fallback exists; a Telnor page
|
||||
contains words a CFE layout rule would otherwise claim, which is why ordering
|
||||
matters.
|
||||
- **Parse amounts by separator position.** A real Telnor bill OCR'd as
|
||||
`$ 649,00`; stripping commas as thousands separators turns that into $64,900.
|
||||
- Two of the three layouts are line-oriented, but the CESPT "RECIBO" is a
|
||||
**table** whose values sit under column headers — that one needs the word
|
||||
boxes, which is why `OcrPage` carries geometry and not just text.
|
||||
- Confirming a document whose matched service had no reference **writes the
|
||||
reference back** (only into an empty field, and only when exactly one blank
|
||||
service of that kind is a candidate), so gas and any other cold start is a
|
||||
one-time cost rather than a permanent queue.
|
||||
- Handwritten folder numbers on the bills (`9`, `405`) are **not** used for
|
||||
matching — Tesseract read `405` as `205`.
|
||||
|
||||
**Open:** whether the CFE charge should be the rounded barcode/headline figure
|
||||
(`$268`, what is paid at the window — what the parser uses today) or the exact
|
||||
breakdown total (`$268.88`). One question for Jorge.
|
||||
|
||||
Reference in New Issue
Block a user