feat(statements): OCR intake for scanned utility bills
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -131,6 +131,72 @@ single-movement form.
|
||||
|
||||
## 2. PDF / OCR auto-capture
|
||||
|
||||
> **BUILT — 2026-08-01.** Implemented and verified end to end against real
|
||||
> scanned statements. `apps/api/src/statements/` holds the module: a swappable
|
||||
> `OcrProvider` seam with a self-hosted Tesseract implementation, per-provider
|
||||
> parsers for CFE / CESPT / Telnor, a scoped matcher, and a review queue that
|
||||
> posts through `BillingService.createBatch` with `source: "OCR"`. Web:
|
||||
> `/recibos` (upload + batch list) and `/recibos/:id` (review queue with the
|
||||
> page image beside the extracted fields). New abilities `statement:ingest` /
|
||||
> `statement:review`, both STAFF.
|
||||
>
|
||||
> **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor
|
||||
> bills) drove every decision below. Against them the shipped parser identifies
|
||||
> the provider on **46/46**, reads an account reference on **43/46**, an amount
|
||||
> on **42/46**, and a due date on **44/46**. Matched against the dev database
|
||||
> that is **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The
|
||||
> remainder are legitimate review cases: one account number shared by two
|
||||
> services, three phone numbers not yet on file, one clave not in the book, and
|
||||
> one page too poorly scanned to read.
|
||||
>
|
||||
> **The OCR-provider question is closed: self-hosted Tesseract.** It clears the
|
||||
> bar for a queue where a human confirms every row, and at 300+ pages/month/
|
||||
> company a per-page API would carry real recurring cost for accuracy that is
|
||||
> not the bottleneck. `OcrProvider` keeps a managed API (Textract, Document
|
||||
> Intelligence, Document AI) a one-line swap in `statements.module.ts` with no
|
||||
> schema change.
|
||||
>
|
||||
> **Four things the samples proved that this spec had wrong or unknown:**
|
||||
>
|
||||
> 1. **Clave catastral ≠ predial — gap 2 below is resolved.** `DATMEX.clave` is
|
||||
> 934 rows of `[A-Z]{2}[0-9]{6}` (`MM000012`, `KH220204`), the exact format
|
||||
> printed as `Cve. Cat.` / `CLAVE CATASTRAL` on real CESPT bills
|
||||
> (`KB078025`, `KA903009`). `DATMEX.predial` — what
|
||||
> `PROPERTY_TAX.accountNumber` actually holds — is 1135 numeric rows with
|
||||
> only **663 distinct values**, so it is not a per-property key at all and
|
||||
> appears on no statement. The clave was never migrated; it now lives on
|
||||
> `Property.cadastralKey` (property-level, because two different services
|
||||
> both print it) and is the matcher's secondary key. Predial is left
|
||||
> untouched. Predial statements match on the clave alone.
|
||||
> 2. **Gas is not a dead end — gap 3 below was wrong.** `DATMEX.gas` has 334
|
||||
> filled rows, of which **160 are real numeric account numbers**
|
||||
> (`900004807`); the other 174 are tank descriptors (`ESTACIONARIO`,
|
||||
> `CILINDRO`). All 334 went to `notes`. The 160 are recovered into
|
||||
> `GAS.meterNumber`; only the descriptor rows start cold.
|
||||
> 3. **Phone is one line per property, not three.** Of 1518 properties, 534
|
||||
> have `phone1`, 18 have `phone2` and exactly **1** has `phone3`. The
|
||||
> secondaries are alternate contacts, so `TELEPHONE` backfills from `phone1`
|
||||
> only rather than fanning out. This answers the open question below.
|
||||
> 4. **Statements arrive bundled, and their printed names are stale.** One PDF
|
||||
> holds many customers, one per page (Telnor's own `Pág 3 de 6` refers to
|
||||
> its internal pagination, not the office's scan). And the name on a utility
|
||||
> bill is the account registrant, not the current owner: a CESPT receipt for
|
||||
> account `5365218` prints `ARNAIZ ROSAS ELSA AURORA` where the office's
|
||||
> book — corroborated by the clave — has `CATT, RANDY`. **The matcher never
|
||||
> reads the name.**
|
||||
>
|
||||
> **Two OCR traps worth keeping in mind if the parsers are ever extended:**
|
||||
> scanned logos read badly (a CESPT header came back as `E BAJA ES PAGO /
|
||||
> EALIFORNIA`), so provider detection falls back to layout anchors — but only
|
||||
> after *every* brand check has run, since a Telnor page contains words a CFE
|
||||
> structural rule would otherwise claim. And amounts must be parsed by
|
||||
> separator position: a real Telnor bill OCR'd as `$ 649,00`, which naive
|
||||
> comma-stripping turns into $64,900.
|
||||
>
|
||||
> **Not covered:** handwritten folder numbers. Staff pencil a customer number on
|
||||
> each bill (`9`, `405`, `406`); Tesseract read `405` as `205`. Handwriting is
|
||||
> a review hint at best and is deliberately not an input to matching.
|
||||
|
||||
### Motivation (from the meeting)
|
||||
|
||||
Each utility company (CFE, water, phone, gas...) sends 300+ individual
|
||||
@@ -810,16 +876,23 @@ action (`customer:purge`) taken well after release — not bundled into
|
||||
|
||||
## Open questions to take back to Jorge (collected)
|
||||
|
||||
- OCR provider/budget for §2 (self-hosted vs. managed API, given 300+
|
||||
pages/month/company).
|
||||
- Whether source PDFs arrive pre-split per customer or as one bundled file
|
||||
needing page-range detection (§2).
|
||||
- Whether "Clave Catastral" and the already-migrated `PREDIAL`-sourced
|
||||
`PROPERTY_TAX.accountNumber` are the same number — blocks OCR matching
|
||||
for predial statements specifically until confirmed (§2).
|
||||
- Whether phone billing is really one service per phone number on file, or
|
||||
one per property regardless of how many numbers are recorded — decides
|
||||
how the new `TELEPHONE` service kind gets backfilled (§2).
|
||||
- ~~OCR provider/budget for §2~~ — **CLOSED**: self-hosted Tesseract, chosen on
|
||||
measured accuracy against real scans (see §2's BUILT note). No per-page cost.
|
||||
- ~~Whether source PDFs arrive pre-split per customer or bundled~~ —
|
||||
**CLOSED**: bundled, one customer per page. Split per page.
|
||||
- ~~Whether "Clave Catastral" and the `PREDIAL`-sourced
|
||||
`PROPERTY_TAX.accountNumber` are the same number~~ — **CLOSED**: they are
|
||||
different. `clave` is the cadastral key and is now on
|
||||
`Property.cadastralKey`; `predial` is not unique and is not printed on
|
||||
statements.
|
||||
- ~~Whether phone billing is one service per number or one per property~~ —
|
||||
**CLOSED**: effectively one (534 / 18 / 1 across phone1/2/3), backfilled
|
||||
from `phone1`.
|
||||
- **Still open (§2):** whether the CFE amount staff should owe is the rounded
|
||||
headline (`$268`, what the barcode encodes and what is paid at the window) or
|
||||
the exact `Total` in the breakdown (`$268.88`). The parser currently takes
|
||||
the barcode figure, which matches what the office actually pays; worth one
|
||||
confirmation from Jorge.
|
||||
- The actual bank name/currency/details for the Seguros USD account, and
|
||||
whether any historical Seguros bank data exists to migrate (§3).
|
||||
- Whether `BankAccount.businessLine` should be enforced or a soft hint
|
||||
|
||||
Reference in New Issue
Block a user