jorgecuadros-platform/docs/STATEMENT_OCR.md

# Utility Statement OCR Capture (receipt capture)

Reads the stack of scanned utility bills the office pays every month, proposes
the customer and the amount for each page, and posts the confirmed pages to the
ledger as one batch against one check. Built 2026-08-01, live under `/recibos`.

This is the **as-built** record. The design and the reasoning behind it are
[`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2, which also carries the
measured results and the four spec corrections the real scans forced. Read that
for *why*; read this for *what is there*.

## The job it replaces

Staff receive 300+ pages per month per service provider — CFE electricity,
CESPT water, Telnor phone, gas, municipal predial, federal zone — and key each
one into the ledger by hand, as a charge against the customer whose property
the bill belongs to. The whole stack is paid with one office check, so the
capture is naturally a batch.

Auto-capture is a **mode of** the existing capture screen, not a separate
feature: it is the same daily job with a scanner instead of a keyboard, and
both modes post through the same ledger path.

## What ships

| Piece | Path |
|---|---|
| API module | `apps/api/src/statements/` (service, controller, DTOs, matcher, parsers) |
| OCR seam | `apps/api/src/statements/ocr/` (interface + Tesseract), bound in `apps/api/src/ocr/ocr.module.ts` |
| Tables | `statement_batches`, `statement_documents` (`20260731235721_statement_ocr_intake`) |
| Web | `components/Captura.tsx` (tab shell), `StatementIntake.tsx` (upload + batch list), `/recibos/:id` (review queue) |
| Abilities | `statement:ingest`, `statement:review` — both **STAFF** |

STAFF is deliberate: the review step is what makes machine capture safe at that
tier, since nothing reaches the ledger unconfirmed.

## The screen

`Captura` is one screen with two ways in:

- `/estado-cuenta/lote` → **manual** tab (`ManualCheckCapture`, key each
  receipt against one check by hand)
- `/recibos` → **automática (OCR)** tab (`StatementIntake`, upload the scans)
- `/recibos/:id` → the batch review queue, page image beside the extracted
  fields

Both end in the same place — charges on customers' ledgers posted against one
check — so they are modes of one screen. Staff pick by what is on the desk that
morning. Either URL renders the same component, so old bookmarks land on the
right tab.

## Pipeline

```
upload PDFs (+ service kind) → store source → render pages → text layer? → parse → match → review → confirm → ledger
```

**A batch is one service kind.** The uploader labels it (ELECTRIC, WATER,
TELEPHONE, GAS, PROPERTY_TAX, FEDERAL_ZONE) and that label is enforced: if the
parser reads a page as a different provider, the page is rejected as
mis-sorted rather than matched. Posting a phone bill as a water charge is the
failure being prevented.

**Processing is not awaited.** 300 pages of OCR is minutes of CPU, far past any
HTTP timeout, so `POST /statements/batches` returns the batch id immediately and
the client polls. That is also what lets the review queue show partial progress.

**One page = one document.** Statements arrive **bundled, one customer per
page** — Telnor's own `Pág 3 de 6` is its internal pagination, not the office's
scan — so every rendered page becomes its own `StatementDocument` and the
parser runs per page. (The policy OCR feature inverts this; see "Sibling
feature" below.)

**One unreadable page must not abandon the other 299.** A page that throws
becomes a single `OCR_FAILED` row and the loop continues.

Both the source PDFs (`statement/{batchId}/source-N.pdf`) and every rendered
page image (`page-N.png`) are stored. The source is the artifact the office
received and the only way to re-run a corrected parser over the original; the
page image is what the reviewer looks at, because "what the parser read" is
only checkable against a picture of the paper.

## The OCR seam

`OcrProvider` (`ocr/ocr.provider.ts`) is the swap point. Four methods:
`available()`, `renderPages()`, `recognize()`, `textPages()`.

Everything above it works in terms of page text and word boxes, so the engine
is replaceable without touching the parsers, the matcher or the schema. The
shipped implementation is **self-hosted Tesseract**, and that choice is
evidence-based rather than assumed — see the spec's measured results. A managed
API (Textract, Document Intelligence, Document AI) fits behind the same
interface with no schema change; at 300+ pages/month/company it would carry
real recurring cost for accuracy that is not the bottleneck.

The binding lives in `apps/api/src/ocr/ocr.module.ts`, extracted out of
`StatementsModule` when [`POLICY_OCR.md`](POLICY_OCR.md) needed the same seam.
`StatementsModule` imports it and binds nothing itself, so the engine decision
is one line in one file for both features.

### Text layer first, OCR as the fallback

**Not every statement is a scan.** The gas company sends born-digital CFDI
invoices whose text layer is already exact and already positioned.
`textPages()` reads it (`pdftotext -bbox-layout`, same poppler package as
`pdftoppm`) and OCR runs only where there is none.

Rasterising a born-digital page and re-recognising it can only lose
information — one sample turned `MEDIDOR: VM01014426` into
`ar (LTR): 014420` — while costing about a minute of CPU for the privilege.
Positions come back in the same pixel space `recognize()` uses, so the parsers'
geometric helpers work unchanged on either source. When the text layer is used
the document's notes say so verbatim: *"texto leído del PDF original, sin
OCR"*.

### Word boxes, not just text

`OcrPage` carries `words[]` with pixel boxes because several real layouts are
**tables**: the CESPT "RECIBO" prints `No. DE CUENTA` as a column header with
the value in the row beneath it, which line-oriented text cannot associate.
Parsers fall back to geometry for exactly those fields.

## Parsers

Eight providers, dispatched by a `BRAND` table checked before a `LAYOUT` table:

| Provider | Service kind |
|---|---|
| `CFE` | ELECTRIC |
| `CESPT` | WATER |
| `TELNOR` | TELEPHONE |
| `GAS TIJUANA` | GAS |
| `PREDIAL TIJUANA` / `PREDIAL ROSARITO` / `PREDIAL ENSENADA` | PROPERTY_TAX |
| `ZONA FEDERAL TIJUANA` | FEDERAL_ZONE |

Three predial parsers rather than one because Tijuana, Rosarito and Ensenada
issue three completely different documents — same tax, nothing else in common.

Rules that are load-bearing and easy to break:

- **Brand before layout, and never interleaved.** Scanned logos read badly (a
  CESPT header came back as `E BAJA ES PAGO / EALIFORNIA`), which is why the
  layout fallback exists — but *every* brand rule runs first, because a Telnor
  page contains words a CFE structural rule would otherwise claim.
- **Tijuana bills predial and zona federal from the same treasury.** Same
  header, same address, same `ATB-541201` RFC, so every predial discriminator
  matches a zona federal page too. The words only that layout prints are
  `Marítimo Terrestre`, so its rule is asked ahead of all three predial ones.
  **Order matters here in a way that is invisible from the code shape.**
- **Parse amounts by separator position.** A real Telnor bill OCR'd as
  `$ 649,00`; stripping commas as thousands separators makes that $64,900.
- **A misread `$` is the dangerous failure, not a missing one.** An Ensenada
  receipt for `$2,203.00` OCR'd as `82,203.00` — the sign read as an 8, which
  would post a charge 37× too large and look entirely ordinary in the ledger.
  Every predial amount therefore requires a literal `$`; a page that cannot
  produce one reports no amount and goes to review.
- **Digit-confusion repair only on fields known to be digits** (`O→0`, `S→5`,
  `B→8`, …), never on free text.
- **The clave catastral is not `[A-Z]{2}[0-9]{6}`.** Position three is a letter
  in fifteen of the 932 stored claves (`MMB01041`, `CGH52121`). Digitising the
  whole tail maps that `B` to an `8` and yields a key matching no property.
- **Barcodes beat printed labels.** Where a provider prints a payment barcode
  it is preferred and the two are cross-checked; disagreement sets
  `crossChecked: false` and forces review, because which of the two was
  misread is a judgement call.

## Matching

`StatementMatcherService`. Two rules govern everything:

**Match on one scoped field, never fuzzily across all identifiers.** Each
service kind has exactly one column its statements print, and only that column
is consulted. A blanket search over accountNumber/meterNumber/route would let a
water account number collide with an unrelated phone number, and the mis-post
would look perfectly ordinary in the ledger.

**Never match on the customer name.** A CESPT receipt for account `5365218`
prints `ARNAIZ ROSAS ELSA AURORA`; the office's book, corroborated by the
clave, has `CATT, RANDY`. The name on a utility bill is the registrant, not the
current owner. Names are shown to the reviewer and are never an input.

### `scopedRefField` — which column each kind actually prints

| Kind | Column | Why |
|---|---|---|
| ELECTRIC, WATER, TELEPHONE, CABLE | `accountNumber` | the legacy column holds the printed number |
| GAS | `meterNumber` | the number lived in free-text notes; `accountNumber` was never populated |
| PROPERTY_TAX | `meterNumber` | `accountNumber` holds `DATMEX.predial`, an office file number that is neither unique nor printed anywhere |
| FEDERAL_ZONE | `meterNumber` | `accountNumber` holds `DATMEX.zfed`, which is a **peso amount**, not a reference |

`scopedRefField` is exported because three places must agree on the answer: the
lookup, the blank-service fill on review, and the write-back on confirm. When
they disagree a reference gets learned into a column nothing searches, and the
same page returns to the review queue every month forever.

The `FEDERAL_ZONE` case is the sharpest instance of a trap this codebase hits
repeatedly (see also `policies.total`): a legacy column whose *name* promises
an identifier and whose *contents* are something else. Three of its 77 values
carry cents and one is negative. Worse than never matching — because every row
already has a value, the `[field]: null` guards on learning and on the
blank-service fill would never fire either.

### The clave catastral is a rescue on some layouts and the primary key on others

CESPT bills print the clave as well as an account number, so it rescues a page
whose account number did not OCR — which happened on real samples. There it
stays a hint.

On Rosarito and Ensenada predial the receipt prints **nothing else**, so a
unique clave hit is a real match and auto-matches. Tijuana predial prints no
clave at all; its only identifier is an 8-digit municipal account carried in a
32-digit payment barcode (`account(8) + DDMMYY + amount(9) + folio(9)`) that
the legacy database never held, so those pages start cold and are taught by the
first confirm.

Multiple hits are always surfaced, never auto-picked — duplicate account
numbers do occur in the legacy data, and the office's own `DUPLICADOS` report
existed for a reason.

## Confirm: what gets written

`confirmBatch` posts through **`BillingService.createBatch`** — the same method
the manual Editor screen uses — rather than writing `Transaction` rows
directly, so OCR-sourced and hand-keyed receipts share one write path, one
validation path and one audit trail.

- `source: "OCR"` and a per-line `captureRef` of the document id feed the
  duplicate-post guard, so a batch confirmed twice cannot double-charge.
- `items[i]` is positionally parallel to `lines[i]` (a documented seam
  guarantee), so the created rows zip straight back onto the documents that
  produced them via `postedTransactionId`.
- **The sign is applied here.** Charges are negative in this ledger; the parser
  reads the printed positive figure, and `-Math.abs()` is applied at the single
  point where a statement becomes a ledger row.
- A missing amount blocks the confirm with the offending page numbers, rather
  than silently posting zero.

### Learning: the cold start is a one-time cost

After posting, `learnAccountRefs` writes each confirmed reference back onto the
`PropertyService` that matched — **only where the field was null**. Never
overwrites a number already on file, which would let one misread page rewrite
good reference data.

This is what turns gas (whose numbers the migration never populated) and
Tijuana predial (whose municipal account the legacy database never held) from a
permanent review queue into a one-time cost: next month's statement for the
same account matches on its own.

### Discarding

Refused once any page is `POSTED` — those pages already wrote ledger rows
against a check, and a "discarded" label on the batch would leave the charges
unexplained. Reject the remaining pages individually instead.

## API surface

| Method | Route | Ability |
|---|---|---|
| `GET` | `/statements/status` (is OCR + storage available) | authenticated |
| `GET` | `/statements/batches`, `/batches/:id`, `/batches/:id/documents` | authenticated |
| `GET` | `/statements/documents/:id/page` (streams the page image) | authenticated |
| `POST` | `/statements/batches` (upload + service kind) | `statement:ingest` |
| `PATCH` | `/statements/documents/:id` (correct a field or the match) | `statement:review` |
| `POST` | `/statements/documents/:id/reject` | `statement:review` |
| `POST` | `/statements/batches/:id/discard` | `statement:review` |
| `POST` | `/statements/batches/:id/confirm` | `statement:review` |

## Requirements

Object storage (`S3_ENDPOINT` + credentials) for the scans, and `tesseract-ocr`
/ `tesseract-ocr-data-spa` / `poppler-utils` in the API image. Both are checked
at upload rather than at the first write — a missing dependency should be a 400
on the request, not a `FAILED` batch minutes later. `GET /statements/status`
reports both and the upload card hides itself unless both hold.

## Tests

- `parsers/statement-parser.spec.ts` — `detectProvider`,
  `normalizeCadastralKey`, and one suite per newer parser
  (`parsePredialTijuana`, `parsePredialRosarito`, `parsePredialEnsenada`,
  `parseGas`, `parseZonaFederal`). Every fixture is a **verbatim OCR excerpt
  from a real receipt**, including the ones that bit: the `82,203.00` Ensenada
  dollar sign, the three-letter clave, the CESPT logo garbage.
- `ocr/tesseract.provider.spec.ts` — `parseBboxLayout`, the
  `pdftotext -bbox-layout` reader that produces the text-layer `OcrPage`
  (positions included, which is what lets the geometric helpers work on
  born-digital input).

Note the CFE / CESPT / Telnor parsers themselves have **no unit suite** — they
predate the gas/predial extension and were verified against the 46-page corpus
end to end rather than in isolation. Worth closing if they are touched.

## Not built

- **Handwritten folder numbers.** Staff pencil a customer number on each bill
  (`9`, `405`); Tesseract read `405` as `205`. Handwriting is a review hint at
  best and is deliberately not an input to matching.
- **Re-running a corrected parser over a stored batch.** The source PDFs are
  kept precisely so this is possible, but nothing exposes it yet.
- **Providers beyond the eight above.** Adding one is a `BRAND` entry, an
  optional `LAYOUT` entry, and a parser function.

## Sibling feature

[`POLICY_OCR.md`](POLICY_OCR.md) — the same pipeline reading carrier policy
PDFs into `Policy` rows, built out of this one. It reuses the seam, the
provider-detection ordering, the digit-confusion map and the
amount-by-separator rule.

**One assumption does not carry over.** Here a page *is* a document, because
statements arrive one customer per page. A policy PDF is one document across
several pages, so that feature concatenates the pages and parses once per file.
If you are porting a change between the two, that is the difference to check
first.