docs: as-built reference for the statement OCR capture
Gives receipt capture the same treatment policy OCR just got: a doc that records what is in the code, separate from the spec that records what was designed. RECEIPT_CAPTURE_SPEC.md §2 had accumulated three BUILT notes totalling ~120 lines of findings, which is the right place for the evidence but the wrong place to look up how the matcher picks a column. docs/STATEMENT_OCR.md covers the pipeline, the OCR seam and its text-layer-first rule, all eight parsers and the ordering constraints between them, the matcher's two governing rules and the scopedRefField table, confirm-through-BillingService, the learning write-back, and the API surface. Weight goes to the things that are load-bearing and invisible from the code shape: brand detection must run to completion before layout because Tijuana bills predial and zona federal off the same treasury header; scopedRefField is exported because three call sites must agree or a reference gets learned into a column nothing searches; FEDERAL_ZONE's accountNumber holds a peso amount, so it fails the null-guards as well as the lookup; a misread `$` is the dangerous failure, not a missing one. Also records that CFE/CESPT/Telnor have no unit suite — they predate the gas/predial extension and were only verified end to end. Cross-linked from the spec, POLICY_OCR.md, PLAN.md, README and RESUME.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -134,7 +134,7 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
|
||||
- **Receipt capture module — DONE** (2026-07-27). The legacy "Editor" replacement, built on the single-movement capture from step 6. Wires up the previously-unused `Transaction.outstanding` (NOPAGO): capture flag on `POST /billing`, `?outstanding=` list filter, `POST /billing/:id/resolve-outstanding` (gated `ledger:create`, not `ledger:void` — resolving *completes* a capture), and exclusion from every balance aggregate exactly as the legacy `SALDOS ULTIMO 0`'s `HAVING NOPAGO = 0` did. Adds `POST /billing/batch` (one `$transaction`, check-level fields shared, per-line customer/amount) and `GET /billing/by-check`, plus the `cheque-count` report replacing `REPORTE CHEQUE COUNT` / `REPORTE POR CHEQUE` / `EDITA CHEQUE ALF|COUNT|NUM` — print/PDF/CSV/XLSX come free from the existing `/reportes/:slug` machinery. Web: `/estado-cuenta/lote` (the actual "Editor" screen, with live reconciliation against the physical check amount), plus an "Estado de pago" filter, a "sin fondos" row tag and a Resolver dialog on `/estado-cuenta`. No new abilities. Verified end-to-end against dev, API + browser.
|
||||
**Two pre-existing bugs found and fixed while building it:** (a) `statement()` filtered `legacySourceTable: { notIn: [...] }`, which compiles to SQL `NOT IN` — and `NULL NOT IN (…)` is NULL, so **every app-captured movement was invisible on the customer statement** (438 rows in the movement browser vs 392 on the statement) while still appearing everywhere else. This would have made the whole receipt-capture feature look broken to staff. Now NULL-safe. (b) The balances *count* query omitted the void filter its own page query applied, so the row count disagreed with the rows.
|
||||
**OCR seam:** `BillingService.createBatch(dto, opts)` is the single multi-row write path and carries three contract guarantees for the step-11 OCR module to post through — `items[i]` maps to `lines[i]` (so `StatementDocument.postedTransactionId` can be zipped back on), `opts.refs[i]` stamps `captureRef` with a duplicate-post guard that a *voided* row deliberately does not block, and `opts.source` is service-level only so an HTTP client cannot label hand-keyed rows as machine-captured. Backed by a new `TransactionCaptureSource` enum (MANUAL/BATCH/OCR) + `captureRef`, both nullable so the 40,136 migrated rows stay NULL rather than being mislabelled.
|
||||
- **PDF/OCR auto-capture — DONE** (2026-08-01). The ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff key in by hand, built in `apps/api/src/statements/` and posting through §1.2's `createBatch` seam with `source: "OCR"` and a per-document `captureRef`. Web: `/recibos` + `/recibos/:id`. Abilities `statement:ingest`/`statement:review` (STAFF — the review step is what makes machine capture safe at that tier). OCR is self-hosted **Tesseract** behind a swappable `OcrProvider` interface; `tesseract-ocr`, `tesseract-ocr-data-spa` and `poppler-utils` were added to the API image.
|
||||
- **PDF/OCR auto-capture — DONE** (2026-08-01). As-built write-up in [`docs/STATEMENT_OCR.md`](docs/STATEMENT_OCR.md); the design and the measured evidence stay in the spec's §2. The ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff key in by hand, built in `apps/api/src/statements/` and posting through §1.2's `createBatch` seam with `source: "OCR"` and a per-document `captureRef`. Web: `/recibos` + `/recibos/:id`. Abilities `statement:ingest`/`statement:review` (STAFF — the review step is what makes machine capture safe at that tier). OCR is self-hosted **Tesseract** behind a swappable `OcrProvider` interface; `tesseract-ocr`, `tesseract-ocr-data-spa` and `poppler-utils` were added to the API image.
|
||||
**Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**.
|
||||
**The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`.
|
||||
**Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost.
|
||||
@@ -168,6 +168,8 @@ Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a r
|
||||
|
||||
**Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown.
|
||||
|
||||
Each of the two OCR intakes now has an as-built doc separate from its spec — `docs/STATEMENT_OCR.md` and `docs/POLICY_OCR.md`. The specs record what was designed and why; those record what is in the code. They share one `OcrProvider` seam (`apps/api/src/ocr/`), so the Tesseract-vs-managed-API decision is one line for both.
|
||||
|
||||
**It also produced a feature nobody planned.** The statement OCR pipeline generalised: the same render→OCR→parse→match→review shape reads **carrier policy PDFs** into `Policy` rows, which is `docs/POLICY_OCR.md` (built 2026-08-01, GMX only so far). It belongs to step 12's subject matter but to step 11's lineage, and it is in no spec — worth knowing before reading `INSURANCE_FEATURES_SPEC.md`, which does not mention it. It also partly overlaps what §4's carrier API was wanted for, and unlike that section it is not blocked on a phone call.
|
||||
|
||||
**Step 12 is one-quarter built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. **§1 renewal emails is done** (2026-08-01/02) and carries a BUILT note recording the three places the build diverged from the spec; §2 liquidación is still the smallest remaining piece, since the per-policy fields are already wired end to end.
|
||||
|
||||
Reference in New Issue
Block a user