From d6501f1d74ce1a9e73491533e2f3bdaba900c761 Mon Sep 17 00:00:00 2001 From: Ricardo Mancinas Date: Sat, 1 Aug 2026 12:52:20 -0700 Subject: [PATCH] feat(recibos): OCR capture for gas butano and municipal predial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds four parsers to the statement intake — GAS TIJUANA plus one per municipality, because Tijuana, Rosarito and Ensenada issue three completely different predial documents — and a text-layer fast path for the born-digital invoices the gas company sends. Measured against a new corpus of 14 documents / 29 pages: provider read on 29/29, amount on 26/29, and 21/29 auto-matched against the dev database (22/29 identified). The eight review cases are all legitimate. Five things the corpus forced: - Not every statement is a scan. The gas invoices are born-digital CFDIs whose text layer is exact; rasterising them only loses information (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new `OcrProvider.textPages` reads the embedded layer via `pdftotext -bbox-layout` — same poppler package as `pdftoppm`, so no new dependency — and OCR stays the fallback for real scans. Poppler's own `` grouping follows text flow rather than the page, so words are regrouped by vertical position; without that, a two-column header leaves every label separated from the value printed beside it. - The clave catastral is not two letters and six digits. Position three is a letter in 15 of the 932 stored claves, and digitising the whole tail mapped a real `MMB01041` to a nonexistent `MM801041`. - Tijuana predial prints no clave at all. Its only identifier is an 8-digit municipal account carried in a 32-digit payment barcode, which the legacy database never held, so it goes in `meterNumber` alongside gas — `accountNumber` holds `DATMEX.predial`, which is not a per-property key and must not be overwritten. Those pages start cold and are taught by the first confirm. - On Rosarito and Ensenada the clave is the primary key, not a fallback: those receipts print nothing else, so a unique hit auto-matches. On a utility bill that merely happens to print one it stays a review hint. - A misread `$` is the dangerous failure. An Ensenada receipt for $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large and look ordinary in the ledger. Predial amounts now require a literal `$` and a page that cannot produce one goes to review. The scoped match field is now one exported function rather than three copies of `kind === "GAS" ? ... : ...`, since the lookup, the blank-service fill and the confirm write-back have to agree or a reference gets learned into a column nothing searches. First tests in this package: 23 specs over the parsers and the text-layer reader, every fixture a verbatim OCR excerpt from a real receipt. Adds the jest config they need and a build tsconfig so they stay out of dist. Co-Authored-By: Claude Opus 5 --- apps/api/jest.config.js | 7 + apps/api/nest-cli.json | 3 +- apps/api/src/statements/ocr/ocr.provider.ts | 30 +- .../statements/ocr/tesseract.provider.spec.ts | 67 ++++ .../src/statements/ocr/tesseract.provider.ts | 155 ++++++++ .../parsers/statement-parser.spec.ts | 194 ++++++++++ .../statements/parsers/statement-parser.ts | 351 +++++++++++++++++- .../statements/statement-matcher.service.ts | 100 +++-- apps/api/src/statements/statements.service.ts | 22 +- apps/api/tsconfig.build.json | 4 + apps/web/src/components/StatementIntake.tsx | 10 +- docs/RECEIPT_CAPTURE_SPEC.md | 55 ++- 12 files changed, 940 insertions(+), 58 deletions(-) create mode 100644 apps/api/jest.config.js create mode 100644 apps/api/src/statements/ocr/tesseract.provider.spec.ts create mode 100644 apps/api/src/statements/parsers/statement-parser.spec.ts create mode 100644 apps/api/tsconfig.build.json diff --git a/apps/api/jest.config.js b/apps/api/jest.config.js new file mode 100644 index 0000000..6fe518f --- /dev/null +++ b/apps/api/jest.config.js @@ -0,0 +1,7 @@ +/** @type {import('jest').Config} */ +module.exports = { + rootDir: "src", + testEnvironment: "node", + testRegex: ".*\\.spec\\.ts$", + transform: { "^.+\\.ts$": "ts-jest" }, +}; diff --git a/apps/api/nest-cli.json b/apps/api/nest-cli.json index f9aa683..2f068a1 100644 --- a/apps/api/nest-cli.json +++ b/apps/api/nest-cli.json @@ -3,6 +3,7 @@ "collection": "@nestjs/schematics", "sourceRoot": "src", "compilerOptions": { - "deleteOutDir": true + "deleteOutDir": true, + "tsConfigPath": "tsconfig.build.json" } } diff --git a/apps/api/src/statements/ocr/ocr.provider.ts b/apps/api/src/statements/ocr/ocr.provider.ts index ba2dfd8..bdcdd8b 100644 --- a/apps/api/src/statements/ocr/ocr.provider.ts +++ b/apps/api/src/statements/ocr/ocr.provider.ts @@ -6,8 +6,10 @@ * The shipped implementation is self-hosted Tesseract (see tesseract.provider). * That choice is evidence-based rather than assumed: run against 46 pages of * real scanned CFE, CESPT and Telnor statements, it identified the provider on - * 46/46 and extracted a usable account reference on 43/46, which is well past - * the bar for a queue whose whole point is that a human confirms every row. A + * 46/46 and extracted a usable account reference on 43/46, and on a later + * corpus of 19 scanned municipal predial receipts it read the provider on + * 19/19 and an identifier on 18/19 — well past the bar for a queue whose whole + * point is that a human confirms every row. A * managed document-extraction API (Textract, Document Intelligence, Document * AI) fits behind this same interface if per-page accuracy ever proves * insufficient, with no schema change — but at 300+ pages/month/company it @@ -31,10 +33,10 @@ export interface OcrPage { /** Full page text, reading order, newline-separated. */ text: string; /** - * Word boxes. Needed because two of the three real layouts are *tables* — - * the CESPT "RECIBO" prints `No. DE CUENTA` as a column header with the - * value in the row beneath it, which line-oriented text cannot associate. - * Parsers fall back to geometry for exactly those fields. + * Word boxes. Needed because several of the real layouts are *tables* — the + * CESPT "RECIBO" prints `No. DE CUENTA` as a column header with the value in + * the row beneath it, which line-oriented text cannot associate. Parsers fall + * back to geometry for exactly those fields. */ words: OcrWord[]; /** Mean word confidence across the page, 0..1. */ @@ -48,6 +50,22 @@ export interface OcrProvider { renderPages(pdf: Buffer): Promise; /** OCR a single rendered page image. */ recognize(pageImage: Buffer): Promise; + /** + * Read a PDF's own text layer, one entry per page, `null` where the page has + * none worth using. + * + * Not every statement is a scan. The gas company e-mails born-digital CFDI + * invoices whose text is already exact and already positioned — running those + * through a rasteriser and a character recogniser can only lose information + * (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`) while + * costing about a minute of CPU per page for the privilege. Where the layer + * exists it is strictly better input for the same parsers, so it is tried + * first and OCR remains the fallback for genuine scans. + * + * Positions are reported in the same pixel space `recognize` uses, so the + * geometric helpers in the parsers work unchanged on either source. + */ + textPages(pdf: Buffer): Promise<(OcrPage | null)[]>; } export const OCR_PROVIDER = Symbol("OCR_PROVIDER"); diff --git a/apps/api/src/statements/ocr/tesseract.provider.spec.ts b/apps/api/src/statements/ocr/tesseract.provider.spec.ts new file mode 100644 index 0000000..71865ec --- /dev/null +++ b/apps/api/src/statements/ocr/tesseract.provider.spec.ts @@ -0,0 +1,67 @@ +import { parseBboxLayout } from "./tesseract.provider"; + +/** + * Shaped like real `pdftotext -bbox-layout` output: the gas invoice lays its + * header out as two columns of independent text flows, so poppler puts a label + * and the value printed beside it in *different* `` elements. Trusting + * that grouping is what left `PERIODO FACTURADO` with no value next to it and + * every period field empty on a batch whose text was perfectly readable. + */ +function word(x: number, y: number, text: string): string { + return `${text}`; +} + +function doc(...lines: string[]): string { + return `${lines + .map((l) => `${l}`) + .join("")}`; +} + +/** Enough words on the page to clear the "is this a real text layer" floor. */ +function padding(): string { + return Array.from({ length: 50 }, (_, i) => word(10, 400 + i * 10, `w${i}`)).join(""); +} + +describe("parseBboxLayout", () => { + it("rejoins a label with the value printed beside it in another flow", () => { + const [page] = parseBboxLayout( + doc( + word(20, 100, "PERIODO") + word(45, 100, "FACTURADO:"), + word(300, 100.4, "20260630-20260630"), + padding(), + ), + 1, + ); + expect(page).not.toBeNull(); + expect(page!.text).toContain("PERIODO FACTURADO: 20260630-20260630"); + }); + + it("keeps genuinely separate lines apart", () => { + const [page] = parseBboxLayout( + doc(word(20, 100, "Cuenta:") + word(80, 100, "0900003463"), word(20, 130, "Nombre:"), padding()), + 1, + ); + expect(page!.text.split("\n")).toContain("Cuenta: 0900003463"); + expect(page!.text.split("\n")).toContain("Nombre:"); + }); + + it("scales point coordinates into the render's pixel space", () => { + // Word boxes have to land in the same coordinate space tesseract reports, + // or the geometric helpers the parsers share silently stop finding values. + const [page] = parseBboxLayout(doc(word(72, 144, "X") + padding()), 300 / 72); + const x = page!.words.find((w) => w.text === "X")!; + expect(x.left).toBeCloseTo(300); + expect(x.top).toBeCloseTo(600); + }); + + it("reports no text layer for a scan carrying a few stray glyphs", () => { + expect(parseBboxLayout(doc(word(10, 10, "3") + word(40, 10, "of") + word(60, 10, "5")), 1)).toEqual([ + null, + ]); + }); + + it("decodes the entities poppler escapes", () => { + const [page] = parseBboxLayout(doc(word(10, 10, "A&B") + padding()), 1); + expect(page!.text).toContain("A&B"); + }); +}); diff --git a/apps/api/src/statements/ocr/tesseract.provider.ts b/apps/api/src/statements/ocr/tesseract.provider.ts index b97673f..e25256c 100644 --- a/apps/api/src/statements/ocr/tesseract.provider.ts +++ b/apps/api/src/statements/ocr/tesseract.provider.ts @@ -105,6 +105,37 @@ export class TesseractOcrProvider implements OcrProvider { }); } + /** + * `pdftotext -bbox-layout` — the same poppler package `pdftoppm` comes from, + * so this costs no extra dependency in the runtime image. + * + * A page is only accepted when it carries a real text layer. Scanned PDFs + * frequently contain a handful of stray glyphs (a scanner watermark, a page + * number stamped by the MFP), and treating those as the page's text would + * hand every parser an almost-empty string and silently take OCR out of the + * loop — so a floor of MIN_TEXT_WORDS words has to be present before the + * layer is believed. + */ + async textPages(pdf: Buffer): Promise<(OcrPage | null)[]> { + await this.require(); + return this.scratch(async (dir) => { + const src = join(dir, "in.pdf"); + await writeFile(src, pdf); + const out = join(dir, "out.html"); + try { + await run("pdftotext", ["-bbox-layout", src, out]); + } catch (err) { + this.logger.warn( + `pdftotext failed; falling back to OCR for this file: ${(err as Error).message}`, + ); + return []; + } + // Points to pixels at the render DPI, so word boxes from either source + // land in one coordinate space and `valueUnder`'s thresholds hold. + return parseBboxLayout(await readFile(out, "utf8"), this.dpi / 72); + }); + } + async recognize(pageImage: Buffer): Promise { await this.require(); return this.scratch(async (dir) => { @@ -136,6 +167,130 @@ export class TesseractOcrProvider implements OcrProvider { } } +/** + * Below this many words a "text layer" is scanner debris, not a document. + * The real born-digital samples carry 400+ words a page; the scanned ones + * carry none at all, so the exact threshold is not delicate. + */ +const MIN_TEXT_WORDS = 40; + +const ENTITIES: Record = { + amp: "&", + lt: "<", + gt: ">", + quot: '"', + apos: "'", +}; + +function decodeEntities(s: string): string { + return s.replace(/&(#x?[0-9a-fA-F]+|[a-z]+);/g, (whole, body: string) => { + if (body[0] === "#") { + const code = + body[1] === "x" || body[1] === "X" + ? parseInt(body.slice(2), 16) + : parseInt(body.slice(1), 10); + return Number.isFinite(code) ? String.fromCodePoint(code) : whole; + } + return ENTITIES[body] ?? whole; + }); +} + +/** + * Turn `pdftotext -bbox-layout`'s XHTML into one OcrPage per PDF page. + * + * Parsed with regexes rather than an XML library on purpose: the output is + * machine-generated by poppler with a fixed element shape (`page` > `flow` > + * `block` > `line` > `word`), and the alternative is a parser dependency in + * the API for one file format read in one place. Only `page` and `word` are + * consulted — see below for why poppler's own `line` grouping is discarded. + * + * `confidence` is 1 for every word: these are the document's own characters, + * not a recognition guess. + */ +export function parseBboxLayout(xhtml: string, scale: number): (OcrPage | null)[] { + const pages: (OcrPage | null)[] = []; + + for (const pageMatch of xhtml.matchAll(/]*>([\s\S]*?)<\/page>/g)) { + const words: OcrWord[] = []; + + for (const w of pageMatch[1].matchAll( + /([\s\S]*?)<\/word>/g, + )) { + const text = decodeEntities(w[5]).trim(); + if (!text) continue; + const left = Number(w[1]) * scale; + const top = Number(w[2]) * scale; + words.push({ + text, + left, + top, + width: Number(w[3]) * scale - left, + height: Number(w[4]) * scale - top, + confidence: 1, + }); + } + + pages.push( + words.length >= MIN_TEXT_WORDS + ? { text: toVisualRows(words), words, confidence: 1 } + : null, + ); + } + + return pages; +} + +/** + * Reassemble words into the rows a reader sees, left to right. + * + * Poppler's own `` grouping cannot be used for this. It groups by text + * flow, and these invoices lay their fields out as two columns of independent + * flows — so `PERIODO FACTURADO:` and the `20260630-20260630` printed beside + * it end up in different `` elements, and every label-then-value pattern + * in the parsers misses a value that is plainly there on the page. Regrouping + * by vertical position restores the adjacency, and matches what tesseract + * hands back for the scanned version of the same layout. + * + * Rows are cut when a word's vertical centre leaves the band established by + * the row's first word, which tolerates the sub-pixel baseline differences + * between fonts on one line without merging two genuinely separate lines. + */ +function toVisualRows(words: OcrWord[]): string { + const centre = (w: OcrWord) => w.top + w.height / 2; + const sorted = [...words].sort((a, b) => centre(a) - centre(b) || a.left - b.left); + + const rows: OcrWord[][] = []; + let current: OcrWord[] = []; + let band = 0; + + for (const w of sorted) { + if (!current.length) { + current = [w]; + band = centre(w); + continue; + } + // Half the word's own height: tall headings and body text both sit within + // their own line's band, and neither reaches into the next one. + if (Math.abs(centre(w) - band) <= Math.max(w.height, current[0].height) / 2) { + current.push(w); + } else { + rows.push(current); + current = [w]; + band = centre(w); + } + } + if (current.length) rows.push(current); + + return rows + .map((r) => + [...r] + .sort((a, b) => a.left - b.left) + .map((w) => w.text) + .join(" "), + ) + .join("\n"); +} + /** * Turn tesseract's TSV into words plus reassembled text. * diff --git a/apps/api/src/statements/parsers/statement-parser.spec.ts b/apps/api/src/statements/parsers/statement-parser.spec.ts new file mode 100644 index 0000000..a9c042d --- /dev/null +++ b/apps/api/src/statements/parsers/statement-parser.spec.ts @@ -0,0 +1,194 @@ +import type { OcrPage } from "../ocr/ocr.provider"; +import { + detectProvider, + normalizeCadastralKey, + parseStatement, +} from "./statement-parser"; + +/** + * Every string in this file is a verbatim excerpt of what the OCR engine + * actually returned for a real receipt — misreads, dropped spaces, mangled + * accents and all. That is the point: these are the specific ways these five + * layouts have been observed to fail, and the assertions pin down what the + * parser is supposed to do about each one. Inventing clean input here would + * test nothing, because clean input was never the problem. + */ +function page(text: string): OcrPage { + return { text, words: [], confidence: 0.9 }; +} + +describe("detectProvider", () => { + it("reads a Rosarito predial receipt as predial, not as a water bill", () => { + // "Clave Catastral" is also a CESPT structural marker, so a predial page + // whose header OCR'd badly must still not be claimed by the CESPT rule. + expect( + detectProvider( + "e | Clave Catastral. KP-128-105 IMPUESTO PREDIAL ea rita\n" + + "TASA | VALOR FISCAL | BIMESTRES | INCISO. | IMPUESTO", + ), + ).toBe("PREDIAL ROSARITO"); + }); + + it("keeps telling the three municipalities apart by their RFC", () => { + expect(detectProvider("R.F.C. ATB-541201-KK2")).toBe("PREDIAL TIJUANA"); + expect(detectProvider("R.F.C. AMP-981201-HJ4")).toBe("PREDIAL ROSARITO"); + expect(detectProvider("MEN-540301-9J5")).toBe("PREDIAL ENSENADA"); + }); + + it("does not let the CFE rule claim a gas bill over 'PERIODO FACTURADO'", () => { + expect( + detectProvider("Orden de Facturación: 000009801640\nPERIODO FACTURADO: 20260630-20260630"), + ).toBe("GAS TIJUANA"); + }); +}); + +describe("normalizeCadastralKey", () => { + it("keeps a letter in the third position instead of digitising it", () => { + // `MMB01041` is a real key on file; mapping its B to 8 produced a key that + // matches no property at all. + expect(normalizeCadastralKey("MM-B01-041", [])).toBe("MMB01041"); + }); + + it("repairs the spurious I tesseract inserts into the prefix", () => { + expect(normalizeCadastralKey("MIM-200-010", [])).toBe("MM200010"); + }); + + it("digitises confusable glyphs from position four onward", () => { + expect(normalizeCadastralKey("KP-1O8-O45", [])).toBe("KP108045"); + }); + + it("flags a prefix it had to truncate", () => { + const notes: string[] = []; + expect(normalizeCadastralKey("KPX-128-106", notes)).toBe("KP128106"); + expect(notes).toHaveLength(1); + }); +}); + +describe("parsePredialTijuana", () => { + const TIJUANA = page( + "Hats | AYUNTAMIENTO DE TIJUANA, BC $2,613.00 23/01/2026\n" + + "y) TELEFONO: 973-7000 R.F.C. ATB-541201-KK2\n" + + "ER AÑO VALOR FISCAL TASA IMPUESTO |CONCEPTO IMPORTE\n" + + "ED ca 2026 1,207,15778 246 2,969.61 1102 - IMPUESTO PREDIAL 2,969.61\n" + + "55164964310126000002613000054192\n" + + "se 0 O (54427 [a] | TOTALAPAGAR: 2,613.00\n" + + "Dc 1097 : FECHA VENCE : 31/ENE/2026", + ); + + it("splits the payment barcode into account, deadline and amount", () => { + const p = parseStatement(TIJUANA); + expect(p.provider).toBe("PREDIAL TIJUANA"); + expect(p.serviceKind).toBe("PROPERTY_TAX"); + expect(p.accountRef).toBe("55164964"); + expect(p.amount).toBe(2613); + expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-01-31"); + expect(p.period).toBe("2026"); + }); + + it("reads the printed total even when the space in the label is lost", () => { + // The real page OCR'd the label as "TOTALAPAGAR:", and it is that reading + // that cross-checks the barcode's amount. + expect(parseStatement(TIJUANA).crossChecked).toBe(true); + }); + + it("refuses to trust a barcode the printed total contradicts", () => { + const p = parseStatement( + page( + "R.F.C. ATB-541201-KK2\n" + + "55164964310126000002613000054192\n" + + "TOTAL A PAGAR: 9,613.00\nFECHA VENCE : 31/ENE/2026", + ), + ); + expect(p.crossChecked).toBe(false); + expect(p.notes.join(" ")).toContain("no coincide"); + }); +}); + +describe("parsePredialRosarito", () => { + it("takes the rounded Total, not the Sub Total printed above it", () => { + const p = parseStatement( + page( + "AYUNTAMIENTO MUNICIPAL DE PLAYAS DE ROSARITO, B.C.\n" + + "Ce Clave Catastral: + JR-400-008 7 | IMPUESTO PREDIAL\n" + + "SUPERFICIE: 228.31 ZONA 30025 “Redondeo IT049 -$0.39 Sub Total $5,409.39\n" + + "¿XTEMPORANEO DESPUES DE: 31/01/2026 Elaboro: MGLG\n" + + "Total | $5,409.00\n" + + "| Periodo por Pagar: 2026/1 2026/6", + ), + ); + expect(p.cadastralKey).toBe("JR400008"); + expect(p.amount).toBe(5409); + expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-01-31"); + expect(p.period).toBe("2026"); + }); + + it("is not fooled by the unspaced 'SubTotal' spelling", () => { + // This exact page read $9,624.85 off a receipt for $9,625.00 while the + // lookbehind still assumed a space. + const p = parseStatement( + page( + "AMP-981201-HJ4 IMPUESTO PREDIAL\n" + + "SUPERFICIE. 367.62 ZONA:30151 | Redondco 17049 $0.15 SubTotal $9,624.85\n" + + ": Total | $9,625.00", + ), + ); + expect(p.amount).toBe(9625); + }); +}); + +describe("parsePredialEnsenada", () => { + const totals = (tail: string) => + page( + "IMPRESION MAQUINA REGISTRADORA ez | MUNICIPIO DE ENSENADA\n" + + "+7] DATOS. DEL.CAUSANTE alta A pe CLAVE MM-200-010 2 CUENTA\n" + + `ES g € S| TOTALES 12,744.47 0.00 0.00 324.56 0.00 13,069.03 ${tail} |`, + ); + + it("reads the paid total off the TOTALES row however the label OCR'd", () => { + expect(parseStatement(totals("TOTA LA A $5,797.00")).amount).toBe(5797); + expect(parseStatement(totals("orAL: M7 z] $14,414.00")).amount).toBe(14414); + expect(parseStatement(totals("| TOTAL: = $6 246.00")).amount).toBe(6246); + }); + + it("reports no amount rather than one whose $ was misread as an 8", () => { + // `TOTAL: A 82,203.00` is a $2,203.00 receipt. Posting $82,203 would look + // entirely ordinary in the ledger, so this page must go to review instead. + const p = parseStatement(totals("TOTAL: A 82,203.00")); + expect(p.amount).toBeNull(); + expect(p.notes.join(" ")).toContain("capturarlo a mano"); + }); + + it("never falls back to the assessed total on the same row", () => { + expect(parseStatement(totals("yo: se TE= 58/4690]")).amount).toBeNull(); + }); +}); + +describe("parseGas", () => { + const gas = (...cuentas: string[]) => + page( + "GTI4608032K2 COMPAÑIA DE GAS DE TIJUANA\n" + + "Fecha de Vencimiento: 2026/08/08\n" + + cuentas.map((c) => `Cuenta: ${c}`).join("\n") + + "\nPERIODO FACTURADO: 20260630-20260630\nTOTAL A PAGAR: $275.82", + ); + + it("strips the printed leading zero to the stored account number", () => { + const p = parseStatement(gas("0900003463", "0900003463", "0900003463")); + expect(p.serviceKind).toBe("GAS"); + expect(p.accountRef).toBe("900003463"); + expect(p.amount).toBe(275.82); + expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-08-08"); + expect(p.period).toBe("2026-06"); + expect(p.crossChecked).toBe(true); + }); + + it("takes the majority reading but still sends a disagreement to review", () => { + const p = parseStatement(gas("0900003463", "0900003463", "0900003468")); + expect(p.accountRef).toBe("900003463"); + expect(p.crossChecked).toBe(false); + }); + + it("claims no cross-check from a single printing", () => { + expect(parseStatement(gas("0900003463")).crossChecked).toBeNull(); + }); +}); diff --git a/apps/api/src/statements/parsers/statement-parser.ts b/apps/api/src/statements/parsers/statement-parser.ts index 17a7b12..6f65bfb 100644 --- a/apps/api/src/statements/parsers/statement-parser.ts +++ b/apps/api/src/statements/parsers/statement-parser.ts @@ -7,7 +7,11 @@ import type { OcrPage, OcrWord } from "../ocr/ocr.provider"; * like with like and never has to know about provider-specific formatting. */ export interface ParsedStatement { - /** "CFE" | "CESPT" | "TELNOR", or null when no parser claimed the page. */ + /** + * "CFE" | "CESPT" | "TELNOR" | "GAS TIJUANA" | "PREDIAL TIJUANA" | + * "PREDIAL ROSARITO" | "PREDIAL ENSENADA", or null when no parser claimed + * the page. + */ provider: string | null; serviceKind: ServiceKind | null; accountRef: string | null; @@ -90,6 +94,16 @@ function firstMatch(text: string, patterns: RegExp[]): string | null { return null; } +/** Every capture of `pattern` across the page, in order. */ +function allMatches(text: string, pattern: RegExp): string[] { + const out: string[] = []; + const re = new RegExp(pattern.source, pattern.flags.includes("g") ? pattern.flags : `${pattern.flags}g`); + for (const m of text.matchAll(re)) { + if (m[1]) out.push(m[1].trim()); + } + return out; +} + const MONTHS: Record = { ENE: 0, FEB: 1, MAR: 2, ABR: 3, MAY: 4, JUN: 5, JUL: 6, AGO: 7, SEP: 8, OCT: 9, NOV: 10, DIC: 11, @@ -104,15 +118,16 @@ export function parseDate(raw: string | null | undefined): Date | null { let m = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/); if (m) return utc(+m[3], +m[2] - 1, +m[1]); - // 22-JUL-2026 / 22 JUN 26 - m = s.match(/^(\d{1,2})[-\s]([A-Z]{3})[A-Z]*[-\s](\d{2,4})$/); + // 22-JUL-2026 / 22 JUN 26 / 31/ENE/2026 (Tijuana predial) + m = s.match(/^(\d{1,2})[-\s/]([A-Z]{3})[A-Z]*[-\s/](\d{2,4})$/); if (m && MONTHS[m[2]] !== undefined) { const y = m[3].length === 2 ? 2000 + +m[3] : +m[3]; return utc(y, MONTHS[m[2]], +m[1]); } - // 2026-07-22 (already normalised, e.g. decoded from a barcode) - m = s.match(/^(\d{4})-(\d{2})-(\d{2})$/); + // 2026-07-22 (already normalised, e.g. decoded from a barcode) and the + // 2026/08/08 the gas bill prints — same field order, different separator. + m = s.match(/^(\d{4})[-/](\d{2})[-/](\d{2})$/); if (m) return utc(+m[1], +m[2] - 1, +m[3]); return null; @@ -174,9 +189,30 @@ const BRAND: [string, RegExp][] = [ ["CFE", /comisi[oó]n federal de electricidad|CFE.?contigo|Suministrador de Servicios/i], ["CESPT", /CESPT|COMISI[OÓ]N ESTATAL DE SERVICIOS/i], ["TELNOR", /TELNOR|TELEFONOS DEL NOROESTE/i], + ["GAS TIJUANA", /COMPA[ÑN][IÍ]?A\s*DE\s*GAS\s*DE\s*TIJUANA|bajagas/i], + // The municipal RFCs are the single most reliable discriminator on a predial + // receipt: they are printed in a clean monospaced run on every layout, they + // never change, and they say which of the three city treasuries issued the + // page — which the wordmarks alone do not, since a Tijuana receipt also + // carries "PLAYAS DE TIJUANA" and a Rosarito one "TIJUANA ENSENADA". + ["PREDIAL TIJUANA", /AYUNTAMIENTO\s*DE\s*TIJUANA|ATB.?541201/i], + ["PREDIAL ROSARITO", /AYUNTAMIENTO\s*MUNICIPAL\s*DE\s*PLAYAS\s*DE\s*ROSARITO|AMP.?981201|rosarito\.gob/i], + ["PREDIAL ENSENADA", /MUNICIPIO\s*DE\s*ENSENADA|MEN.?540301/i], ]; +/** + * The predial rules come first because a Rosarito receipt prints "Clave + * Catastral" as a boxed label — the very string the CESPT structural rule + * looks for — so a page whose municipal header failed to OCR would otherwise + * be claimed as a water bill and matched against the wrong column entirely. + * "IMPUESTO PREDIAL" appears on all three municipal layouts and on none of the + * utility ones, so it is the safe first question to ask. + */ const LAYOUT: [string, RegExp][] = [ + ["PREDIAL TIJUANA", /IMPUESTO\s*PREDIAL[\s\S]*?(?:CERTIFICACION\s*DE\s*CAJA|PASEO\s*DEL\s*CENTENARIO|PAGA\s*TU\s*PREDIAL)/i], + ["PREDIAL ENSENADA", /(?:IMPUESTO\s*PREDIAL[\s\S]*?TRANSPENINSULAR)|(?:IMPRESION\s*MAQUINA\s*REGISTRADORA)/i], + ["PREDIAL ROSARITO", /IMPUESTO\s*PREDIAL/i], + ["GAS TIJUANA", /Orden\s*de\s*Facturaci[oó]n|FACTOR\s*DE\s*PRESI[OÓ]N|GAS\s*LP/i], ["CFE", /NO\.?\s*DE\s*SERVICIO|L[IÍ]MITE\s*DE\s*PAGO|PERIODO\s*FACTURADO/i], ["CESPT", /SALDO\s+CORRIENTE|CLAVE\s*CATASTRAL|No\.?\s*DE\s*CUENTA/i], ["TELNOR", /Mes\s*de\s*Facturaci[oó]n|Pagar\s*antes\s*de/i], @@ -364,10 +400,315 @@ function parseTelnor(page: OcrPage): ParsedStatement { }; } +// --- GAS (Compañía de Gas de Tijuana / bajagas) ------------------------------ + +/** + * These arrive as born-digital CFDI PDFs rather than scans, so the text layer + * (see `TesseractOcrProvider.textPages`) usually reads them exactly and the + * patterns below only have to be tolerant enough for the scanned case. + * + * The account number is printed three times — supply address, fiscal data, and + * the payment stub at the foot — which is a free cross-check: three readings + * that agree are near-certainly right, and any disagreement means one of them + * was misread and the page deserves a human glance. + * + * `Cuenta` is what the matcher compares, not `Contrato`. The migration + * recovered gas references out of `PropertyService.notes` into `meterNumber` + * and what sat there is the 9-digit account (`900003463`), printed here with a + * leading zero as `0900003463`. + */ +function parseGas(page: OcrPage): ParsedStatement { + const text = page.text; + const notes: string[] = []; + + const seen = allMatches(text, /Cuenta\s*[:;.]?\s*([0-9OIlSBD]{6,12})/i).map((s) => + toDigits(s).replace(/^0+/, ""), + ); + const distinct = [...new Set(seen.filter(Boolean))]; + + let accountRef: string | null = null; + let crossChecked: boolean | null = null; + if (distinct.length === 1) { + accountRef = distinct[0]; + if (seen.length > 1) crossChecked = true; + } else if (distinct.length > 1) { + // Majority wins — the stub and the two address blocks print the same + // number, so a single divergent reading is the misread one. It still goes + // to review: `crossChecked: false` is what keeps the batch from + // auto-matching a number one of three readings disagreed with. + const tally = new Map(); + for (const s of seen) tally.set(s, (tally.get(s) ?? 0) + 1); + accountRef = [...tally.entries()].sort((a, b) => b[1] - a[1])[0][0]; + crossChecked = false; + notes.push(`el número de cuenta se leyó de ${distinct.length} formas distintas (${distinct.join(", ")})`); + } + + const amount = money( + firstMatch(text, [ + /TOTAL\s*A\s*PAGAR\s*[:;.]?\s*\$\s*([\d,]+\.\d{2})/i, + /Total\s*a\s*pagar\s*[:;.]?\s*\$\s*([\d,]+\.\d{2})/i, + ]), + ); + + // `20260630-20260630` — the range the bill was cut for. Both ends are the + // same reading date on every sample, so the period is reported as the ISO + // month rather than a range no ledger row would ever be searched by. + const facturado = firstMatch(text, [/PERIODO\s*FACTURADO\s*[:;.]?\s*(\d{8})\s*-\s*\d{8}/i]); + const period = facturado ? `${facturado.slice(0, 4)}-${facturado.slice(4, 6)}` : null; + + return { + provider: "GAS TIJUANA", + serviceKind: "GAS", + accountRef: accountRef || null, + cadastralKey: null, + amount, + dueDate: parseDate( + firstMatch(text, [/Fecha\s*de\s*Vencimiento\s*[:;.]?\s*(\d{4}\s*\/\s*\d{2}\s*\/\s*\d{2})/i])?.replace( + /\s/g, + "", + ), + ), + period, + crossChecked, + notes, + }; +} + +// --- PREDIAL (municipal property tax) --------------------------------------- + +/** + * Normalise a printed clave catastral to the eight-character form + * `Property.cadastralKey` holds. The municipalities print it grouped + * (`KP-128-106`, `MM-B01-041`); the stored value drops the separators + * (`KP128106`, `MMB01041`). + * + * The shape is *not* two letters and six digits, which is the assumption that + * has to be resisted here. Across the 932 distinct claves on file, characters + * four through eight are digits without exception, but the third is a digit in + * 917 of them and one of `A`, `B`, `H`, `T` in the other fifteen. Running the + * whole tail through `toDigits` — which maps `B` to `8` — is what turned a real + * `MMB01041` into a nonexistent `MM801041`, so only positions four onward get + * that treatment and a letter in the third position is kept as printed. + * + * That leaves a genuine ambiguity at that one position: a `B` there might be a + * misread `8`, and 34 stored claves do carry an `8` there against six with a + * `B`. It is left as read rather than guessed, because a page that fails to + * match lands in the review queue where a human fixes it in seconds, while a + * page that matches the wrong property posts a charge to the wrong customer. + * + * The two-letter prefix is the other fragile part. Tesseract inserts a spurious + * `I` into letter pairs with some regularity — a real `MM-200-010` came back as + * `MIM-200-010` — so a run longer than two letters has its `I`/`L` dropped + * first, which recovers exactly that case. Anything still not two letters is + * truncated and flagged, because a wrong prefix silently matches the wrong + * property or, more often, nothing at all. + */ +export function normalizeCadastralKey( + raw: string, + notes: string[], +): string | null { + const m = raw.match(/^([A-Za-z|]{2,5})[-\s]?([A-Za-z0-9|]{3})[-\s]?([0-9OIlSBD]{3})$/); + if (!m) return null; + + let letters = m[1].toUpperCase().replace(/[^A-Z]/g, ""); + if (letters.length > 2) { + const stripped = letters.replace(/[IL]/g, ""); + if (stripped.length === 2) { + letters = stripped; + } else { + letters = letters.slice(0, 2); + notes.push(`la clave catastral se leyó como "${m[1]}"; se tomó "${letters}"`); + } + } + if (letters.length !== 2) return null; + + const third = m[2][0].toUpperCase(); + const tail = + (/[A-Z]/.test(third) ? third : toDigits(third)) + + toDigits(m[2].slice(1)) + + toDigits(m[3]); + + return tail.length === 6 ? letters + tail : null; +} + +/** The grouped clave as printed, anchored to its label when one survived OCR. */ +const GROUPED_CLAVE = "[A-Z|]{2,5}-[A-Z0-9OIlSBD]{3}-[0-9OIlSBD]{3}"; + +function findCadastralKey(text: string, notes: string[]): string | null { + const labelled = firstMatch(text, [ + new RegExp(`Clave\\s*Catastral\\s*[^A-Z0-9]{0,8}(${GROUPED_CLAVE})`, "i"), + new RegExp(`CLAVE\\s*[^A-Z0-9]{0,8}(${GROUPED_CLAVE})`, "i"), + ]); + if (labelled) return normalizeCadastralKey(labelled, notes); + + // Ensenada's label ("CLAVE") lands inside a table header that OCRs into + // noise more often than not, so the bare grouped shape is accepted as a + // fallback. It is distinctive enough — two letters and two three-character + // groups joined by hyphens appears nowhere else on these pages. + const bare = firstMatch(text, [new RegExp(`\\b(${GROUPED_CLAVE})\\b`)]); + return bare ? normalizeCadastralKey(bare, notes) : null; +} + +/** + * Tijuana: a "CERTIFICACIÓN DE CAJA" whose payment barcode is one 32-digit run + * of `account(8) + due date(DDMMYY) + amount(9) + folio(9)`, verified against + * all five sample pages. Municipal totals are whole pesos (the receipt itself + * carries a "Redondeo" line), so the barcode amount needs no decimal point. + * + * No clave catastral is printed anywhere on this layout — the 8-digit + * municipal account is the only identifier, and it is not a number the legacy + * database ever held. Until a reviewer confirms one, every Tijuana page lands + * in review; confirming teaches the matcher (see `learnAccountRefs`) so the + * same property matches itself next year. + */ +function parsePredialTijuana(page: OcrPage): ParsedStatement { + const text = page.text; + const notes: string[] = []; + + const barcode = text.match(/(? /TOTALES/i.test(l)) ?? ""; + const figures = allMatches(totalsRow, /\$\s*(\d[\d,.\s]*\.\d{2})/); + const amount = figures.length ? money(figures[figures.length - 1]) : null; + if (amount == null) { + notes.push("no se pudo leer el importe con certeza; capturarlo a mano"); + } + + return { + provider: "PREDIAL ENSENADA", + serviceKind: "PROPERTY_TAX", + accountRef: null, + cadastralKey: findCadastralKey(text, notes), + amount, + // This layout prints no payment deadline at all — it is a receipt for a + // payment already made at the municipal window. + dueDate: null, + period: firstMatch(text, [/A[ÑN]O\s*[\s\S]{0,60}?\b(20\d{2})\b/i]), + crossChecked: null, + notes, + }; +} + const PARSERS: Record ParsedStatement> = { CFE: parseCfe, CESPT: parseCespt, TELNOR: parseTelnor, + "GAS TIJUANA": parseGas, + "PREDIAL TIJUANA": parsePredialTijuana, + "PREDIAL ROSARITO": parsePredialRosarito, + "PREDIAL ENSENADA": parsePredialEnsenada, }; const EMPTY: ParsedStatement = { diff --git a/apps/api/src/statements/statement-matcher.service.ts b/apps/api/src/statements/statement-matcher.service.ts index fc9496f..6d86d7f 100644 --- a/apps/api/src/statements/statement-matcher.service.ts +++ b/apps/api/src/statements/statement-matcher.service.ts @@ -32,30 +32,46 @@ export interface MatchResult { * person. Names are displayed for the reviewer to sanity-check, and are never * an input to matching. */ +/** + * Which `PropertyService` column a given kind's statements actually print. + * + * Exported because the same answer governs three places that must agree: the + * lookup here, the blank-service fill on review, and the write-back on confirm. + * When they disagree, a reference gets learned into a column nothing searches, + * and the same page returns to the review queue every month forever. + * + * `meterNumber` is doing double duty for the two kinds whose printed reference + * DATMEX never held in `accountNumber`: + * - GAS, where the number lived in free-text notes, and + * - PROPERTY_TAX, where `accountNumber` holds DATMEX.predial — a 3-4 digit + * office file number that is neither unique nor printed on any statement. + * The Tijuana municipal receipt prints an 8-digit account and no clave + * catastral at all, so it needs a column of its own; overwriting the legacy + * predial numbers to make room would destroy the only link back to the + * original records. + */ +export function scopedRefField( + kind: ServiceKind, +): "accountNumber" | "meterNumber" | null { + switch (kind) { + case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu + case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua + case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono + case "FEDERAL_ZONE": + case "CABLE": + return "accountNumber"; + case "GAS": // bajagas "Cuenta" -> recovered from notes into meterNumber + case "PROPERTY_TAX": // Tijuana's 8-digit municipal account + return "meterNumber"; + default: + return null; + } +} + @Injectable() export class StatementMatcherService { constructor(private readonly prisma: PrismaService) {} - /** Which PropertyService column a given kind's statements actually print. */ - private fieldFor(kind: ServiceKind): "accountNumber" | "meterNumber" | null { - switch (kind) { - case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu - case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua - case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono - case "FEDERAL_ZONE": - case "CABLE": - return "accountNumber"; - case "GAS": // no account column in DATMEX; the number lived in notes - return "meterNumber"; - // PROPERTY_TAX deliberately has no scoped column: what its - // accountNumber holds is DATMEX.predial, which is neither unique nor - // printed on any statement. Predial bills match on the clave catastral - // alone — see matchByCadastralKey. - default: - return null; - } - } - async match(parsed: ParsedStatement, expectedKind: ServiceKind): Promise { const kind = parsed.serviceKind ?? expectedKind; @@ -68,33 +84,39 @@ export class StatementMatcherService { ); } - const field = this.fieldFor(kind); + const field = scopedRefField(kind); if (field && parsed.accountRef) { const hit = await this.byServiceField(kind, field, parsed.accountRef); if (hit) return hit; } - // Secondary key. The clave catastral is printed on CESPT bills as well as - // predial ones, so it rescues a page whose account number did not OCR — - // which happened on real samples, where the clave read cleanly and the - // account number did not. + // The clave catastral is printed on CESPT bills as well as predial ones, so + // it rescues a page whose account number did not OCR — which happened on + // real samples, where the clave read cleanly and the account number did + // not. On the Rosarito and Ensenada predial layouts it is not a rescue at + // all but the only identifier the receipt carries, so a unique hit there is + // as good as any account-number match and is treated as one. if (parsed.cadastralKey) { - const hit = await this.byCadastralKey(kind, parsed.cadastralKey); + const primary = kind === "PROPERTY_TAX" && !parsed.accountRef; + const hit = await this.byCadastralKey(kind, parsed.cadastralKey, primary); if (hit) return hit; } if (!field && !parsed.cadastralKey) { + return this.unmatched(`no hay campo de búsqueda definido para ${kind}`); + } + if (!parsed.accountRef && !parsed.cadastralKey) { return this.unmatched( kind === "PROPERTY_TAX" - ? "el predial sólo se puede identificar por clave catastral y no se leyó ninguna" - : `no hay campo de búsqueda definido para ${kind}`, + ? "no se leyó ni la clave catastral ni la cuenta municipal" + : "no se pudo leer la referencia de la cuenta", ); } return this.unmatched( parsed.accountRef ? `no se encontró ningún servicio de ${kind} con la referencia ${parsed.accountRef}` - : "no se pudo leer la referencia de la cuenta", + : `no se encontró ninguna propiedad con la clave catastral ${parsed.cadastralKey}`, ); } @@ -145,6 +167,8 @@ export class StatementMatcherService { private async byCadastralKey( kind: ServiceKind, key: string, + /** True when the clave is the identifier the statement was issued against. */ + primary: boolean, ): Promise { const props = await this.prisma.property.findMany({ where: { cadastralKey: key }, @@ -174,15 +198,21 @@ export class StatementMatcherService { }; } - // The clave identifies the property with certainty, but it is a *secondary* - // key: it was not the number the statement was issued against. Left for - // review so the confirm also teaches the matcher the account number, rather - // than the same page needing the fallback again next month. + // When the clave is the *secondary* key — a utility bill that also happens + // to print it — the page is left for review, because the clave was not the + // number the statement was issued against and confirming is what teaches + // the matcher the account number for next month. When it is the primary key + // (Rosarito and Ensenada predial, which print nothing else), a unique hit + // is a real match and there is no second number to learn. return { propertyServiceId: candidates[0].propertyServiceId ?? null, customerId: candidates[0].customerId, - note: `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`, - confident: false, + note: primary + ? `coincidencia exacta por clave catastral ${key}` + : `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`, + // A clave with no service row of the right kind behind it still needs a + // human: there is nothing to attach the posting to. + confident: primary && candidates[0].propertyServiceId != null, candidates, }; } diff --git a/apps/api/src/statements/statements.service.ts b/apps/api/src/statements/statements.service.ts index f4497b5..3fb9463 100644 --- a/apps/api/src/statements/statements.service.ts +++ b/apps/api/src/statements/statements.service.ts @@ -16,7 +16,7 @@ import { BillingService } from "../billing/billing.service"; import type { UploadedFileLike } from "../storage/upload-file"; import { OCR_PROVIDER, type OcrProvider } from "./ocr/ocr.provider"; import { parseStatement } from "./parsers/statement-parser"; -import { StatementMatcherService } from "./statement-matcher.service"; +import { StatementMatcherService, scopedRefField } from "./statement-matcher.service"; import type { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto"; /** @@ -127,14 +127,23 @@ export class StatementsService { await this.storage.put(sourceKey, file.buffer, "application/pdf"); const pages = await this.ocr.renderPages(file.buffer); - for (const image of pages) { + // Page images are still rendered and stored for every file, text layer or + // not: the review screen shows the reviewer the page, and "what the + // parser read" is only checkable against a picture of the paper. + const textLayer = await this.ocr.textPages(file.buffer).catch(() => []); + + for (const [index, image] of pages.entries()) { pageNumber += 1; const storageKey = `statement/${batchId}/page-${pageNumber}.png`; await this.storage.put(storageKey, image, "image/png"); try { - const ocr = await this.ocr.recognize(image); + const embedded = textLayer[index] ?? null; + const ocr = embedded ?? (await this.ocr.recognize(image)); const parsed = parseStatement(ocr); + if (embedded) { + parsed.notes.unshift("texto leído del PDF original, sin OCR"); + } const match = await this.matcher.match(parsed, serviceKind); const notes = [...parsed.notes, match.note].filter(Boolean); @@ -295,8 +304,8 @@ export class StatementsService { where: { id: doc.batchId }, select: { serviceKind: true }, }); - if (batch) { - const field = batch.serviceKind === "GAS" ? "meterNumber" : "accountNumber"; + const field = batch && scopedRefField(batch.serviceKind); + if (batch && field) { const blank = await this.prisma.propertyService.findMany({ where: { kind: batch.serviceKind, @@ -434,7 +443,8 @@ export class StatementsService { docs: { matchedPropertyServiceId: string | null; extractedAccountRef: string | null }[], kind: ServiceKind, ) { - const field = kind === "GAS" ? "meterNumber" : "accountNumber"; + const field = scopedRefField(kind); + if (!field) return; for (const d of docs) { if (!d.matchedPropertyServiceId || !d.extractedAccountRef) continue; await this.prisma.propertyService.updateMany({ diff --git a/apps/api/tsconfig.build.json b/apps/api/tsconfig.build.json new file mode 100644 index 0000000..9aa5801 --- /dev/null +++ b/apps/api/tsconfig.build.json @@ -0,0 +1,4 @@ +{ + "extends": "./tsconfig.json", + "exclude": ["node_modules", "dist", "**/*.spec.ts"] +} diff --git a/apps/web/src/components/StatementIntake.tsx b/apps/web/src/components/StatementIntake.tsx index 0404fb5..429c5ad 100644 --- a/apps/web/src/components/StatementIntake.tsx +++ b/apps/web/src/components/StatementIntake.tsx @@ -28,9 +28,15 @@ import type { ServiceKind, StatementBatch, StatementBatchStatus } from "@/lib/ty */ /** The kinds the parsers actually recognise today. */ -const SUPPORTED: ServiceKind[] = ["ELECTRIC", "WATER", "TELEPHONE"]; +const SUPPORTED: ServiceKind[] = [ + "ELECTRIC", + "WATER", + "TELEPHONE", + "GAS", + "PROPERTY_TAX", +]; /** Uploadable, but every page will land in review until a parser learns it. */ -const OTHER_KINDS: ServiceKind[] = ["GAS", "PROPERTY_TAX", "FEDERAL_ZONE", "CABLE"]; +const OTHER_KINDS: ServiceKind[] = ["FEDERAL_ZONE", "CABLE"]; const STATUS_LABEL: Record = { UPLOADED: "Recibido", diff --git a/docs/RECEIPT_CAPTURE_SPEC.md b/docs/RECEIPT_CAPTURE_SPEC.md index c21e065..dedd548 100644 --- a/docs/RECEIPT_CAPTURE_SPEC.md +++ b/docs/RECEIPT_CAPTURE_SPEC.md @@ -134,7 +134,8 @@ single-movement form. > **BUILT — 2026-08-01.** Implemented and verified end to end against real > scanned statements. `apps/api/src/statements/` holds the module: a swappable > `OcrProvider` seam with a self-hosted Tesseract implementation, per-provider -> parsers for CFE / CESPT / Telnor, a scoped matcher, and a review queue that +> parsers for CFE / CESPT / Telnor / gas / predial, a scoped matcher, and a +> review queue that > posts through `BillingService.createBatch` with `source: "OCR"`. Web: > the "Captura automática (OCR)" tab of the Captura screen (upload + batch > list) and `/recibos/:id` (review queue with the page image beside the @@ -207,6 +208,54 @@ single-movement form. > each bill (`9`, `405`, `406`); Tesseract read `405` as `205`. Handwriting is > a review hint at best and is deliberately not an input to matching. +> **EXTENDED — gas and predial, 2026-08-01.** A second corpus (14 documents, +> 29 pages: five municipal predial batches and ten gas invoices) added four +> parsers — `GAS TIJUANA` plus one per municipality, because Tijuana, Rosarito +> and Ensenada issue three completely different documents. End to end against +> the dev database that is **21/29 auto-matched, 22/29 identified**, with the +> provider read on 29/29 and an amount on 26/29. +> +> The eight review cases are all legitimate: five Tijuana pages whose municipal +> account is not yet on file (see below), one clave not in the book, one page +> too poorly scanned to read a clave at all, and one gas account shared by two +> services. Excluding the structural Tijuana case, that is 21/24. +> +> **Five things this corpus proved:** +> +> 1. **Not every statement is a scan.** The gas company sends born-digital CFDI +> invoices whose text layer is exact. Rasterising and re-recognising those +> can only lose information — one sample turned `MEDIDOR: VM01014426` into +> `ar (LTR): 014420` — so `OcrProvider.textPages` reads the embedded layer +> first (`pdftotext -bbox-layout`, same poppler package as `pdftoppm`) and +> OCR stays the fallback for real scans. Page images are still rendered and +> stored either way, because the reviewer needs to see the paper. +> 2. **The clave catastral is not two letters and six digits.** Positions four +> through eight are digits in all 932 stored claves, but the third is a +> letter in fifteen of them (`MMB01041`, `CGH52121`). Digitising the whole +> tail maps that `B` to an `8` and produces a key matching no property. +> 3. **Tijuana predial prints no clave catastral at all.** Its only identifier +> is an 8-digit municipal account, carried in a 32-digit payment barcode +> (`account(8) + DDMMYY + amount(9) + folio(9)`) that the legacy database +> never held. It goes in `PROPERTY_TAX.meterNumber` — the same column gas +> uses, and for the same reason: `accountNumber` holds `DATMEX.predial`, +> which is not a per-property key and overwriting it would destroy the only +> link back to the original records. So Tijuana pages start cold and are +> taught by the first confirm, exactly like gas. +> 4. **On Rosarito and Ensenada the clave is the primary key, not a fallback.** +> Those receipts print nothing else, so a unique clave hit there is a real +> match and auto-matches; on a utility bill that merely happens to print one +> it stays a review hint, as before. +> 5. **A misread `$` is the dangerous failure, not a missing one.** An Ensenada +> receipt for `$2,203.00` OCR'd as `82,203.00` — the dollar sign read as an +> 8, which would post a charge 37× too large and look entirely ordinary in +> the ledger. Every predial amount therefore requires a literal `$`, and a +> page that cannot produce one reports no amount and goes to review. Two of +> the 29 pages take that path, which is the correct outcome for both. +> +> Regression cover for all of the above lives in +> `statement-parser.spec.ts` and `tesseract.provider.spec.ts`; every fixture in +> them is a verbatim OCR excerpt from a real receipt. + ### Motivation (from the meeting) Each utility company (CFE, water, phone, gas...) sends 300+ individual @@ -330,8 +379,8 @@ Per the meeting notes' own field list: | Agua — Número de cuenta | `WATER` | `accountNumber` | `AGUA` | ✅ populated today | | Zona Fed — Número de Zona Federal | `FEDERAL_ZONE` | `accountNumber` | `ZFED` | ✅ populated today | | Tel — Número de teléfono | `TELEPHONE` *(new)* | `accountNumber` | `Property.phone1/2/3` (currently on `Property`, not `PropertyService`) | ⚠️ schema gap — see below | -| Impuesto — Clave Catastral | `PROPERTY_TAX` | `accountNumber` | migrated from `PREDIAL`, **not** `CLAVE` | ⚠️ needs verification — see below | -| Gas — Número de medidor | `GAS` | `meterNumber` | not populated — folded into free-text `notes` today | ⚠️ data gap — see below | +| Impuesto — Clave Catastral | `PROPERTY_TAX` | `Property.cadastralKey`, plus `meterNumber` for Tijuana's municipal account | `CLAVE`; `PREDIAL` is left on `accountNumber` and never matched against | ✅ built — see the 2026-08-01 extension note | +| Gas — Número de medidor | `GAS` | `meterNumber` | not populated — folded into free-text `notes` today | ✅ 160/334 recovered from `notes` | Confidence rule of thumb once a field is confirmed populated, tune after seeing real statements: