rmancinasandClaude Opus 5
d6501f1d74
Adds four parsers to the statement intake — GAS TIJUANA plus one per municipality, because Tijuana, Rosarito and Ensenada issue three completely different predial documents — and a text-layer fast path for the born-digital invoices the gas company sends. Measured against a new corpus of 14 documents / 29 pages: provider read on 29/29, amount on 26/29, and 21/29 auto-matched against the dev database (22/29 identified). The eight review cases are all legitimate. Five things the corpus forced: - Not every statement is a scan. The gas invoices are born-digital CFDIs whose text layer is exact; rasterising them only loses information (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new `OcrProvider.textPages` reads the embedded layer via `pdftotext -bbox-layout` — same poppler package as `pdftoppm`, so no new dependency — and OCR stays the fallback for real scans. Poppler's own `<line>` grouping follows text flow rather than the page, so words are regrouped by vertical position; without that, a two-column header leaves every label separated from the value printed beside it. - The clave catastral is not two letters and six digits. Position three is a letter in 15 of the 932 stored claves, and digitising the whole tail mapped a real `MMB01041` to a nonexistent `MM801041`. - Tijuana predial prints no clave at all. Its only identifier is an 8-digit municipal account carried in a 32-digit payment barcode, which the legacy database never held, so it goes in `meterNumber` alongside gas — `accountNumber` holds `DATMEX.predial`, which is not a per-property key and must not be overwritten. Those pages start cold and are taught by the first confirm. - On Rosarito and Ensenada the clave is the primary key, not a fallback: those receipts print nothing else, so a unique hit auto-matches. On a utility bill that merely happens to print one it stays a review hint. - A misread `$` is the dangerous failure. An Ensenada receipt for $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large and look ordinary in the ledger. Predial amounts now require a literal `$` and a page that cannot produce one goes to review. The scoped match field is now one exported function rather than three copies of `kind === "GAS" ? ... : ...`, since the lookup, the blank-service fill and the confirm write-back have to agree or a reference gets learned into a column nothing searches. First tests in this package: 23 specs over the parsers and the text-layer reader, every fixture a verbatim OCR excerpt from a real receipt. Adds the jest config they need and a build tsconfig so they stay out of dist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
72 lines
3.0 KiB
TypeScript
72 lines
3.0 KiB
TypeScript
/**
|
|
* The OCR seam. Everything above this interface works in terms of page text and
|
|
* word boxes, so the concrete engine is swappable without touching the parsers,
|
|
* the matcher, or the schema.
|
|
*
|
|
* The shipped implementation is self-hosted Tesseract (see tesseract.provider).
|
|
* That choice is evidence-based rather than assumed: run against 46 pages of
|
|
* real scanned CFE, CESPT and Telnor statements, it identified the provider on
|
|
* 46/46 and extracted a usable account reference on 43/46, and on a later
|
|
* corpus of 19 scanned municipal predial receipts it read the provider on
|
|
* 19/19 and an identifier on 18/19 — well past the bar for a queue whose whole
|
|
* point is that a human confirms every row. A
|
|
* managed document-extraction API (Textract, Document Intelligence, Document
|
|
* AI) fits behind this same interface if per-page accuracy ever proves
|
|
* insufficient, with no schema change — but at 300+ pages/month/company it
|
|
* would carry a real recurring cost for accuracy that is not currently the
|
|
* bottleneck.
|
|
*/
|
|
|
|
/** One OCR'd word, with where it sits on the page. */
|
|
export interface OcrWord {
|
|
text: string;
|
|
/** Pixel box in the rendered page image. */
|
|
left: number;
|
|
top: number;
|
|
width: number;
|
|
height: number;
|
|
/** Engine confidence for this word, 0..1. */
|
|
confidence: number;
|
|
}
|
|
|
|
export interface OcrPage {
|
|
/** Full page text, reading order, newline-separated. */
|
|
text: string;
|
|
/**
|
|
* Word boxes. Needed because several of the real layouts are *tables* — the
|
|
* CESPT "RECIBO" prints `No. DE CUENTA` as a column header with the value in
|
|
* the row beneath it, which line-oriented text cannot associate. Parsers fall
|
|
* back to geometry for exactly those fields.
|
|
*/
|
|
words: OcrWord[];
|
|
/** Mean word confidence across the page, 0..1. */
|
|
confidence: number;
|
|
}
|
|
|
|
export interface OcrProvider {
|
|
/** True when the engine is actually usable in this deployment. */
|
|
available(): Promise<boolean>;
|
|
/** Split a PDF into one rendered page image per page. */
|
|
renderPages(pdf: Buffer): Promise<Buffer[]>;
|
|
/** OCR a single rendered page image. */
|
|
recognize(pageImage: Buffer): Promise<OcrPage>;
|
|
/**
|
|
* Read a PDF's own text layer, one entry per page, `null` where the page has
|
|
* none worth using.
|
|
*
|
|
* Not every statement is a scan. The gas company e-mails born-digital CFDI
|
|
* invoices whose text is already exact and already positioned — running those
|
|
* through a rasteriser and a character recogniser can only lose information
|
|
* (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`) while
|
|
* costing about a minute of CPU per page for the privilege. Where the layer
|
|
* exists it is strictly better input for the same parsers, so it is tried
|
|
* first and OCR remains the fallback for genuine scans.
|
|
*
|
|
* Positions are reported in the same pixel space `recognize` uses, so the
|
|
* geometric helpers in the parsers work unchanged on either source.
|
|
*/
|
|
textPages(pdf: Buffer): Promise<(OcrPage | null)[]>;
|
|
}
|
|
|
|
export const OCR_PROVIDER = Symbol("OCR_PROVIDER");
|