rmancinasandClaude Opus 5
d6501f1d74
Adds four parsers to the statement intake — GAS TIJUANA plus one per municipality, because Tijuana, Rosarito and Ensenada issue three completely different predial documents — and a text-layer fast path for the born-digital invoices the gas company sends. Measured against a new corpus of 14 documents / 29 pages: provider read on 29/29, amount on 26/29, and 21/29 auto-matched against the dev database (22/29 identified). The eight review cases are all legitimate. Five things the corpus forced: - Not every statement is a scan. The gas invoices are born-digital CFDIs whose text layer is exact; rasterising them only loses information (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new `OcrProvider.textPages` reads the embedded layer via `pdftotext -bbox-layout` — same poppler package as `pdftoppm`, so no new dependency — and OCR stays the fallback for real scans. Poppler's own `<line>` grouping follows text flow rather than the page, so words are regrouped by vertical position; without that, a two-column header leaves every label separated from the value printed beside it. - The clave catastral is not two letters and six digits. Position three is a letter in 15 of the 932 stored claves, and digitising the whole tail mapped a real `MMB01041` to a nonexistent `MM801041`. - Tijuana predial prints no clave at all. Its only identifier is an 8-digit municipal account carried in a 32-digit payment barcode, which the legacy database never held, so it goes in `meterNumber` alongside gas — `accountNumber` holds `DATMEX.predial`, which is not a per-property key and must not be overwritten. Those pages start cold and are taught by the first confirm. - On Rosarito and Ensenada the clave is the primary key, not a fallback: those receipts print nothing else, so a unique hit auto-matches. On a utility bill that merely happens to print one it stays a review hint. - A misread `$` is the dangerous failure. An Ensenada receipt for $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large and look ordinary in the ledger. Predial amounts now require a literal `$` and a page that cannot produce one goes to review. The scoped match field is now one exported function rather than three copies of `kind === "GAS" ? ... : ...`, since the lookup, the blank-service fill and the confirm write-back have to agree or a reference gets learned into a column nothing searches. First tests in this package: 23 specs over the parsers and the text-layer reader, every fixture a verbatim OCR excerpt from a real receipt. Adds the jest config they need and a build tsconfig so they stay out of dist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
68 lines
2.6 KiB
TypeScript
68 lines
2.6 KiB
TypeScript
import { parseBboxLayout } from "./tesseract.provider";
|
|
|
|
/**
|
|
* Shaped like real `pdftotext -bbox-layout` output: the gas invoice lays its
|
|
* header out as two columns of independent text flows, so poppler puts a label
|
|
* and the value printed beside it in *different* `<line>` elements. Trusting
|
|
* that grouping is what left `PERIODO FACTURADO` with no value next to it and
|
|
* every period field empty on a batch whose text was perfectly readable.
|
|
*/
|
|
function word(x: number, y: number, text: string): string {
|
|
return `<word xMin="${x}" yMin="${y}" xMax="${x + 20}" yMax="${y + 8}">${text}</word>`;
|
|
}
|
|
|
|
function doc(...lines: string[]): string {
|
|
return `<doc><page width="612" height="792">${lines
|
|
.map((l) => `<flow><block><line>${l}</line></block></flow>`)
|
|
.join("")}</page></doc>`;
|
|
}
|
|
|
|
/** Enough words on the page to clear the "is this a real text layer" floor. */
|
|
function padding(): string {
|
|
return Array.from({ length: 50 }, (_, i) => word(10, 400 + i * 10, `w${i}`)).join("");
|
|
}
|
|
|
|
describe("parseBboxLayout", () => {
|
|
it("rejoins a label with the value printed beside it in another flow", () => {
|
|
const [page] = parseBboxLayout(
|
|
doc(
|
|
word(20, 100, "PERIODO") + word(45, 100, "FACTURADO:"),
|
|
word(300, 100.4, "20260630-20260630"),
|
|
padding(),
|
|
),
|
|
1,
|
|
);
|
|
expect(page).not.toBeNull();
|
|
expect(page!.text).toContain("PERIODO FACTURADO: 20260630-20260630");
|
|
});
|
|
|
|
it("keeps genuinely separate lines apart", () => {
|
|
const [page] = parseBboxLayout(
|
|
doc(word(20, 100, "Cuenta:") + word(80, 100, "0900003463"), word(20, 130, "Nombre:"), padding()),
|
|
1,
|
|
);
|
|
expect(page!.text.split("\n")).toContain("Cuenta: 0900003463");
|
|
expect(page!.text.split("\n")).toContain("Nombre:");
|
|
});
|
|
|
|
it("scales point coordinates into the render's pixel space", () => {
|
|
// Word boxes have to land in the same coordinate space tesseract reports,
|
|
// or the geometric helpers the parsers share silently stop finding values.
|
|
const [page] = parseBboxLayout(doc(word(72, 144, "X") + padding()), 300 / 72);
|
|
const x = page!.words.find((w) => w.text === "X")!;
|
|
expect(x.left).toBeCloseTo(300);
|
|
expect(x.top).toBeCloseTo(600);
|
|
});
|
|
|
|
it("reports no text layer for a scan carrying a few stray glyphs", () => {
|
|
expect(parseBboxLayout(doc(word(10, 10, "3") + word(40, 10, "of") + word(60, 10, "5")), 1)).toEqual([
|
|
null,
|
|
]);
|
|
});
|
|
|
|
it("decodes the entities poppler escapes", () => {
|
|
const [page] = parseBboxLayout(doc(word(10, 10, "A&B") + padding()), 1);
|
|
expect(page!.text).toContain("A&B");
|
|
});
|
|
});
|