feat(recibos): OCR capture for gas butano and municipal predial
Adds four parsers to the statement intake — GAS TIJUANA plus one per municipality, because Tijuana, Rosarito and Ensenada issue three completely different predial documents — and a text-layer fast path for the born-digital invoices the gas company sends. Measured against a new corpus of 14 documents / 29 pages: provider read on 29/29, amount on 26/29, and 21/29 auto-matched against the dev database (22/29 identified). The eight review cases are all legitimate. Five things the corpus forced: - Not every statement is a scan. The gas invoices are born-digital CFDIs whose text layer is exact; rasterising them only loses information (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new `OcrProvider.textPages` reads the embedded layer via `pdftotext -bbox-layout` — same poppler package as `pdftoppm`, so no new dependency — and OCR stays the fallback for real scans. Poppler's own `<line>` grouping follows text flow rather than the page, so words are regrouped by vertical position; without that, a two-column header leaves every label separated from the value printed beside it. - The clave catastral is not two letters and six digits. Position three is a letter in 15 of the 932 stored claves, and digitising the whole tail mapped a real `MMB01041` to a nonexistent `MM801041`. - Tijuana predial prints no clave at all. Its only identifier is an 8-digit municipal account carried in a 32-digit payment barcode, which the legacy database never held, so it goes in `meterNumber` alongside gas — `accountNumber` holds `DATMEX.predial`, which is not a per-property key and must not be overwritten. Those pages start cold and are taught by the first confirm. - On Rosarito and Ensenada the clave is the primary key, not a fallback: those receipts print nothing else, so a unique hit auto-matches. On a utility bill that merely happens to print one it stays a review hint. - A misread `$` is the dangerous failure. An Ensenada receipt for $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large and look ordinary in the ledger. Predial amounts now require a literal `$` and a page that cannot produce one goes to review. The scoped match field is now one exported function rather than three copies of `kind === "GAS" ? ... : ...`, since the lookup, the blank-service fill and the confirm write-back have to agree or a reference gets learned into a column nothing searches. First tests in this package: 23 specs over the parsers and the text-layer reader, every fixture a verbatim OCR excerpt from a real receipt. Adds the jest config they need and a build tsconfig so they stay out of dist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -16,7 +16,7 @@ import { BillingService } from "../billing/billing.service";
|
||||
import type { UploadedFileLike } from "../storage/upload-file";
|
||||
import { OCR_PROVIDER, type OcrProvider } from "./ocr/ocr.provider";
|
||||
import { parseStatement } from "./parsers/statement-parser";
|
||||
import { StatementMatcherService } from "./statement-matcher.service";
|
||||
import { StatementMatcherService, scopedRefField } from "./statement-matcher.service";
|
||||
import type { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto";
|
||||
|
||||
/**
|
||||
@@ -127,14 +127,23 @@ export class StatementsService {
|
||||
await this.storage.put(sourceKey, file.buffer, "application/pdf");
|
||||
|
||||
const pages = await this.ocr.renderPages(file.buffer);
|
||||
for (const image of pages) {
|
||||
// Page images are still rendered and stored for every file, text layer or
|
||||
// not: the review screen shows the reviewer the page, and "what the
|
||||
// parser read" is only checkable against a picture of the paper.
|
||||
const textLayer = await this.ocr.textPages(file.buffer).catch(() => []);
|
||||
|
||||
for (const [index, image] of pages.entries()) {
|
||||
pageNumber += 1;
|
||||
const storageKey = `statement/${batchId}/page-${pageNumber}.png`;
|
||||
await this.storage.put(storageKey, image, "image/png");
|
||||
|
||||
try {
|
||||
const ocr = await this.ocr.recognize(image);
|
||||
const embedded = textLayer[index] ?? null;
|
||||
const ocr = embedded ?? (await this.ocr.recognize(image));
|
||||
const parsed = parseStatement(ocr);
|
||||
if (embedded) {
|
||||
parsed.notes.unshift("texto leído del PDF original, sin OCR");
|
||||
}
|
||||
const match = await this.matcher.match(parsed, serviceKind);
|
||||
|
||||
const notes = [...parsed.notes, match.note].filter(Boolean);
|
||||
@@ -295,8 +304,8 @@ export class StatementsService {
|
||||
where: { id: doc.batchId },
|
||||
select: { serviceKind: true },
|
||||
});
|
||||
if (batch) {
|
||||
const field = batch.serviceKind === "GAS" ? "meterNumber" : "accountNumber";
|
||||
const field = batch && scopedRefField(batch.serviceKind);
|
||||
if (batch && field) {
|
||||
const blank = await this.prisma.propertyService.findMany({
|
||||
where: {
|
||||
kind: batch.serviceKind,
|
||||
@@ -434,7 +443,8 @@ export class StatementsService {
|
||||
docs: { matchedPropertyServiceId: string | null; extractedAccountRef: string | null }[],
|
||||
kind: ServiceKind,
|
||||
) {
|
||||
const field = kind === "GAS" ? "meterNumber" : "accountNumber";
|
||||
const field = scopedRefField(kind);
|
||||
if (!field) return;
|
||||
for (const d of docs) {
|
||||
if (!d.matchedPropertyServiceId || !d.extractedAccountRef) continue;
|
||||
await this.prisma.propertyService.updateMany({
|
||||
|
||||
Reference in New Issue
Block a user