feat(recibos): OCR capture for gas butano and municipal predial
Adds four parsers to the statement intake — GAS TIJUANA plus one per municipality, because Tijuana, Rosarito and Ensenada issue three completely different predial documents — and a text-layer fast path for the born-digital invoices the gas company sends. Measured against a new corpus of 14 documents / 29 pages: provider read on 29/29, amount on 26/29, and 21/29 auto-matched against the dev database (22/29 identified). The eight review cases are all legitimate. Five things the corpus forced: - Not every statement is a scan. The gas invoices are born-digital CFDIs whose text layer is exact; rasterising them only loses information (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new `OcrProvider.textPages` reads the embedded layer via `pdftotext -bbox-layout` — same poppler package as `pdftoppm`, so no new dependency — and OCR stays the fallback for real scans. Poppler's own `<line>` grouping follows text flow rather than the page, so words are regrouped by vertical position; without that, a two-column header leaves every label separated from the value printed beside it. - The clave catastral is not two letters and six digits. Position three is a letter in 15 of the 932 stored claves, and digitising the whole tail mapped a real `MMB01041` to a nonexistent `MM801041`. - Tijuana predial prints no clave at all. Its only identifier is an 8-digit municipal account carried in a 32-digit payment barcode, which the legacy database never held, so it goes in `meterNumber` alongside gas — `accountNumber` holds `DATMEX.predial`, which is not a per-property key and must not be overwritten. Those pages start cold and are taught by the first confirm. - On Rosarito and Ensenada the clave is the primary key, not a fallback: those receipts print nothing else, so a unique hit auto-matches. On a utility bill that merely happens to print one it stays a review hint. - A misread `$` is the dangerous failure. An Ensenada receipt for $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large and look ordinary in the ledger. Predial amounts now require a literal `$` and a page that cannot produce one goes to review. The scoped match field is now one exported function rather than three copies of `kind === "GAS" ? ... : ...`, since the lookup, the blank-service fill and the confirm write-back have to agree or a reference gets learned into a column nothing searches. First tests in this package: 23 specs over the parsers and the text-layer reader, every fixture a verbatim OCR excerpt from a real receipt. Adds the jest config they need and a build tsconfig so they stay out of dist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -32,30 +32,46 @@ export interface MatchResult {
|
||||
* person. Names are displayed for the reviewer to sanity-check, and are never
|
||||
* an input to matching.
|
||||
*/
|
||||
/**
|
||||
* Which `PropertyService` column a given kind's statements actually print.
|
||||
*
|
||||
* Exported because the same answer governs three places that must agree: the
|
||||
* lookup here, the blank-service fill on review, and the write-back on confirm.
|
||||
* When they disagree, a reference gets learned into a column nothing searches,
|
||||
* and the same page returns to the review queue every month forever.
|
||||
*
|
||||
* `meterNumber` is doing double duty for the two kinds whose printed reference
|
||||
* DATMEX never held in `accountNumber`:
|
||||
* - GAS, where the number lived in free-text notes, and
|
||||
* - PROPERTY_TAX, where `accountNumber` holds DATMEX.predial — a 3-4 digit
|
||||
* office file number that is neither unique nor printed on any statement.
|
||||
* The Tijuana municipal receipt prints an 8-digit account and no clave
|
||||
* catastral at all, so it needs a column of its own; overwriting the legacy
|
||||
* predial numbers to make room would destroy the only link back to the
|
||||
* original records.
|
||||
*/
|
||||
export function scopedRefField(
|
||||
kind: ServiceKind,
|
||||
): "accountNumber" | "meterNumber" | null {
|
||||
switch (kind) {
|
||||
case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu
|
||||
case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua
|
||||
case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono
|
||||
case "FEDERAL_ZONE":
|
||||
case "CABLE":
|
||||
return "accountNumber";
|
||||
case "GAS": // bajagas "Cuenta" -> recovered from notes into meterNumber
|
||||
case "PROPERTY_TAX": // Tijuana's 8-digit municipal account
|
||||
return "meterNumber";
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@Injectable()
|
||||
export class StatementMatcherService {
|
||||
constructor(private readonly prisma: PrismaService) {}
|
||||
|
||||
/** Which PropertyService column a given kind's statements actually print. */
|
||||
private fieldFor(kind: ServiceKind): "accountNumber" | "meterNumber" | null {
|
||||
switch (kind) {
|
||||
case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu
|
||||
case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua
|
||||
case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono
|
||||
case "FEDERAL_ZONE":
|
||||
case "CABLE":
|
||||
return "accountNumber";
|
||||
case "GAS": // no account column in DATMEX; the number lived in notes
|
||||
return "meterNumber";
|
||||
// PROPERTY_TAX deliberately has no scoped column: what its
|
||||
// accountNumber holds is DATMEX.predial, which is neither unique nor
|
||||
// printed on any statement. Predial bills match on the clave catastral
|
||||
// alone — see matchByCadastralKey.
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async match(parsed: ParsedStatement, expectedKind: ServiceKind): Promise<MatchResult> {
|
||||
const kind = parsed.serviceKind ?? expectedKind;
|
||||
|
||||
@@ -68,33 +84,39 @@ export class StatementMatcherService {
|
||||
);
|
||||
}
|
||||
|
||||
const field = this.fieldFor(kind);
|
||||
const field = scopedRefField(kind);
|
||||
|
||||
if (field && parsed.accountRef) {
|
||||
const hit = await this.byServiceField(kind, field, parsed.accountRef);
|
||||
if (hit) return hit;
|
||||
}
|
||||
|
||||
// Secondary key. The clave catastral is printed on CESPT bills as well as
|
||||
// predial ones, so it rescues a page whose account number did not OCR —
|
||||
// which happened on real samples, where the clave read cleanly and the
|
||||
// account number did not.
|
||||
// The clave catastral is printed on CESPT bills as well as predial ones, so
|
||||
// it rescues a page whose account number did not OCR — which happened on
|
||||
// real samples, where the clave read cleanly and the account number did
|
||||
// not. On the Rosarito and Ensenada predial layouts it is not a rescue at
|
||||
// all but the only identifier the receipt carries, so a unique hit there is
|
||||
// as good as any account-number match and is treated as one.
|
||||
if (parsed.cadastralKey) {
|
||||
const hit = await this.byCadastralKey(kind, parsed.cadastralKey);
|
||||
const primary = kind === "PROPERTY_TAX" && !parsed.accountRef;
|
||||
const hit = await this.byCadastralKey(kind, parsed.cadastralKey, primary);
|
||||
if (hit) return hit;
|
||||
}
|
||||
|
||||
if (!field && !parsed.cadastralKey) {
|
||||
return this.unmatched(`no hay campo de búsqueda definido para ${kind}`);
|
||||
}
|
||||
if (!parsed.accountRef && !parsed.cadastralKey) {
|
||||
return this.unmatched(
|
||||
kind === "PROPERTY_TAX"
|
||||
? "el predial sólo se puede identificar por clave catastral y no se leyó ninguna"
|
||||
: `no hay campo de búsqueda definido para ${kind}`,
|
||||
? "no se leyó ni la clave catastral ni la cuenta municipal"
|
||||
: "no se pudo leer la referencia de la cuenta",
|
||||
);
|
||||
}
|
||||
return this.unmatched(
|
||||
parsed.accountRef
|
||||
? `no se encontró ningún servicio de ${kind} con la referencia ${parsed.accountRef}`
|
||||
: "no se pudo leer la referencia de la cuenta",
|
||||
: `no se encontró ninguna propiedad con la clave catastral ${parsed.cadastralKey}`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -145,6 +167,8 @@ export class StatementMatcherService {
|
||||
private async byCadastralKey(
|
||||
kind: ServiceKind,
|
||||
key: string,
|
||||
/** True when the clave is the identifier the statement was issued against. */
|
||||
primary: boolean,
|
||||
): Promise<MatchResult | null> {
|
||||
const props = await this.prisma.property.findMany({
|
||||
where: { cadastralKey: key },
|
||||
@@ -174,15 +198,21 @@ export class StatementMatcherService {
|
||||
};
|
||||
}
|
||||
|
||||
// The clave identifies the property with certainty, but it is a *secondary*
|
||||
// key: it was not the number the statement was issued against. Left for
|
||||
// review so the confirm also teaches the matcher the account number, rather
|
||||
// than the same page needing the fallback again next month.
|
||||
// When the clave is the *secondary* key — a utility bill that also happens
|
||||
// to print it — the page is left for review, because the clave was not the
|
||||
// number the statement was issued against and confirming is what teaches
|
||||
// the matcher the account number for next month. When it is the primary key
|
||||
// (Rosarito and Ensenada predial, which print nothing else), a unique hit
|
||||
// is a real match and there is no second number to learn.
|
||||
return {
|
||||
propertyServiceId: candidates[0].propertyServiceId ?? null,
|
||||
customerId: candidates[0].customerId,
|
||||
note: `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`,
|
||||
confident: false,
|
||||
note: primary
|
||||
? `coincidencia exacta por clave catastral ${key}`
|
||||
: `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`,
|
||||
// A clave with no service row of the right kind behind it still needs a
|
||||
// human: there is nothing to attach the posting to.
|
||||
confident: primary && candidates[0].propertyServiceId != null,
|
||||
candidates,
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user