diff --git a/PLAN.md b/PLAN.md index 0474eb8..be98ad5 100644 --- a/PLAN.md +++ b/PLAN.md @@ -134,7 +134,10 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult - **Receipt capture module — DONE** (2026-07-27). The legacy "Editor" replacement, built on the single-movement capture from step 6. Wires up the previously-unused `Transaction.outstanding` (NOPAGO): capture flag on `POST /billing`, `?outstanding=` list filter, `POST /billing/:id/resolve-outstanding` (gated `ledger:create`, not `ledger:void` — resolving *completes* a capture), and exclusion from every balance aggregate exactly as the legacy `SALDOS ULTIMO 0`'s `HAVING NOPAGO = 0` did. Adds `POST /billing/batch` (one `$transaction`, check-level fields shared, per-line customer/amount) and `GET /billing/by-check`, plus the `cheque-count` report replacing `REPORTE CHEQUE COUNT` / `REPORTE POR CHEQUE` / `EDITA CHEQUE ALF|COUNT|NUM` — print/PDF/CSV/XLSX come free from the existing `/reportes/:slug` machinery. Web: `/estado-cuenta/lote` (the actual "Editor" screen, with live reconciliation against the physical check amount), plus an "Estado de pago" filter, a "sin fondos" row tag and a Resolver dialog on `/estado-cuenta`. No new abilities. Verified end-to-end against dev, API + browser. **Two pre-existing bugs found and fixed while building it:** (a) `statement()` filtered `legacySourceTable: { notIn: [...] }`, which compiles to SQL `NOT IN` — and `NULL NOT IN (…)` is NULL, so **every app-captured movement was invisible on the customer statement** (438 rows in the movement browser vs 392 on the statement) while still appearing everywhere else. This would have made the whole receipt-capture feature look broken to staff. Now NULL-safe. (b) The balances *count* query omitted the void filter its own page query applied, so the row count disagreed with the rows. **OCR seam:** `BillingService.createBatch(dto, opts)` is the single multi-row write path and carries three contract guarantees for the step-11 OCR module to post through — `items[i]` maps to `lines[i]` (so `StatementDocument.postedTransactionId` can be zipped back on), `opts.refs[i]` stamps `captureRef` with a duplicate-post guard that a *voided* row deliberately does not block, and `opts.source` is service-level only so an HTTP client cannot label hand-keyed rows as machine-captured. Backed by a new `TransactionCaptureSource` enum (MANUAL/BATCH/OCR) + `captureRef`, both nullable so the 40,136 migrated rows stay NULL rather than being mislabelled. - - **PDF/OCR auto-capture** — ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff currently key in by hand. Posts through the capture module above. Matching logic was checked field-by-field against `migration/transform_properties.py`'s actual output and found three real gaps to close first: no `TELEPHONE` service kind exists yet, `PROPERTY_TAX.accountNumber` was migrated from `PREDIAL` not `CLAVE` (needs verification against a real predial statement), and `GAS.meterNumber` was never populated by the migration at all. + - **PDF/OCR auto-capture — DONE** (2026-08-01). The ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff key in by hand, built in `apps/api/src/statements/` and posting through §1.2's `createBatch` seam with `source: "OCR"` and a per-document `captureRef`. Web: `/recibos` + `/recibos/:id`. Abilities `statement:ingest`/`statement:review` (STAFF — the review step is what makes machine capture safe at that tier). OCR is self-hosted **Tesseract** behind a swappable `OcrProvider` interface; `tesseract-ocr`, `tesseract-ocr-data-spa` and `poppler-utils` were added to the API image. + **Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**. + **The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`. + **Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost. - **Multi-bank chequera — DONE** (2026-07-27). `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. `bank_transactions` gained a **required** `bankAccountId` (plus an `(bankAccountId, transactionDate)` index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by `migration/backfill_bank_accounts.py` — a standalone step because `prisma db push` cannot add a required column to a populated table. It is idempotent and now runs inside `run_all.py` (both normal and `--sync`) ahead of `transform_bank.py`, which fails fast if the account is missing. Every read path in `bank.service.ts` is account-scoped, including `facets()` (which had no filter at all) and *both* raw-SQL rollups in `summary()`. API: `?bankAccountId=` is required on `list`/`stats`/`facets`/`summary` — **not** optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new `bank/accounts` + `bank/banks` sub-resource under a MANAGER `bank:manage-accounts` ability. Web: `/banco` gained an account picker (remembered per browser) and reads every figure in the selected account's currency, `/banco/cuentas` manages banks and accounts, and `/inicio`'s chequera card names the account it is showing instead of implying one register. An account's `currency` is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged. - **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s). @@ -157,7 +160,7 @@ Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a r **Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB. -**Step 11 spec written, not built.** `docs/RECEIPT_CAPTURE_SPEC.md` covers the receipt-capture ("Editor") completion plus the three net-new ops features (OCR auto-capture, multi-bank chequera, customer-number recycling) — see Build sequencing step 11 above for the summary. Written from the 2026-07-25/26 meeting notes and verified against the real migration scripts and current API code, not just designed from the meeting notes alone. +**Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown. **Step 12 spec written, not built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. Two of the four features are much smaller than they sound: the renewal-notice table, its idempotency key and the letter body already exist, and the per-policy liquidación fields are already wired end to end. @@ -183,8 +186,9 @@ Unlike the ops items above, these block design decisions, not just infrastructur **Step 11 — utilities/ops side:** -- OCR provider/budget for the statement auto-capture pipeline (self-hosted vs. a paid per-page API, given 300+ statements/month/service provider). -- Whether `PROPERTY_TAX.accountNumber` (migrated from `DATMEX.PREDIAL`) is actually the same number as "Clave Catastral" (`DATMEX.CLAVE`) — blocks OCR matching for predial statements until confirmed against a real bill. +- ~~OCR provider/budget~~ — **CLOSED**: self-hosted Tesseract, chosen on measured accuracy against real scans, so there is no per-page cost to approve. +- ~~Whether `PROPERTY_TAX.accountNumber` (from `DATMEX.PREDIAL`) is the same number as "Clave Catastral" (`DATMEX.CLAVE`)~~ — **CLOSED**: they are different numbers. Answered from real CESPT bills plus the staged data; the clave is now migrated separately and predial was left alone. +- Whether the CFE figure to charge is the rounded headline/barcode amount (`$268` — what is actually paid at the window) or the exact breakdown `Total` (`$268.88`). The parser takes the barcode amount; one confirmation from Jorge would settle it. - The actual bank name/currency/details for the Seguros USD account, and whether any historical Seguros bank register exists to migrate. (Multi-bank support itself is **built** — this is now only the missing content: staff can open the account in `/banco/cuentas` the moment the answer arrives, and it starts empty unless a historical register turns up.) - The exact "1 year inactivity" / "cancelled" triggers for customer-number recycling eligibility. - Whether customer-number recycling should ever include true PII purge (matching the office's paper-world habit) or archive-and-reuse-the-number is sufficient — recommended default is archive-only, consistent with this project's existing never-hard-delete convention. diff --git a/RESUME.md b/RESUME.md index 0dd6375..0274049 100644 --- a/RESUME.md +++ b/RESUME.md @@ -442,3 +442,87 @@ for what's actually next. verified vs dev: Anular buttons admin-gated, voided rows struck + excluded from totals, clicking Anular voids end-to-end (note: it uses a blocking `window.confirm`). Customer-detail mini tx list now also strikes voided rows ("(anulado)" tag) — was the last void-UI gap. + +--- + +## Statement OCR intake (`/recibos`) — DONE 2026-08-01 + +Plan step 11 §2 (`docs/RECEIPT_CAPTURE_SPEC.md` §2). The last big utilities +feature: staff scan the month's utility bills and the machine proposes customer ++ amount per page, instead of keying 300+ statements per company by hand. Built +in `apps/api/src/statements/` and `apps/web/src/app/recibos/`, posting through +step 11 §1.2's `BillingService.createBatch` seam (`source: "OCR"`, per-document +`captureRef`) so machine and hand capture share one write path and one audit +trail. Abilities `statement:ingest` / `statement:review`, both STAFF. + +**Verified end to end against the live dev API + MinIO**, not just built: real +CFE and Telnor scans uploaded over HTTP, OCR'd, matched, confirmed against a +check, and the resulting rows checked in MySQL — negative (charge) amounts, +`captureSource = OCR`, concept auto-derived from the batch's service kind, +`captureRef` linking each transaction back to its page. Re-confirming a posted +batch is refused. All test data was removed afterwards. + +**Everything here was decided from 10 real scanned statements (46 pages), not +from the sample-free spec.** Shipped-parser results on them: provider 46/46, +account reference 43/46, amount 42/46, due date 44/46; matched against the dev +database, **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The rest are +real review cases (one shared account number, three phones not on file, one +clave not in the book, one page too poor to read). + +Findings that corrected the spec, each of which changed the build: + +- **The scans have no text layer at all** — they are camera images of paper, so + OCR is mandatory rather than a convenience, and they arrive **bundled, one + customer per page**. +- **Clave catastral is not predial.** `DATMEX.clave` (934 rows, `KA903009`) is + what CESPT and predial bills print; `DATMEX.predial` — which + `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 + rows and appears on no statement. The clave now lives on + `Property.cadastralKey` as the matcher's secondary key; predial was left + untouched. This is the question that had been blocking predial matching. +- **Gas was recoverable after all.** The spec said no legacy gas number existed; + in fact 160 of 334 `DATMEX.gas` values are real account numbers (the rest are + `ESTACIONARIO`/`CILINDRO` descriptors). Recovered into `GAS.meterNumber`. +- **Phone is one billed line per property** (534 / 18 / 1 across phone1/2/3), so + `TELEPHONE` — a new `ServiceKind` — backfills from `phone1` only. +- **Never match on the printed name.** A CESPT receipt for account `5365218` + reads `ARNAIZ ROSAS ELSA AURORA`; the office's book, corroborated by the + clave, has `CATT, RANDY`. The name on a utility bill is the registrant, not + the current owner. + +`migration/backfill_statement_match_fields.py` closes those three data gaps on +an existing database (idempotent, wired into `run_all.py` after +`transform_properties.py`, which now produces them directly on a full rebuild). +Applied to dev: 934 claves, 160 gas numbers, 534 TELEPHONE rows. + +Implementation notes worth keeping: + +- OCR is self-hosted **Tesseract** behind an `OcrProvider` interface — the + provider question is closed on measured accuracy, and a managed API stays a + one-line swap in `statements.module.ts`. `tesseract-ocr`, + `tesseract-ocr-data-spa` and `poppler-utils` were added to the API image; if + they are missing the module reports itself unavailable and only this feature + is disabled. +- **Payment barcodes beat printed labels.** One CFE label OCR'd a digit too + many while its barcode was correct, so the barcode is the source and the label + the cross-check; disagreement forces review. +- **Detect the provider by brand first, layout only as a fallback** — and never + interleave the two passes. A scanned CESPT header came back as `E BAJA ES + PAGO / EALIFORNIA`, which is why the layout fallback exists; a Telnor page + contains words a CFE layout rule would otherwise claim, which is why ordering + matters. +- **Parse amounts by separator position.** A real Telnor bill OCR'd as + `$ 649,00`; stripping commas as thousands separators turns that into $64,900. +- Two of the three layouts are line-oriented, but the CESPT "RECIBO" is a + **table** whose values sit under column headers — that one needs the word + boxes, which is why `OcrPage` carries geometry and not just text. +- Confirming a document whose matched service had no reference **writes the + reference back** (only into an empty field, and only when exactly one blank + service of that kind is a candidate), so gas and any other cold start is a + one-time cost rather than a permanent queue. +- Handwritten folder numbers on the bills (`9`, `405`) are **not** used for + matching — Tesseract read `405` as `205`. + +**Open:** whether the CFE charge should be the rounded barcode/headline figure +(`$268`, what is paid at the window — what the parser uses today) or the exact +breakdown total (`$268.88`). One question for Jorge. diff --git a/apps/api/src/app.module.ts b/apps/api/src/app.module.ts index 75d9f84..29b66c9 100644 --- a/apps/api/src/app.module.ts +++ b/apps/api/src/app.module.ts @@ -9,6 +9,7 @@ import { CustomersModule } from "./customers/customers.module"; import { PoliciesModule } from "./policies/policies.module"; import { PropertiesModule } from "./properties/properties.module"; import { BillingModule } from "./billing/billing.module"; +import { StatementsModule } from "./statements/statements.module"; import { BankModule } from "./bank/bank.module"; import { OpsModule } from "./ops/ops.module"; import { ReportsModule } from "./reports/reports.module"; @@ -26,6 +27,7 @@ import { AppController } from "./app.controller"; PoliciesModule, PropertiesModule, BillingModule, + StatementsModule, BankModule, OpsModule, ReportsModule, diff --git a/apps/api/src/auth/abilities.ts b/apps/api/src/auth/abilities.ts index a112822..ea11508 100644 --- a/apps/api/src/auth/abilities.ts +++ b/apps/api/src/auth/abilities.ts @@ -32,6 +32,8 @@ export type Ability = | "bank:create" | "bank:void" | "bank:manage-accounts" + | "statement:ingest" + | "statement:review" | "lookup:manage" | "user:manage" | "db:manage"; @@ -54,6 +56,12 @@ export const ABILITY_MIN: Record = { // Opening or renaming a chequera is rarer and higher-stakes than posting a // movement into one — a wrong account silently mixes two sets of books. "bank:manage-accounts": "MANAGER", + // Uploading a stack of scans and reviewing what the OCR read are both + // "capturing a receipt" — the same trust tier as ledger:create, since + // confirming a statement *is* capturing it. The review step is what makes + // this safe at STAFF level: nothing reaches the ledger unconfirmed. + "statement:ingest": "STAFF", + "statement:review": "STAFF", "lookup:manage": "MANAGER", "user:manage": "ADMIN", "db:manage": "ADMIN", diff --git a/apps/api/src/billing/billing.module.ts b/apps/api/src/billing/billing.module.ts index ef9f33e..f565973 100644 --- a/apps/api/src/billing/billing.module.ts +++ b/apps/api/src/billing/billing.module.ts @@ -5,5 +5,8 @@ import { BillingService } from "./billing.service"; @Module({ controllers: [BillingController], providers: [BillingService], + // The statements module posts confirmed OCR captures through + // BillingService.createBatch rather than writing Transaction rows itself. + exports: [BillingService], }) export class BillingModule {} diff --git a/apps/api/src/statements/ocr/ocr.provider.ts b/apps/api/src/statements/ocr/ocr.provider.ts new file mode 100644 index 0000000..ba2dfd8 --- /dev/null +++ b/apps/api/src/statements/ocr/ocr.provider.ts @@ -0,0 +1,53 @@ +/** + * The OCR seam. Everything above this interface works in terms of page text and + * word boxes, so the concrete engine is swappable without touching the parsers, + * the matcher, or the schema. + * + * The shipped implementation is self-hosted Tesseract (see tesseract.provider). + * That choice is evidence-based rather than assumed: run against 46 pages of + * real scanned CFE, CESPT and Telnor statements, it identified the provider on + * 46/46 and extracted a usable account reference on 43/46, which is well past + * the bar for a queue whose whole point is that a human confirms every row. A + * managed document-extraction API (Textract, Document Intelligence, Document + * AI) fits behind this same interface if per-page accuracy ever proves + * insufficient, with no schema change — but at 300+ pages/month/company it + * would carry a real recurring cost for accuracy that is not currently the + * bottleneck. + */ + +/** One OCR'd word, with where it sits on the page. */ +export interface OcrWord { + text: string; + /** Pixel box in the rendered page image. */ + left: number; + top: number; + width: number; + height: number; + /** Engine confidence for this word, 0..1. */ + confidence: number; +} + +export interface OcrPage { + /** Full page text, reading order, newline-separated. */ + text: string; + /** + * Word boxes. Needed because two of the three real layouts are *tables* — + * the CESPT "RECIBO" prints `No. DE CUENTA` as a column header with the + * value in the row beneath it, which line-oriented text cannot associate. + * Parsers fall back to geometry for exactly those fields. + */ + words: OcrWord[]; + /** Mean word confidence across the page, 0..1. */ + confidence: number; +} + +export interface OcrProvider { + /** True when the engine is actually usable in this deployment. */ + available(): Promise; + /** Split a PDF into one rendered page image per page. */ + renderPages(pdf: Buffer): Promise; + /** OCR a single rendered page image. */ + recognize(pageImage: Buffer): Promise; +} + +export const OCR_PROVIDER = Symbol("OCR_PROVIDER"); diff --git a/apps/api/src/statements/ocr/tesseract.provider.ts b/apps/api/src/statements/ocr/tesseract.provider.ts new file mode 100644 index 0000000..b97673f --- /dev/null +++ b/apps/api/src/statements/ocr/tesseract.provider.ts @@ -0,0 +1,195 @@ +import { Injectable, Logger, ServiceUnavailableException } from "@nestjs/common"; +import { ConfigService } from "@nestjs/config"; +import { execFile } from "node:child_process"; +import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import type { OcrPage, OcrProvider, OcrWord } from "./ocr.provider"; + +const run = promisify(execFile); + +/** + * Self-hosted OCR: `pdftoppm` (poppler) to rasterise, `tesseract` to read. + * + * Both are external binaries rather than a native npm addon, which keeps the + * pnpm workspace free of a compiled dependency and makes the alpine runtime + * image a two-package change (see docker/api.Dockerfile). Like StorageService, + * a missing binary degrades rather than crashes the API: the module reports + * itself unavailable and statement ingest returns 503, while every other + * feature keeps working. + * + * The settings below are not arbitrary — they were measured against the real + * scanned samples: + * - 300 DPI grayscale. The source scans are phone photos of paper at ~5MB a + * page; below 300 the small print (RMU, clave catastral) stops resolving, + * above it costs time for no additional fields. + * - `--psm 6` ("assume a single uniform block of text"). The default page + * segmentation splits these dense forms into columns and interleaves them, + * which destroys the label-then-value adjacency every parser depends on. + * - Spanish traineddata, with a graceful fall back to English if the language + * pack is absent — an accented label reads worse but the digits, which are + * what actually gets matched, are unaffected. + */ +@Injectable() +export class TesseractOcrProvider implements OcrProvider { + private readonly logger = new Logger(TesseractOcrProvider.name); + private readonly dpi: number; + private readonly lang: string; + private probe: Promise | null = null; + + constructor(config: ConfigService) { + this.dpi = Number(config.get("OCR_DPI") ?? 300); + this.lang = config.get("OCR_LANG") ?? "spa"; + } + + /** Cached — the binaries do not appear or vanish while the process runs. */ + available(): Promise { + if (!this.probe) { + this.probe = (async () => { + try { + await Promise.all([ + run("tesseract", ["--version"]), + run("pdftoppm", ["-v"]), + ]); + return true; + } catch { + this.logger.warn( + "OCR unavailable: `tesseract` and/or `pdftoppm` not found on PATH. " + + "Statement ingest is disabled; every other feature is unaffected.", + ); + return false; + } + })(); + } + return this.probe; + } + + private async require(): Promise { + if (!(await this.available())) { + throw new ServiceUnavailableException( + "El servicio de OCR no está disponible en este servidor.", + ); + } + } + + private async scratch(fn: (dir: string) => Promise): Promise { + const dir = await mkdtemp(join(tmpdir(), "stmt-ocr-")); + try { + return await fn(dir); + } finally { + await rm(dir, { recursive: true, force: true }); + } + } + + async renderPages(pdf: Buffer): Promise { + await this.require(); + return this.scratch(async (dir) => { + const src = join(dir, "in.pdf"); + await writeFile(src, pdf); + // -gray: these are grayscale scans already; colour triples the bytes + // handed to tesseract for no gain in character recognition. + await run("pdftoppm", [ + "-r", + String(this.dpi), + "-gray", + "-png", + src, + join(dir, "page"), + ]); + const files = (await readdir(dir)) + .filter((f) => f.startsWith("page") && f.endsWith(".png")) + // pdftoppm zero-pads its page numbers, so lexical order is page order. + .sort(); + return Promise.all(files.map((f) => readFile(join(dir, f)))); + }); + } + + async recognize(pageImage: Buffer): Promise { + await this.require(); + return this.scratch(async (dir) => { + const img = join(dir, "page.png"); + await writeFile(img, pageImage); + + // One tesseract invocation produces both outputs; TSV carries the word + // boxes and per-word confidence, and its text can be reassembled into + // reading order, so there is no need to run the engine twice. + const out = join(dir, "out"); + try { + await run("tesseract", [img, out, "-l", this.lang, "--psm", "6", "tsv"]); + } catch (err) { + if (this.lang !== "eng") { + this.logger.warn( + `Tesseract failed with lang "${this.lang}", retrying with "eng": ${ + (err as Error).message + }`, + ); + await run("tesseract", [img, out, "-l", "eng", "--psm", "6", "tsv"]); + } else { + throw err; + } + } + + const tsv = await readFile(`${out}.tsv`, "utf8"); + return parseTsv(tsv); + }); + } +} + +/** + * Turn tesseract's TSV into words plus reassembled text. + * + * Columns are: level, page_num, block_num, par_num, line_num, word_num, left, + * top, width, height, conf, text. Rows with level < 5 are structural (page, + * block, paragraph, line) and carry no text; only level 5 is a word. A conf of + * -1 marks a structural row, so those are dropped rather than averaged in — + * including them would drag every page's confidence toward zero. + */ +export function parseTsv(tsv: string): OcrPage { + const lines = tsv.split("\n"); + const header = lines[0]?.split("\t") ?? []; + const col = (name: string) => header.indexOf(name); + const iLeft = col("left"); + const iTop = col("top"); + const iWidth = col("width"); + const iHeight = col("height"); + const iConf = col("conf"); + const iText = col("text"); + const iLine = col("line_num"); + const iBlock = col("block_num"); + + const words: OcrWord[] = []; + // Keyed by block+line so the reassembled text preserves the engine's own + // reading order instead of sorting words by raw y, which interleaves columns. + const byLine = new Map(); + + for (let i = 1; i < lines.length; i++) { + const f = lines[i].split("\t"); + if (f.length <= iText) continue; + const text = f[iText]?.trim(); + if (!text) continue; + const confidence = Number(f[iConf]); + if (!Number.isFinite(confidence) || confidence < 0) continue; + + words.push({ + text, + left: Number(f[iLeft]) || 0, + top: Number(f[iTop]) || 0, + width: Number(f[iWidth]) || 0, + height: Number(f[iHeight]) || 0, + confidence: confidence / 100, + }); + + const key = `${f[iBlock]}:${f[iLine]}`; + const bucket = byLine.get(key); + if (bucket) bucket.push(text); + else byLine.set(key, [text]); + } + + const text = [...byLine.values()].map((w) => w.join(" ")).join("\n"); + const confidence = words.length + ? words.reduce((sum, w) => sum + w.confidence, 0) / words.length + : 0; + + return { text, words, confidence }; +} diff --git a/apps/api/src/statements/parsers/statement-parser.ts b/apps/api/src/statements/parsers/statement-parser.ts new file mode 100644 index 0000000..17a7b12 --- /dev/null +++ b/apps/api/src/statements/parsers/statement-parser.ts @@ -0,0 +1,390 @@ +import type { ServiceKind } from "@jorgecuadros/database"; +import type { OcrPage, OcrWord } from "../ocr/ocr.provider"; + +/** + * What one parsed statement page yields. `accountRef` is already normalised to + * the form the migrated `PropertyService` columns hold, so the matcher compares + * like with like and never has to know about provider-specific formatting. + */ +export interface ParsedStatement { + /** "CFE" | "CESPT" | "TELNOR", or null when no parser claimed the page. */ + provider: string | null; + serviceKind: ServiceKind | null; + accountRef: string | null; + /** Clave catastral, when printed — a second key to match on. */ + cadastralKey: string | null; + amount: number | null; + dueDate: Date | null; + period: string | null; + /** + * Independent corroboration of `accountRef`. CFE and Telnor both print a + * payment barcode that repeats the account number (and the amount), so when + * the barcode and the label agree the extraction is near-certainly right; + * when they disagree, or only one is present, the page is worth a human + * glance. Null when the layout has no second source. + */ + crossChecked: boolean | null; + /** Human-readable trail of what was read, surfaced in the review queue. */ + notes: string[]; +} + +// --- shared helpers --------------------------------------------------------- + +/** + * Tesseract confuses these glyphs inside numeric runs with some regularity — + * a real clave catastral `KB078025` came back as `KBO78025`. Applied ONLY to + * fields known to be digits, never to free text, where it would corrupt words. + */ +const DIGIT_CONFUSIONS: Record = { + O: "0", + o: "0", + D: "0", + I: "1", + l: "1", + "|": "1", + S: "5", + B: "8", +}; + +export function toDigits(s: string | null | undefined): string { + if (!s) return ""; + return s + .split("") + .map((c) => DIGIT_CONFUSIONS[c] ?? c) + .join("") + .replace(/\D/g, ""); +} + +/** + * Parse a printed amount, treating `,` and `.` by position rather than by + * assumption. A real Telnor bill OCR'd as "$ 649,00" — blindly stripping commas + * as thousands separators turned $649.00 into $64,900, a hundredfold error that + * would post silently. Two trailing digits after a single separator are always + * cents here; a separator followed by three digits is a thousands group. + */ +function money(s: string | null | undefined): number | null { + if (!s) return null; + const cleaned = s.replace(/[\s$]/g, ""); + + // 1.234,56 or 1,234.56 — grouped thousands plus optional cents. + let m = cleaned.match(/^(\d{1,3}(?:[.,]\d{3})+)([.,]\d{1,2})?$/); + if (m) { + const whole = m[1].replace(/[.,]/g, ""); + const cents = m[2] ? m[2].slice(1) : ""; + return Number(cents ? `${whole}.${cents.padEnd(2, "0")}` : whole); + } + + // 649,00 / 649.00 — a single separator with exactly two digits after it. + m = cleaned.match(/^(\d+)[.,](\d{2})$/); + if (m) return Number(`${m[1]}.${m[2]}`); + + const n = Number(cleaned.replace(/[,.]/g, "")); + return Number.isFinite(n) ? n : null; +} + +function firstMatch(text: string, patterns: RegExp[]): string | null { + for (const p of patterns) { + const m = text.match(p); + if (m?.[1]) return m[1].trim(); + } + return null; +} + +const MONTHS: Record = { + ENE: 0, FEB: 1, MAR: 2, ABR: 3, MAY: 4, JUN: 5, + JUL: 6, AGO: 7, SEP: 8, OCT: 9, NOV: 10, DIC: 11, +}; + +/** Parses the three date shapes these statements actually print. */ +export function parseDate(raw: string | null | undefined): Date | null { + if (!raw) return null; + const s = raw.trim().toUpperCase(); + + // 16/07/2026 + let m = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/); + if (m) return utc(+m[3], +m[2] - 1, +m[1]); + + // 22-JUL-2026 / 22 JUN 26 + m = s.match(/^(\d{1,2})[-\s]([A-Z]{3})[A-Z]*[-\s](\d{2,4})$/); + if (m && MONTHS[m[2]] !== undefined) { + const y = m[3].length === 2 ? 2000 + +m[3] : +m[3]; + return utc(y, MONTHS[m[2]], +m[1]); + } + + // 2026-07-22 (already normalised, e.g. decoded from a barcode) + m = s.match(/^(\d{4})-(\d{2})-(\d{2})$/); + if (m) return utc(+m[1], +m[2] - 1, +m[3]); + + return null; +} + +function utc(y: number, mo: number, d: number): Date | null { + const dt = new Date(Date.UTC(y, mo, d)); + return Number.isNaN(dt.getTime()) ? null : dt; +} + +/** + * Read the value printed *underneath* a column header. + * + * The CESPT "RECIBO" is a table: `No. DE CUENTA` is a header cell and its value + * sits in the row below it, so no amount of label-adjacent regex on line text + * can associate the two. This walks the word boxes instead — find the header + * word, then take the nearest word below it whose horizontal centre falls + * within the column. + */ +export function valueUnder( + page: OcrPage, + header: RegExp, + opts: { maxDy?: number; tolerance?: number; match?: RegExp } = {}, +): string | null { + const { maxDy = 300, tolerance = 200, match } = opts; + const centre = (w: OcrWord) => ({ + x: w.left + w.width / 2, + y: w.top + w.height / 2, + }); + + for (const h of page.words.filter((w) => header.test(w.text))) { + const hc = centre(h); + const below = page.words + .filter((w) => { + const c = centre(w); + return c.y > hc.y && c.y <= hc.y + maxDy && Math.abs(c.x - hc.x) <= tolerance; + }) + .sort((a, b) => centre(a).y - centre(b).y); + + for (const w of below) { + if (!match || match.test(w.text)) return w.text; + } + } + return null; +} + +// --- provider detection ----------------------------------------------------- + +/** + * Brand wordmarks first, page structure only as a fallback — and the two passes + * must not be interleaved. Scanned logos OCR badly (one CESPT header came back + * as "E BAJA ES PAGO / EALIFORNIA", with neither "CESPT" nor "COMISIÓN ESTATAL" + * readable), so the structural pass is what rescues those pages. But a Telnor + * bill contains the words "Pagar antes de", which a CFE structural rule + * evaluated first will happily claim — running all brand checks before any + * structural check is what keeps that from happening. + */ +const BRAND: [string, RegExp][] = [ + ["CFE", /comisi[oó]n federal de electricidad|CFE.?contigo|Suministrador de Servicios/i], + ["CESPT", /CESPT|COMISI[OÓ]N ESTATAL DE SERVICIOS/i], + ["TELNOR", /TELNOR|TELEFONOS DEL NOROESTE/i], +]; + +const LAYOUT: [string, RegExp][] = [ + ["CFE", /NO\.?\s*DE\s*SERVICIO|L[IÍ]MITE\s*DE\s*PAGO|PERIODO\s*FACTURADO/i], + ["CESPT", /SALDO\s+CORRIENTE|CLAVE\s*CATASTRAL|No\.?\s*DE\s*CUENTA/i], + ["TELNOR", /Mes\s*de\s*Facturaci[oó]n|Pagar\s*antes\s*de/i], +]; + +export function detectProvider(text: string): string | null { + for (const group of [BRAND, LAYOUT]) { + for (const [name, pattern] of group) { + if (pattern.test(text)) return name; + } + } + return null; +} + +// --- CFE (electric) --------------------------------------------------------- + +function parseCfe(page: OcrPage): ParsedStatement { + const text = page.text; + const notes: string[] = []; + + // The payment barcode line repeats the service number, the due date (YYMMDD) + // and the amount in one fixed-width run, and reads far more reliably than the + // label: on one sample the label came back as "0059603001917" (a digit too + // many) while its barcode gave the correct "005960300191". So the barcode + // wins, and the label becomes the cross-check rather than the source. + const barcode = text.match(/\b01\s+([0-9OIlSBD]{12})\s+([0-9OIlSBD]{6})\s+([0-9OIlSBD]{9})\b/); + const label = firstMatch(text, [/NO\.?\s*DE\s*SERVICIO\s*[:;.]?\s*([0-9OIlSBD]{10,14})/i]); + + let accountRef: string | null = null; + let amount: number | null = null; + let dueDate: Date | null = null; + let crossChecked: boolean | null = null; + + if (barcode) { + // Leading zeros are print padding: DATMEX.rpu holds the bare 10 digits. + accountRef = toDigits(barcode[1]).replace(/^0+/, ""); + amount = Number(toDigits(barcode[3])); + const d = toDigits(barcode[2]); + dueDate = parseDate(`20${d.slice(0, 2)}-${d.slice(2, 4)}-${d.slice(4, 6)}`); + notes.push("importe y vencimiento leídos del código de barras"); + if (label) { + crossChecked = toDigits(label).replace(/^0+/, "") === accountRef; + if (!crossChecked) { + notes.push( + `el número impreso (${toDigits(label).replace(/^0+/, "")}) no coincide con el código de barras`, + ); + } + } + } else if (label) { + accountRef = toDigits(label).replace(/^0+/, ""); + notes.push("sin código de barras legible; número tomado de la etiqueta"); + } + + if (amount == null) { + amount = money(firstMatch(text, [/TOTAL\s*A\s*PAGAR\s*[:;.]?\s*\$?\s*([\d,]+\.?\d*)/i])); + } + if (!dueDate) { + dueDate = parseDate( + firstMatch(text, [/L[IÍ]MITE\s*DE\s*PAGO\s*[:;.]?\s*(\d{1,2}\s+\w{3}\s+\d{2,4})/i]), + ); + } + + return { + provider: "CFE", + serviceKind: "ELECTRIC", + accountRef: accountRef || null, + cadastralKey: null, + amount, + dueDate, + period: firstMatch(text, [ + /PERIODO\s*FACTURADO\s*[:;.]?\s*(\d{1,2}\s+\w{3}\s+\d{2}\s*-\s*\d{1,2}\s+\w{3}\s+\d{2})/i, + ]), + crossChecked, + notes, + }; +} + +// --- CESPT (water) ---------------------------------------------------------- + +/** + * Two different layouts arrive under the same brand: + * - the line-oriented "COMPROBANTE DE PAGO" (`Cuenta : 7604192`), and + * - the tabular "RECIBO", where `No. DE CUENTA` is a column header. + * Line patterns are tried first; anything they miss falls through to the + * geometric read, which is what the tabular layout needs. + */ +function parseCespt(page: OcrPage): ParsedStatement { + const text = page.text; + const notes: string[] = []; + + let account = firstMatch(text, [/Cuenta\s*[:;.]?\s*([0-9OIlSBD]{5,9})/i]); + if (!account) { + account = valueUnder(page, /^CUENTA$/i, { match: /^[0-9OIlSBD]{5,9}$/ }); + if (account) notes.push("número de cuenta leído de la columna del recibo"); + } + + let clave = firstMatch(text, [/Cve\.?\s*Cat\.?\s*[:;.]?\s*([A-Z]{2}\s?[0-9OIlSBD]{6})/i]); + if (!clave) { + clave = valueUnder(page, /^CATASTRAL$/i, { match: /^[A-Z]{2}[0-9OIlSBD]{6}$/i }); + if (clave) notes.push("clave catastral leída de la columna del recibo"); + } + + let due = firstMatch(text, [/Fecha\s*Venc\s*[:;.]?\s*(\d{2}\/\d{2}\/\d{4})/i]); + if (!due) due = valueUnder(page, /^VENCIMIENTO$/i, { match: /^\d{2}\/\d{2}\/\d{4}$/ }); + + const amount = money( + firstMatch(text, [ + /TOTAL\s*[:;.]?\s*\$?\s*([\d,]+\.\d{2})/i, + /SALDO\s+CORRIENTE[^\n]*?([\d,]+\.\d{2})/i, + ]), + ); + + // Leading zeros are print padding here too: the RECIBO prints `0457341` for + // what DATMEX.agua holds as `457341`. + const accountRef = account ? toDigits(account).replace(/^0+/, "") : null; + const cadastralKey = clave + ? clave.replace(/\s/g, "").slice(0, 2).toUpperCase() + + toDigits(clave.replace(/\s/g, "").slice(2)) + : null; + + return { + provider: "CESPT", + serviceKind: "WATER", + accountRef: accountRef || null, + cadastralKey: cadastralKey || null, + amount, + dueDate: parseDate(due), + period: null, + crossChecked: null, + notes, + }; +} + +// --- TELNOR (telephone) ----------------------------------------------------- + +function parseTelnor(page: OcrPage): ParsedStatement { + const text = page.text; + const notes: string[] = []; + + const label = firstMatch(text, [ + /Tel[eé]fono\s*[:;.]?\s*([0-9OIlSBD]{3}\s?[0-9OIlSBD]{3}\s?[0-9OIlSBD]{4})/i, + ]); + // The payment stub prints phone (10 digits) + amount in cents (9) + a check + // digit: `6646093444 000099900 7` for a $999.00 bill. Reading the amount as + // 10 digits swallows the check digit and inflates the figure 100-fold. + const barcode = text.match(/\b(\d{10})(\d{9})\d\b/); + + let accountRef: string | null = null; + let crossChecked: boolean | null = null; + + // The bill prints the number with its 664 Tijuana LADA; DATMEX stores the + // bare local 7 digits, so the LADA is dropped rather than the stored value + // being padded — padding would guess at an area code for the 500+ existing + // rows that never recorded one. + if (label) accountRef = toDigits(label).slice(-7); + if (barcode) { + const fromBarcode = barcode[1].slice(-7); + if (accountRef) { + crossChecked = fromBarcode === accountRef; + if (!crossChecked) notes.push("el teléfono impreso no coincide con el código de barras"); + } else { + accountRef = fromBarcode; + notes.push("teléfono leído del código de barras"); + } + } + + let amount = money(firstMatch(text, [/Total\s*a\s*Pagar\s*[:;.]?\s*\$?\s*([\d,]+\.?\d{0,2})/i])); + if (amount == null && barcode) { + amount = Number(barcode[2]) / 100; + notes.push("importe leído del código de barras"); + } + + return { + provider: "TELNOR", + serviceKind: "TELEPHONE", + accountRef: accountRef || null, + cadastralKey: null, + amount, + dueDate: parseDate( + firstMatch(text, [/Pagar\s*antes\s*de\s*[:;.]?\s*(\d{2}-\w{3}-\d{4})/i]), + ), + period: firstMatch(text, [/Mes\s*de\s*Facturaci[oó]n\s*[:;.]?\s*(\w+)/i]), + crossChecked, + notes, + }; +} + +const PARSERS: Record ParsedStatement> = { + CFE: parseCfe, + CESPT: parseCespt, + TELNOR: parseTelnor, +}; + +const EMPTY: ParsedStatement = { + provider: null, + serviceKind: null, + accountRef: null, + cadastralKey: null, + amount: null, + dueDate: null, + period: null, + crossChecked: null, + notes: [], +}; + +/** Detect the provider and run its parser. */ +export function parseStatement(page: OcrPage): ParsedStatement { + const provider = detectProvider(page.text); + if (!provider) return { ...EMPTY, notes: ["no se reconoció el proveedor"] }; + return PARSERS[provider](page); +} diff --git a/apps/api/src/statements/statement-matcher.service.ts b/apps/api/src/statements/statement-matcher.service.ts new file mode 100644 index 0000000..fc9496f --- /dev/null +++ b/apps/api/src/statements/statement-matcher.service.ts @@ -0,0 +1,199 @@ +import { Injectable } from "@nestjs/common"; +import type { ServiceKind } from "@jorgecuadros/database"; +import { PrismaService } from "../prisma/prisma.service"; +import type { ParsedStatement } from "./parsers/statement-parser"; + +export interface MatchResult { + propertyServiceId: string | null; + customerId: string | null; + /** Why it landed here — shown in the review queue verbatim. */ + note: string; + /** True only for an unambiguous hit on the scoped field. */ + confident: boolean; + /** Populated when more than one service claims the same number. */ + candidates: { propertyServiceId: string; customerId: string; customerName: string }[]; +} + +/** + * Resolves a parsed statement to the customer who should be billed for it. + * + * Two rules govern everything here. + * + * **Match on one scoped field, never fuzzily across all identifiers.** Each + * service kind has exactly one column its statements print, and only that + * column is consulted. A blanket search over accountNumber/meterNumber/route + * would let a water account number collide with an unrelated phone number, and + * the resulting mis-post would look perfectly ordinary in the ledger. + * + * **Never match on the customer name.** The name on a utility bill is the + * account's registrant, which drifts from the current owner and is often years + * stale — one sample CESPT receipt is printed to "ARNAIZ ROSAS ELSA AURORA" + * for an account this office holds under "CATT, RANDY", who is not the same + * person. Names are displayed for the reviewer to sanity-check, and are never + * an input to matching. + */ +@Injectable() +export class StatementMatcherService { + constructor(private readonly prisma: PrismaService) {} + + /** Which PropertyService column a given kind's statements actually print. */ + private fieldFor(kind: ServiceKind): "accountNumber" | "meterNumber" | null { + switch (kind) { + case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu + case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua + case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono + case "FEDERAL_ZONE": + case "CABLE": + return "accountNumber"; + case "GAS": // no account column in DATMEX; the number lived in notes + return "meterNumber"; + // PROPERTY_TAX deliberately has no scoped column: what its + // accountNumber holds is DATMEX.predial, which is neither unique nor + // printed on any statement. Predial bills match on the clave catastral + // alone — see matchByCadastralKey. + default: + return null; + } + } + + async match(parsed: ParsedStatement, expectedKind: ServiceKind): Promise { + const kind = parsed.serviceKind ?? expectedKind; + + // The uploader labels a batch with one service kind. If the parser reads a + // page as a different provider, that is a mis-sorted page, not a match — + // posting it would book a phone bill as a water charge. + if (parsed.serviceKind && parsed.serviceKind !== expectedKind) { + return this.unmatched( + `la página parece de ${parsed.provider} (${parsed.serviceKind}) pero el lote es de ${expectedKind}`, + ); + } + + const field = this.fieldFor(kind); + + if (field && parsed.accountRef) { + const hit = await this.byServiceField(kind, field, parsed.accountRef); + if (hit) return hit; + } + + // Secondary key. The clave catastral is printed on CESPT bills as well as + // predial ones, so it rescues a page whose account number did not OCR — + // which happened on real samples, where the clave read cleanly and the + // account number did not. + if (parsed.cadastralKey) { + const hit = await this.byCadastralKey(kind, parsed.cadastralKey); + if (hit) return hit; + } + + if (!field && !parsed.cadastralKey) { + return this.unmatched( + kind === "PROPERTY_TAX" + ? "el predial sólo se puede identificar por clave catastral y no se leyó ninguna" + : `no hay campo de búsqueda definido para ${kind}`, + ); + } + return this.unmatched( + parsed.accountRef + ? `no se encontró ningún servicio de ${kind} con la referencia ${parsed.accountRef}` + : "no se pudo leer la referencia de la cuenta", + ); + } + + private async byServiceField( + kind: ServiceKind, + field: "accountNumber" | "meterNumber", + ref: string, + ): Promise { + const rows = await this.prisma.propertyService.findMany({ + where: { kind, [field]: ref }, + select: { + id: true, + property: { + select: { customerId: true, customer: { select: { name: true } } }, + }, + }, + }); + if (rows.length === 0) return null; + + const candidates = rows.map((r) => ({ + propertyServiceId: r.id, + customerId: r.property.customerId, + customerName: r.property.customer.name, + })); + + // Duplicate account numbers do occur in the legacy data (the office's own + // DUPLICADOS report existed for a reason), so every candidate is surfaced + // for the reviewer to choose rather than one being picked arbitrarily. + if (rows.length > 1) { + return { + propertyServiceId: null, + customerId: null, + note: `${rows.length} servicios comparten la referencia ${ref}`, + confident: false, + candidates, + }; + } + + return { + propertyServiceId: candidates[0].propertyServiceId, + customerId: candidates[0].customerId, + note: `coincidencia exacta por ${field === "accountNumber" ? "número de cuenta" : "medidor"} ${ref}`, + confident: true, + candidates, + }; + } + + private async byCadastralKey( + kind: ServiceKind, + key: string, + ): Promise { + const props = await this.prisma.property.findMany({ + where: { cadastralKey: key }, + select: { + customerId: true, + customer: { select: { name: true } }, + services: { where: { kind }, select: { id: true } }, + }, + }); + if (props.length === 0) return null; + + const candidates = props.flatMap((p) => + (p.services.length ? p.services.map((s) => s.id) : [null]).map((sid) => ({ + propertyServiceId: sid as string, + customerId: p.customerId, + customerName: p.customer.name, + })), + ); + + if (candidates.length > 1) { + return { + propertyServiceId: null, + customerId: null, + note: `${candidates.length} propiedades comparten la clave catastral ${key}`, + confident: false, + candidates, + }; + } + + // The clave identifies the property with certainty, but it is a *secondary* + // key: it was not the number the statement was issued against. Left for + // review so the confirm also teaches the matcher the account number, rather + // than the same page needing the fallback again next month. + return { + propertyServiceId: candidates[0].propertyServiceId ?? null, + customerId: candidates[0].customerId, + note: `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`, + confident: false, + candidates, + }; + } + + private unmatched(note: string): MatchResult { + return { + propertyServiceId: null, + customerId: null, + note, + confident: false, + candidates: [], + }; + } +} diff --git a/apps/api/src/statements/statement.dto.ts b/apps/api/src/statements/statement.dto.ts new file mode 100644 index 0000000..f55c72f --- /dev/null +++ b/apps/api/src/statements/statement.dto.ts @@ -0,0 +1,52 @@ +import { + IsBoolean, + IsEnum, + IsInt, + IsNumber, + IsOptional, + IsString, + MinLength, +} from "class-validator"; +import { Currency, ServiceKind, StatementDocumentStatus } from "@jorgecuadros/database"; + +export class CreateStatementBatchDto { + @IsEnum(ServiceKind) serviceKind!: ServiceKind; + @IsOptional() @IsString() label?: string; +} + +/** Staff correction of one document's extracted fields or its match. */ +export class ReviewDocumentDto { + @IsOptional() @IsString() accountRef?: string; + @IsOptional() @IsNumber() amount?: number; + @IsOptional() @IsString() period?: string; + @IsOptional() @IsString() dueDate?: string; + @IsOptional() @IsString() matchedPropertyServiceId?: string; + @IsOptional() @IsString() matchedCustomerId?: string; + // Restricted to the review-reachable states: a client cannot declare a + // document POSTED, because only a successful ledger write may do that. + @IsOptional() + @IsEnum(StatementDocumentStatus) + status?: Extract; +} + +/** + * Post a batch's confirmed documents. The check-level fields are shared by + * every line, exactly as on the manual batch-capture screen — an OCR batch is + * still "these receipts, paid by this check". + */ +export class ConfirmBatchDto { + @IsString() @MinLength(1) checkNumber!: string; + @IsString() @MinLength(1) transactionDate!: string; + @IsOptional() @IsEnum(Currency) currency?: Currency; + /** Overrides the concept derived from the batch's service kind. */ + @IsOptional() @IsString() typeId?: string; + /** Post as outstanding (sin fondos) — captured but not yet funded. */ + @IsOptional() @IsBoolean() outstanding?: boolean; + /** Also post documents a reviewer explicitly marked CONFIRMED. */ + @IsOptional() @IsBoolean() includeReviewed?: boolean; +} + +export class ListBatchesQuery { + @IsOptional() @IsInt() page?: number; + @IsOptional() @IsInt() pageSize?: number; +} diff --git a/apps/api/src/statements/statements.controller.ts b/apps/api/src/statements/statements.controller.ts new file mode 100644 index 0000000..33f66e3 --- /dev/null +++ b/apps/api/src/statements/statements.controller.ts @@ -0,0 +1,155 @@ +import { + Body, + Controller, + Get, + Param, + Patch, + Post, + Query, + Req, + Res, + StreamableFile, + UploadedFiles, + UseGuards, + UseInterceptors, +} from "@nestjs/common"; +import { FilesInterceptor } from "@nestjs/platform-express"; +import type { ServiceKind, StatementDocumentStatus } from "@jorgecuadros/database"; +import type { Request, Response } from "express"; +import { AuthenticatedGuard } from "../auth/authenticated.guard"; +import { AbilityGuard } from "../auth/ability.guard"; +import { RequireAbility } from "../auth/require-ability.decorator"; +import { AuditService } from "../common/audit.service"; +import type { UploadedFileLike } from "../storage/upload-file"; +import { StatementsService } from "./statements.service"; +import { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto"; + +/** + * Statement OCR intake (RECEIPT_CAPTURE_SPEC §2). + * + * Nothing here writes to the ledger directly — confirming a batch delegates to + * BillingService, so an OCR-captured charge is indistinguishable from a + * hand-keyed one except for its `captureSource`. + */ +@Controller("statements") +@UseGuards(AuthenticatedGuard, AbilityGuard) +export class StatementsController { + constructor( + private readonly statements: StatementsService, + private readonly audit: AuditService, + ) {} + + private actingId(req: Request): string { + return (req.user as { id: string } | undefined)?.id ?? ""; + } + + /** Whether this deployment can OCR at all — the UI hides upload without it. */ + @Get("status") + async status() { + return { ocrAvailable: await this.statements.ocrAvailable() }; + } + + @Get("batches") + listBatches(@Query("page") page?: string, @Query("pageSize") pageSize?: string) { + return this.statements.listBatches( + Math.max(1, Number(page) || 1), + Math.min(100, Math.max(1, Number(pageSize) || 25)), + ); + } + + @Get("batches/:id") + getBatch(@Param("id") id: string) { + return this.statements.getBatch(id); + } + + @Get("batches/:id/documents") + listDocuments(@Param("id") id: string, @Query("status") status?: string) { + return this.statements.listDocuments( + id, + (status || undefined) as StatementDocumentStatus | undefined, + ); + } + + /** The rendered page, so a reviewer can compare it against what was read. */ + @Get("documents/:id/page") + async pageImage(@Param("id") id: string, @Res({ passthrough: true }) res: Response) { + const { stream, contentType, contentLength } = await this.statements.pageImage(id); + res.set({ + "Content-Type": contentType ?? "image/png", + ...(contentLength ? { "Content-Length": String(contentLength) } : {}), + }); + return new StreamableFile(stream); + } + + // --- writes --------------------------------------------------------------- + + @Post("batches") + @RequireAbility("statement:ingest") + @UseInterceptors( + // A month of one company's statements is a handful of multi-page scans; + // 25 files at 50MB covers that with room to spare. + FilesInterceptor("files", 25, { limits: { fileSize: 50 * 1024 * 1024 } }), + ) + async createBatch( + @UploadedFiles() files: UploadedFileLike[] | undefined, + @Query("serviceKind") serviceKind: ServiceKind, + @Query("label") label: string | undefined, + @Req() req: Request, + ) { + const batch = await this.statements.createBatch( + files ?? [], + serviceKind, + this.actingId(req), + label, + ); + void this.audit.log(this.actingId(req), "statement.batch.create", { + batchId: batch.id, + serviceKind, + fileCount: batch.fileCount, + }); + return batch; + } + + @Patch("documents/:id") + @RequireAbility("statement:review") + async review( + @Param("id") id: string, + @Body() dto: ReviewDocumentDto, + @Req() req: Request, + ) { + const doc = await this.statements.review(id, dto, this.actingId(req)); + void this.audit.log(this.actingId(req), "statement.document.review", { + documentId: id, + status: doc.status, + }); + return doc; + } + + @Post("documents/:id/reject") + @RequireAbility("statement:review") + async reject(@Param("id") id: string, @Req() req: Request) { + const doc = await this.statements.reject(id, this.actingId(req)); + void this.audit.log(this.actingId(req), "statement.document.reject", { + documentId: id, + }); + return doc; + } + + /** Post every matched document in the batch, against one check. */ + @Post("batches/:id/confirm") + @RequireAbility("statement:review") + async confirm( + @Param("id") id: string, + @Body() dto: ConfirmBatchDto, + @Req() req: Request, + ) { + const result = await this.statements.confirmBatch(id, dto, this.actingId(req)); + void this.audit.log(this.actingId(req), "statement.batch.confirm", { + batchId: id, + posted: result.posted, + total: result.total, + checkNumber: dto.checkNumber, + }); + return result; + } +} diff --git a/apps/api/src/statements/statements.module.ts b/apps/api/src/statements/statements.module.ts new file mode 100644 index 0000000..c832adf --- /dev/null +++ b/apps/api/src/statements/statements.module.ts @@ -0,0 +1,23 @@ +import { Module } from "@nestjs/common"; +import { BillingModule } from "../billing/billing.module"; +import { StatementsController } from "./statements.controller"; +import { StatementsService } from "./statements.service"; +import { StatementMatcherService } from "./statement-matcher.service"; +import { OCR_PROVIDER } from "./ocr/ocr.provider"; +import { TesseractOcrProvider } from "./ocr/tesseract.provider"; + +/** + * The concrete OCR engine is bound here and nowhere else — everything + * downstream depends on the OcrProvider interface, so swapping Tesseract for a + * managed extraction API is a one-line change in this file. + */ +@Module({ + imports: [BillingModule], + controllers: [StatementsController], + providers: [ + StatementsService, + StatementMatcherService, + { provide: OCR_PROVIDER, useClass: TesseractOcrProvider }, + ], +}) +export class StatementsModule {} diff --git a/apps/api/src/statements/statements.service.ts b/apps/api/src/statements/statements.service.ts new file mode 100644 index 0000000..edee42a --- /dev/null +++ b/apps/api/src/statements/statements.service.ts @@ -0,0 +1,457 @@ +import { + BadRequestException, + Inject, + Injectable, + Logger, + NotFoundException, +} from "@nestjs/common"; +import { + Prisma, + type ServiceKind, + type StatementDocumentStatus, +} from "@jorgecuadros/database"; +import { PrismaService } from "../prisma/prisma.service"; +import { StorageService } from "../storage/storage.service"; +import { BillingService } from "../billing/billing.service"; +import type { UploadedFileLike } from "../storage/upload-file"; +import { OCR_PROVIDER, type OcrProvider } from "./ocr/ocr.provider"; +import { parseStatement } from "./parsers/statement-parser"; +import { StatementMatcherService } from "./statement-matcher.service"; +import type { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto"; + +/** + * Default ledger concept per service kind. The names are the legacy + * `TYPE OF TRX` values already in `type_transactions`, resolved by name once + * per confirm rather than hard-coded as ids, which differ per environment. + */ +const CONCEPT_BY_KIND: Partial> = { + ELECTRIC: "ELECTRIC", + WATER: "WATER", + TELEPHONE: "TELEPHONE", + GAS: "GAS BUTANO", + PROPERTY_TAX: "PROPERTY TAXES", + FEDERAL_ZONE: "FEDERAL ZONE", + CABLE: "CABLE", +}; + +/** Statuses a document can still be worked on from. */ +const OPEN: StatementDocumentStatus[] = ["NEEDS_REVIEW", "MATCHED", "CONFIRMED"]; + +@Injectable() +export class StatementsService { + private readonly logger = new Logger(StatementsService.name); + + constructor( + private readonly prisma: PrismaService, + private readonly storage: StorageService, + private readonly billing: BillingService, + private readonly matcher: StatementMatcherService, + @Inject(OCR_PROVIDER) private readonly ocr: OcrProvider, + ) {} + + ocrAvailable(): Promise { + return this.ocr.available(); + } + + // --- ingest --------------------------------------------------------------- + + /** + * Accept a batch of scanned PDFs and start processing. + * + * Processing is kicked off but deliberately not awaited: 300 pages of OCR is + * minutes of CPU, far past any sane HTTP timeout. The caller gets the batch + * id immediately and polls its status, which is also what lets the review + * queue show partial progress. + */ + async createBatch( + files: UploadedFileLike[], + serviceKind: ServiceKind, + uploadedById: string, + label?: string, + ) { + if (!files?.length) throw new BadRequestException("No se recibió ningún archivo."); + if (!(await this.ocr.available())) { + throw new BadRequestException( + "El servidor no tiene OCR instalado; no se pueden procesar recibos.", + ); + } + + const batch = await this.prisma.statementBatch.create({ + data: { serviceKind, uploadedById, label, fileCount: files.length }, + }); + + // Buffers are held for the background pass; the request's own copies would + // otherwise be garbage once the response is sent. + const copies = files.map((f) => ({ buffer: f.buffer, name: f.originalname })); + void this.process(batch.id, copies, serviceKind).catch(async (err) => { + this.logger.error(`Batch ${batch.id} failed: ${(err as Error).message}`); + await this.prisma.statementBatch.update({ + where: { id: batch.id }, + data: { status: "FAILED", error: (err as Error).message }, + }); + }); + + return batch; + } + + /** Render → OCR → parse → match, one document row per page. */ + private async process( + batchId: string, + files: { buffer: Buffer; name?: string }[], + serviceKind: ServiceKind, + ) { + await this.prisma.statementBatch.update({ + where: { id: batchId }, + data: { status: "PROCESSING" }, + }); + + let pageNumber = 0; + for (const file of files) { + // The source PDF is kept as well as the page images: it is the artifact + // the office actually received, and the only way to re-run a corrected + // parser over the original later. + const sourceKey = `statement/${batchId}/source-${pageNumber + 1}.pdf`; + await this.storage.put(sourceKey, file.buffer, "application/pdf"); + + const pages = await this.ocr.renderPages(file.buffer); + for (const image of pages) { + pageNumber += 1; + const storageKey = `statement/${batchId}/page-${pageNumber}.png`; + await this.storage.put(storageKey, image, "image/png"); + + try { + const ocr = await this.ocr.recognize(image); + const parsed = parseStatement(ocr); + const match = await this.matcher.match(parsed, serviceKind); + + const notes = [...parsed.notes, match.note].filter(Boolean); + // A confident field match is only trusted when nothing contradicts + // it: a barcode that disagrees with the printed number means one of + // the two was misread, and which one is a judgement call. + const trusted = match.confident && parsed.crossChecked !== false; + + await this.prisma.statementDocument.create({ + data: { + batchId, + pageNumber, + storageKey, + status: trusted ? "MATCHED" : "NEEDS_REVIEW", + ocrRawText: ocr.text, + ocrConfidence: new Prisma.Decimal(ocr.confidence.toFixed(3)), + provider: parsed.provider, + extractedAccountRef: parsed.accountRef, + extractedAmount: + parsed.amount != null ? new Prisma.Decimal(parsed.amount) : null, + extractedPeriod: parsed.period, + extractedDueDate: parsed.dueDate, + extractedCadastralKey: parsed.cadastralKey, + matchedPropertyServiceId: match.propertyServiceId, + matchedCustomerId: match.customerId, + matchNote: notes.join("; ").slice(0, 190), + }, + }); + } catch (err) { + // One unreadable page must not abandon the other 299. + await this.prisma.statementDocument.create({ + data: { + batchId, + pageNumber, + storageKey, + status: "OCR_FAILED", + matchNote: (err as Error).message.slice(0, 190), + }, + }); + } + } + } + + await this.prisma.statementBatch.update({ + where: { id: batchId }, + data: { status: "READY_FOR_REVIEW" }, + }); + } + + // --- reads ---------------------------------------------------------------- + + async listBatches(page: number, pageSize: number) { + const [total, items] = await this.prisma.$transaction([ + this.prisma.statementBatch.count(), + this.prisma.statementBatch.findMany({ + orderBy: { createdAt: "desc" }, + skip: (page - 1) * pageSize, + take: pageSize, + include: { + uploadedBy: { select: { name: true } }, + _count: { select: { documents: true } }, + }, + }), + ]); + return { items, total, page, pageSize, pageCount: Math.ceil(total / pageSize) }; + } + + async getBatch(id: string) { + const batch = await this.prisma.statementBatch.findUnique({ + where: { id }, + include: { uploadedBy: { select: { name: true } } }, + }); + if (!batch) throw new NotFoundException("Lote no encontrado."); + + const counts = await this.prisma.statementDocument.groupBy({ + by: ["status"], + where: { batchId: id }, + _count: { _all: true }, + }); + const totals = await this.prisma.statementDocument.aggregate({ + where: { batchId: id, status: { in: OPEN } }, + _sum: { extractedAmount: true }, + }); + + return { + ...batch, + byStatus: Object.fromEntries(counts.map((c) => [c.status, c._count._all])), + pendingTotal: totals._sum.extractedAmount?.toFixed(2) ?? "0.00", + }; + } + + async listDocuments(batchId: string, status?: StatementDocumentStatus) { + return this.prisma.statementDocument.findMany({ + where: { batchId, ...(status ? { status } : {}) }, + orderBy: { pageNumber: "asc" }, + include: { + matchedCustomer: { select: { id: true, name: true } }, + matchedPropertyService: { + select: { + id: true, + kind: true, + accountNumber: true, + meterNumber: true, + property: { select: { id: true, addressLine1: true } }, + }, + }, + }, + }); + } + + /** The rendered page image, so a reviewer can read what the parser read. */ + async pageImage(documentId: string) { + const doc = await this.prisma.statementDocument.findUnique({ + where: { id: documentId }, + select: { storageKey: true }, + }); + if (!doc) throw new NotFoundException("Documento no encontrado."); + return this.storage.getStream(doc.storageKey); + } + + // --- review --------------------------------------------------------------- + + /** Staff correction of an extracted field or of the match itself. */ + async review(id: string, dto: ReviewDocumentDto, reviewedById: string) { + const doc = await this.prisma.statementDocument.findUnique({ where: { id } }); + if (!doc) throw new NotFoundException("Documento no encontrado."); + if (doc.status === "POSTED") { + throw new BadRequestException("Este documento ya fue registrado."); + } + + // Changing the service implies its owner; deriving the customer here rather + // than trusting a client-supplied pair is what stops a page being posted to + // one customer's ledger against another customer's service. + let matchedCustomerId = doc.matchedCustomerId; + let matchedPropertyServiceId = dto.matchedPropertyServiceId ?? undefined; + + if (dto.matchedPropertyServiceId) { + const svc = await this.prisma.propertyService.findUnique({ + where: { id: dto.matchedPropertyServiceId }, + select: { property: { select: { customerId: true } } }, + }); + if (!svc) throw new BadRequestException("Servicio no encontrado."); + matchedCustomerId = svc.property.customerId; + } else if (dto.matchedCustomerId) { + matchedCustomerId = dto.matchedCustomerId; + + // A reviewer picks a *customer*, not one of their service rows. Without + // a service the posting still works, but the confirmed reference has + // nowhere to be written back, so the same account would land in review + // again next month — which is exactly the behaviour that is supposed to + // make gas (whose numbers the migration never populated) a one-time cost. + // So: if the batch's service kind resolves to exactly one of that + // customer's services that has no reference yet, attach it. Exactly one + // — with two candidates there is no way to tell which meter or line the + // bill belongs to, and guessing would write a real number onto the wrong + // service. + const batch = await this.prisma.statementBatch.findUnique({ + where: { id: doc.batchId }, + select: { serviceKind: true }, + }); + if (batch) { + const field = batch.serviceKind === "GAS" ? "meterNumber" : "accountNumber"; + const blank = await this.prisma.propertyService.findMany({ + where: { + kind: batch.serviceKind, + [field]: null, + property: { customerId: matchedCustomerId }, + }, + select: { id: true }, + take: 2, + }); + if (blank.length === 1) matchedPropertyServiceId = blank[0].id; + } + } + + return this.prisma.statementDocument.update({ + where: { id }, + data: { + extractedAccountRef: dto.accountRef ?? undefined, + extractedAmount: + dto.amount != null ? new Prisma.Decimal(dto.amount) : undefined, + extractedPeriod: dto.period ?? undefined, + extractedDueDate: dto.dueDate ? new Date(dto.dueDate) : undefined, + matchedPropertyServiceId, + matchedCustomerId, + status: dto.status ?? "MATCHED", + reviewedById, + reviewedAt: new Date(), + }, + }); + } + + async reject(id: string, reviewedById: string) { + const doc = await this.prisma.statementDocument.findUnique({ where: { id } }); + if (!doc) throw new NotFoundException("Documento no encontrado."); + if (doc.status === "POSTED") { + throw new BadRequestException("Este documento ya fue registrado."); + } + return this.prisma.statementDocument.update({ + where: { id }, + data: { status: "REJECTED", reviewedById, reviewedAt: new Date() }, + }); + } + + // --- posting -------------------------------------------------------------- + + /** + * Post every confirmable document in a batch to the ledger. + * + * This goes through `BillingService.createBatch` — the same method the manual + * "Editor" screen uses — rather than writing `Transaction` rows directly, so + * OCR-sourced and hand-keyed receipts share one write path, one validation + * path and one audit trail. `source: "OCR"` and a per-line `captureRef` of + * the document id give the duplicate-post guard something to key on, so a + * batch confirmed twice cannot double-charge anyone. + */ + async confirmBatch(batchId: string, dto: ConfirmBatchDto, reviewedById: string) { + const batch = await this.prisma.statementBatch.findUnique({ + where: { id: batchId }, + }); + if (!batch) throw new NotFoundException("Lote no encontrado."); + + const docs = await this.prisma.statementDocument.findMany({ + where: { + batchId, + status: { in: dto.includeReviewed ? ["MATCHED", "CONFIRMED"] : ["MATCHED"] }, + matchedCustomerId: { not: null }, + }, + orderBy: { pageNumber: "asc" }, + }); + if (!docs.length) { + throw new BadRequestException("No hay documentos listos para registrar."); + } + + const missing = docs.filter((d) => d.extractedAmount == null); + if (missing.length) { + throw new BadRequestException( + `Falta el importe en ${missing.length} documento(s): página(s) ` + + missing.map((d) => d.pageNumber).join(", "), + ); + } + + const typeId = dto.typeId ?? (await this.conceptFor(batch.serviceKind)); + + const result = await this.billing.createBatch( + { + domain: "UTILITY", + transactionDate: dto.transactionDate, + checkNumber: dto.checkNumber, + currency: dto.currency ?? "MXN", + typeId, + lines: docs.map((d) => ({ + customerId: d.matchedCustomerId!, + // Charges are negative in this ledger: a negative amount is what the + // customer owes. The parser reads the printed (positive) figure, so + // the sign is applied here, at the single point where a statement + // becomes a ledger row. + amount: -Math.abs(Number(d.extractedAmount)), + reference: d.extractedAccountRef ?? undefined, + period: d.extractedPeriod ?? undefined, + outstanding: dto.outstanding ?? false, + })), + }, + { source: "OCR", refs: docs.map((d) => d.id) }, + ); + + // `items[i]` is positionally parallel to `lines[i]` (seam guarantee 1), so + // the created rows zip straight back onto the documents that produced them. + await this.prisma.$transaction( + docs.map((d, i) => + this.prisma.statementDocument.update({ + where: { id: d.id }, + data: { + status: "POSTED", + postedTransactionId: result.items[i].id, + reviewedById, + reviewedAt: new Date(), + }, + }), + ), + ); + + // Teach the matcher. When a document was matched by clave catastral or by + // hand because the scoped field was blank, writing the reference back means + // next month's statement for the same account matches on its own — this is + // what turns gas (whose numbers the migration never populated) from a + // permanent review queue into a one-time cost. + await this.learnAccountRefs(docs, batch.serviceKind); + + await this.closeIfDone(batchId); + + return { posted: result.count, total: result.total, checkNumber: dto.checkNumber }; + } + + /** Write a confirmed reference onto a service that had none. */ + private async learnAccountRefs( + docs: { matchedPropertyServiceId: string | null; extractedAccountRef: string | null }[], + kind: ServiceKind, + ) { + const field = kind === "GAS" ? "meterNumber" : "accountNumber"; + for (const d of docs) { + if (!d.matchedPropertyServiceId || !d.extractedAccountRef) continue; + await this.prisma.propertyService.updateMany({ + // Only fills a hole — never overwrites a number already on file, which + // would let one misread page rewrite good reference data. + where: { id: d.matchedPropertyServiceId, [field]: null }, + data: { [field]: d.extractedAccountRef }, + }); + } + } + + private async closeIfDone(batchId: string) { + const open = await this.prisma.statementDocument.count({ + where: { batchId, status: { in: OPEN } }, + }); + if (open === 0) { + await this.prisma.statementBatch.update({ + where: { id: batchId }, + data: { status: "COMPLETED", completedAt: new Date() }, + }); + } + } + + private async conceptFor(kind: ServiceKind): Promise { + const name = CONCEPT_BY_KIND[kind]; + if (!name) return undefined; + const row = await this.prisma.typeTransaction.findFirst({ + where: { nameEn: name }, + select: { id: true }, + }); + return row?.id; + } +} diff --git a/apps/web/src/app/recibos/[id]/page.tsx b/apps/web/src/app/recibos/[id]/page.tsx new file mode 100644 index 0000000..cf26b90 --- /dev/null +++ b/apps/web/src/app/recibos/[id]/page.tsx @@ -0,0 +1,478 @@ +"use client"; + +import { useCallback, useEffect, useMemo, useState } from "react"; +import Link from "next/link"; +import { AppShell } from "@/components/AppShell"; +import { CustomerPicker } from "@/components/CustomerPicker"; +import { + confirmStatementBatch, + getStatementBatch, + listStatementDocuments, + rejectStatementDocument, + reviewStatementDocument, + statementPageUrl, +} from "@/lib/api"; +import { useCan } from "@/lib/abilities"; +import { formatDate, formatMoney, serviceKindLabel } from "@/lib/labels"; +import type { + ConfirmBatchInput, + StatementBatchDetail, + StatementDocument, + StatementDocumentStatus, +} from "@/lib/types"; + +/** + * Review queue for one batch of scanned bills. + * + * The reviewer's job is to answer one question per page — "is this the right + * customer for this amount?" — so the page image sits next to the extracted + * fields and every row can be corrected in place. Rows the matcher is sure + * about are pre-approved and can be posted in bulk; everything else is listed + * first, because that is the work. + */ + +const STATUS_LABEL: Record = { + PENDING_OCR: "En proceso", + OCR_FAILED: "No se pudo leer", + NEEDS_REVIEW: "Requiere revisión", + MATCHED: "Identificado", + CONFIRMED: "Confirmado", + POSTED: "Registrado", + REJECTED: "Descartado", +}; + +/** Rows still needing a decision, listed before the settled ones. */ +const OPEN_FIRST: StatementDocumentStatus[] = [ + "NEEDS_REVIEW", + "OCR_FAILED", + "MATCHED", + "CONFIRMED", + "POSTED", + "REJECTED", + "PENDING_OCR", +]; + +export default function RecibosBatchPage({ params }: { params: { id: string } }) { + return ( + + + + ); +} + +function BatchReview({ id }: { id: string }) { + const canReview = useCan("statement:review"); + const [batch, setBatch] = useState(null); + const [docs, setDocs] = useState([]); + const [error, setError] = useState(null); + const [loading, setLoading] = useState(true); + + const load = useCallback(async () => { + try { + const [b, d] = await Promise.all([ + getStatementBatch(id), + listStatementDocuments(id), + ]); + setBatch(b); + setDocs(d); + setError(null); + } catch (e) { + setError((e as Error)?.message ?? "No se pudo cargar el lote."); + } finally { + setLoading(false); + } + }, [id]); + + useEffect(() => { + void load(); + }, [load]); + + const processing = batch?.status === "PROCESSING" || batch?.status === "UPLOADED"; + useEffect(() => { + if (!processing) return; + const t = setInterval(() => void load(), 4000); + return () => clearInterval(t); + }, [processing, load]); + + const sorted = useMemo( + () => + [...docs].sort( + (a, b) => + OPEN_FIRST.indexOf(a.status) - OPEN_FIRST.indexOf(b.status) || + a.pageNumber - b.pageNumber, + ), + [docs], + ); + + const readyCount = docs.filter( + (d) => d.status === "MATCHED" && d.matchedCustomer, + ).length; + + if (loading) return
Cargando…
; + if (!batch) return
{error ?? "No encontrado."}
; + + return ( +
+
+
+

+ Recibos — {serviceKindLabel(batch.serviceKind)} + {batch.label ? ` · ${batch.label}` : ""} +

+

+ {formatDate(batch.createdAt)} · {docs.length} página(s) ·{" "} + {STATUS_LABEL_BATCH[batch.status] ?? batch.status} +

+
+ + Volver + +
+ + {error &&
{error}
} + + {processing && ( +
+ Leyendo los recibos… esta pantalla se actualiza sola. +
+ )} + + + + {canReview && readyCount > 0 && ( + + )} + +
+ {sorted.map((doc) => ( + + ))} +
+
+ ); +} + +const STATUS_LABEL_BATCH: Record = { + UPLOADED: "Recibido", + PROCESSING: "Procesando", + READY_FOR_REVIEW: "Listo para revisar", + COMPLETED: "Registrado", + FAILED: "Falló", +}; + +function SummaryCard({ + batch, + readyCount, +}: { + batch: StatementBatchDetail; + readyCount: number; +}) { + const entries = Object.entries(batch.byStatus) as [StatementDocumentStatus, number][]; + return ( +
+
+ {entries.map(([status, count]) => ( +
+
{STATUS_LABEL[status] ?? status}
+
{count}
+
+ ))} +
+
Importe pendiente
+
+ {formatMoney(batch.pendingTotal, "MXN")} +
+
+
+
Listos para registrar
+
{readyCount}
+
+
+
+ ); +} + +/** + * Posting is by check, exactly as on the manual capture screen — an OCR batch + * is still "these bills, paid with this check", so the same fields are asked + * for and the same ledger path is used. + */ +function ConfirmCard({ + batchId, + readyCount, + onDone, + setError, +}: { + batchId: string; + readyCount: number; + onDone: () => void; + setError: (m: string | null) => void; +}) { + const [checkNumber, setCheckNumber] = useState(""); + const [transactionDate, setTransactionDate] = useState( + new Date().toISOString().slice(0, 10), + ); + const [outstanding, setOutstanding] = useState(false); + const [includeReviewed, setIncludeReviewed] = useState(true); + const [busy, setBusy] = useState(false); + const [result, setResult] = useState(null); + + async function submit() { + if (!checkNumber.trim()) return; + setBusy(true); + setError(null); + try { + const input: ConfirmBatchInput = { + checkNumber: checkNumber.trim(), + transactionDate, + outstanding, + includeReviewed, + }; + const r = await confirmStatementBatch(batchId, input); + setResult( + `Se registraron ${r.posted} movimiento(s) por ${formatMoney(r.total, "MXN")} con el cheque ${r.checkNumber}.`, + ); + setCheckNumber(""); + onDone(); + } catch (e) { + setError((e as Error)?.message ?? "No se pudo registrar el lote."); + } finally { + setBusy(false); + } + } + + return ( +
+

+ Registrar {readyCount} recibo(s) +

+
+ + + + + +
+ {result && ( +
+ {result} +
+ )} +

+ Se registran como cargos del cliente, por la misma vía que la captura + manual. Un lote registrado dos veces no duplica cobros. +

+
+ ); +} + +function DocumentRow({ + doc, + canReview, + onChange, +}: { + doc: StatementDocument; + canReview: boolean; + onChange: () => void; +}) { + const [open, setOpen] = useState( + doc.status === "NEEDS_REVIEW" || doc.status === "OCR_FAILED", + ); + const [amount, setAmount] = useState(doc.extractedAmount ?? ""); + const [accountRef, setAccountRef] = useState(doc.extractedAccountRef ?? ""); + const [customerId, setCustomerId] = useState(doc.matchedCustomer?.id ?? ""); + const [customerName, setCustomerName] = useState(doc.matchedCustomer?.name ?? ""); + const [busy, setBusy] = useState(false); + const [err, setErr] = useState(null); + + const settled = doc.status === "POSTED" || doc.status === "REJECTED"; + + async function save(status: "MATCHED" | "CONFIRMED") { + setBusy(true); + setErr(null); + try { + await reviewStatementDocument(doc.id, { + accountRef: accountRef.trim() || undefined, + amount: amount ? Number(amount) : undefined, + matchedCustomerId: customerId || undefined, + status, + }); + onChange(); + } catch (e) { + setErr((e as Error)?.message ?? "No se pudo guardar."); + } finally { + setBusy(false); + } + } + + async function reject() { + setBusy(true); + setErr(null); + try { + await rejectStatementDocument(doc.id); + onChange(); + } catch (e) { + setErr((e as Error)?.message ?? "No se pudo descartar."); + } finally { + setBusy(false); + } + } + + return ( +
+
+
+ Página {doc.pageNumber}{" "} + {STATUS_LABEL[doc.status] ?? doc.status}{" "} + {doc.provider && {doc.provider}} +
+ {doc.matchedCustomer ? ( + + {doc.matchedCustomer.name} + + ) : ( + "Sin cliente asignado" + )} + {doc.extractedAccountRef && ` · cuenta ${doc.extractedAccountRef}`} + {doc.extractedCadastralKey && ` · clave ${doc.extractedCadastralKey}`} +
+ {doc.matchNote && ( +
+ {doc.matchNote} +
+ )} +
+
+ + {doc.extractedAmount + ? formatMoney(doc.extractedAmount, "MXN") + : "sin importe"} + + +
+
+ + {open && ( +
+ {/* The scan itself — the reviewer's source of truth, not the OCR. */} + {`Recibo + + {canReview && !settled && ( +
+ + +
+ Cliente + { + setCustomerId(cid); + setCustomerName(name); + }} + /> +
+ + +
+ )} + + {err &&
{err}
} +
+ )} +
+ ); +} diff --git a/apps/web/src/app/recibos/page.tsx b/apps/web/src/app/recibos/page.tsx new file mode 100644 index 0000000..0f6b605 --- /dev/null +++ b/apps/web/src/app/recibos/page.tsx @@ -0,0 +1,270 @@ +"use client"; + +import { useCallback, useEffect, useState } from "react"; +import Link from "next/link"; +import { AppShell } from "@/components/AppShell"; +import { + getStatementStatus, + listStatementBatches, + uploadStatementBatch, +} from "@/lib/api"; +import { useCan } from "@/lib/abilities"; +import { formatDate, SERVICE_KIND_LABELS, serviceKindLabel } from "@/lib/labels"; +import type { ServiceKind, StatementBatch, StatementBatchStatus } from "@/lib/types"; + +/** + * Statement OCR intake (docs/RECEIPT_CAPTURE_SPEC.md §2). + * + * Each utility company mails 300+ paper bills a month, one per customer, which + * staff otherwise key in by hand through the "Captura" screen. Here they scan + * the stack, and the machine proposes customer + amount for every page; a + * human still confirms before anything reaches the ledger. + * + * One batch = one service kind, because the matcher is scoped per kind: a + * water account number and a phone number are compared against different + * columns, and mixing them in one upload is how a bill gets posted to the + * wrong customer. + */ + +/** The kinds the parsers actually recognise today. */ +const SUPPORTED: ServiceKind[] = ["ELECTRIC", "WATER", "TELEPHONE"]; +/** Uploadable, but every page will land in review until a parser learns it. */ +const OTHER_KINDS: ServiceKind[] = ["GAS", "PROPERTY_TAX", "FEDERAL_ZONE", "CABLE"]; + +const STATUS_LABEL: Record = { + UPLOADED: "Recibido", + PROCESSING: "Procesando…", + READY_FOR_REVIEW: "Listo para revisar", + COMPLETED: "Registrado", + FAILED: "Falló", +}; + +export default function RecibosPage() { + return ( + + + + ); +} + +function Recibos() { + const canIngest = useCan("statement:ingest"); + const [batches, setBatches] = useState([]); + const [ocrAvailable, setOcrAvailable] = useState(null); + const [loading, setLoading] = useState(true); + const [error, setError] = useState(null); + + const load = useCallback(async () => { + try { + const [list, status] = await Promise.all([ + listStatementBatches(), + getStatementStatus(), + ]); + setBatches(list.items); + setOcrAvailable(status.ocrAvailable); + setError(null); + } catch (e) { + setError((e as Error)?.message ?? "No se pudieron cargar los lotes."); + } finally { + setLoading(false); + } + }, []); + + useEffect(() => { + void load(); + }, [load]); + + // A batch of 300 pages takes minutes to OCR, so the list refreshes itself + // while anything is still working rather than making staff reload. + const working = batches.some( + (b) => b.status === "PROCESSING" || b.status === "UPLOADED", + ); + useEffect(() => { + if (!working) return; + const t = setInterval(() => void load(), 4000); + return () => clearInterval(t); + }, [working, load]); + + return ( +
+
+
+

Recibos (OCR)

+

+ Escanea los recibos del mes y el sistema propone cliente e importe + para cada página. Nada se registra sin tu confirmación. +

+
+
+ + {ocrAvailable === false && ( +
+ Este servidor no tiene OCR instalado, así que no se pueden procesar + recibos. El resto del sistema funciona con normalidad. +
+ )} + + {canIngest && ocrAvailable && } + + {error &&
{error}
} + +
+

+ Lotes +

+ {loading ? ( +
Cargando…
+ ) : batches.length === 0 ? ( +
Todavía no hay lotes de recibos.
+ ) : ( +
+ + + + + + + + + + + + + {batches.map((b) => ( + + + + + + + + + + ))} + +
FechaServicioReferenciaEstadoPáginasSubido por +
{formatDate(b.createdAt)}{serviceKindLabel(b.serviceKind)}{b.label || "—"} + + {b.error && ( +
+ {b.error} +
+ )} + 		{b._count?.documents ?? 0}{b.uploadedBy?.name ?? "—"} + + Revisar + +
+
+ )} +
+
+ ); +} + +function StatusTag({ status }: { status: StatementBatchStatus }) { + return {STATUS_LABEL[status] ?? status}; +} + +function UploadCard({ onDone }: { onDone: () => void }) { + const [files, setFiles] = useState([]); + const [serviceKind, setServiceKind] = useState("ELECTRIC"); + const [label, setLabel] = useState(""); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + + async function submit() { + if (!files.length) return; + setBusy(true); + setError(null); + try { + await uploadStatementBatch(files, serviceKind, label.trim() || undefined); + setFiles([]); + setLabel(""); + onDone(); + } catch (e) { + setError((e as Error)?.message ?? "No se pudo subir el lote."); + } finally { + setBusy(false); + } + } + + const unsupported = !SUPPORTED.includes(serviceKind); + + return ( +
+

+ Subir recibos escaneados +

+
+ + + + + + + +
+ + {unsupported && ( +
+ Todavía no hay lectura automática para{" "} + {SERVICE_KIND_LABELS[serviceKind] ?? serviceKind}: cada página quedará + para revisión manual. Al confirmarlas se guarda el número de cuenta, + así que los recibos del mes siguiente sí se reconocerán solos. +
+ )} + +

+ Un lote es de un solo servicio. Cada página del PDF se trata como un + recibo distinto, salvo que el proveedor imprima varias hojas por cliente. +

+
+ ); +} diff --git a/apps/web/src/components/AppShell.tsx b/apps/web/src/components/AppShell.tsx index a1df301..40f18fe 100644 --- a/apps/web/src/components/AppShell.tsx +++ b/apps/web/src/components/AppShell.tsx @@ -47,6 +47,9 @@ const NAV: NavEntry[] = [ // Daily data-entry screen (the legacy "Editor"). Hidden from VIEWER, who // can't capture anyway — the page itself also refuses. { href: "/estado-cuenta/lote", label: "Captura", ability: "ledger:create" }, + // Same daily job as "Captura", entered from a stack of scanned bills + // instead of a keyboard. + { href: "/recibos", label: "Recibos (OCR)", ability: "statement:ingest" }, { href: "/estado-cuenta", label: "Estado de cuenta" }, { href: "/banco", label: "Chequera" }, ], diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts index ae1e698..75be613 100644 --- a/apps/web/src/lib/api.ts +++ b/apps/web/src/lib/api.ts @@ -16,6 +16,8 @@ import type { BankStats, BankSummary, BatchCreateInput, + ConfirmBatchInput, + ConfirmBatchResult, BatchCreateResponse, BillingFacets, BillingStats, @@ -27,6 +29,11 @@ import type { CreateMovementInput, UpdateBankAccountInput, ResolveOutstandingInput, + ReviewDocumentInput, + StatementBatch, + StatementBatchDetail, + StatementDocument, + StatementDocumentStatus, CustomerDetail, CustomerInput, CustomerListResponse, @@ -895,3 +902,93 @@ export function reportDownloadUrl( const tail = qs.toString(); return `${API_ORIGIN}/reports/${slug}/${format}${tail ? `?${tail}` : ""}`; } + +/* ------------------------------------- Statement OCR intake (recibos) */ + +/** Whether this deployment has the OCR binaries — upload is hidden without. */ +export function getStatementStatus(): Promise<{ ocrAvailable: boolean }> { + return apiFetch("/statements/status"); +} + +export function listStatementBatches( + page = 1, + pageSize = 25, +): Promise<{ + items: StatementBatch[]; + total: number; + page: number; + pageSize: number; + pageCount: number; +}> { + return apiFetch(`/statements/batches?page=${page}&pageSize=${pageSize}`); +} + +export function getStatementBatch(id: string): Promise { + return apiFetch(`/statements/batches/${id}`); +} + +export function listStatementDocuments( + batchId: string, + status?: StatementDocumentStatus, +): Promise { + const q = status ? `?status=${status}` : ""; + return apiFetch(`/statements/batches/${batchId}/documents${q}`); +} + +/** Multi-file upload — one batch is usually several multi-page scans. */ +export async function uploadStatementBatch( + files: File[], + serviceKind: ServiceKind, + label?: string, +): Promise { + const body = new FormData(); + for (const f of files) body.append("files", f, f.name); + const qs = new URLSearchParams({ serviceKind }); + if (label) qs.set("label", label); + + const res = await fetch(`${API_ORIGIN}/statements/batches?${qs}`, { + method: "POST", + credentials: "include", + body, + }); + if (!res.ok) { + let message = `Error ${res.status}`; + try { + const b = await res.json(); + if (b?.message) message = b.message; + } catch { + /* non-JSON error body */ + } + throw new Error(message); + } + return res.json(); +} + +export function reviewStatementDocument( + id: string, + input: ReviewDocumentInput, +): Promise { + return apiFetch(`/statements/documents/${id}`, { + method: "PATCH", + body: JSON.stringify(input), + }); +} + +export function rejectStatementDocument(id: string): Promise { + return apiFetch(`/statements/documents/${id}/reject`, { method: "POST" }); +} + +export function confirmStatementBatch( + batchId: string, + input: ConfirmBatchInput, +): Promise { + return apiFetch(`/statements/batches/${batchId}/confirm`, { + method: "POST", + body: JSON.stringify(input), + }); +} + +/** The rendered page image. A plain — the cookie rides along. */ +export function statementPageUrl(documentId: string): string { + return `${API_ORIGIN}/statements/documents/${documentId}/page`; +} diff --git a/apps/web/src/lib/labels.ts b/apps/web/src/lib/labels.ts index 4dd623b..f1e72da 100644 --- a/apps/web/src/lib/labels.ts +++ b/apps/web/src/lib/labels.ts @@ -45,6 +45,7 @@ export const SERVICE_KIND_LABELS: Record = { PROPERTY_TAX: "Predial", FEDERAL_ZONE: "Zona Federal", ALARM: "Alarma", + TELEPHONE: "Teléfono", OTHER: "Otro", }; @@ -57,6 +58,7 @@ export const SERVICE_KIND_GLYPH: Record = { WATER: "≈", ELECTRIC: "⚡", GAS: "◐", + TELEPHONE: "☎", CABLE: "▤", PROPERTY_TAX: "⌂", FEDERAL_ZONE: "⇲", diff --git a/apps/web/src/lib/types.ts b/apps/web/src/lib/types.ts index a1ac4ec..40b9865 100644 --- a/apps/web/src/lib/types.ts +++ b/apps/web/src/lib/types.ts @@ -20,6 +20,8 @@ export type Ability = | "bank:create" | "bank:void" | "bank:manage-accounts" + | "statement:ingest" + | "statement:review" | "lookup:manage" | "user:manage" | "db:manage"; @@ -134,6 +136,7 @@ export type ServiceKind = | "PROPERTY_TAX" | "FEDERAL_ZONE" | "ALARM" + | "TELEPHONE" | "OTHER" | string; @@ -1204,3 +1207,89 @@ export interface ReportRunResult { export interface ReportCatalog { items: ReportDef[]; } + +/* ------------------------------------- Statement OCR intake (recibos) */ + +export type StatementBatchStatus = + | "UPLOADED" + | "PROCESSING" + | "READY_FOR_REVIEW" + | "COMPLETED" + | "FAILED"; + +export type StatementDocumentStatus = + | "PENDING_OCR" + | "OCR_FAILED" + | "NEEDS_REVIEW" + | "MATCHED" + | "CONFIRMED" + | "POSTED" + | "REJECTED"; + +export interface StatementBatch { + id: string; + serviceKind: ServiceKind; + status: StatementBatchStatus; + label: string | null; + fileCount: number; + error: string | null; + createdAt: string; + completedAt: string | null; + uploadedBy?: { name: string }; + _count?: { documents: number }; +} + +export interface StatementBatchDetail extends StatementBatch { + byStatus: Partial>; + /** Sum of the amounts still awaiting posting. */ + pendingTotal: string; +} + +export interface StatementDocument { + id: string; + pageNumber: number; + status: StatementDocumentStatus; + provider: string | null; + ocrConfidence: string | null; + extractedAccountRef: string | null; + extractedAmount: string | null; + extractedPeriod: string | null; + extractedDueDate: string | null; + extractedCadastralKey: string | null; + matchNote: string | null; + matchedCustomer: { id: string; name: string } | null; + matchedPropertyService: { + id: string; + kind: ServiceKind; + accountNumber: string | null; + meterNumber: string | null; + property: { id: string; addressLine1: string | null }; + } | null; + postedTransactionId: string | null; +} + +export interface ReviewDocumentInput { + accountRef?: string; + amount?: number; + period?: string; + dueDate?: string; + matchedPropertyServiceId?: string; + matchedCustomerId?: string; + status?: "MATCHED" | "NEEDS_REVIEW" | "CONFIRMED"; +} + +/** Check-level fields shared by every line posted from a batch. */ +export interface ConfirmBatchInput { + checkNumber: string; + transactionDate: string; + currency?: Currency; + typeId?: string; + outstanding?: boolean; + includeReviewed?: boolean; +} + +export interface ConfirmBatchResult { + posted: number; + total: string; + checkNumber: string; +} diff --git a/docker/api.Dockerfile b/docker/api.Dockerfile index c5c47fb..b7da029 100644 --- a/docker/api.Dockerfile +++ b/docker/api.Dockerfile @@ -46,7 +46,17 @@ ENV NODE_ENV=production # ERROR 1045: Plugin caching_sha2_password could not be loaded # That breaks the pre-migrate deploy backup AND the whole "Operaciones" admin # panel (backup, restore, sync, re-import all shell out to these binaries). +# +# tesseract-ocr + tesseract-ocr-data-spa + poppler-utils drive the statement +# OCR intake (RECEIPT_CAPTURE_SPEC §2): poppler's `pdftoppm` rasterises each +# scanned page and tesseract reads it, with the Spanish traineddata for the +# accented labels on CFE/CESPT/Telnor bills. These are external binaries rather +# than a native npm addon so the pnpm workspace stays free of a compiled +# dependency. If they are absent the API still boots — the statements module +# reports itself unavailable and only that feature is disabled — but statement +# ingest is the point of shipping them. RUN apk add --no-cache python3 mdbtools mysql-client mariadb-connector-c openssl \ + tesseract-ocr tesseract-ocr-data-spa poppler-utils \ && apk add --no-cache --virtual .pybuild python3-dev build-base \ && rm -rf /var/cache/apk/* diff --git a/docs/RECEIPT_CAPTURE_SPEC.md b/docs/RECEIPT_CAPTURE_SPEC.md index 6e611d0..5486a74 100644 --- a/docs/RECEIPT_CAPTURE_SPEC.md +++ b/docs/RECEIPT_CAPTURE_SPEC.md @@ -131,6 +131,72 @@ single-movement form. ## 2. PDF / OCR auto-capture +> **BUILT — 2026-08-01.** Implemented and verified end to end against real +> scanned statements. `apps/api/src/statements/` holds the module: a swappable +> `OcrProvider` seam with a self-hosted Tesseract implementation, per-provider +> parsers for CFE / CESPT / Telnor, a scoped matcher, and a review queue that +> posts through `BillingService.createBatch` with `source: "OCR"`. Web: +> `/recibos` (upload + batch list) and `/recibos/:id` (review queue with the +> page image beside the extracted fields). New abilities `statement:ingest` / +> `statement:review`, both STAFF. +> +> **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor +> bills) drove every decision below. Against them the shipped parser identifies +> the provider on **46/46**, reads an account reference on **43/46**, an amount +> on **42/46**, and a due date on **44/46**. Matched against the dev database +> that is **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The +> remainder are legitimate review cases: one account number shared by two +> services, three phone numbers not yet on file, one clave not in the book, and +> one page too poorly scanned to read. +> +> **The OCR-provider question is closed: self-hosted Tesseract.** It clears the +> bar for a queue where a human confirms every row, and at 300+ pages/month/ +> company a per-page API would carry real recurring cost for accuracy that is +> not the bottleneck. `OcrProvider` keeps a managed API (Textract, Document +> Intelligence, Document AI) a one-line swap in `statements.module.ts` with no +> schema change. +> +> **Four things the samples proved that this spec had wrong or unknown:** +> +> 1. **Clave catastral ≠ predial — gap 2 below is resolved.** `DATMEX.clave` is +> 934 rows of `[A-Z]{2}[0-9]{6}` (`MM000012`, `KH220204`), the exact format +> printed as `Cve. Cat.` / `CLAVE CATASTRAL` on real CESPT bills +> (`KB078025`, `KA903009`). `DATMEX.predial` — what +> `PROPERTY_TAX.accountNumber` actually holds — is 1135 numeric rows with +> only **663 distinct values**, so it is not a per-property key at all and +> appears on no statement. The clave was never migrated; it now lives on +> `Property.cadastralKey` (property-level, because two different services +> both print it) and is the matcher's secondary key. Predial is left +> untouched. Predial statements match on the clave alone. +> 2. **Gas is not a dead end — gap 3 below was wrong.** `DATMEX.gas` has 334 +> filled rows, of which **160 are real numeric account numbers** +> (`900004807`); the other 174 are tank descriptors (`ESTACIONARIO`, +> `CILINDRO`). All 334 went to `notes`. The 160 are recovered into +> `GAS.meterNumber`; only the descriptor rows start cold. +> 3. **Phone is one line per property, not three.** Of 1518 properties, 534 +> have `phone1`, 18 have `phone2` and exactly **1** has `phone3`. The +> secondaries are alternate contacts, so `TELEPHONE` backfills from `phone1` +> only rather than fanning out. This answers the open question below. +> 4. **Statements arrive bundled, and their printed names are stale.** One PDF +> holds many customers, one per page (Telnor's own `Pág 3 de 6` refers to +> its internal pagination, not the office's scan). And the name on a utility +> bill is the account registrant, not the current owner: a CESPT receipt for +> account `5365218` prints `ARNAIZ ROSAS ELSA AURORA` where the office's +> book — corroborated by the clave — has `CATT, RANDY`. **The matcher never +> reads the name.** +> +> **Two OCR traps worth keeping in mind if the parsers are ever extended:** +> scanned logos read badly (a CESPT header came back as `E BAJA ES PAGO / +> EALIFORNIA`), so provider detection falls back to layout anchors — but only +> after *every* brand check has run, since a Telnor page contains words a CFE +> structural rule would otherwise claim. And amounts must be parsed by +> separator position: a real Telnor bill OCR'd as `$ 649,00`, which naive +> comma-stripping turns into $64,900. +> +> **Not covered:** handwritten folder numbers. Staff pencil a customer number on +> each bill (`9`, `405`, `406`); Tesseract read `405` as `205`. Handwriting is +> a review hint at best and is deliberately not an input to matching. + ### Motivation (from the meeting) Each utility company (CFE, water, phone, gas...) sends 300+ individual @@ -810,16 +876,23 @@ action (`customer:purge`) taken well after release — not bundled into ## Open questions to take back to Jorge (collected) -- OCR provider/budget for §2 (self-hosted vs. managed API, given 300+ - pages/month/company). -- Whether source PDFs arrive pre-split per customer or as one bundled file - needing page-range detection (§2). -- Whether "Clave Catastral" and the already-migrated `PREDIAL`-sourced - `PROPERTY_TAX.accountNumber` are the same number — blocks OCR matching - for predial statements specifically until confirmed (§2). -- Whether phone billing is really one service per phone number on file, or - one per property regardless of how many numbers are recorded — decides - how the new `TELEPHONE` service kind gets backfilled (§2). +- ~~OCR provider/budget for §2~~ — **CLOSED**: self-hosted Tesseract, chosen on + measured accuracy against real scans (see §2's BUILT note). No per-page cost. +- ~~Whether source PDFs arrive pre-split per customer or bundled~~ — + **CLOSED**: bundled, one customer per page. Split per page. +- ~~Whether "Clave Catastral" and the `PREDIAL`-sourced + `PROPERTY_TAX.accountNumber` are the same number~~ — **CLOSED**: they are + different. `clave` is the cadastral key and is now on + `Property.cadastralKey`; `predial` is not unique and is not printed on + statements. +- ~~Whether phone billing is one service per number or one per property~~ — + **CLOSED**: effectively one (534 / 18 / 1 across phone1/2/3), backfilled + from `phone1`. +- **Still open (§2):** whether the CFE amount staff should owe is the rounded + headline (`$268`, what the barcode encodes and what is paid at the window) or + the exact `Total` in the breakdown (`$268.88`). The parser currently takes + the barcode figure, which matches what the office actually pays; worth one + confirmation from Jorge. - The actual bank name/currency/details for the Seguros USD account, and whether any historical Seguros bank data exists to migrate (§3). - Whether `BankAccount.businessLine` should be enforced or a soft hint diff --git a/migration/backfill_statement_match_fields.py b/migration/backfill_statement_match_fields.py new file mode 100644 index 0000000..d01d9ed --- /dev/null +++ b/migration/backfill_statement_match_fields.py @@ -0,0 +1,155 @@ +""" +Closes the three data gaps the statement-OCR matcher depends on +(docs/RECEIPT_CAPTURE_SPEC.md §2, "Matching logic"). + +OCR matching is only as good as the field it matches against, and a +field-by-field check of real scanned statements against what +`transform_properties.py` actually loaded turned up three mismatches. This +script fixes them on an existing database; `transform_properties.py` has been +updated in the same commit so a full re-migration produces them directly. + + 1. CLAVE CATASTRAL — printed on both the CESPT water bill ("Cve. Cat.: + KB078025") and the predial statement, and held in `DATMEX.clave` (934 + rows, format `[A-Z]{2}[0-9]{6}`). It was never migrated. What + `PROPERTY_TAX.accountNumber` carries instead is `DATMEX.predial`, a + different, purely numeric column that is *not* unique — 663 distinct + values across 1135 filled rows — and appears on no statement. So predial + is left exactly where it is, and the clave lands on `Property` (it is a + property-level key, which is why two different services both print it). + + 2. GAS — `GAS.meterNumber` is empty for all 334 rows because the transform + put `DATMEX.gas` into `notes`. That column is mixed: 160 rows hold a real + numeric account/meter number, the remaining 174 hold a tank descriptor + ("ESTACIONARIO", "CILINDRO"). The numeric ones are recoverable now; the + descriptors legitimately have no number, so those statements still start + cold and get their number from the first human confirmation. + + 3. TELEPHONE — no such `ServiceKind` existed, so a Telnor bill had nothing to + match against. One service row is created per property with a `phone1`. + Only phone1: 534 properties have one, 18 have a phone2 and exactly 1 has a + phone3, so the secondaries are alternate contacts rather than separately + billed lines. + +Idempotent — re-running updates nothing it has already done, and it never +overwrites a value a human has since corrected. + + ./.venv/bin/python backfill_statement_match_fields.py --env dev +""" + +from __future__ import annotations + +import uuid +from pathlib import Path + +import pandas as pd + +from dbenv import connect +from sync import parse_mode + +STG = Path(__file__).parent / "output" / "stg_utilities" +NULL = "∅" + + +def s(v): + if v is None or pd.isna(v): + return None + v = str(v).strip() + return None if v in ("", NULL, "0") else v + + +def main(): + # `--sync` is accepted and ignored — the script is idempotent either way. + env, _sync_mode = parse_mode() + conn = connect(env) + c = conn.cursor() + print(f"[statement-match-fields] target env: {env}") + + # --- 1. clave catastral -> properties.cadastralKey ---------------------- + # Joined on provenance, the same key transform_properties.py writes, so a + # property that was re-created by a later sync still lines up. + dm = pd.read_parquet(STG / "datmex.parquet") + claves = [] + for _, row in dm.iterrows(): + clave = s(row["clave"]) + if clave: + claves.append((clave, str(int(row["_row_num"])))) + + updated = 0 + for clave, legacy_id in claves: + c.execute( + "UPDATE properties SET cadastralKey = %s " + "WHERE legacySourceTable = 'DATMEX' AND legacyId = %s AND cadastralKey IS NULL", + (clave, legacy_id), + ) + updated += c.rowcount + print(f" cadastralKey: set on {updated} propert(ies) ({len(claves)} in source)") + + # --- 2. gas account numbers out of notes -> GAS.meterNumber ------------- + # REGEXP rather than a Python loop: the value is already sitting in the + # notes column, so this is one pass over 334 rows inside the database. + c.execute( + "UPDATE property_services SET meterNumber = notes " + "WHERE kind = 'GAS' AND meterNumber IS NULL " + "AND notes REGEXP '^[0-9]{5,}$'" + ) + print(f" GAS.meterNumber: recovered {c.rowcount} account number(s) from notes") + + # --- 3. TELEPHONE service rows ------------------------------------------ + # Digits only, matching how the transform now writes them: a scanned Telnor + # bill prints "664 609 3444" and reduces to the stored local 6093444 once + # the LADA is stripped, which is the matcher's job, not this script's. + c.execute( + "SELECT p.id, p.phone1 FROM properties p " + "WHERE p.phone1 IS NOT NULL AND p.phone1 <> '' " + "AND NOT EXISTS (SELECT 1 FROM property_services ps " + " WHERE ps.propertyId = p.id AND ps.kind = 'TELEPHONE')" + ) + rows = c.fetchall() + made = [] + for pid, phone in rows: + digits = "".join(ch for ch in str(phone) if ch.isdigit()) + if digits: + made.append((str(uuid.uuid4()), pid, digits)) + if made: + c.executemany( + "INSERT INTO property_services " + "(id, propertyId, kind, accountNumber, active, notes) " + "VALUES (%s, %s, 'TELEPHONE', %s, 1, 'from DATMEX.telefono')", + made, + ) + print(f" TELEPHONE: created {len(made)} service row(s)") + + conn.commit() + + # --- validation --------------------------------------------------------- + c.execute("SELECT COUNT(*) FROM properties WHERE cadastralKey IS NOT NULL") + n_clave = c.fetchone()[0] + c.execute( + "SELECT COUNT(*) FROM property_services WHERE kind='GAS' AND meterNumber IS NOT NULL" + ) + n_gas = c.fetchone()[0] + c.execute("SELECT COUNT(*) FROM property_services WHERE kind='TELEPHONE'") + n_tel = c.fetchone()[0] + + # A clave that is not unique would silently make the secondary match key + # ambiguous, which is worse than not having one — surface it rather than + # letting the matcher discover it a statement at a time. + c.execute( + "SELECT COUNT(*) FROM (SELECT cadastralKey FROM properties " + "WHERE cadastralKey IS NOT NULL GROUP BY cadastralKey HAVING COUNT(*) > 1) d" + ) + dupe_claves = c.fetchone()[0] + + print("=== Statement match fields ready ===") + print(f" properties with cadastralKey : {n_clave}") + print(f" GAS services with meterNumber: {n_gas}") + print(f" TELEPHONE services : {n_tel}") + print(f" duplicated cadastralKey values: {dupe_claves}" + + (" (matcher treats these as ambiguous)" if dupe_claves else "")) + assert n_clave > 0 and n_tel > 0, "backfill produced nothing — check staging output" + print(" validation: OK") + conn.close() + + +if __name__ == "__main__": + main() diff --git a/migration/run_all.py b/migration/run_all.py index a7443ab..d685626 100644 --- a/migration/run_all.py +++ b/migration/run_all.py @@ -41,6 +41,12 @@ PY = sys.executable # the venv python running this orchestrator STEPS = [ "transform_customers.py", "transform_properties.py", + # Statement-OCR match fields. transform_properties.py now produces these + # directly, so on a full rebuild this is a no-op that re-asserts they are + # there; on a database predating the OCR module it is what fills them in. + # Must follow transform_properties.py, which truncates both tables it + # touches. + "backfill_statement_match_fields.py", "transform_policies.py", "transform_transactions.py", "prune_empty_customers.py", @@ -54,6 +60,12 @@ STEPS = [ SYNC_STEPS = [ "transform_customers.py", "transform_properties.py", + # Statement-OCR match fields. transform_properties.py now produces these + # directly, so on a full rebuild this is a no-op that re-asserts they are + # there; on a database predating the OCR module it is what fills them in. + # Must follow transform_properties.py, which truncates both tables it + # touches. + "backfill_statement_match_fields.py", "transform_policies.py", "transform_transactions.py", # Manual-safe prune: drops legacy-owned empties that the customer upsert diff --git a/migration/transform_properties.py b/migration/transform_properties.py index 476ba9b..c09977c 100644 --- a/migration/transform_properties.py +++ b/migration/transform_properties.py @@ -147,7 +147,7 @@ def main(): props.append(( pid, cust_id, s(row["direccion"]), ", ".join(addr2_parts) or None, s(row["telefono"]), s(row["telefono2"]), s(row["telefono3"]), - s(row["zona"]), "DATMEX", legacy_id, + s(row["zona"]), s(row["clave"]), "DATMEX", legacy_id, )) @@ -178,9 +178,16 @@ def main(): svc("ELECTRIC", account=s(row[rc]), notes=s(row["luz_tipo"]) if rc == "rpu" else None, active=flag("electric")) - # GAS + # GAS — the column mixes two things: an account/meter number for 160 of + # the 334 filled rows, and a tank descriptor ("ESTACIONARIO", + # "CILINDRO") for the rest. Only the numeric form can be matched + # against a scanned gas statement, so it is promoted to meterNumber; + # the descriptor stays a note, as before. if s(row["gas"]) or flag("gas1", False): - svc("GAS", due=s(row["gas_vence"]), notes=s(row["gas"]), active=flag("gas1")) + gas_val = s(row["gas"]) + gas_meter = gas_val if gas_val and gas_val.isdigit() and len(gas_val) >= 5 else None + svc("GAS", meter=gas_meter, due=s(row["gas_vence"]), + notes=s(row["gas"]), active=flag("gas1")) # CABLE if s(row["cable_num"]) or (s_keep0(row["cable_sky"]) or "0") in _TRUE: svc("CABLE", account=s(row["cable_num"]), route=s(row["cia_cable"]), @@ -194,6 +201,19 @@ def main(): if s(row["zfed"]) or flag("federalzone", False): svc("FEDERAL_ZONE", account=s(row["zfed"]), notes=s(row["zfed_t"]), active=flag("federalzone")) + # TELEPHONE — DATMEX never had a phone *service*, only the contact + # numbers unpivoted into Property.phone1/2/3 above, even though the + # legacy ledger billed phone as its own transaction type. A Telnor bill + # can only be matched against a service row, so the primary number + # becomes one. Only phone1: of 1518 properties, 534 have phone1, 18 + # phone2 and exactly 1 phone3 — the secondaries are alternate contacts, + # not additional billed lines. Stored as the bare local number, which is + # how DATMEX holds it and what a printed bill reduces to once the 664 + # Tijuana LADA is stripped. + tel = s(row["telefono"]) + if tel: + svc("TELEPHONE", account="".join(ch for ch in tel if ch.isdigit()) or None, + notes="from DATMEX.telefono") # ALARM if s(row["alarm_system"]): svc("ALARM", notes=s(row["alarm_system"])) @@ -218,7 +238,7 @@ def main(): cur.execute("DELETE ps FROM property_services ps JOIN properties p ON p.id=ps.propertyId WHERE p.legacyId IS NOT NULL") cur.execute("DELETE ta FROM trust_accounts ta JOIN properties p ON p.id=ta.propertyId WHERE p.legacyId IS NOT NULL") cur.executemany( - "INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),archivedAt=NULL", props) + "INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),cadastralKey=VALUES(cadastralKey),archivedAt=NULL", props) delete_missing(cur, "properties", ("legacySourceTable", "legacyId"), prop_keys, "WHERE legacyId IS NOT NULL") cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services) cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts) @@ -227,7 +247,7 @@ def main(): for t in ("property_services", "service_documents", "trust_accounts", "properties"): cur.execute(f"TRUNCATE TABLE {t}") cur.execute("SET FOREIGN_KEY_CHECKS=1") - cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props) + cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props) cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services) cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts) conn.commit() diff --git a/packages/database/prisma/migrations/20260731235721_statement_ocr_intake/migration.sql b/packages/database/prisma/migrations/20260731235721_statement_ocr_intake/migration.sql new file mode 100644 index 0000000..355a3e5 --- /dev/null +++ b/packages/database/prisma/migrations/20260731235721_statement_ocr_intake/migration.sql @@ -0,0 +1,79 @@ +-- AlterTable +ALTER TABLE `properties` ADD COLUMN `cadastralKey` VARCHAR(191) NULL; + +-- AlterTable +ALTER TABLE `property_services` MODIFY `kind` ENUM('WATER', 'ELECTRIC', 'GAS', 'CABLE', 'PROPERTY_TAX', 'FEDERAL_ZONE', 'ALARM', 'TELEPHONE', 'OTHER') NOT NULL; + +-- CreateTable +CREATE TABLE `statement_batches` ( + `id` VARCHAR(191) NOT NULL, + `serviceKind` ENUM('WATER', 'ELECTRIC', 'GAS', 'CABLE', 'PROPERTY_TAX', 'FEDERAL_ZONE', 'ALARM', 'TELEPHONE', 'OTHER') NOT NULL, + `status` ENUM('UPLOADED', 'PROCESSING', 'READY_FOR_REVIEW', 'COMPLETED', 'FAILED') NOT NULL DEFAULT 'UPLOADED', + `uploadedById` VARCHAR(191) NOT NULL, + `label` VARCHAR(191) NULL, + `fileCount` INTEGER NOT NULL DEFAULT 0, + `error` TEXT NULL, + `createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), + `completedAt` DATETIME(3) NULL, + + INDEX `statement_batches_status_createdAt_idx`(`status`, `createdAt`), + PRIMARY KEY (`id`) +) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; + +-- CreateTable +CREATE TABLE `statement_documents` ( + `id` VARCHAR(191) NOT NULL, + `batchId` VARCHAR(191) NOT NULL, + `pageNumber` INTEGER NOT NULL, + `storageKey` VARCHAR(191) NOT NULL, + `status` ENUM('PENDING_OCR', 'OCR_FAILED', 'NEEDS_REVIEW', 'MATCHED', 'CONFIRMED', 'POSTED', 'REJECTED') NOT NULL DEFAULT 'PENDING_OCR', + `ocrRawText` TEXT NULL, + `ocrConfidence` DECIMAL(4, 3) NULL, + `provider` VARCHAR(191) NULL, + `extractedAccountRef` VARCHAR(191) NULL, + `extractedAmount` DECIMAL(12, 2) NULL, + `extractedPeriod` VARCHAR(191) NULL, + `extractedDueDate` DATETIME(3) NULL, + `extractedCadastralKey` VARCHAR(191) NULL, + `matchedPropertyServiceId` VARCHAR(191) NULL, + `matchedCustomerId` VARCHAR(191) NULL, + `matchNote` VARCHAR(191) NULL, + `reviewedById` VARCHAR(191) NULL, + `reviewedAt` DATETIME(3) NULL, + `postedTransactionId` VARCHAR(191) NULL, + `createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), + + UNIQUE INDEX `statement_documents_postedTransactionId_key`(`postedTransactionId`), + INDEX `statement_documents_status_idx`(`status`), + INDEX `statement_documents_matchedCustomerId_idx`(`matchedCustomerId`), + UNIQUE INDEX `statement_documents_batchId_pageNumber_key`(`batchId`, `pageNumber`), + PRIMARY KEY (`id`) +) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; + +-- CreateIndex +CREATE INDEX `properties_cadastralKey_idx` ON `properties`(`cadastralKey`); + +-- CreateIndex +CREATE INDEX `property_services_kind_accountNumber_idx` ON `property_services`(`kind`, `accountNumber`); + +-- CreateIndex +CREATE INDEX `property_services_kind_meterNumber_idx` ON `property_services`(`kind`, `meterNumber`); + +-- AddForeignKey +ALTER TABLE `statement_batches` ADD CONSTRAINT `statement_batches_uploadedById_fkey` FOREIGN KEY (`uploadedById`) REFERENCES `users`(`id`) ON DELETE RESTRICT ON UPDATE CASCADE; + +-- AddForeignKey +ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_batchId_fkey` FOREIGN KEY (`batchId`) REFERENCES `statement_batches`(`id`) ON DELETE CASCADE ON UPDATE CASCADE; + +-- AddForeignKey +ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_matchedPropertyServiceId_fkey` FOREIGN KEY (`matchedPropertyServiceId`) REFERENCES `property_services`(`id`) ON DELETE SET NULL ON UPDATE CASCADE; + +-- AddForeignKey +ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_matchedCustomerId_fkey` FOREIGN KEY (`matchedCustomerId`) REFERENCES `customers`(`id`) ON DELETE SET NULL ON UPDATE CASCADE; + +-- AddForeignKey +ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_reviewedById_fkey` FOREIGN KEY (`reviewedById`) REFERENCES `users`(`id`) ON DELETE SET NULL ON UPDATE CASCADE; + +-- AddForeignKey +ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_postedTransactionId_fkey` FOREIGN KEY (`postedTransactionId`) REFERENCES `transactions`(`id`) ON DELETE SET NULL ON UPDATE CASCADE; + diff --git a/packages/database/prisma/schema.prisma b/packages/database/prisma/schema.prisma index 9d85813..2c189f7 100644 --- a/packages/database/prisma/schema.prisma +++ b/packages/database/prisma/schema.prisma @@ -53,6 +53,12 @@ enum ServiceKind { PROPERTY_TAX FEDERAL_ZONE ALARM + /// Telephone was never unpivoted out of DATMEX — the numbers sat on + /// `Property.phone1/2/3` as contact fields even though the legacy ledger + /// billed phone as its own transaction type. OCR matching needs a real + /// service row to match a Telnor bill against, so it becomes one; see + /// `migration/backfill_statement_match_fields.py`. + TELEPHONE OTHER } @@ -115,6 +121,8 @@ model Customer { vehicles Vehicle[] transactions Transaction[] + statementDocuments StatementDocument[] + @@map("customers") } @@ -373,6 +381,14 @@ model Property { phone2 String? phone3 String? zone String? + /// Clave catastral (DATMEX.clave) — the cadastral key, format `KA903009`. + /// Property-level, not per-service: it is printed on both the CESPT water + /// bill and the predial statement, which is exactly why it is a useful + /// secondary match key when a bill's account number does not OCR cleanly. + /// Distinct from the numeric DATMEX.predial that `PROPERTY_TAX.accountNumber` + /// carries — that column is not unique (663 distinct across 1135 rows) and + /// is not what any statement prints. + cadastralKey String? // Soft-delete marker (see Customer.archivedAt). archivedAt DateTime? legacySourceTable String? @@ -383,6 +399,8 @@ model Property { documents ServiceDocument[] trustAccount TrustAccount? + @@index([cadastralKey]) + @@unique([legacySourceTable, legacyId]) @@map("properties") } @@ -401,6 +419,13 @@ model PropertyService { active Boolean @default(true) notes String? @db.Text + statementDocuments StatementDocument[] + + // The OCR matcher looks a service up by (kind, accountNumber) — always + // scoped to one kind, never fuzzily across every identifier column, so a + // water account number cannot collide with an unrelated phone number. + @@index([kind, accountNumber]) + @@index([kind, meterNumber]) @@map("property_services") } @@ -415,6 +440,120 @@ model ServiceDocument { @@map("service_documents") } +// --------------------------------------------------------------------------- +// Statement OCR intake (RECEIPT_CAPTURE_SPEC §2) +// +// Each utility company mails 300+ paper statements a month, one per customer, +// which staff key in by hand. These two tables are the intake side of removing +// that: a batch of scanned PDFs is split per page, OCR'd, matched to a +// PropertyService by its scoped account number, and queued for review. Nothing +// here writes to the ledger — confirming a document posts it through +// `BillingService.createBatch`, the same path hand-keyed batches take. +// +// Everything ingested is a CHARGE (a bill awaiting payment), never a proof of +// payment: the office scans what it must pay, and settles it by check through +// the existing capture flow. +// --------------------------------------------------------------------------- + +enum StatementBatchStatus { + UPLOADED + PROCESSING + READY_FOR_REVIEW + COMPLETED + FAILED +} + +enum StatementDocumentStatus { + PENDING_OCR + OCR_FAILED + /// No confident match, or the extraction itself was low-confidence. + NEEDS_REVIEW + /// Confident auto-match, awaiting a human confirm. + MATCHED + /// Staff confirmed; not yet posted. + CONFIRMED + POSTED + /// Duplicate, unreadable, or wrong batch. + REJECTED +} + +/// One upload session — e.g. "October CFE statements". +model StatementBatch { + id String @id @default(uuid()) + /// What kind of service every statement in this batch bills. The parser + /// still detects the provider per page and flags any page that disagrees, + /// rather than trusting the uploader's label. + serviceKind ServiceKind + status StatementBatchStatus @default(UPLOADED) + uploadedById String + uploadedBy User @relation("StatementBatchUploader", fields: [uploadedById], references: [id]) + label String? + fileCount Int @default(0) + /// Set when the pipeline fails as a whole (bad PDF, OCR binaries missing). + error String? @db.Text + createdAt DateTime @default(now()) + completedAt DateTime? + + documents StatementDocument[] + + @@index([status, createdAt]) + @@map("statement_batches") +} + +/// One statement — one customer, one period — after splitting the batch. +model StatementDocument { + id String @id @default(uuid()) + batchId String + batch StatementBatch @relation(fields: [batchId], references: [id], onDelete: Cascade) + /// 1-based page of the source PDF this was split from. + pageNumber Int + /// The rendered page image in object storage. The source PDF is kept too, so + /// a reviewer can always see exactly what the parser read. + storageKey String + status StatementDocumentStatus @default(PENDING_OCR) + + /// Raw OCR text, kept even after a manual correction so a mismatch between + /// what the machine read and what staff entered stays auditable. + ocrRawText String? @db.Text + /// Mean per-word confidence reported by the OCR engine, 0..1. + ocrConfidence Decimal? @db.Decimal(4, 3) + /// Which parser claimed the page ("CFE", "CESPT", "TELNOR"). + provider String? + + // Extracted, then staff-corrected in place. `extractedAccountRef` is already + // normalised for matching (CFE leading zeros stripped, Telnor LADA removed). + extractedAccountRef String? + extractedAmount Decimal? @db.Decimal(12, 2) + extractedPeriod String? + extractedDueDate DateTime? + /// Clave catastral when the statement prints one — a second key to match on + /// when the account number is unreadable. + extractedCadastralKey String? + + matchedPropertyServiceId String? + matchedPropertyService PropertyService? @relation(fields: [matchedPropertyServiceId], references: [id]) + matchedCustomerId String? + matchedCustomer Customer? @relation(fields: [matchedCustomerId], references: [id]) + /// Why this landed where it did — "exact account match", "no candidate", + /// "2 candidates". Shown in the review queue so staff can trust or distrust + /// the suggestion without opening the image. + matchNote String? + + reviewedById String? + reviewedBy User? @relation("StatementDocumentReviewer", fields: [reviewedById], references: [id]) + reviewedAt DateTime? + + postedTransactionId String? @unique + postedTransaction Transaction? @relation(fields: [postedTransactionId], references: [id]) + + createdAt DateTime @default(now()) + + @@unique([batchId, pageNumber]) + @@index([status]) + @@index([matchedCustomerId]) + @@map("statement_documents") +} + /// From TRUSTVENCE. model TrustAccount { id String @id @default(uuid()) @@ -483,6 +622,9 @@ model Transaction { legacyId String? createdAt DateTime @default(now()) + /// Set only on OCR-posted rows — the statement page this came from. + statementDocument StatementDocument? + @@index([customerId, transactionDate]) // By-check reconciliation (billing.byCheck / the cheque-count report) looks // rows up by check number alone — the legacy EDITA CHEQUE COUNT lookup. @@ -600,6 +742,9 @@ model User { updatedAt DateTime @updatedAt activityLogs ActivityLog[] + statementBatches StatementBatch[] @relation("StatementBatchUploader") + statementsReviewed StatementDocument[] @relation("StatementDocumentReviewer") + @@map("users") }