diff --git a/PLAN.md b/PLAN.md index 03fdc0c..7a49db9 100644 --- a/PLAN.md +++ b/PLAN.md @@ -138,6 +138,9 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult **Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**. **The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`. **Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost. + - **Policy OCR capture — DONE** (2026-08-01), **unplanned — it came out of building the bullet above.** Full write-up in [`docs/POLICY_OCR.md`](docs/POLICY_OCR.md). Once the receipt pipeline existed it was obvious the same render→OCR→parse→match→review shape fits the *other* stack of paper this office keys in by hand: the carrier policy PDFs behind every `Policy` row. Built in `apps/api/src/policy-ocr/` with a GMX parser, `policy_ocr_batches`/`policy_ocr_documents`, and abilities `policy:ingest`/`policy:ocr-review` (STAFF, same trust tier and same reason). Web: `/polizas/captura` is the "automática" tab of the policy-creation screen (`/polizas/nuevo` is the manual one, both render `PolicyCaptura.tsx`) with the review queue at `/polizas/captura/[id]`. The `OcrProvider` seam was **extracted out of `StatementsModule` into its own `OcrModule`** to make this possible — that was blocking, not cosmetic; `StatementsModule` now imports it and binds nothing. + **The statement pipeline's core assumption inverts here.** Utility statements arrive bundled *one customer per page*, so there a page is a document; a GMX certificate is one policy across two pages (header on 1, coverage table on 2), so the pipeline concatenates the pages and runs the parser and matcher **once per file**. `PolicyOcrDocument.pageNumber` is therefore the file ordinal in the batch, and `storageKey` points at the **source PDF** (the review screen embeds the exact artifact the office received) rather than at a page image. Matching is on `Policy.policyNumber` alone and never the printed insured name — the same registrant-vs-owner drift that rules names out on the utility side. Zero hits means a new policy and confirm creates it; more than one is surfaced, never auto-picked. + **The GMX certificate carries no premium at all** — the figure lives on a separate `recibo` PDF — so the premium fields stay null with a note saying why, confirm never overwrites an existing premium with null, and the optional ledger write is gated on staff ticking `postPremium` *and* a premium actually parsing. 8/8 parser tests against one real document (`HC_Folio_000767_Traduccion.pdf`). GMX is the only carrier implemented; the dispatcher is a pattern table, so a second one is a parser function and two entries. - **Multi-bank chequera — DONE** (2026-07-27). `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. `bank_transactions` gained a **required** `bankAccountId` (plus an `(bankAccountId, transactionDate)` index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by `migration/backfill_bank_accounts.py` — a standalone step because `prisma db push` cannot add a required column to a populated table. It is idempotent and now runs inside `run_all.py` (both normal and `--sync`) ahead of `transform_bank.py`, which fails fast if the account is missing. Every read path in `bank.service.ts` is account-scoped, including `facets()` (which had no filter at all) and *both* raw-SQL rollups in `summary()`. API: `?bankAccountId=` is required on `list`/`stats`/`facets`/`summary` — **not** optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new `bank/accounts` + `bank/banks` sub-resource under a MANAGER `bank:manage-accounts` ability. Web: `/banco` gained an account picker (remembered per browser) and reads every figure in the selected account's currency, `/banco/cuentas` manages banks and accounts, and `/inicio`'s chequera card names the account it is showing instead of implying one register. An account's `currency` is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged. - **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s). @@ -165,6 +168,8 @@ Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a r **Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown. +**It also produced a feature nobody planned.** The statement OCR pipeline generalised: the same render→OCR→parse→match→review shape reads **carrier policy PDFs** into `Policy` rows, which is `docs/POLICY_OCR.md` (built 2026-08-01, GMX only so far). It belongs to step 12's subject matter but to step 11's lineage, and it is in no spec — worth knowing before reading `INSURANCE_FEATURES_SPEC.md`, which does not mention it. It also partly overlaps what §4's carrier API was wanted for, and unlike that section it is not blocked on a phone call. + **Step 12 is one-quarter built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. **§1 renewal emails is done** (2026-08-01/02) and carries a BUILT note recording the three places the build diverged from the spec; §2 liquidación is still the smallest remaining piece, since the per-policy fields are already wired end to end. **Notifications are one screen, not two features.** The four legacy mass-email jobs (`docs/MASS_EMAIL_NOTIFICATIONS.md`) and the insurance renewal avisos both mean "tell a customer something by email", so they are tabs of `/notificaciones` over one `email_notification_log`, with one shared flags panel and one schedule editor. `app_settings` + `SettingsService` (db → env → default) is the operator-config seam they introduced: summary recipients and both sweep cadences live there, so changing any of them is a save, not a redeploy. Credentials stay in the environment. diff --git a/README.md b/README.md index 0bed228..1ce07c9 100644 --- a/README.md +++ b/README.md @@ -43,11 +43,19 @@ API feature modules: `auth`, `users`, `customers`, `policies`, `properties`, `billing`, `bank`, `reports`, `notifications`, `renewals`, `mail`, `statements`, `policy-ocr`, `ocr`, `storage`, `settings`, `ops`. -Web routes: `/inicio`, `/clientes`, `/polizas`, `/servicios`, `/estado-cuenta`, -`/banco` (chequera), `/recibos` (OCR capture), `/notificaciones` (mass email + +Web routes: `/inicio`, `/clientes`, `/polizas` (+ `/polizas/captura`, policy +PDF OCR capture), `/servicios`, `/estado-cuenta`, `/banco` (chequera), +`/recibos` (utility statement OCR capture), `/notificaciones` (mass email + renewal avisos; `/renovaciones` is an alias onto its Pólizas tab), `/reportes`, `/catalogos`, `/operaciones` (DB ingest/backup, ADMIN), `/usuarios`, `/login`. +Two OCR intakes share one `OcrProvider` seam (`src/ocr/`, Tesseract today): +utility statements → ledger rows ([`docs/RECEIPT_CAPTURE_SPEC.md`](docs/RECEIPT_CAPTURE_SPEC.md) §2) +and carrier policy PDFs → `Policy` rows ([`docs/POLICY_OCR.md`](docs/POLICY_OCR.md)). +Both need `tesseract-ocr`, `tesseract-ocr-data-spa`, `poppler-utils` and object +storage; each reports its own availability and disables only itself if either +is missing. + --- ## Prerequisites diff --git a/RESUME.md b/RESUME.md index aec3cc4..e31ba59 100644 --- a/RESUME.md +++ b/RESUME.md @@ -527,6 +527,82 @@ Implementation notes worth keeping: (`$268`, what is paid at the window — what the parser uses today) or the exact breakdown total (`$268.88`). One question for Jorge. +## Policy OCR capture (`/polizas/captura`) — DONE 2026-08-01, unplanned + +**This feature was not in any spec.** It is what the statement OCR work above +turned into once the pipeline existed. Having built render → OCR → parse → +match → review for CFE/CESPT/Telnor receipts, the same shape obviously fits +the *other* stack of paper this office keys in by hand every week: the carrier +policy PDFs behind every `Policy` row. Full write-up in `docs/POLICY_OCR.md`. + +The pipeline was reused rather than copied. `OcrModule` was **extracted out of +`StatementsModule`** in the same commit so `PolicyOcrModule` could inject +`OCR_PROVIDER` without taking on the statement pipeline — that extraction was +blocking, not tidying; the policy module could not resolve the provider at all +until it existed. `StatementsModule` imports it now and binds nothing itself, +so the Tesseract-vs-managed-API decision stays one line in one file for both +features. + +Screens mirror Captura exactly: `/polizas/nuevo` is the manual tab, +`/polizas/captura` the automática one, both rendering `PolicyCaptura.tsx`, with +the batch review queue at `/polizas/captura/[id]`. Abilities `policy:ingest` / +`policy:ocr-review`, both STAFF — same trust tier as statement OCR, and for the +same reason: nothing reaches the books unconfirmed. + +**The statement pipeline's central assumption inverts here, and that is the +thing to remember.** Utility statements arrive bundled *one customer per page*, +so there a page is a document and the parser runs per page. A policy PDF is the +opposite: the GMX certificate is one policy spread across two pages (contract +header on page 1, the per-coverage table on page 2). So every page's text is +concatenated and the parser and matcher run **once per file**. Consequences: +`PolicyOcrDocument.pageNumber` is repurposed as the file ordinal within the +batch (the `(batchId, pageNumber)` unique constraint still holds), `ocrConfidence` +is the mean across the file's pages, and a file that fails to parse yields +exactly one `OCR_FAILED` row. + +`storageKey` points at the **source PDF**, not a rendered page image, so the +review screen embeds the exact artifact the office received and gets the +browser's native PDF scrolling, zoom and text selection for free. The page PNGs +are still written for future re-OCR, but nothing treats them as the document's +identity. (The statement side is the reverse, because there a page *is* the +document.) + +Findings worth keeping: + +- **The GMX certificate has no premium on it at all.** Not intermittently + missing — the figure lives on GMX's separate `recibo` PDF. The parser leaves + the premium fields null and pushes a note saying so, confirm never overwrites + an existing `Policy.netPremium` with null, and the optional ledger write is + gated on staff ticking `postPremium` *and* a premium actually parsing. + Without that second gate a premium-less certificate would book a $0 charge on + every confirm. +- **Match on `Policy.policyNumber`, never the printed insured name.** Same + registrant-vs-current-owner drift that rules names out on the utility side. + Zero hits means a new policy and confirm creates the row under a picked + customer; more than one hit is surfaced for a human, never auto-picked — + duplicate numbers across related parties do occur. +- Deductible and loss participation are stored as **strings** (`"5%"`, + `"USD 1,000"`): they are printed as a mix of percentages, amounts and free + text, and normalising them would lose the distinction. +- Carrier-portal PDFs are usually **born-digital**, so the text layer wins and + no OCR runs at all most of the time — same precedence rule as the statement + pipeline. +- The digit-confusion map and the amount-by-separator-position parser are + **duplicated on purpose** rather than imported, to keep the module + self-contained. Fix a bug in one, check the other. + +8/8 parser tests, all against verbatim text from one real document +(`HC_Folio_000767_Traduccion.pdf`). + +**Open:** GMX is the only carrier implemented — the dispatcher is a +`[provider, pattern]` table plus a parser map, so a second carrier is a +function and two entries, but no other layout has been seen. Reading the +premium off the separate `recibo` PDF and pairing it to its certificate is the +obvious next piece; it is what would let `postPremium` stop being a manual +tick. And nothing versions a re-issued policy — confirm updates the existing +row, so there is no record that this is the 2027 issue of that number. + + ## Notificaciones (`/notificaciones`) — DONE 2026-08-01 → 08-02 Two features that were spec'd separately turned out to be one screen. The four diff --git a/docs/INSURANCE_FEATURES_SPEC.md b/docs/INSURANCE_FEATURES_SPEC.md index 6eb2c31..0c49d8e 100644 --- a/docs/INSURANCE_FEATURES_SPEC.md +++ b/docs/INSURANCE_FEATURES_SPEC.md @@ -11,6 +11,15 @@ Companion doc: [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) covers the Utility Management half of the same meeting (PLAN.md step 11). This doc is the insurance half (PLAN.md step 12). +> **A fifth insurance feature exists that this spec never proposed.** +> [`POLICY_OCR.md`](POLICY_OCR.md) — OCR capture of carrier policy PDFs into +> `Policy` rows, built 2026-08-01. It came out of the *utility* statement OCR +> work in [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2, not from +> this meeting, which is why it is documented on its own rather than folded in +> here. It is relevant to §4: an OCR path that turns a carrier PDF into a +> `Policy` row already covers part of what a carrier API was wanted for, and +> unlike the API it is not blocked on Grupo Valore returning a phone call. + ## Why these four features are one spec The meeting produced four insurance asks. They are specified together because diff --git a/docs/POLICY_OCR.md b/docs/POLICY_OCR.md new file mode 100644 index 0000000..175886e --- /dev/null +++ b/docs/POLICY_OCR.md @@ -0,0 +1,247 @@ +# Insurance Policy OCR Capture + +Reads an insurance policy PDF the office downloads from a carrier portal, +proposes the `Policy` row it should become, and lets staff confirm. Built +2026-08-01 (`5e9cb12`), live under `/polizas/captura`. + +## Why this exists — it was not planned + +This feature is **not in any spec**. It came out of building the utility +statement OCR intake in [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) +§2: once there was a working render → OCR → parse → match → review pipeline +for CFE/CESPT/Telnor receipts, it was obvious the same shape applies to the +*other* stack of paper this office keys in by hand every week — the carrier +policy PDFs behind every `Policy` row. + +The two are the same job with a different document on the scanner. Keeping +that recognition cheap is the whole point of how it was built: the pipeline +was **reused, not copied**. + +- `OcrModule` (`apps/api/src/ocr/ocr.module.ts`) was extracted out of + `StatementsModule` in this same commit, purely so `PolicyOcrModule` can + inject `OCR_PROVIDER` without dragging in the statement pipeline. + `StatementsModule` now imports it and binds nothing itself. That extraction + was **blocking**: without it the policy module could not resolve the + provider at all. +- The engine stays Tesseract behind the same swappable seam, so a managed + extraction API remains a one-line change in one file for both features. +- The intake screen is a *mode of* the existing policy-creation screen, the + same way OCR receipt capture is a mode of Captura — not a new menu entry. + +## What ships + +| Piece | Path | +|---|---| +| API module | `apps/api/src/policy-ocr/` (service, controller, DTOs, matcher, parser) | +| Shared OCR seam | `apps/api/src/ocr/ocr.module.ts` | +| Tables | `policy_ocr_batches`, `policy_ocr_documents` (`20260801000000_policy_ocr_intake`) | +| Web | `components/PolicyCaptura.tsx` (tab shell), `PolicyOcrIntake.tsx` (upload), `PolicyOcrReview.tsx` (review queue) | +| Abilities | `policy:ingest`, `policy:ocr-review` — both **STAFF** | + +Abilities are STAFF for the same reason statement OCR is: nothing reaches the +books unconfirmed, and the review step is what makes machine capture safe at +that tier. + +## The screen + +`PolicyCaptura` is one screen with two ways in, mirroring `Captura.tsx`: + +- `/polizas/nuevo` → **manual** tab (`PolicyForm`, every field by hand) +- `/polizas/captura` → **automática** tab (`PolicyOcrIntake`, drop a PDF) +- `/polizas/captura/[id]` → the batch review queue + +Both modes end at the same place — a `Policy` row on a customer's file — so +they are modes of one screen rather than two menu entries. Either URL renders +the same component, so the tab toggle works from either entry point and old +bookmarks land on the right tab. + +## Pipeline + +``` +upload PDF → store source → render pages → text layer? → parse → match → review → confirm +``` + +1. **Store the source.** `policy-ocr/{batchId}/source-N.pdf`, before anything + else touches it. +2. **Render + read.** Every page is rendered to + `policy-ocr/{batchId}/page-M.png`. Text-layer wins when the PDF has one + (cheap, exact); the rendered image is OCR'd only when it does not — the + same precedence rule as the statement pipeline. Carrier-portal PDFs are + usually born-digital, so most of the time no OCR runs at all. +3. **Parse.** Provider detected by brand signal first + (`GMX`, `Grupo Mexicano de Seguros`, `gmx.com.mx`, + `JUNTOS EL RIESGO ES MENOR`), layout patterns only as fallback — the same + ordering rule the statement parser needed. +4. **Match.** Against `Policy.policyNumber`. +5. **Review + confirm.** Nothing is written to `Policy` until a human + confirms. + +### One PDF = one policy + +This is the sharpest difference from statement OCR, and it inverts that +feature's core assumption. + +Utility statements arrive **bundled, one customer per page** — so there, one +page is one document and the parser runs per page. A policy PDF is the +opposite: the GMX certificate is a 2-page document where page 1 carries the +contract header and page 2 carries the per-coverage table, and **both pages +describe the same policy**. So the pipeline concatenates every page's text +(`\n\n` between pages, which also keeps `ocrRawText` readable for debugging) +and runs the parser and the matcher exactly **once per file**. + +Consequences worth knowing before touching this code: + +- `PolicyOcrDocument.pageNumber` is repurposed as the **file ordinal within + the batch** (1, 2, 3…), not a page index. The + `(batchId, pageNumber)` unique constraint still holds, and one batch still + carries many policies — one per uploaded file. +- Parser regexes are anchored across the whole concatenated text (`^From$`, + `^Currency\s+…`), which is why the page-boundary blank line matters. +- `ocrConfidence` on the row is the **mean** across the file's pages. +- A file that fails to parse produces exactly one `OCR_FAILED` row — the right + granularity, and the page PNGs stay on disk for a re-run after a parser fix. + +### `storageKey` is the source PDF, not a page image + +`PolicyOcrDocument.storageKey` points at `source-N.pdf`. The review screen +embeds that file directly, so the reviewer looks at the **exact artifact the +office received** and gets the browser's native PDF scrolling, zoom and text +selection for free. Rendered PNGs are still written for future re-OCR or an +image-based audit, but nothing points at them as the document's identity. + +(The statement side does the opposite — there `storageKey` is the page image, +because a page *is* the document.) + +## Matching: policy number only, never the insured name + +`PolicyMatcherService` matches on `Policy.policyNumber` and nothing else. + +The certificate's "Insured" line is the account's registrant, which drifts +from the customer the office actually holds the file under — the same finding +the statement matcher is built around (a CESPT receipt reading +`ARNAIZ ROSAS ELSA AURORA` for a customer this office holds as `CATT, RANDY`). +Names are shown to the reviewer as a sanity check and never feed matching. + +| Rows on `policyNumber` | Result | +|---|---| +| exactly 1 | `MATCHED`, confident — the only unambiguous hit | +| 0 | new policy: review offers a customer picker, confirm **creates** the row | +| >1 | surfaced as candidates, human picks | + +More than one hit is never auto-resolved. Duplicate policy numbers across +customers do occur (one group policy bound by two related parties), and +picking arbitrarily would silently book the wrong coverage against the wrong +person. + +## What the parser reads, and the field it cannot + +`ParsedPolicy` fields are all nullable on purpose: each carrier prints a +different subset, and the matcher and review queue both work better with +"field was read" vs "field was not" than with a guess. + +Read from the GMX certificate: policy number, insured name, additional +insured, broker (→ `Policy.agentName`), legal address, ZIP, `policyFrom` / +`policyTo` / `policyDate`, currency, premium-payment cadence, and the full +per-coverage table (risk, insured amount, deductible, loss participation) +preserved verbatim. + +> **The GMX certificate carries no premium.** Not "sometimes missing" — the +> document does not have the figure. It lives on GMX's **separate `recibo` +> PDF**. The parser leaves `netPremium` / `policyFee` / `brokerFee` / `total` +> null and pushes a note onto the row — +> *"esta página no trae prima; revisar el recibo de GMX por separado"* — so +> the reviewer sees why the field is empty rather than assuming a read +> failure. + +This is also why confirm never overwrites an existing `Policy.netPremium` +with null: the certificate not carrying a premium is not evidence that the +premium is gone. + +Deductible and loss participation are stored as **strings** (`"5%"`, `"20%"`, +`"USD 1,000"`) — they are printed as a mix of percentages, currency amounts +and free text, and normalising them would lose the distinction. + +## Confirm: what actually gets written + +Per confirmed document, in order: + +1. **The `Policy` row** — updated if a policy was matched, created under the + picked customer if not. Only non-null `extracted*` fields are written; null + never overwrites existing data. +2. **A `PolicyDocument`** — the source PDF is streamed into the policy's + storage namespace and attached, so the paperwork stays with the policy. +3. **Optionally a `Transaction`** — `INSURANCE` domain, negative amount + (a charge), `captureSource: "OCR"`, `captureRef` = the document id. + +The ledger write is **opt-in twice over**: staff must tick `postPremium` +*and* a premium must have parsed to a positive number. Without that gate the +premium-less certificate above would silently book a $0 charge on every +confirm. + +`createdPolicyId` and `postedTransactionId` are unique columns on the +document row, so a double-confirm cannot re-apply — and a `POSTED` document +is refused outright. + +Discarding a batch is refused once any page is `POSTED`: a partly-applied +batch has already written `Policy` (and possibly `Transaction`) rows, and +hiding the paperwork behind a "discarded" label would leave those rows +unexplained. Reject the remaining pages individually instead. + +## API surface + +| Method | Route | Ability | +|---|---|---| +| `GET` | `/policy-ocr/status` (is OCR + storage available) | authenticated | +| `GET` | `/policy-ocr/batches`, `/batches/:id`, `/batches/:id/documents` | authenticated | +| `GET` | `/policy-ocr/documents/:id/page` (streams the source PDF) | authenticated | +| `POST` | `/policy-ocr/batches` (upload) | `policy:ingest` | +| `PATCH` | `/policy-ocr/documents/:id` (edit the extracted fields) | `policy:ocr-review` | +| `POST` | `/policy-ocr/documents/:id/reject` | `policy:ocr-review` | +| `POST` | `/policy-ocr/batches/:id/discard` | `policy:ocr-review` | +| `POST` | `/policy-ocr/batches/:id/confirm` | `policy:ocr-review` | + +## Requirements + +Same as statement OCR: object storage (`S3_ENDPOINT` + credentials) for the +source PDFs and page images, and `tesseract-ocr` / `tesseract-ocr-data-spa` / +`poppler-utils` in the API image. `GET /policy-ocr/status` reports both; if +either is missing the feature reports itself unavailable and only this +feature is disabled. + +## Tests + +`apps/api/src/policy-ocr/parsers/policy-parser.spec.ts` — 8 cases, all +against verbatim text extracted from one real document, +`HC_Folio_000767_Traduccion.pdf`: provider detection from the wordmark and +from the footer URL, the header fields, every coverage row off the second +page, the deductible/loss-participation strings, the missing-premium note, +the broker line with the agent-number parens absent, and a page with no GMX +signal at all (which must yield no provider rather than a bad guess). + +## Not built + +- **Only GMX.** The dispatcher (`detectPolicyProvider`) is a table of + `[provider, pattern]` pairs plus a `parsers` map, so adding ANA or Qualitas + is a parser function and two entries — but no other carrier's layout has + been seen yet, and guessing at one produces a parser nobody can verify. +- **The `recibo` PDF.** Reading the premium off GMX's separate receipt + document, and pairing it to the certificate it belongs to, is the obvious + next piece. It is what would let `postPremium` stop being a manual tick. +- **Renewals from OCR.** A re-issued policy arrives as a new certificate with + the same number; confirm updates the existing row rather than versioning + it. Nothing tracks "this is the 2027 issue of that policy". + +## Related + +- [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2 — the utility + statement pipeline this was lifted from, and the origin of three rules the + policy parser applies: detect the provider by brand before layout, only + ever apply the Tesseract digit-confusion map (`O→0`, `S→5`, `B→8`, …) to + fields known to be digits, and parse amounts by separator *position* rather + than assuming `,` is thousands. + + Those last two are **duplicated on purpose**, not imported: the module is + kept self-contained, since sharing a helper would couple two unrelated + domains through it. If you fix a bug in one, check the other. +- [`INSURANCE_FEATURES_SPEC.md`](INSURANCE_FEATURES_SPEC.md) — the four + insurance features that *were* planned. This is not one of them. diff --git a/docs/RECEIPT_CAPTURE_SPEC.md b/docs/RECEIPT_CAPTURE_SPEC.md index f8bc1b3..0da9f0c 100644 --- a/docs/RECEIPT_CAPTURE_SPEC.md +++ b/docs/RECEIPT_CAPTURE_SPEC.md @@ -151,6 +151,21 @@ single-movement form. > blobs. `GET /statements/status` reports `ocrAvailable` and `storageAvailable`, > and the upload card hides itself unless both hold. > +> **This pipeline turned out to generalise, and a second feature came out of +> it.** Once render → OCR → parse → match → review existed for utility +> receipts, the same shape obviously fit the *other* stack of paper this +> office keys in by hand — carrier policy PDFs. That is +> [`POLICY_OCR.md`](POLICY_OCR.md), built 2026-08-01, and it is **not in any +> spec**; it was a revelation from doing this one. The `OcrProvider` seam was +> lifted out of `StatementsModule` into its own `OcrModule` so the policy +> module could inject it without taking on the statement pipeline — +> `StatementsModule` imports it now and binds nothing itself. The engine +> choice stays a one-line change in one file, for both features. +> +> One assumption does **not** carry over: statements arrive bundled *one +> customer per page*, so here a page is a document. A policy PDF is one +> document across several pages. See that doc's "One PDF = one policy". +> > **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor > bills) drove every decision below. Against them the shipped parser identifies > the provider on **46/46**, reads an account reference on **43/46**, an amount