rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document policy OCR capture, the feature no spec proposed

Browse Source 
Policy OCR shipped 2026-08-01 (5e9cb12) and was documented nowhere. It is
not in INSURANCE_FEATURES_SPEC.md because it did not come from that
meeting — it came out of building the utility statement OCR pipeline in
RECEIPT_CAPTURE_SPEC.md §2 and noticing the same shape fits carrier
policy PDFs. A reader had no way to find that lineage.

New docs/POLICY_OCR.md covers it end to end, with weight on the three
things that are not obvious from the statement side:

- **One PDF = one policy.** Statements arrive bundled one customer per
  page, so there a page is a document. A GMX certificate is one policy
  across two pages, so the pages are concatenated and the parser runs
  once per file — which is why `pageNumber` is a file ordinal and
  `storageKey` is the source PDF, not a page image.
- **The GMX certificate carries no premium at all** — it lives on a
  separate recibo PDF. Hence the null-preserving confirm and the
  double-gated ledger write.
- **OcrModule was extracted out of StatementsModule to make this
  possible**, and that was blocking rather than cosmetic.

Cross-referenced from RECEIPT_CAPTURE_SPEC.md §2 (where it came from),
INSURANCE_FEATURES_SPEC.md (which never proposed it, and whose §4 carrier
API it partly overlaps), PLAN.md step 11, README and RESUME.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-08-02 12:56:25 -07:00
co-authored by Claude Opus 5
parent 6331481f82
commit 872a661051
6 changed files with 362 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/POLICY_OCR.md
+247
View File
@@ -0,0 +1,247 @@
# Insurance Policy OCR Capture
Reads an insurance policy PDF the office downloads from a carrier portal,
proposes the `Policy` row it should become, and lets staff confirm. Built
2026-08-01 (`5e9cb12`), live under `/polizas/captura`.
## Why this exists — it was not planned
This feature is **not in any spec**. It came out of building the utility
statement OCR intake in [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md)
§2: once there was a working render → OCR → parse → match → review pipeline
for CFE/CESPT/Telnor receipts, it was obvious the same shape applies to the
*other* stack of paper this office keys in by hand every week — the carrier
policy PDFs behind every `Policy` row.
The two are the same job with a different document on the scanner. Keeping
that recognition cheap is the whole point of how it was built: the pipeline
was **reused, not copied**.
- `OcrModule` (`apps/api/src/ocr/ocr.module.ts`) was extracted out of
`StatementsModule` in this same commit, purely so `PolicyOcrModule` can
inject `OCR_PROVIDER` without dragging in the statement pipeline.
`StatementsModule` now imports it and binds nothing itself. That extraction
was **blocking**: without it the policy module could not resolve the
provider at all.
- The engine stays Tesseract behind the same swappable seam, so a managed
extraction API remains a one-line change in one file for both features.
- The intake screen is a *mode of* the existing policy-creation screen, the
same way OCR receipt capture is a mode of Captura — not a new menu entry.
## What ships
| Piece | Path |
|---|---|
| API module | `apps/api/src/policy-ocr/` (service, controller, DTOs, matcher, parser) |
| Shared OCR seam | `apps/api/src/ocr/ocr.module.ts` |
| Tables | `policy_ocr_batches`, `policy_ocr_documents` (`20260801000000_policy_ocr_intake`) |
| Web | `components/PolicyCaptura.tsx` (tab shell), `PolicyOcrIntake.tsx` (upload), `PolicyOcrReview.tsx` (review queue) |
| Abilities | `policy:ingest`, `policy:ocr-review` — both **STAFF** |
Abilities are STAFF for the same reason statement OCR is: nothing reaches the
books unconfirmed, and the review step is what makes machine capture safe at
that tier.
## The screen
`PolicyCaptura` is one screen with two ways in, mirroring `Captura.tsx`:
- `/polizas/nuevo` → **manual** tab (`PolicyForm`, every field by hand)
- `/polizas/captura` → **automática** tab (`PolicyOcrIntake`, drop a PDF)
- `/polizas/captura/[id]` → the batch review queue
Both modes end at the same place — a `Policy` row on a customer's file — so
they are modes of one screen rather than two menu entries. Either URL renders
the same component, so the tab toggle works from either entry point and old
bookmarks land on the right tab.
## Pipeline
```
upload PDF → store source → render pages → text layer? → parse → match → review → confirm
```
1. **Store the source.** `policy-ocr/{batchId}/source-N.pdf`, before anything
else touches it.
2. **Render + read.** Every page is rendered to
`policy-ocr/{batchId}/page-M.png`. Text-layer wins when the PDF has one
(cheap, exact); the rendered image is OCR'd only when it does not — the
same precedence rule as the statement pipeline. Carrier-portal PDFs are
usually born-digital, so most of the time no OCR runs at all.
3. **Parse.** Provider detected by brand signal first
(`GMX`, `Grupo Mexicano de Seguros`, `gmx.com.mx`,
`JUNTOS EL RIESGO ES MENOR`), layout patterns only as fallback — the same
ordering rule the statement parser needed.
4. **Match.** Against `Policy.policyNumber`.
5. **Review + confirm.** Nothing is written to `Policy` until a human
confirms.
### One PDF = one policy
This is the sharpest difference from statement OCR, and it inverts that
feature's core assumption.
Utility statements arrive **bundled, one customer per page** — so there, one
page is one document and the parser runs per page. A policy PDF is the
opposite: the GMX certificate is a 2-page document where page 1 carries the
contract header and page 2 carries the per-coverage table, and **both pages
describe the same policy**. So the pipeline concatenates every page's text
(`\n\n` between pages, which also keeps `ocrRawText` readable for debugging)
and runs the parser and the matcher exactly **once per file**.
Consequences worth knowing before touching this code:
- `PolicyOcrDocument.pageNumber` is repurposed as the **file ordinal within
the batch** (1, 2, 3…), not a page index. The
`(batchId, pageNumber)` unique constraint still holds, and one batch still
carries many policies — one per uploaded file.
- Parser regexes are anchored across the whole concatenated text (`^From$`,
`^Currency\s+…`), which is why the page-boundary blank line matters.
- `ocrConfidence` on the row is the **mean** across the file's pages.
- A file that fails to parse produces exactly one `OCR_FAILED` row — the right
granularity, and the page PNGs stay on disk for a re-run after a parser fix.
### `storageKey` is the source PDF, not a page image
`PolicyOcrDocument.storageKey` points at `source-N.pdf`. The review screen
embeds that file directly, so the reviewer looks at the **exact artifact the
office received** and gets the browser's native PDF scrolling, zoom and text
selection for free. Rendered PNGs are still written for future re-OCR or an
image-based audit, but nothing points at them as the document's identity.
(The statement side does the opposite — there `storageKey` is the page image,
because a page *is* the document.)
## Matching: policy number only, never the insured name
`PolicyMatcherService` matches on `Policy.policyNumber` and nothing else.
The certificate's "Insured" line is the account's registrant, which drifts
from the customer the office actually holds the file under — the same finding
the statement matcher is built around (a CESPT receipt reading
`ARNAIZ ROSAS ELSA AURORA` for a customer this office holds as `CATT, RANDY`).
Names are shown to the reviewer as a sanity check and never feed matching.
| Rows on `policyNumber` | Result |
|---|---|
| exactly 1 | `MATCHED`, confident — the only unambiguous hit |
| 0 | new policy: review offers a customer picker, confirm **creates** the row |
| >1 | surfaced as candidates, human picks |
More than one hit is never auto-resolved. Duplicate policy numbers across
customers do occur (one group policy bound by two related parties), and
picking arbitrarily would silently book the wrong coverage against the wrong
person.
## What the parser reads, and the field it cannot
`ParsedPolicy` fields are all nullable on purpose: each carrier prints a
different subset, and the matcher and review queue both work better with
"field was read" vs "field was not" than with a guess.
Read from the GMX certificate: policy number, insured name, additional
insured, broker (→ `Policy.agentName`), legal address, ZIP, `policyFrom` /
`policyTo` / `policyDate`, currency, premium-payment cadence, and the full
per-coverage table (risk, insured amount, deductible, loss participation)
preserved verbatim.
> **The GMX certificate carries no premium.** Not "sometimes missing" — the
> document does not have the figure. It lives on GMX's **separate `recibo`
> PDF**. The parser leaves `netPremium` / `policyFee` / `brokerFee` / `total`
> null and pushes a note onto the row —
> *"esta página no trae prima; revisar el recibo de GMX por separado"* — so
> the reviewer sees why the field is empty rather than assuming a read
> failure.
This is also why confirm never overwrites an existing `Policy.netPremium`
with null: the certificate not carrying a premium is not evidence that the
premium is gone.
Deductible and loss participation are stored as **strings** (`"5%"`, `"20%"`,
`"USD 1,000"`) — they are printed as a mix of percentages, currency amounts
and free text, and normalising them would lose the distinction.
## Confirm: what actually gets written
Per confirmed document, in order:
1. **The `Policy` row** — updated if a policy was matched, created under the
picked customer if not. Only non-null `extracted*` fields are written; null
never overwrites existing data.
2. **A `PolicyDocument`** — the source PDF is streamed into the policy's
storage namespace and attached, so the paperwork stays with the policy.
3. **Optionally a `Transaction`** — `INSURANCE` domain, negative amount
(a charge), `captureSource: "OCR"`, `captureRef` = the document id.
The ledger write is **opt-in twice over**: staff must tick `postPremium`
*and* a premium must have parsed to a positive number. Without that gate the
premium-less certificate above would silently book a $0 charge on every
confirm.
`createdPolicyId` and `postedTransactionId` are unique columns on the
document row, so a double-confirm cannot re-apply — and a `POSTED` document
is refused outright.
Discarding a batch is refused once any page is `POSTED`: a partly-applied
batch has already written `Policy` (and possibly `Transaction`) rows, and
hiding the paperwork behind a "discarded" label would leave those rows
unexplained. Reject the remaining pages individually instead.
## API surface
| Method | Route | Ability |
|---|---|---|
| `GET` | `/policy-ocr/status` (is OCR + storage available) | authenticated |
| `GET` | `/policy-ocr/batches`, `/batches/:id`, `/batches/:id/documents` | authenticated |
| `GET` | `/policy-ocr/documents/:id/page` (streams the source PDF) | authenticated |
| `POST` | `/policy-ocr/batches` (upload) | `policy:ingest` |
| `PATCH` | `/policy-ocr/documents/:id` (edit the extracted fields) | `policy:ocr-review` |
| `POST` | `/policy-ocr/documents/:id/reject` | `policy:ocr-review` |
| `POST` | `/policy-ocr/batches/:id/discard` | `policy:ocr-review` |
| `POST` | `/policy-ocr/batches/:id/confirm` | `policy:ocr-review` |
## Requirements
Same as statement OCR: object storage (`S3_ENDPOINT` + credentials) for the
source PDFs and page images, and `tesseract-ocr` / `tesseract-ocr-data-spa` /
`poppler-utils` in the API image. `GET /policy-ocr/status` reports both; if
either is missing the feature reports itself unavailable and only this
feature is disabled.
## Tests
`apps/api/src/policy-ocr/parsers/policy-parser.spec.ts` — 8 cases, all
against verbatim text extracted from one real document,
`HC_Folio_000767_Traduccion.pdf`: provider detection from the wordmark and
from the footer URL, the header fields, every coverage row off the second
page, the deductible/loss-participation strings, the missing-premium note,
the broker line with the agent-number parens absent, and a page with no GMX
signal at all (which must yield no provider rather than a bad guess).
## Not built
- **Only GMX.** The dispatcher (`detectPolicyProvider`) is a table of
`[provider, pattern]` pairs plus a `parsers` map, so adding ANA or Qualitas
is a parser function and two entries — but no other carrier's layout has
been seen yet, and guessing at one produces a parser nobody can verify.
- **The `recibo` PDF.** Reading the premium off GMX's separate receipt
document, and pairing it to the certificate it belongs to, is the obvious
next piece. It is what would let `postPremium` stop being a manual tick.
- **Renewals from OCR.** A re-issued policy arrives as a new certificate with
the same number; confirm updates the existing row rather than versioning
it. Nothing tracks "this is the 2027 issue of that policy".
## Related
- [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2 — the utility
statement pipeline this was lifted from, and the origin of three rules the
policy parser applies: detect the provider by brand before layout, only
ever apply the Tesseract digit-confusion map (`O→0`, `S→5`, `B→8`, …) to
fields known to be digits, and parse amounts by separator *position* rather
than assuming `,` is thousands.
Those last two are **duplicated on purpose**, not imported: the module is
kept self-contained, since sharing a helper would couple two unrelated
domains through it. If you fix a bug in one, check the other.
- [`INSURANCE_FEATURES_SPEC.md`](INSURANCE_FEATURES_SPEC.md) — the four
insurance features that *were* planned. This is not one of them.