docs: document policy OCR capture, the feature no spec proposed
Policy OCR shipped 2026-08-01 (5e9cb12) and was documented nowhere. It is
not in INSURANCE_FEATURES_SPEC.md because it did not come from that
meeting — it came out of building the utility statement OCR pipeline in
RECEIPT_CAPTURE_SPEC.md §2 and noticing the same shape fits carrier
policy PDFs. A reader had no way to find that lineage.
New docs/POLICY_OCR.md covers it end to end, with weight on the three
things that are not obvious from the statement side:
- **One PDF = one policy.** Statements arrive bundled one customer per
page, so there a page is a document. A GMX certificate is one policy
across two pages, so the pages are concatenated and the parser runs
once per file — which is why `pageNumber` is a file ordinal and
`storageKey` is the source PDF, not a page image.
- **The GMX certificate carries no premium at all** — it lives on a
separate recibo PDF. Hence the null-preserving confirm and the
double-gated ledger write.
- **OcrModule was extracted out of StatementsModule to make this
possible**, and that was blocking rather than cosmetic.
Cross-referenced from RECEIPT_CAPTURE_SPEC.md §2 (where it came from),
INSURANCE_FEATURES_SPEC.md (which never proposed it, and whose §4 carrier
API it partly overlaps), PLAN.md step 11, README and RESUME.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -138,6 +138,9 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
|
|||||||
**Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**.
|
**Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**.
|
||||||
**The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`.
|
**The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`.
|
||||||
**Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost.
|
**Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost.
|
||||||
|
- **Policy OCR capture — DONE** (2026-08-01), **unplanned — it came out of building the bullet above.** Full write-up in [`docs/POLICY_OCR.md`](docs/POLICY_OCR.md). Once the receipt pipeline existed it was obvious the same render→OCR→parse→match→review shape fits the *other* stack of paper this office keys in by hand: the carrier policy PDFs behind every `Policy` row. Built in `apps/api/src/policy-ocr/` with a GMX parser, `policy_ocr_batches`/`policy_ocr_documents`, and abilities `policy:ingest`/`policy:ocr-review` (STAFF, same trust tier and same reason). Web: `/polizas/captura` is the "automática" tab of the policy-creation screen (`/polizas/nuevo` is the manual one, both render `PolicyCaptura.tsx`) with the review queue at `/polizas/captura/[id]`. The `OcrProvider` seam was **extracted out of `StatementsModule` into its own `OcrModule`** to make this possible — that was blocking, not cosmetic; `StatementsModule` now imports it and binds nothing.
|
||||||
|
**The statement pipeline's core assumption inverts here.** Utility statements arrive bundled *one customer per page*, so there a page is a document; a GMX certificate is one policy across two pages (header on 1, coverage table on 2), so the pipeline concatenates the pages and runs the parser and matcher **once per file**. `PolicyOcrDocument.pageNumber` is therefore the file ordinal in the batch, and `storageKey` points at the **source PDF** (the review screen embeds the exact artifact the office received) rather than at a page image. Matching is on `Policy.policyNumber` alone and never the printed insured name — the same registrant-vs-owner drift that rules names out on the utility side. Zero hits means a new policy and confirm creates it; more than one is surfaced, never auto-picked.
|
||||||
|
**The GMX certificate carries no premium at all** — the figure lives on a separate `recibo` PDF — so the premium fields stay null with a note saying why, confirm never overwrites an existing premium with null, and the optional ledger write is gated on staff ticking `postPremium` *and* a premium actually parsing. 8/8 parser tests against one real document (`HC_Folio_000767_Traduccion.pdf`). GMX is the only carrier implemented; the dispatcher is a pattern table, so a second one is a parser function and two entries.
|
||||||
- **Multi-bank chequera — DONE** (2026-07-27). `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. `bank_transactions` gained a **required** `bankAccountId` (plus an `(bankAccountId, transactionDate)` index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by `migration/backfill_bank_accounts.py` — a standalone step because `prisma db push` cannot add a required column to a populated table. It is idempotent and now runs inside `run_all.py` (both normal and `--sync`) ahead of `transform_bank.py`, which fails fast if the account is missing. Every read path in `bank.service.ts` is account-scoped, including `facets()` (which had no filter at all) and *both* raw-SQL rollups in `summary()`. API: `?bankAccountId=` is required on `list`/`stats`/`facets`/`summary` — **not** optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new `bank/accounts` + `bank/banks` sub-resource under a MANAGER `bank:manage-accounts` ability. Web: `/banco` gained an account picker (remembered per browser) and reads every figure in the selected account's currency, `/banco/cuentas` manages banks and accounts, and `/inicio`'s chequera card names the account it is showing instead of implying one register. An account's `currency` is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged.
|
- **Multi-bank chequera — DONE** (2026-07-27). `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. `bank_transactions` gained a **required** `bankAccountId` (plus an `(bankAccountId, transactionDate)` index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by `migration/backfill_bank_accounts.py` — a standalone step because `prisma db push` cannot add a required column to a populated table. It is idempotent and now runs inside `run_all.py` (both normal and `--sync`) ahead of `transform_bank.py`, which fails fast if the account is missing. Every read path in `bank.service.ts` is account-scoped, including `facets()` (which had no filter at all) and *both* raw-SQL rollups in `summary()`. API: `?bankAccountId=` is required on `list`/`stats`/`facets`/`summary` — **not** optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new `bank/accounts` + `bank/banks` sub-resource under a MANAGER `bank:manage-accounts` ability. Web: `/banco` gained an account picker (remembered per browser) and reads every figure in the selected account's currency, `/banco/cuentas` manages banks and accounts, and `/inicio`'s chequera card names the account it is showing instead of implying one register. An account's `currency` is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged.
|
||||||
- **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s).
|
- **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s).
|
||||||
|
|
||||||
@@ -165,6 +168,8 @@ Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a r
|
|||||||
|
|
||||||
**Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown.
|
**Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown.
|
||||||
|
|
||||||
|
**It also produced a feature nobody planned.** The statement OCR pipeline generalised: the same render→OCR→parse→match→review shape reads **carrier policy PDFs** into `Policy` rows, which is `docs/POLICY_OCR.md` (built 2026-08-01, GMX only so far). It belongs to step 12's subject matter but to step 11's lineage, and it is in no spec — worth knowing before reading `INSURANCE_FEATURES_SPEC.md`, which does not mention it. It also partly overlaps what §4's carrier API was wanted for, and unlike that section it is not blocked on a phone call.
|
||||||
|
|
||||||
**Step 12 is one-quarter built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. **§1 renewal emails is done** (2026-08-01/02) and carries a BUILT note recording the three places the build diverged from the spec; §2 liquidación is still the smallest remaining piece, since the per-policy fields are already wired end to end.
|
**Step 12 is one-quarter built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. **§1 renewal emails is done** (2026-08-01/02) and carries a BUILT note recording the three places the build diverged from the spec; §2 liquidación is still the smallest remaining piece, since the per-policy fields are already wired end to end.
|
||||||
|
|
||||||
**Notifications are one screen, not two features.** The four legacy mass-email jobs (`docs/MASS_EMAIL_NOTIFICATIONS.md`) and the insurance renewal avisos both mean "tell a customer something by email", so they are tabs of `/notificaciones` over one `email_notification_log`, with one shared flags panel and one schedule editor. `app_settings` + `SettingsService` (db → env → default) is the operator-config seam they introduced: summary recipients and both sweep cadences live there, so changing any of them is a save, not a redeploy. Credentials stay in the environment.
|
**Notifications are one screen, not two features.** The four legacy mass-email jobs (`docs/MASS_EMAIL_NOTIFICATIONS.md`) and the insurance renewal avisos both mean "tell a customer something by email", so they are tabs of `/notificaciones` over one `email_notification_log`, with one shared flags panel and one schedule editor. `app_settings` + `SettingsService` (db → env → default) is the operator-config seam they introduced: summary recipients and both sweep cadences live there, so changing any of them is a save, not a redeploy. Credentials stay in the environment.
|
||||||
|
|||||||
@@ -43,11 +43,19 @@ API feature modules: `auth`, `users`, `customers`, `policies`, `properties`,
|
|||||||
`billing`, `bank`, `reports`, `notifications`, `renewals`, `mail`, `statements`,
|
`billing`, `bank`, `reports`, `notifications`, `renewals`, `mail`, `statements`,
|
||||||
`policy-ocr`, `ocr`, `storage`, `settings`, `ops`.
|
`policy-ocr`, `ocr`, `storage`, `settings`, `ops`.
|
||||||
|
|
||||||
Web routes: `/inicio`, `/clientes`, `/polizas`, `/servicios`, `/estado-cuenta`,
|
Web routes: `/inicio`, `/clientes`, `/polizas` (+ `/polizas/captura`, policy
|
||||||
`/banco` (chequera), `/recibos` (OCR capture), `/notificaciones` (mass email +
|
PDF OCR capture), `/servicios`, `/estado-cuenta`, `/banco` (chequera),
|
||||||
|
`/recibos` (utility statement OCR capture), `/notificaciones` (mass email +
|
||||||
renewal avisos; `/renovaciones` is an alias onto its Pólizas tab), `/reportes`,
|
renewal avisos; `/renovaciones` is an alias onto its Pólizas tab), `/reportes`,
|
||||||
`/catalogos`, `/operaciones` (DB ingest/backup, ADMIN), `/usuarios`, `/login`.
|
`/catalogos`, `/operaciones` (DB ingest/backup, ADMIN), `/usuarios`, `/login`.
|
||||||
|
|
||||||
|
Two OCR intakes share one `OcrProvider` seam (`src/ocr/`, Tesseract today):
|
||||||
|
utility statements → ledger rows ([`docs/RECEIPT_CAPTURE_SPEC.md`](docs/RECEIPT_CAPTURE_SPEC.md) §2)
|
||||||
|
and carrier policy PDFs → `Policy` rows ([`docs/POLICY_OCR.md`](docs/POLICY_OCR.md)).
|
||||||
|
Both need `tesseract-ocr`, `tesseract-ocr-data-spa`, `poppler-utils` and object
|
||||||
|
storage; each reports its own availability and disables only itself if either
|
||||||
|
is missing.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|||||||
@@ -527,6 +527,82 @@ Implementation notes worth keeping:
|
|||||||
(`$268`, what is paid at the window — what the parser uses today) or the exact
|
(`$268`, what is paid at the window — what the parser uses today) or the exact
|
||||||
breakdown total (`$268.88`). One question for Jorge.
|
breakdown total (`$268.88`). One question for Jorge.
|
||||||
|
|
||||||
|
## Policy OCR capture (`/polizas/captura`) — DONE 2026-08-01, unplanned
|
||||||
|
|
||||||
|
**This feature was not in any spec.** It is what the statement OCR work above
|
||||||
|
turned into once the pipeline existed. Having built render → OCR → parse →
|
||||||
|
match → review for CFE/CESPT/Telnor receipts, the same shape obviously fits
|
||||||
|
the *other* stack of paper this office keys in by hand every week: the carrier
|
||||||
|
policy PDFs behind every `Policy` row. Full write-up in `docs/POLICY_OCR.md`.
|
||||||
|
|
||||||
|
The pipeline was reused rather than copied. `OcrModule` was **extracted out of
|
||||||
|
`StatementsModule`** in the same commit so `PolicyOcrModule` could inject
|
||||||
|
`OCR_PROVIDER` without taking on the statement pipeline — that extraction was
|
||||||
|
blocking, not tidying; the policy module could not resolve the provider at all
|
||||||
|
until it existed. `StatementsModule` imports it now and binds nothing itself,
|
||||||
|
so the Tesseract-vs-managed-API decision stays one line in one file for both
|
||||||
|
features.
|
||||||
|
|
||||||
|
Screens mirror Captura exactly: `/polizas/nuevo` is the manual tab,
|
||||||
|
`/polizas/captura` the automática one, both rendering `PolicyCaptura.tsx`, with
|
||||||
|
the batch review queue at `/polizas/captura/[id]`. Abilities `policy:ingest` /
|
||||||
|
`policy:ocr-review`, both STAFF — same trust tier as statement OCR, and for the
|
||||||
|
same reason: nothing reaches the books unconfirmed.
|
||||||
|
|
||||||
|
**The statement pipeline's central assumption inverts here, and that is the
|
||||||
|
thing to remember.** Utility statements arrive bundled *one customer per page*,
|
||||||
|
so there a page is a document and the parser runs per page. A policy PDF is the
|
||||||
|
opposite: the GMX certificate is one policy spread across two pages (contract
|
||||||
|
header on page 1, the per-coverage table on page 2). So every page's text is
|
||||||
|
concatenated and the parser and matcher run **once per file**. Consequences:
|
||||||
|
`PolicyOcrDocument.pageNumber` is repurposed as the file ordinal within the
|
||||||
|
batch (the `(batchId, pageNumber)` unique constraint still holds), `ocrConfidence`
|
||||||
|
is the mean across the file's pages, and a file that fails to parse yields
|
||||||
|
exactly one `OCR_FAILED` row.
|
||||||
|
|
||||||
|
`storageKey` points at the **source PDF**, not a rendered page image, so the
|
||||||
|
review screen embeds the exact artifact the office received and gets the
|
||||||
|
browser's native PDF scrolling, zoom and text selection for free. The page PNGs
|
||||||
|
are still written for future re-OCR, but nothing treats them as the document's
|
||||||
|
identity. (The statement side is the reverse, because there a page *is* the
|
||||||
|
document.)
|
||||||
|
|
||||||
|
Findings worth keeping:
|
||||||
|
|
||||||
|
- **The GMX certificate has no premium on it at all.** Not intermittently
|
||||||
|
missing — the figure lives on GMX's separate `recibo` PDF. The parser leaves
|
||||||
|
the premium fields null and pushes a note saying so, confirm never overwrites
|
||||||
|
an existing `Policy.netPremium` with null, and the optional ledger write is
|
||||||
|
gated on staff ticking `postPremium` *and* a premium actually parsing.
|
||||||
|
Without that second gate a premium-less certificate would book a $0 charge on
|
||||||
|
every confirm.
|
||||||
|
- **Match on `Policy.policyNumber`, never the printed insured name.** Same
|
||||||
|
registrant-vs-current-owner drift that rules names out on the utility side.
|
||||||
|
Zero hits means a new policy and confirm creates the row under a picked
|
||||||
|
customer; more than one hit is surfaced for a human, never auto-picked —
|
||||||
|
duplicate numbers across related parties do occur.
|
||||||
|
- Deductible and loss participation are stored as **strings** (`"5%"`,
|
||||||
|
`"USD 1,000"`): they are printed as a mix of percentages, amounts and free
|
||||||
|
text, and normalising them would lose the distinction.
|
||||||
|
- Carrier-portal PDFs are usually **born-digital**, so the text layer wins and
|
||||||
|
no OCR runs at all most of the time — same precedence rule as the statement
|
||||||
|
pipeline.
|
||||||
|
- The digit-confusion map and the amount-by-separator-position parser are
|
||||||
|
**duplicated on purpose** rather than imported, to keep the module
|
||||||
|
self-contained. Fix a bug in one, check the other.
|
||||||
|
|
||||||
|
8/8 parser tests, all against verbatim text from one real document
|
||||||
|
(`HC_Folio_000767_Traduccion.pdf`).
|
||||||
|
|
||||||
|
**Open:** GMX is the only carrier implemented — the dispatcher is a
|
||||||
|
`[provider, pattern]` table plus a parser map, so a second carrier is a
|
||||||
|
function and two entries, but no other layout has been seen. Reading the
|
||||||
|
premium off the separate `recibo` PDF and pairing it to its certificate is the
|
||||||
|
obvious next piece; it is what would let `postPremium` stop being a manual
|
||||||
|
tick. And nothing versions a re-issued policy — confirm updates the existing
|
||||||
|
row, so there is no record that this is the 2027 issue of that number.
|
||||||
|
|
||||||
|
|
||||||
## Notificaciones (`/notificaciones`) — DONE 2026-08-01 → 08-02
|
## Notificaciones (`/notificaciones`) — DONE 2026-08-01 → 08-02
|
||||||
|
|
||||||
Two features that were spec'd separately turned out to be one screen. The four
|
Two features that were spec'd separately turned out to be one screen. The four
|
||||||
|
|||||||
@@ -11,6 +11,15 @@ Companion doc: [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) covers the
|
|||||||
Utility Management half of the same meeting (PLAN.md step 11). This doc is the
|
Utility Management half of the same meeting (PLAN.md step 11). This doc is the
|
||||||
insurance half (PLAN.md step 12).
|
insurance half (PLAN.md step 12).
|
||||||
|
|
||||||
|
> **A fifth insurance feature exists that this spec never proposed.**
|
||||||
|
> [`POLICY_OCR.md`](POLICY_OCR.md) — OCR capture of carrier policy PDFs into
|
||||||
|
> `Policy` rows, built 2026-08-01. It came out of the *utility* statement OCR
|
||||||
|
> work in [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2, not from
|
||||||
|
> this meeting, which is why it is documented on its own rather than folded in
|
||||||
|
> here. It is relevant to §4: an OCR path that turns a carrier PDF into a
|
||||||
|
> `Policy` row already covers part of what a carrier API was wanted for, and
|
||||||
|
> unlike the API it is not blocked on Grupo Valore returning a phone call.
|
||||||
|
|
||||||
## Why these four features are one spec
|
## Why these four features are one spec
|
||||||
|
|
||||||
The meeting produced four insurance asks. They are specified together because
|
The meeting produced four insurance asks. They are specified together because
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
# Insurance Policy OCR Capture
|
||||||
|
|
||||||
|
Reads an insurance policy PDF the office downloads from a carrier portal,
|
||||||
|
proposes the `Policy` row it should become, and lets staff confirm. Built
|
||||||
|
2026-08-01 (`5e9cb12`), live under `/polizas/captura`.
|
||||||
|
|
||||||
|
## Why this exists — it was not planned
|
||||||
|
|
||||||
|
This feature is **not in any spec**. It came out of building the utility
|
||||||
|
statement OCR intake in [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md)
|
||||||
|
§2: once there was a working render → OCR → parse → match → review pipeline
|
||||||
|
for CFE/CESPT/Telnor receipts, it was obvious the same shape applies to the
|
||||||
|
*other* stack of paper this office keys in by hand every week — the carrier
|
||||||
|
policy PDFs behind every `Policy` row.
|
||||||
|
|
||||||
|
The two are the same job with a different document on the scanner. Keeping
|
||||||
|
that recognition cheap is the whole point of how it was built: the pipeline
|
||||||
|
was **reused, not copied**.
|
||||||
|
|
||||||
|
- `OcrModule` (`apps/api/src/ocr/ocr.module.ts`) was extracted out of
|
||||||
|
`StatementsModule` in this same commit, purely so `PolicyOcrModule` can
|
||||||
|
inject `OCR_PROVIDER` without dragging in the statement pipeline.
|
||||||
|
`StatementsModule` now imports it and binds nothing itself. That extraction
|
||||||
|
was **blocking**: without it the policy module could not resolve the
|
||||||
|
provider at all.
|
||||||
|
- The engine stays Tesseract behind the same swappable seam, so a managed
|
||||||
|
extraction API remains a one-line change in one file for both features.
|
||||||
|
- The intake screen is a *mode of* the existing policy-creation screen, the
|
||||||
|
same way OCR receipt capture is a mode of Captura — not a new menu entry.
|
||||||
|
|
||||||
|
## What ships
|
||||||
|
|
||||||
|
| Piece | Path |
|
||||||
|
|---|---|
|
||||||
|
| API module | `apps/api/src/policy-ocr/` (service, controller, DTOs, matcher, parser) |
|
||||||
|
| Shared OCR seam | `apps/api/src/ocr/ocr.module.ts` |
|
||||||
|
| Tables | `policy_ocr_batches`, `policy_ocr_documents` (`20260801000000_policy_ocr_intake`) |
|
||||||
|
| Web | `components/PolicyCaptura.tsx` (tab shell), `PolicyOcrIntake.tsx` (upload), `PolicyOcrReview.tsx` (review queue) |
|
||||||
|
| Abilities | `policy:ingest`, `policy:ocr-review` — both **STAFF** |
|
||||||
|
|
||||||
|
Abilities are STAFF for the same reason statement OCR is: nothing reaches the
|
||||||
|
books unconfirmed, and the review step is what makes machine capture safe at
|
||||||
|
that tier.
|
||||||
|
|
||||||
|
## The screen
|
||||||
|
|
||||||
|
`PolicyCaptura` is one screen with two ways in, mirroring `Captura.tsx`:
|
||||||
|
|
||||||
|
- `/polizas/nuevo` → **manual** tab (`PolicyForm`, every field by hand)
|
||||||
|
- `/polizas/captura` → **automática** tab (`PolicyOcrIntake`, drop a PDF)
|
||||||
|
- `/polizas/captura/[id]` → the batch review queue
|
||||||
|
|
||||||
|
Both modes end at the same place — a `Policy` row on a customer's file — so
|
||||||
|
they are modes of one screen rather than two menu entries. Either URL renders
|
||||||
|
the same component, so the tab toggle works from either entry point and old
|
||||||
|
bookmarks land on the right tab.
|
||||||
|
|
||||||
|
## Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
upload PDF → store source → render pages → text layer? → parse → match → review → confirm
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **Store the source.** `policy-ocr/{batchId}/source-N.pdf`, before anything
|
||||||
|
else touches it.
|
||||||
|
2. **Render + read.** Every page is rendered to
|
||||||
|
`policy-ocr/{batchId}/page-M.png`. Text-layer wins when the PDF has one
|
||||||
|
(cheap, exact); the rendered image is OCR'd only when it does not — the
|
||||||
|
same precedence rule as the statement pipeline. Carrier-portal PDFs are
|
||||||
|
usually born-digital, so most of the time no OCR runs at all.
|
||||||
|
3. **Parse.** Provider detected by brand signal first
|
||||||
|
(`GMX`, `Grupo Mexicano de Seguros`, `gmx.com.mx`,
|
||||||
|
`JUNTOS EL RIESGO ES MENOR`), layout patterns only as fallback — the same
|
||||||
|
ordering rule the statement parser needed.
|
||||||
|
4. **Match.** Against `Policy.policyNumber`.
|
||||||
|
5. **Review + confirm.** Nothing is written to `Policy` until a human
|
||||||
|
confirms.
|
||||||
|
|
||||||
|
### One PDF = one policy
|
||||||
|
|
||||||
|
This is the sharpest difference from statement OCR, and it inverts that
|
||||||
|
feature's core assumption.
|
||||||
|
|
||||||
|
Utility statements arrive **bundled, one customer per page** — so there, one
|
||||||
|
page is one document and the parser runs per page. A policy PDF is the
|
||||||
|
opposite: the GMX certificate is a 2-page document where page 1 carries the
|
||||||
|
contract header and page 2 carries the per-coverage table, and **both pages
|
||||||
|
describe the same policy**. So the pipeline concatenates every page's text
|
||||||
|
(`\n\n` between pages, which also keeps `ocrRawText` readable for debugging)
|
||||||
|
and runs the parser and the matcher exactly **once per file**.
|
||||||
|
|
||||||
|
Consequences worth knowing before touching this code:
|
||||||
|
|
||||||
|
- `PolicyOcrDocument.pageNumber` is repurposed as the **file ordinal within
|
||||||
|
the batch** (1, 2, 3…), not a page index. The
|
||||||
|
`(batchId, pageNumber)` unique constraint still holds, and one batch still
|
||||||
|
carries many policies — one per uploaded file.
|
||||||
|
- Parser regexes are anchored across the whole concatenated text (`^From$`,
|
||||||
|
`^Currency\s+…`), which is why the page-boundary blank line matters.
|
||||||
|
- `ocrConfidence` on the row is the **mean** across the file's pages.
|
||||||
|
- A file that fails to parse produces exactly one `OCR_FAILED` row — the right
|
||||||
|
granularity, and the page PNGs stay on disk for a re-run after a parser fix.
|
||||||
|
|
||||||
|
### `storageKey` is the source PDF, not a page image
|
||||||
|
|
||||||
|
`PolicyOcrDocument.storageKey` points at `source-N.pdf`. The review screen
|
||||||
|
embeds that file directly, so the reviewer looks at the **exact artifact the
|
||||||
|
office received** and gets the browser's native PDF scrolling, zoom and text
|
||||||
|
selection for free. Rendered PNGs are still written for future re-OCR or an
|
||||||
|
image-based audit, but nothing points at them as the document's identity.
|
||||||
|
|
||||||
|
(The statement side does the opposite — there `storageKey` is the page image,
|
||||||
|
because a page *is* the document.)
|
||||||
|
|
||||||
|
## Matching: policy number only, never the insured name
|
||||||
|
|
||||||
|
`PolicyMatcherService` matches on `Policy.policyNumber` and nothing else.
|
||||||
|
|
||||||
|
The certificate's "Insured" line is the account's registrant, which drifts
|
||||||
|
from the customer the office actually holds the file under — the same finding
|
||||||
|
the statement matcher is built around (a CESPT receipt reading
|
||||||
|
`ARNAIZ ROSAS ELSA AURORA` for a customer this office holds as `CATT, RANDY`).
|
||||||
|
Names are shown to the reviewer as a sanity check and never feed matching.
|
||||||
|
|
||||||
|
| Rows on `policyNumber` | Result |
|
||||||
|
|---|---|
|
||||||
|
| exactly 1 | `MATCHED`, confident — the only unambiguous hit |
|
||||||
|
| 0 | new policy: review offers a customer picker, confirm **creates** the row |
|
||||||
|
| >1 | surfaced as candidates, human picks |
|
||||||
|
|
||||||
|
More than one hit is never auto-resolved. Duplicate policy numbers across
|
||||||
|
customers do occur (one group policy bound by two related parties), and
|
||||||
|
picking arbitrarily would silently book the wrong coverage against the wrong
|
||||||
|
person.
|
||||||
|
|
||||||
|
## What the parser reads, and the field it cannot
|
||||||
|
|
||||||
|
`ParsedPolicy` fields are all nullable on purpose: each carrier prints a
|
||||||
|
different subset, and the matcher and review queue both work better with
|
||||||
|
"field was read" vs "field was not" than with a guess.
|
||||||
|
|
||||||
|
Read from the GMX certificate: policy number, insured name, additional
|
||||||
|
insured, broker (→ `Policy.agentName`), legal address, ZIP, `policyFrom` /
|
||||||
|
`policyTo` / `policyDate`, currency, premium-payment cadence, and the full
|
||||||
|
per-coverage table (risk, insured amount, deductible, loss participation)
|
||||||
|
preserved verbatim.
|
||||||
|
|
||||||
|
> **The GMX certificate carries no premium.** Not "sometimes missing" — the
|
||||||
|
> document does not have the figure. It lives on GMX's **separate `recibo`
|
||||||
|
> PDF**. The parser leaves `netPremium` / `policyFee` / `brokerFee` / `total`
|
||||||
|
> null and pushes a note onto the row —
|
||||||
|
> *"esta página no trae prima; revisar el recibo de GMX por separado"* — so
|
||||||
|
> the reviewer sees why the field is empty rather than assuming a read
|
||||||
|
> failure.
|
||||||
|
|
||||||
|
This is also why confirm never overwrites an existing `Policy.netPremium`
|
||||||
|
with null: the certificate not carrying a premium is not evidence that the
|
||||||
|
premium is gone.
|
||||||
|
|
||||||
|
Deductible and loss participation are stored as **strings** (`"5%"`, `"20%"`,
|
||||||
|
`"USD 1,000"`) — they are printed as a mix of percentages, currency amounts
|
||||||
|
and free text, and normalising them would lose the distinction.
|
||||||
|
|
||||||
|
## Confirm: what actually gets written
|
||||||
|
|
||||||
|
Per confirmed document, in order:
|
||||||
|
|
||||||
|
1. **The `Policy` row** — updated if a policy was matched, created under the
|
||||||
|
picked customer if not. Only non-null `extracted*` fields are written; null
|
||||||
|
never overwrites existing data.
|
||||||
|
2. **A `PolicyDocument`** — the source PDF is streamed into the policy's
|
||||||
|
storage namespace and attached, so the paperwork stays with the policy.
|
||||||
|
3. **Optionally a `Transaction`** — `INSURANCE` domain, negative amount
|
||||||
|
(a charge), `captureSource: "OCR"`, `captureRef` = the document id.
|
||||||
|
|
||||||
|
The ledger write is **opt-in twice over**: staff must tick `postPremium`
|
||||||
|
*and* a premium must have parsed to a positive number. Without that gate the
|
||||||
|
premium-less certificate above would silently book a $0 charge on every
|
||||||
|
confirm.
|
||||||
|
|
||||||
|
`createdPolicyId` and `postedTransactionId` are unique columns on the
|
||||||
|
document row, so a double-confirm cannot re-apply — and a `POSTED` document
|
||||||
|
is refused outright.
|
||||||
|
|
||||||
|
Discarding a batch is refused once any page is `POSTED`: a partly-applied
|
||||||
|
batch has already written `Policy` (and possibly `Transaction`) rows, and
|
||||||
|
hiding the paperwork behind a "discarded" label would leave those rows
|
||||||
|
unexplained. Reject the remaining pages individually instead.
|
||||||
|
|
||||||
|
## API surface
|
||||||
|
|
||||||
|
| Method | Route | Ability |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET` | `/policy-ocr/status` (is OCR + storage available) | authenticated |
|
||||||
|
| `GET` | `/policy-ocr/batches`, `/batches/:id`, `/batches/:id/documents` | authenticated |
|
||||||
|
| `GET` | `/policy-ocr/documents/:id/page` (streams the source PDF) | authenticated |
|
||||||
|
| `POST` | `/policy-ocr/batches` (upload) | `policy:ingest` |
|
||||||
|
| `PATCH` | `/policy-ocr/documents/:id` (edit the extracted fields) | `policy:ocr-review` |
|
||||||
|
| `POST` | `/policy-ocr/documents/:id/reject` | `policy:ocr-review` |
|
||||||
|
| `POST` | `/policy-ocr/batches/:id/discard` | `policy:ocr-review` |
|
||||||
|
| `POST` | `/policy-ocr/batches/:id/confirm` | `policy:ocr-review` |
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
Same as statement OCR: object storage (`S3_ENDPOINT` + credentials) for the
|
||||||
|
source PDFs and page images, and `tesseract-ocr` / `tesseract-ocr-data-spa` /
|
||||||
|
`poppler-utils` in the API image. `GET /policy-ocr/status` reports both; if
|
||||||
|
either is missing the feature reports itself unavailable and only this
|
||||||
|
feature is disabled.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
`apps/api/src/policy-ocr/parsers/policy-parser.spec.ts` — 8 cases, all
|
||||||
|
against verbatim text extracted from one real document,
|
||||||
|
`HC_Folio_000767_Traduccion.pdf`: provider detection from the wordmark and
|
||||||
|
from the footer URL, the header fields, every coverage row off the second
|
||||||
|
page, the deductible/loss-participation strings, the missing-premium note,
|
||||||
|
the broker line with the agent-number parens absent, and a page with no GMX
|
||||||
|
signal at all (which must yield no provider rather than a bad guess).
|
||||||
|
|
||||||
|
## Not built
|
||||||
|
|
||||||
|
- **Only GMX.** The dispatcher (`detectPolicyProvider`) is a table of
|
||||||
|
`[provider, pattern]` pairs plus a `parsers` map, so adding ANA or Qualitas
|
||||||
|
is a parser function and two entries — but no other carrier's layout has
|
||||||
|
been seen yet, and guessing at one produces a parser nobody can verify.
|
||||||
|
- **The `recibo` PDF.** Reading the premium off GMX's separate receipt
|
||||||
|
document, and pairing it to the certificate it belongs to, is the obvious
|
||||||
|
next piece. It is what would let `postPremium` stop being a manual tick.
|
||||||
|
- **Renewals from OCR.** A re-issued policy arrives as a new certificate with
|
||||||
|
the same number; confirm updates the existing row rather than versioning
|
||||||
|
it. Nothing tracks "this is the 2027 issue of that policy".
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [`RECEIPT_CAPTURE_SPEC.md`](RECEIPT_CAPTURE_SPEC.md) §2 — the utility
|
||||||
|
statement pipeline this was lifted from, and the origin of three rules the
|
||||||
|
policy parser applies: detect the provider by brand before layout, only
|
||||||
|
ever apply the Tesseract digit-confusion map (`O→0`, `S→5`, `B→8`, …) to
|
||||||
|
fields known to be digits, and parse amounts by separator *position* rather
|
||||||
|
than assuming `,` is thousands.
|
||||||
|
|
||||||
|
Those last two are **duplicated on purpose**, not imported: the module is
|
||||||
|
kept self-contained, since sharing a helper would couple two unrelated
|
||||||
|
domains through it. If you fix a bug in one, check the other.
|
||||||
|
- [`INSURANCE_FEATURES_SPEC.md`](INSURANCE_FEATURES_SPEC.md) — the four
|
||||||
|
insurance features that *were* planned. This is not one of them.
|
||||||
@@ -151,6 +151,21 @@ single-movement form.
|
|||||||
> blobs. `GET /statements/status` reports `ocrAvailable` and `storageAvailable`,
|
> blobs. `GET /statements/status` reports `ocrAvailable` and `storageAvailable`,
|
||||||
> and the upload card hides itself unless both hold.
|
> and the upload card hides itself unless both hold.
|
||||||
>
|
>
|
||||||
|
> **This pipeline turned out to generalise, and a second feature came out of
|
||||||
|
> it.** Once render → OCR → parse → match → review existed for utility
|
||||||
|
> receipts, the same shape obviously fit the *other* stack of paper this
|
||||||
|
> office keys in by hand — carrier policy PDFs. That is
|
||||||
|
> [`POLICY_OCR.md`](POLICY_OCR.md), built 2026-08-01, and it is **not in any
|
||||||
|
> spec**; it was a revelation from doing this one. The `OcrProvider` seam was
|
||||||
|
> lifted out of `StatementsModule` into its own `OcrModule` so the policy
|
||||||
|
> module could inject it without taking on the statement pipeline —
|
||||||
|
> `StatementsModule` imports it now and binds nothing itself. The engine
|
||||||
|
> choice stays a one-line change in one file, for both features.
|
||||||
|
>
|
||||||
|
> One assumption does **not** carry over: statements arrive bundled *one
|
||||||
|
> customer per page*, so here a page is a document. A policy PDF is one
|
||||||
|
> document across several pages. See that doc's "One PDF = one policy".
|
||||||
|
>
|
||||||
> **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor
|
> **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor
|
||||||
> bills) drove every decision below. Against them the shipped parser identifies
|
> bills) drove every decision below. Against them the shipped parser identifies
|
||||||
> the provider on **46/46**, reads an account reference on **43/46**, an amount
|
> the provider on **46/46**, reads an account reference on **43/46**, an amount
|
||||||
|
|||||||
Reference in New Issue
Block a user