Compare commits

...
21 Commits
Author SHA1 Message Date
gitea-actions 905fa31e47 chore(release): v1.0.6
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m37s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m11s
Cut by rmancinas via the "Cut release" workflow. Pushing the tag triggers build.yml; deploy separately with tag=1.0.6.
2026-08-02 02:06:12 +00:00
rmancinasandClaude Opus 5 5e9cb12fba feat(polizas): OCR capture for insurance policy PDFs
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m43s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m0s
Mirrors the utility statement intake on the insurance side: a policy_ocr
batch/document pair of tables, a GMX parser, a matcher keyed on
Policy.policyNumber, and a "Captura" screen under /polizas that proposes
policy -> customer for staff to confirm.

Lifts the OCR seam out of StatementsModule into its own OcrModule so
PolicyOcrModule can inject OCR_PROVIDER without taking on the rest of
the statement pipeline; StatementsModule now imports it and binds
nothing itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:07:29 -07:00
rmancinasandClaude Opus 5 5bce0e4c94 feat(recibos): OCR capture for zona federal (ZOFEMAT Tijuana)
Adds the ZONA FEDERAL TIJUANA parser to the statement intake, measured
against 8 pages of real "Zona Federal Marítimo Terrestre" receipts — the
federal maritime-zone occupancy fee the municipality bills on beachfront
lots. Provider read on 8/8, amount on 8/8 (each verified against the
paper), concession clave on 6/8, period on 8/8, deadline on 2/8.

Four things the corpus forced:

- Tijuana bills predial and zona federal from the same treasury: same
  header, same Paseo del Centenario address, same ATB-541201 RFC. Every
  predial discriminator matches a zona federal page too, so whichever
  rule is asked first wins it. The only words exclusive to this layout
  are "Marítimo Terrestre", so its brand rule is asked ahead of all
  three predial ones — and its structural rule, anchored on the stub's
  "Derechos de ocupación", ahead of theirs.

- FEDERAL_ZONE.accountNumber is an amount, not a reference. It holds
  DATMEX.zfed, whose 77 values include 246.06, 2369.09, 22653.94 and a
  negative -1679, while the concession claves these receipts are keyed
  by appear nowhere in the database. Matching on that column could never
  hit — and because every row already has a value, the `[field]: null`
  guards on learnAccountRefs and on the review blank-service fill would
  never fire either, so every page would return to the queue every
  bimester forever. The clave moves to meterNumber, joining gas and
  Tijuana predial, and the first confirm teaches the match.

- The payable figure is not the printed subtotal. The municipality
  rounds to whole pesos and prints the difference on its own "Ajuste Ley
  Hacienda Mpal" line (-$0.05 against a 591.05 subtotal, $0.21 against
  2,872.79). The "Total a pagar" box carrying the rounded figure sits on
  a grey fill and OCR'd on 1 of 8 pages; the SubTotal row read on 8 of
  8. So the amount is the rounded subtotal, cross-checked against the
  printed box wherever it survives — where it did, it agreed.

- The clave is 2 digits, a letter and 3 digits (12-T -012), not the
  cadastral shape, and the letter is kept as printed: toDigits maps D to
  0, which turns a real 14-D -014 into 140014. It is printed twice,
  which rescued a page whose heading was struck through by the office's
  own highlighter — the failure mode behind both missing claves.

Deriving the deadline from the bimester is deliberately not attempted:
it is the 17th of the month after the bimester closes on a current bill,
but four of these eight are late (a $1,000 Multa) and print a
recalculated date, so a derived date would be wrong on exactly the pages
a human most wants to see.

Re-ran the earlier corpora (25 pages: predial Tijuana/Rosarito/Ensenada,
CFE, CESPT, Telnor) through detection to confirm the new rules steal
nothing — all 25 still read as their original provider, including the
five Tijuana predial pages that share the RFC.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:07:21 -07:00
rmancinasandClaude Opus 5 d6501f1d74 feat(recibos): OCR capture for gas butano and municipal predial
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m50s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m8s
Adds four parsers to the statement intake — GAS TIJUANA plus one per
municipality, because Tijuana, Rosarito and Ensenada issue three
completely different predial documents — and a text-layer fast path for
the born-digital invoices the gas company sends.

Measured against a new corpus of 14 documents / 29 pages: provider read
on 29/29, amount on 26/29, and 21/29 auto-matched against the dev
database (22/29 identified). The eight review cases are all legitimate.

Five things the corpus forced:

- Not every statement is a scan. The gas invoices are born-digital CFDIs
  whose text layer is exact; rasterising them only loses information (one
  sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`). The new
  `OcrProvider.textPages` reads the embedded layer via `pdftotext
  -bbox-layout` — same poppler package as `pdftoppm`, so no new
  dependency — and OCR stays the fallback for real scans. Poppler's own
  `<line>` grouping follows text flow rather than the page, so words are
  regrouped by vertical position; without that, a two-column header
  leaves every label separated from the value printed beside it.

- The clave catastral is not two letters and six digits. Position three
  is a letter in 15 of the 932 stored claves, and digitising the whole
  tail mapped a real `MMB01041` to a nonexistent `MM801041`.

- Tijuana predial prints no clave at all. Its only identifier is an
  8-digit municipal account carried in a 32-digit payment barcode, which
  the legacy database never held, so it goes in `meterNumber` alongside
  gas — `accountNumber` holds `DATMEX.predial`, which is not a
  per-property key and must not be overwritten. Those pages start cold
  and are taught by the first confirm.

- On Rosarito and Ensenada the clave is the primary key, not a fallback:
  those receipts print nothing else, so a unique hit auto-matches. On a
  utility bill that merely happens to print one it stays a review hint.

- A misread `$` is the dangerous failure. An Ensenada receipt for
  $2,203.00 OCR'd as `82,203.00`, which would post a charge 37x too large
  and look ordinary in the ledger. Predial amounts now require a literal
  `$` and a page that cannot produce one goes to review.

The scoped match field is now one exported function rather than three
copies of `kind === "GAS" ? ... : ...`, since the lookup, the
blank-service fill and the confirm write-back have to agree or a
reference gets learned into a column nothing searches.

First tests in this package: 23 specs over the parsers and the text-layer
reader, every fixture a verbatim OCR excerpt from a real receipt. Adds
the jest config they need and a build tsconfig so they stay out of dist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 12:52:20 -07:00
rmancinas 216309190c feat(recibos): live OCR progress bar on review page
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m51s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m35s
Backend already returns per-status counts via byStatus; render a real
progress bar (X% / N de M / en cola) while PENDING_OCR pages remain,
using the existing progress-track CSS. Falls back to indeterminate
when no docs have been reported yet.
2026-08-01 02:43:19 -07:00
rmancinasandClaude Opus 5 e589bda28b ci(build): skip the redundant master build when a release is cut
Build and Push Images / Build jorgecuadros-api (push) Canceled after 1m10s
Build and Push Images / Build jorgecuadros-web (push) Canceled after 1m8s
release.yml pushes the release commit and its tag in a single `git push`,
so Gitea created two build.yml runs for the same commit. Only the tag run
matters: it emits the X.Y.Z and X.Y image tags, and since it is the same
commit it publishes `latest` and `sha-<short>` as well. The master run was
pure duplicate work that had to be waited out or cancelled by hand.

Guard the build job with an `if` that skips a branch push whose head commit
message starts with `chore(release):`. Ordinary pushes to master are
unaffected, and tag pushes and manual dispatches always build.

The skipped master run keeps the release commit's sha, which would have let
release.yml's "Verify build.yml started" check go green on it alone even if
the tag run were never created — the exact failure that check exists to
catch. It now also requires the run's ref to be the tag, falling back to the
sha match only when the API reports no ref.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 02:28:09 -07:00
gitea-actions 98f7aa8a2d chore(release): v1.0.5
Build and Push Images / Build jorgecuadros-api (push) Canceled after 0s
Build and Push Images / Build jorgecuadros-web (push) Canceled after 0s
Cut by rmancinas via the "Cut release" workflow. Pushing the tag triggers build.yml; deploy separately with tag=1.0.5.
2026-08-01 09:22:56 +00:00
rmancinasandClaude Opus 5 898cf48c80 fix(migration): re-import died on the last step because blob_extract required deploy/.env.prod
Every transform resolves its target through dbenv.database_url(), which lets a
DATABASE_URL in the process environment win — that is how the API container
drives a re-import against its own database with no deploy/ directory present.
blob_extract.py was the one step that bypassed it and called load_env()
directly for the MinIO credentials, so the "Operaciones" re-import loaded all
the data and then exited 1 on:

  missing /repo/deploy/.env.prod — deploy the 'prod' DB stack and write its
  .env first

Give the S3 settings the same resolution as the DB URL: load_env() now returns
{} for an absent file, and setting()/require() layer the process environment on
top of it. blob_extract reads S3_ENDPOINT / S3_BUCKET and accepts either
S3_ACCESS_KEY/S3_SECRET_KEY or MINIO_ROOT_USER/MINIO_ROOT_PASSWORD, matching
the fallback order in storage.service.ts and the vars the api service already
sets in deploy/galactus/jorgecuadros-app.compose.yml. A genuinely missing
setting still fails fast, now naming the variable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 02:21:38 -07:00
gitea-actions 70fe425043 chore(release): v1.0.4
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m2s
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m55s
Cut by rmancinas via the "Cut release" workflow. Pushing the tag triggers build.yml; deploy separately with tag=1.0.4.
2026-08-01 09:09:15 +00:00
rmancinasandClaude Opus 5 567b033c46 fix(docker): re-import failed because the Access CLI tools were never installed
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m37s
Build and Push Images / Build jorgecuadros-api (push) Successful in 3m22s
The API image installed Alpine's `mdbtools` package, which ships only the
shared library. The command-line tools that migration/extract.py actually
shells out to -- `mdb-tables` and `mdb-export` -- are in the separate
`mdbtools-utils` subpackage, so the build succeeded and the re-import in the
"Operaciones" admin panel failed at run time with:

    RuntimeError: mdbtools not found on PATH (need mdb-tables and mdb-export)

Install `mdbtools-utils` instead; it pulls the library in as a dependency.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 02:07:22 -07:00
rmancinasandClaude Opus 5 1934470d53 ci(release): dispatch the fallback build with a fully qualified ref
Gitea's workflow dispatch API 404s on a bare `v1.0.3` and accepts only
`refs/tags/v1.0.3`, so the fallback added in fdbe9fd would have failed
the release instead of rescuing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 02:00:23 -07:00
rmancinasandClaude Opus 5 fdbe9fdb88 ci(release): fail the release when the build never starts
Gitea creates workflow runs from the post-receive hook. When that hook
errors the refs still land, git prints `remote: error: Internal Server
Error` and exits 0 — a post-receive failure does not fail a push. v1.0.3
was cut exactly that way: tag pushed, no build run created, no images
published, and the release step green. It surfaced two steps later as a
404 when the deploy tried to pull 1.0.3.

Capture the push output and warn on `remote: error`, then verify a
build.yml run actually exists for the new commit, dispatching it against
the tag if not. Fail the release if that does not take either, so a
release that publishes nothing is red instead of green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 01:58:10 -07:00
gitea-actions e082113640 chore(release): v1.0.3
Cut by rmancinas via the "Cut release" workflow. Pushing the tag triggers build.yml; deploy separately with tag=1.0.3.
2026-08-01 08:45:42 +00:00
rmancinasandClaude Opus 5 860d483bad fix(ops): backup failed on the MariaDB client shipped in the API image
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m7s
Build and Push Images / Build jorgecuadros-web (push) Successful in 2m7s
Every backup on galactus died with:

  mysqldump: unknown variable 'set-gtid-purged=OFF'
  respaldo incompleto eliminado

Alpine's mysql-client is MariaDB's, so `mysqldump` inside the API
container is a shim over `mariadb-dump`, which has no --set-gtid-purged.
That took out BACKUP and, because they take a safety dump first, SYNC
and REIMPORT too.

Probe `mysqldump --help` and pass the flag only when it is advertised,
calling `mariadb-dump` directly otherwise — MariaDB writes no GTID state
unless asked with --gtid, so there is nothing to suppress. Testing
whether mariadb-dump merely exists would be wrong: on a host carrying
both clients it would shadow a perfectly good MySQL mysqldump.

The probe uses a command substitution rather than `--help | grep -q`
because PIPEFAIL is in effect for these commands and grep closing the
pipe early would report a supported flag as unsupported.

pre-migrate-backup.mjs is unaffected — it dumps from a real mysql:8.4
image, not from the API container.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 01:43:25 -07:00
rmancinasandClaude Opus 5 783ec83464 feat(ops): show upload percent, speed and ETA for ingest files
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m46s
Build and Push Images / Build jorgecuadros-api (push) Successful in 3m3s
The ingest upload used fetch(), which cannot report request-body
progress, so the only feedback was a static "Cargando…" label — no way
to tell a stalled 2 GB upload from a working one.

Switch uploadFile() to XMLHttpRequest and expose an optional onProgress
callback reporting loaded/total bytes, a smoothed transfer rate and a
remaining-time estimate. The Operaciones ingest table renders a progress
bar row under the file being uploaded. Once the bytes are all sent the
server still has to write the file, so that tail reads "Procesando en el
servidor…" rather than parking at 100%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 01:38:22 -07:00
gitea-actions a9b4aab7ec chore(release): v1.0.2
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m4s
Build and Push Images / Build jorgecuadros-web (push) Successful in 2m12s
Cut by rmancinas via the "Cut release" workflow. Pushing the tag triggers build.yml; deploy separately with tag=1.0.2.
2026-08-01 08:23:07 +00:00
rmancinasandClaude Opus 5 a8afd87c3f ci: add a "Cut release" dispatch workflow
Stamps every package.json, commits chore(release): vX.Y.Z, tags and pushes
both refs in one dispatch — patch/minor/major, or an explicit number. Cutting
a release from a laptop is how a manifest bump gets forgotten or a tag lands
on an unpushed commit; the only input here is the number.

Guards: refuses a version that already exists as a tag (releases are
immutable), a no-op bump, a leading `v`, and a malformed number. Checkout is
full-depth because the duplicate-tag check is meaningless against a shallow
clone.

Pushes with a RELEASE_TOKEN PAT rather than the built-in Actions token —
whether a push made with that token re-triggers build.yml depends on the Gitea
version, and a release that quietly publishes no images is worse than one that
fails outright.

Builds and deploys stay separate: the tag push triggers build.yml, and
deploying remains a deliberate dispatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 01:21:13 -07:00
rmancinasandClaude Opus 5 b59abda895 feat(captura): fold recibo OCR into Captura as an automatic mode
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m46s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m17s
Scanning a stack of bills and keying them in are the same daily job, ending
in the same ledger path, so OCR intake becomes a mode of the capture screen
instead of a second menu entry:

- components/Captura.tsx holds the mode switch; the manual check form moves
  verbatim to components/ManualCheckCapture.tsx and the OCR intake to
  components/StatementIntake.tsx.
- /estado-cuenta/lote opens on manual, /recibos on automatic — both render
  Captura, so batch-review links and old bookmarks still land right.
- Nav drops "Recibos (OCR)"; "Captura" covers both, with a NavLink.aliases
  field so /recibos still highlights it.

Also fixes the "El almacenamiento de documentos no está configurado" failure
staff hit on upload. Uploading with no object storage configured used to
succeed, then die on the first put minutes later, leaving a FAILED batch
whose only explanation was that string. createBatch now refuses up front,
GET /statements/status reports storageAvailable alongside ocrAvailable, and
the intake tab explains the situation instead of offering an upload that
cannot work. S3_* documented in .env.example (deploy stacks already set it).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 01:04:09 -07:00
rmancinasandClaude Opus 5 4d5008b545 feat(statements): OCR intake for scanned utility bills
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m41s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m18s
Staff key 300+ utility statements per company per month by hand. This adds
the ingest -> split -> OCR -> match -> review pipeline that proposes customer
and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the
existing BillingService.createBatch seam with source=OCR and a per-document
captureRef so machine and hand capture share one write path and audit trail.

Everything was designed against 10 real scanned statements (46 pages of CFE,
CESPT and Telnor bills) rather than from the sample-free spec. The scans have
no text layer at all — they are camera images — so OCR is mandatory, and they
arrive bundled one customer per page. Measured on those pages the parser
identifies the provider 46/46 and reads an account reference 43/46; against
the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with
the rest genuine review cases. That closes the OCR-provider question in favour
of self-hosted Tesseract: it clears the bar for a queue where a human confirms
every row, and OcrProvider keeps a managed API a one-line swap.

The samples corrected three things the spec had wrong or unknown:

- Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and
  predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds,
  has 663 distinct values across 1135 rows and appears on no statement. The
  clave now lives on Property.cadastralKey as the matcher's secondary key;
  predial is left untouched. This had been blocking predial matching.
- Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers
  (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber.
- Phone is one billed line per property (534/18/1 across phone1/2/3), so the
  new TELEPHONE ServiceKind backfills from phone1 only, not three rows.

Matching is scoped to one column per service kind and never reads the customer
name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this
office holds under CATT, RANDY, because the printed name is the registrant,
not the current owner. Where a provider prints a payment barcode it beats the
printed label (one CFE label OCR'd a digit too many while its barcode was
correct) and the two cross-check, with disagreement forcing review.

Confirming a document whose service had no reference writes it back, so gas
and any other cold start is a one-time cost rather than a permanent queue.

Verified end to end against the live dev API and MinIO: real scans uploaded
over HTTP, matched, confirmed against a check, and the resulting rows checked
in MySQL (negative amounts, captureSource=OCR, concept derived from the batch
kind, captureRef linking back to each page). Re-confirming a posted batch is
refused. Test data was removed afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:42:35 -07:00
rmancinasandClaude Opus 5 121952fdc1 chore(release): v1.0.1
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m6s
Build and Push Images / Build jorgecuadros-web (push) Successful in 2m22s
v1.0.0's images were built from db2bd54, which predates the full-hash
footer. Deploying 1.0.0 would ship the abbreviated footer, so the version
that actually goes to galactus is 1.0.1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 16:38:06 -07:00
rmancinasandClaude Opus 5 15f533b984 fix(web): show the full commit hash in the build footer
The footer abbreviated to 7 characters, so the line read
"v master · 19f0319". That line exists to be pasted into `git show` or
compared against a registry tag, and an abbreviation makes both a manual
step — while the full 40-char value was already baked into the image
(build.yml passes `github.sha` whole, and /version returns it untouched).

`shortSha` had no other caller, so it goes with it.

The span gets `overflow-wrap: anywhere` and `min-width: 0`: hex offers no
break opportunity, and the api/web mismatch branch renders two of these
hashes side by side, which would otherwise push a phone into horizontal
scroll. Measured at a simulated 360px with both hashes present — the span
wraps, and documentElement.scrollWidth stays equal to clientWidth.

Verified in the browser against the dev database: footer renders
"v1.0.0 · db2bd54c0ffee1234567890abcdef0123456789a", hash length 40.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 16:37:48 -07:00
66 changed files with 8817 additions and 782 deletions
+9
View File
@@ -4,6 +4,15 @@ SESSION_SECRET=change-me-to-a-random-string
WEB_ORIGIN=http://localhost:3000
NEXT_PUBLIC_API_ORIGIN=http://localhost:3001
# Object storage (MinIO / S3) for document blobs and scanned receipt pages.
# Without S3_ENDPOINT + credentials the API still boots, but every document
# upload/download and the whole recibo OCR intake are disabled. Credentials fall
# back to MINIO_ROOT_USER / MINIO_ROOT_PASSWORD when the S3_* pair is unset.
S3_ENDPOINT=http://localhost:9000
S3_BUCKET=jorgecuadros-documents
S3_ACCESS_KEY=
S3_SECRET_KEY=
# Login the "Operaciones" screen runs mysqldump/mysql as. Optional locally: when
# unset it falls back to the DATABASE_URL credentials, which a dev MySQL usually
# grants enough for. Required in any deployment, where the application user has
+10
View File
@@ -37,6 +37,16 @@ env:
jobs:
build:
name: Build ${{ matrix.image }}
# release.yml pushes the release commit and its tag in a single `git push`,
# so Gitea creates two runs for the same commit: one for master, one for the
# tag. Only the tag run matters — it is the one that emits the X.Y.Z / X.Y
# image tags, and it publishes `latest` and `sha-<short>` too, since it is
# the same commit. Skip the branch run rather than racing or cancelling it.
# Ordinary pushes to master (any message but `chore(release):`) still build.
if: >-
github.event_name != 'push' ||
startsWith(github.ref, 'refs/tags/') ||
!startsWith(github.event.head_commit.message, 'chore(release):')
runs-on: docker
container:
image: docker:27-dind
+270
View File
@@ -0,0 +1,270 @@
# Cut a release: stamp the version across every package.json, commit, tag, push.
#
# This does NOT build and does NOT deploy. Pushing the `vX.Y.Z` tag is what
# triggers build.yml, which publishes `X.Y.Z`, `X.Y`, `sha-<short>` and `latest`
# image tags. Deploying stays a separate, deliberate act: once the build is
# green, dispatch deploy-galactus.yml with `tag=X.Y.Z` (no leading v — the tag
# carries the `v`, the image tag does not).
#
# Why a workflow instead of three local commands: the release commit is the one
# thing that must be identical every time, and cutting it from a laptop is how
# a manifest bump gets forgotten or a tag lands on an unpushed commit. Here the
# only input is the number.
#
# Prereqs (once):
# - Repo secret RELEASE_TOKEN: a Gitea personal access token with
# write:repository on this repo. The built-in Actions token is deliberately
# NOT used — whether a push made with it re-triggers build.yml depends on the
# Gitea version, and a release that silently publishes no images is worse
# than one that fails. A PAT push is an ordinary push and always triggers.
# If build.yml somehow does not start, it has workflow_dispatch: run it
# against the new tag by hand.
name: Cut release
on:
workflow_dispatch:
inputs:
bump:
description: "Which part to bump (choose 'explicit' to type the number)"
type: choice
required: true
default: "minor"
options:
- patch
- minor
- major
- explicit
version:
description: "Exact version when bump=explicit (x.y.z, no leading v)"
required: false
default: ""
jobs:
release:
name: Release
runs-on: docker
container:
image: node:20-alpine
steps:
- name: Install tools
run: apk add --no-cache git
- name: Preflight — RELEASE_TOKEN
env:
RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eu
if [ -z "${RELEASE_TOKEN:-}" ]; then
echo "::error::Secret RELEASE_TOKEN is not set. Create a Gitea PAT with"
echo "::error::write:repository and add it as a repo secret named RELEASE_TOKEN."
exit 1
fi
# Full history + tags: the duplicate-tag check below is meaningless
# against a shallow clone, which has none of them.
- uses: actions/checkout@v4
with:
fetch-depth: 0
ref: master
token: ${{ secrets.RELEASE_TOKEN }}
- name: Resolve the new version
id: ver
env:
BUMP: ${{ github.event.inputs.bump }}
EXPLICIT: ${{ github.event.inputs.version }}
run: |
set -eu
CURRENT=$(node -p "require('./package.json').version")
echo "current: $CURRENT"
if [ "$BUMP" = "explicit" ]; then
NEXT="$EXPLICIT"
if [ -z "$NEXT" ]; then
echo "::error::bump=explicit requires the version input."
exit 1
fi
else
NEXT=$(node -e '
const [cur, part] = process.argv.slice(1);
const m = /^(\d+)\.(\d+)\.(\d+)/.exec(cur);
if (!m) { console.error(`unparseable current version: ${cur}`); process.exit(1); }
let [maj, min, pat] = m.slice(1).map(Number);
if (part === "major") { maj += 1; min = 0; pat = 0; }
else if (part === "minor") { min += 1; pat = 0; }
else { pat += 1; }
process.stdout.write(`${maj}.${min}.${pat}`);
' "$CURRENT" "$BUMP")
fi
# set-version.mjs validates the shape too, but failing here keeps the
# working tree clean when the input is a typo.
case "$NEXT" in
v*) echo "::error::Version must not carry a leading 'v' (got $NEXT)."; exit 1 ;;
esac
if ! printf '%s' "$NEXT" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$'; then
echo "::error::Invalid version: $NEXT (expected x.y.z)."
exit 1
fi
if [ "$NEXT" = "$CURRENT" ]; then
echo "::error::$NEXT is already the current version."
exit 1
fi
if git rev-parse -q --verify "refs/tags/v$NEXT" >/dev/null; then
echo "::error::Tag v$NEXT already exists. Releases are immutable — pick a new number."
exit 1
fi
echo "next: $NEXT"
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
- name: Stamp the version across every manifest
run: node scripts/set-version.mjs "${{ steps.ver.outputs.version }}"
# A release whose only content is the version bump means the dispatch was
# a mistake — set-version.mjs already refused a no-op above, so an empty
# diff here means the manifests were somehow already at this number.
- name: Commit, tag, push
env:
RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}
VERSION: ${{ steps.ver.outputs.version }}
ACTOR: ${{ github.actor }}
run: |
set -eu
if git diff --quiet; then
echo "::error::No manifest changed. Nothing to release."
exit 1
fi
git config user.name "gitea-actions"
git config user.email "actions@git.mancinas.io"
git commit -a \
-m "chore(release): v${VERSION}" \
-m "Cut by ${ACTOR} via the \"Cut release\" workflow. Pushing the tag triggers build.yml; deploy separately with tag=${VERSION}."
git tag -a "v${VERSION}" -m "v${VERSION}"
# Re-point at an authenticated remote. The token is a secret, so Gitea
# masks it in the log; nothing here echoes the URL regardless.
git remote set-url origin \
"$(printf '%s' "${GITHUB_SERVER_URL}" | sed "s#://#://x-access-token:${RELEASE_TOKEN}@#")/${GITHUB_REPOSITORY}.git"
# One push for both refs: a commit that lands without its tag builds
# nothing and looks like a successful release.
#
# The output is captured because a failing *post-receive* hook does not
# fail the push: git prints `remote: error: ...`, updates both refs and
# exits 0. That is how v1.0.3 was cut — the hook 500'd, so Gitea never
# created the build run, and this step went green anyway.
if ! git push origin "HEAD:master" "refs/tags/v${VERSION}" 2>push.log; then
cat push.log
echo "::error::Push failed. Nothing was released."
exit 1
fi
cat push.log
if grep -q '^remote: error' push.log; then
echo "::warning::The remote's post-receive hook errored. Both refs landed,"
echo "::warning::but Gitea most likely created no workflow run for them."
echo "::warning::The next step checks and dispatches build.yml if needed."
fi
echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
id: push
# Gitea creates workflow runs from the post-receive hook, so a hook error
# silently costs you the build: the tag exists, no image is ever published,
# and the failure only surfaces later as a 404 when deploy pulls the image.
# Confirm the run exists; dispatch it if it does not; fail loudly if that
# does not work either.
- name: Verify build.yml started
env:
RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}
VERSION: ${{ steps.ver.outputs.version }}
SHA: ${{ steps.push.outputs.sha }}
run: |
node -e '
const base = `${process.env.GITHUB_SERVER_URL}/api/v1/repos/${process.env.GITHUB_REPOSITORY}`;
const headers = { Authorization: `token ${process.env.RELEASE_TOKEN}` };
const sha = process.env.SHA;
const tag = `v${process.env.VERSION}`;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// The master push and the tag push carry the SAME commit, so a sha
// match alone is not enough: build.yml skips the master run by
// design, and that skipped run would satisfy a sha-only check even
// if the tag run were never created. When the API reports a ref for
// the run, require it to be the tag; when it reports none, fall back
// to the sha match rather than failing a release over a field name.
const isTagRun = (r) => {
const ref = r.head_branch || r.ref || "";
return !ref || ref === tag || ref === `refs/tags/${tag}`;
};
const started = async () => {
const res = await fetch(`${base}/actions/runs?limit=30`, { headers });
if (!res.ok) throw new Error(`runs query failed: HTTP ${res.status}`);
const body = await res.json();
return (body.workflow_runs || []).some(
(r) =>
r.head_sha === sha &&
String(r.path || "").includes("build.yml") &&
isTagRun(r),
);
};
// The hook fires synchronously with the push, so a run that is coming
// is usually already there; the retries cover a busy instance.
const poll = async (attempts) => {
for (let i = 0; i < attempts; i++) {
if (await started()) return true;
await sleep(10_000);
}
return started();
};
(async () => {
if (await poll(3)) {
console.log(`build.yml is running for ${sha}.`);
return;
}
console.log(`No build.yml run for ${sha}. Dispatching against ${tag}.`);
const res = await fetch(
`${base}/actions/workflows/build.yml/dispatches`,
{
method: "POST",
headers: { ...headers, "Content-Type": "application/json" },
// Must be the tag, not master: metadata-action only emits the
// X.Y.Z and X.Y image tags when the ref is a semver tag. And
// it must be the fully qualified ref — Gitea 404s on `v1.0.3`.
body: JSON.stringify({ ref: `refs/tags/${tag}` }),
},
);
if (!res.ok) console.log(`Dispatch returned HTTP ${res.status}.`);
if (await poll(3)) {
console.log(`build.yml is running for ${sha}.`);
return;
}
console.log(`::error::${tag} is pushed but nothing is building it, and`);
console.log(`::error::the dispatch did not take. Run "Build and Push Images"`);
console.log(`::error::by hand with ref=${tag} (the tag, not master), then`);
console.log(`::error::deploy. Check the Gitea server log for the`);
console.log(`::error::post-receive error while you are at it.`);
process.exit(1);
})();
'
- name: Summary
env:
VERSION: ${{ steps.ver.outputs.version }}
run: |
set -eu
echo "Released v${VERSION}."
echo ""
echo "build.yml is now building git.mancinas.io/rmancinas/jorgecuadros-{api,web}:${VERSION}."
echo "When it is green, dispatch 'Deploy to galactus' with:"
echo " tag=${VERSION} scope=app bootstrap=false skip_migrate=false"
+8 -4
View File
@@ -134,7 +134,10 @@ Given the amount of near-duplicate/overlapping data across snapshot tables (mult
- **Receipt capture module — DONE** (2026-07-27). The legacy "Editor" replacement, built on the single-movement capture from step 6. Wires up the previously-unused `Transaction.outstanding` (NOPAGO): capture flag on `POST /billing`, `?outstanding=` list filter, `POST /billing/:id/resolve-outstanding` (gated `ledger:create`, not `ledger:void` — resolving *completes* a capture), and exclusion from every balance aggregate exactly as the legacy `SALDOS ULTIMO 0`'s `HAVING NOPAGO = 0` did. Adds `POST /billing/batch` (one `$transaction`, check-level fields shared, per-line customer/amount) and `GET /billing/by-check`, plus the `cheque-count` report replacing `REPORTE CHEQUE COUNT` / `REPORTE POR CHEQUE` / `EDITA CHEQUE ALF|COUNT|NUM` — print/PDF/CSV/XLSX come free from the existing `/reportes/:slug` machinery. Web: `/estado-cuenta/lote` (the actual "Editor" screen, with live reconciliation against the physical check amount), plus an "Estado de pago" filter, a "sin fondos" row tag and a Resolver dialog on `/estado-cuenta`. No new abilities. Verified end-to-end against dev, API + browser.
**Two pre-existing bugs found and fixed while building it:** (a) `statement()` filtered `legacySourceTable: { notIn: [...] }`, which compiles to SQL `NOT IN` — and `NULL NOT IN (…)` is NULL, so **every app-captured movement was invisible on the customer statement** (438 rows in the movement browser vs 392 on the statement) while still appearing everywhere else. This would have made the whole receipt-capture feature look broken to staff. Now NULL-safe. (b) The balances *count* query omitted the void filter its own page query applied, so the row count disagreed with the rows.
**OCR seam:** `BillingService.createBatch(dto, opts)` is the single multi-row write path and carries three contract guarantees for the step-11 OCR module to post through — `items[i]` maps to `lines[i]` (so `StatementDocument.postedTransactionId` can be zipped back on), `opts.refs[i]` stamps `captureRef` with a duplicate-post guard that a *voided* row deliberately does not block, and `opts.source` is service-level only so an HTTP client cannot label hand-keyed rows as machine-captured. Backed by a new `TransactionCaptureSource` enum (MANUAL/BATCH/OCR) + `captureRef`, both nullable so the 40,136 migrated rows stay NULL rather than being mislabelled.
- **PDF/OCR auto-capture** — ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff currently key in by hand. Posts through the capture module above. Matching logic was checked field-by-field against `migration/transform_properties.py`'s actual output and found three real gaps to close first: no `TELEPHONE` service kind exists yet, `PROPERTY_TAX.accountNumber` was migrated from `PREDIAL` not `CLAVE` (needs verification against a real predial statement), and `GAS.meterNumber` was never populated by the migration at all.
- **PDF/OCR auto-capture — DONE** (2026-08-01). The ingest→split→OCR→match→review pipeline for the 300+/month/service-provider statements staff key in by hand, built in `apps/api/src/statements/` and posting through §1.2's `createBatch` seam with `source: "OCR"` and a per-document `captureRef`. Web: `/recibos` + `/recibos/:id`. Abilities `statement:ingest`/`statement:review` (STAFF — the review step is what makes machine capture safe at that tier). OCR is self-hosted **Tesseract** behind a swappable `OcrProvider` interface; `tesseract-ocr`, `tesseract-ocr-data-spa` and `poppler-utils` were added to the API image.
**Every decision was driven by 10 real scans (46 pages).** Shipped-parser results on them: provider 46/46, account ref 43/46, amount 42/46, due date 44/46 — and against the dev database **39/46 (85%) exact auto-match, 40/46 (87%) identified**, the rest genuine review cases. The scans are pure images (no text layer), so OCR is mandatory, and they arrive **bundled one customer per page**.
**The three gaps are closed, and two of them were mis-stated in the spec.** (a) `TELEPHONE` now exists and is backfilled from `Property.phone1` only — coverage is 534/18/1 across phone1/2/3, so phone is one billed line per property, not three. (b) **Clave catastral ≠ predial**: `DATMEX.clave` (934 rows, `KA903009`) is what CESPT and predial bills actually print, while `predial` — what `PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135 rows and appears on no statement; the clave now lives on `Property.cadastralKey` as the matcher's secondary key and predial is left untouched. (c) Gas was **not** a dead end: 160 of the 334 `DATMEX.gas` values are real account numbers (the rest are `ESTACIONARIO`/`CILINDRO` descriptors), all recovered into `GAS.meterNumber`.
**Matching is scoped per service kind and never reads the customer name** — a CESPT receipt prints `ARNAIZ ROSAS ELSA AURORA` for an account this office holds under `CATT, RANDY`, because the name on a utility bill is the registrant, not the current owner. Normalisation is per provider: CFE strips leading zeros off `NO. DE SERVICIO`, Telnor strips the 664 LADA down to the stored local 7 digits. Where a provider prints a payment barcode it is preferred over the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two are cross-checked, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost.
- **Multi-bank chequera — DONE** (2026-07-27). `Bank`/`BankAccount` models so Seguros (US bank) and Utilities (Mexican bank, currently SCOTHIA) can each have their own register. `bank_transactions` gained a **required** `bankAccountId` (plus an `(bankAccountId, transactionDate)` index, since every read is now filtered by account and ordered by date), and all 22,669 existing rows were backfilled onto a seeded "Utilities — Scotiabank (MXN)" account by `migration/backfill_bank_accounts.py` — a standalone step because `prisma db push` cannot add a required column to a populated table. It is idempotent and now runs inside `run_all.py` (both normal and `--sync`) ahead of `transform_bank.py`, which fails fast if the account is missing. Every read path in `bank.service.ts` is account-scoped, including `facets()` (which had no filter at all) and *both* raw-SQL rollups in `summary()`. API: `?bankAccountId=` is required on `list`/`stats`/`facets`/`summary`**not** optional-with-an-all-accounts-default, since summing an MXN and a USD register repeats exactly the currency-collapsing mistake the billing module exists to prevent — plus a new `bank/accounts` + `bank/banks` sub-resource under a MANAGER `bank:manage-accounts` ability. Web: `/banco` gained an account picker (remembered per browser) and reads every figure in the selected account's currency, `/banco/cuentas` manages banks and accounts, and `/inicio`'s chequera card names the account it is showing instead of implying one register. An account's `currency` is immutable after creation by design — its booked movements are denominated in it. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged.
- **Customer-number recycling** — promotes the legacy `NUM id` (currently only inside `customer_legacy_refs`) into a first-class, reusable `Customer.customerNumber`, automates *finding* candidates for reuse (cancelled / 1-year-inactive), and auto-assigns the lowest free number at creation — the search is automated, the release/reuse decision stays a human action. Backfill needs care: ~140 utilities rows and all insurance-only customers have no real legacy number (synthetic `rownum_N`/`insrow_N` placeholders in `transform_customers.py`, not real `NUM id`s).
@@ -157,7 +160,7 @@ Repo scaffolded at `jorgecuadros-platform/`: npm workspaces, NestJS API with a r
**Portal live DB now in hand.** `utility_dbo.sql` (1.3 GB, 55 tables) and the portal codebase `my-jorgecuadros-web` (PHP/`mysqli`, Gitea repo, themed classic/modern, ~397 PHP files, core in `scripts/functions.php`) are both on disk — resolving the long-standing "`utility_dbo` schema unknown" blocker. Sync-relevant tables identified: statements/money (`utility_bills`, `accounting`, `email_alert_log`), customer/property (`home_owners`, `home_index`, `condominium`, `management`, `hoa_management`, `trust_assist`), portal-facing policy views (`fm2`/`fm3`/`fmt`, `full_coverage`, `mx_liability`, `usa_liability`), and portal write points (`peticion_gas`, PayPal payments, `notifications_settings`, `verification_codes`). A second dump, `jorgecuadros.sql` (38 MB, 11 tables — `pagos`/`pagosemail`/`PROPANO`/`TRUSTVENCE`/etc.), appears to be an older/partial export, not the portal live DB.
**Step 11 spec written, not built.** `docs/RECEIPT_CAPTURE_SPEC.md` covers the receipt-capture ("Editor") completion plus the three net-new ops features (OCR auto-capture, multi-bank chequera, customer-number recycling) — see Build sequencing step 11 above for the summary. Written from the 2026-07-25/26 meeting notes and verified against the real migration scripts and current API code, not just designed from the meeting notes alone.
**Step 11 is now three-quarters built.** Receipt capture, the multi-bank chequera and PDF/OCR auto-capture are all done and verified; only customer-number recycling remains unbuilt. `docs/RECEIPT_CAPTURE_SPEC.md` carries a BUILT note per section recording what shipped and, for §2, the four things real scanned statements proved the spec had wrong or unknown.
**Step 12 spec written, not built.** `docs/INSURANCE_FEATURES_SPEC.md` covers the insurance half of the same meeting (renewal emails, liquidación batch, certificate + portal delivery, carrier APIs) — see Build sequencing step 12 above. Verified the same way, plus a live query of the dev DB for the counts it quotes (email coverage, pending liquidación, installment fill rates) and of the staged Parquet for the legacy settlement-slot usage. Two of the four features are much smaller than they sound: the renewal-notice table, its idempotency key and the letter body already exist, and the per-policy liquidación fields are already wired end to end.
@@ -183,8 +186,9 @@ Unlike the ops items above, these block design decisions, not just infrastructur
**Step 11 — utilities/ops side:**
- OCR provider/budget for the statement auto-capture pipeline (self-hosted vs. a paid per-page API, given 300+ statements/month/service provider).
- Whether `PROPERTY_TAX.accountNumber` (migrated from `DATMEX.PREDIAL`) is actually the same number as "Clave Catastral" (`DATMEX.CLAVE`) — blocks OCR matching for predial statements until confirmed against a real bill.
- ~~OCR provider/budget~~ — **CLOSED**: self-hosted Tesseract, chosen on measured accuracy against real scans, so there is no per-page cost to approve.
- ~~Whether `PROPERTY_TAX.accountNumber` (from `DATMEX.PREDIAL`) is the same number as "Clave Catastral" (`DATMEX.CLAVE`)~~**CLOSED**: they are different numbers. Answered from real CESPT bills plus the staged data; the clave is now migrated separately and predial was left alone.
- Whether the CFE figure to charge is the rounded headline/barcode amount (`$268` — what is actually paid at the window) or the exact breakdown `Total` (`$268.88`). The parser takes the barcode amount; one confirmation from Jorge would settle it.
- The actual bank name/currency/details for the Seguros USD account, and whether any historical Seguros bank register exists to migrate. (Multi-bank support itself is **built** — this is now only the missing content: staff can open the account in `/banco/cuentas` the moment the answer arrives, and it starts empty unless a historical register turns up.)
- The exact "1 year inactivity" / "cancelled" triggers for customer-number recycling eligibility.
- Whether customer-number recycling should ever include true PII purge (matching the office's paper-world habit) or archive-and-reuse-the-number is sufficient — recommended default is archive-only, consistent with this project's existing never-hard-delete convention.
+84
View File
@@ -442,3 +442,87 @@ for what's actually next.
verified vs dev: Anular buttons admin-gated, voided rows struck + excluded from totals,
clicking Anular voids end-to-end (note: it uses a blocking `window.confirm`). Customer-detail
mini tx list now also strikes voided rows ("(anulado)" tag) — was the last void-UI gap.
---
## Statement OCR intake (`/recibos`) — DONE 2026-08-01
Plan step 11 §2 (`docs/RECEIPT_CAPTURE_SPEC.md` §2). The last big utilities
feature: staff scan the month's utility bills and the machine proposes customer
+ amount per page, instead of keying 300+ statements per company by hand. Built
in `apps/api/src/statements/` and `apps/web/src/app/recibos/`, posting through
step 11 §1.2's `BillingService.createBatch` seam (`source: "OCR"`, per-document
`captureRef`) so machine and hand capture share one write path and one audit
trail. Abilities `statement:ingest` / `statement:review`, both STAFF.
**Verified end to end against the live dev API + MinIO**, not just built: real
CFE and Telnor scans uploaded over HTTP, OCR'd, matched, confirmed against a
check, and the resulting rows checked in MySQL — negative (charge) amounts,
`captureSource = OCR`, concept auto-derived from the batch's service kind,
`captureRef` linking each transaction back to its page. Re-confirming a posted
batch is refused. All test data was removed afterwards.
**Everything here was decided from 10 real scanned statements (46 pages), not
from the sample-free spec.** Shipped-parser results on them: provider 46/46,
account reference 43/46, amount 42/46, due date 44/46; matched against the dev
database, **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The rest are
real review cases (one shared account number, three phones not on file, one
clave not in the book, one page too poor to read).
Findings that corrected the spec, each of which changed the build:
- **The scans have no text layer at all** — they are camera images of paper, so
OCR is mandatory rather than a convenience, and they arrive **bundled, one
customer per page**.
- **Clave catastral is not predial.** `DATMEX.clave` (934 rows, `KA903009`) is
what CESPT and predial bills print; `DATMEX.predial` — which
`PROPERTY_TAX.accountNumber` holds — has only 663 distinct values across 1135
rows and appears on no statement. The clave now lives on
`Property.cadastralKey` as the matcher's secondary key; predial was left
untouched. This is the question that had been blocking predial matching.
- **Gas was recoverable after all.** The spec said no legacy gas number existed;
in fact 160 of 334 `DATMEX.gas` values are real account numbers (the rest are
`ESTACIONARIO`/`CILINDRO` descriptors). Recovered into `GAS.meterNumber`.
- **Phone is one billed line per property** (534 / 18 / 1 across phone1/2/3), so
`TELEPHONE` — a new `ServiceKind` — backfills from `phone1` only.
- **Never match on the printed name.** A CESPT receipt for account `5365218`
reads `ARNAIZ ROSAS ELSA AURORA`; the office's book, corroborated by the
clave, has `CATT, RANDY`. The name on a utility bill is the registrant, not
the current owner.
`migration/backfill_statement_match_fields.py` closes those three data gaps on
an existing database (idempotent, wired into `run_all.py` after
`transform_properties.py`, which now produces them directly on a full rebuild).
Applied to dev: 934 claves, 160 gas numbers, 534 TELEPHONE rows.
Implementation notes worth keeping:
- OCR is self-hosted **Tesseract** behind an `OcrProvider` interface — the
provider question is closed on measured accuracy, and a managed API stays a
one-line swap in `statements.module.ts`. `tesseract-ocr`,
`tesseract-ocr-data-spa` and `poppler-utils` were added to the API image; if
they are missing the module reports itself unavailable and only this feature
is disabled.
- **Payment barcodes beat printed labels.** One CFE label OCR'd a digit too
many while its barcode was correct, so the barcode is the source and the label
the cross-check; disagreement forces review.
- **Detect the provider by brand first, layout only as a fallback** — and never
interleave the two passes. A scanned CESPT header came back as `E BAJA ES
PAGO / EALIFORNIA`, which is why the layout fallback exists; a Telnor page
contains words a CFE layout rule would otherwise claim, which is why ordering
matters.
- **Parse amounts by separator position.** A real Telnor bill OCR'd as
`$ 649,00`; stripping commas as thousands separators turns that into $64,900.
- Two of the three layouts are line-oriented, but the CESPT "RECIBO" is a
**table** whose values sit under column headers — that one needs the word
boxes, which is why `OcrPage` carries geometry and not just text.
- Confirming a document whose matched service had no reference **writes the
reference back** (only into an empty field, and only when exactly one blank
service of that kind is a candidate), so gas and any other cold start is a
one-time cost rather than a permanent queue.
- Handwritten folder numbers on the bills (`9`, `405`) are **not** used for
matching — Tesseract read `405` as `205`.
**Open:** whether the CFE charge should be the rounded barcode/headline figure
(`$268`, what is paid at the window — what the parser uses today) or the exact
breakdown total (`$268.88`). One question for Jorge.
+7
View File
@@ -0,0 +1,7 @@
/** @type {import('jest').Config} */
module.exports = {
rootDir: "src",
testEnvironment: "node",
testRegex: ".*\\.spec\\.ts$",
transform: { "^.+\\.ts$": "ts-jest" },
};
+2 -1
View File
@@ -3,6 +3,7 @@
"collection": "@nestjs/schematics",
"sourceRoot": "src",
"compilerOptions": {
"deleteOutDir": true
"deleteOutDir": true,
"tsConfigPath": "tsconfig.build.json"
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@jorgecuadros/api",
"version": "1.0.0",
"version": "1.0.6",
"private": true,
"scripts": {
"build": "nest build",
+4
View File
@@ -9,6 +9,8 @@ import { CustomersModule } from "./customers/customers.module";
import { PoliciesModule } from "./policies/policies.module";
import { PropertiesModule } from "./properties/properties.module";
import { BillingModule } from "./billing/billing.module";
import { StatementsModule } from "./statements/statements.module";
import { PolicyOcrModule } from "./policy-ocr/policy-ocr.module";
import { BankModule } from "./bank/bank.module";
import { OpsModule } from "./ops/ops.module";
import { ReportsModule } from "./reports/reports.module";
@@ -26,6 +28,8 @@ import { AppController } from "./app.controller";
PoliciesModule,
PropertiesModule,
BillingModule,
StatementsModule,
PolicyOcrModule,
BankModule,
OpsModule,
ReportsModule,
+14
View File
@@ -24,6 +24,8 @@ export type Ability =
| "policy:create"
| "policy:update"
| "policy:delete"
| "policy:ingest"
| "policy:ocr-review"
| "property:create"
| "property:update"
| "property:delete"
@@ -32,6 +34,8 @@ export type Ability =
| "bank:create"
| "bank:void"
| "bank:manage-accounts"
| "statement:ingest"
| "statement:review"
| "lookup:manage"
| "user:manage"
| "db:manage";
@@ -44,6 +48,10 @@ export const ABILITY_MIN: Record<Ability, Role> = {
"policy:create": "STAFF",
"policy:update": "STAFF",
"policy:delete": "MANAGER",
// Insurance OCR intake is the same trust tier as statement OCR: STAFF can
// upload + confirm, nothing reaches the books unconfirmed.
"policy:ingest": "STAFF",
"policy:ocr-review": "STAFF",
"property:create": "STAFF",
"property:update": "STAFF",
"property:delete": "MANAGER",
@@ -54,6 +62,12 @@ export const ABILITY_MIN: Record<Ability, Role> = {
// Opening or renaming a chequera is rarer and higher-stakes than posting a
// movement into one — a wrong account silently mixes two sets of books.
"bank:manage-accounts": "MANAGER",
// Uploading a stack of scans and reviewing what the OCR read are both
// "capturing a receipt" — the same trust tier as ledger:create, since
// confirming a statement *is* capturing it. The review step is what makes
// this safe at STAFF level: nothing reaches the ledger unconfirmed.
"statement:ingest": "STAFF",
"statement:review": "STAFF",
"lookup:manage": "MANAGER",
"user:manage": "ADMIN",
"db:manage": "ADMIN",
+3
View File
@@ -5,5 +5,8 @@ import { BillingService } from "./billing.service";
@Module({
controllers: [BillingController],
providers: [BillingService],
// The statements module posts confirmed OCR captures through
// BillingService.createBatch rather than writing Transaction rows itself.
exports: [BillingService],
})
export class BillingModule {}
+18
View File
@@ -0,0 +1,18 @@
import { Module } from "@nestjs/common";
import { OCR_PROVIDER } from "../statements/ocr/ocr.provider";
import { TesseractOcrProvider } from "../statements/ocr/tesseract.provider";
/**
* Lifts the OCR seam out of StatementsModule so other modules (today:
* PolicyOcrModule) can inject OCR_PROVIDER without taking on the rest of
* the statement intake. StatementsModule itself imports this and gets the
* provider the same way.
*
* The concrete engine is still bound here — Tesseract today, a managed
* extraction API later is a one-line change in this file.
*/
@Module({
providers: [{ provide: OCR_PROVIDER, useClass: TesseractOcrProvider }],
exports: [OCR_PROVIDER],
})
export class OcrModule {}
+26 -5
View File
@@ -262,9 +262,26 @@ export class OpsService implements OnModuleInit {
* deploy/scripts/pre-migrate-backup.mjs — the two write into the same volume
* and both are listed as restore points by this same screen.
*
* --set-gtid-purged=OFF: the production server is the replication SOURCE with
* GTID on, so without it every dump embeds SET @@GLOBAL.GTID_PURGED and is
* unrestorable onto the very server it came from.
* The dumper is probed at runtime rather than assumed. This command runs
* inside the API image, whose `mysql-client` is Alpine's — i.e. MariaDB's —
* where `mysqldump` is a deprecation-warning shim over `mariadb-dump` that
* rejects --set-gtid-purged outright:
* mysqldump: unknown variable 'set-gtid-purged=OFF'
* which failed every backup, including the safety backups SYNC and REIMPORT
* take first. MariaDB's dumper emits no GTID state unless asked (--gtid), so
* there is nothing to suppress there; the flag is passed only when the dumper
* on PATH advertises it, and the real binary is called directly only in the
* MariaDB case (calling `mariadb-dump` whenever it merely exists would pick
* it over a MySQL `mysqldump` earlier in PATH on a host carrying both).
*
* The probe is a command substitution, not `--help | grep -q`: PIPEFAIL is in
* effect and grep closing the pipe early would make a supported flag look
* unsupported.
*
* --set-gtid-purged=OFF (MySQL only): the production server is the
* replication SOURCE with GTID on, so without it every dump embeds
* SET @@GLOBAL.GTID_PURGED and is unrestorable onto the very server it came
* from.
*
* The table-count assertion is not belt-and-braces: `gzip -t` passes on the
* ~372-byte output of a mysqldump that died on its first statement, so a
@@ -277,8 +294,12 @@ export class OpsService implements OnModuleInit {
*/
private dumpCommand(flags: string, db: string, out: string): string {
return (
`( mysqldump ${flags} --single-transaction --routines --triggers ` +
`--no-tablespaces --set-gtid-purged=OFF ${db} | gzip -c > ${out} && ` +
`DUMP=mysqldump; GTID=; ` +
`case "$(mysqldump --help 2>/dev/null || true)" in ` +
`*set-gtid-purged*) GTID=--set-gtid-purged=OFF;; ` +
`*) command -v mariadb-dump >/dev/null 2>&1 && DUMP=mariadb-dump;; esac; ` +
`( $DUMP ${flags} --single-transaction --routines --triggers ` +
`--no-tablespaces $GTID ${db} | gzip -c > ${out} && ` +
`gzip -t ${out} && ` +
`TABLAS=$(gunzip -c ${out} | grep -c 'CREATE TABLE') && ` +
`echo "tablas capturadas: $TABLAS" && ` +
@@ -0,0 +1,147 @@
import type { OcrPage } from "../../statements/ocr/ocr.provider";
import {
detectPolicyProvider,
parsePolicy,
type ParsedCoverage,
} from "./policy-parser";
/**
* Verbatim excerpts of what the GMX portal's translation PDF actually
* rendered through pdftotext — same convention as the statement parser
* tests, where invented-clean input would test nothing because clean input
* is not the failure mode.
*/
function page(text: string): OcrPage {
return { text, words: [], confidence: 0.95 };
}
describe("detectPolicyProvider", () => {
it("claims GMX from the brand wordmark on the letterhead", () => {
expect(
detectPolicyProvider(
"Grupo Mexicano de Seguros, S.A. de C.V.\nTecoyotitla 412, Edificio GMX",
),
).toBe("GMX");
});
it("claims GMX from the 'gmx.com.mx' footer URL", () => {
expect(detectPolicyProvider("JUNTOS EL RIESGO ES MENOR\nwww.gmx.com.mx")).toBe("GMX");
});
});
describe("parsePolicy / GMX", () => {
// Verbatim text extracted from ~/Downloads/HC_Folio_000767_Traduccion.pdf via
// `pdftotext -layout`. Two pages joined by "\n\n".
const GMX_FULL = page(
"Multiple Policy\nHome\n" +
"Policy 007-037-07005947-0000-02 in accordance with the enclosed clauses, to insurance:\n" +
"Insured JON ASHLEY STRABALA\n" +
"Additional insured VIVIAN\n" +
"Legal address BONAMPACK No. EXT26 No.INT 0 COL. Punta Bandera, Tijuana, Baja California, C.P. 22550\n" +
"ZIP 22550 Income Tax No. XEXX-010101-000\n" +
"Broker (1176) Jorge Humberto Cuadros\n" +
"Term 12 months\n" +
"From 19/07/2026\n" +
"To 19/07/2027 at twelve hours (noon) Mexico City time.\n" +
"Currency DOLARES Premium payment CONTADO\n" +
"Free translation from the Spanish Insurance contract. The English text is just copy given by courtesy. In case of a dispute, the Spanish will prevail over the English version.\n" +
"Agreed clauses:\n" +
"•The insured and GMX Hereby declared...\n" +
"From the above, the present contract shall not be considered under the condition mentioned within article 36-B from the Insurance Companies General Law. Therefore it shall not be required its registration before the Comision National de Seguros y Fianzas.\n" +
"July 23, 2026\n" +
"Authority sign.\n" +
"Grupo Mexicano de Seguros, S.A. de C.V.\n" +
"Tecoyotitla 412, Edificio GMX\n" +
"JUNTOS EL RIESGO ES MENOR\n" +
"www.gmx.com.mx\n\n" +
"Risk Insured Amount Deductible Loss Participation\n" +
"Building $350,000.00 Not applies Not applies\n" +
"Contents $60,000.00 Not applies Not applies\n" +
"ADDITIONAL RISK\n" +
"Risk Insured Amount Deductible Loss Participation\n" +
"Debris removal Building $35,000.00 Not applies Not applies\n" +
"Debris removal Contents $6,000.00 Not applies Not applies\n" +
"Outdoors Constructions $10,000.00 5% 10%\n" +
"Coverage Extention Covered Not applies Not applies\n" +
"All Risk Covered Not applies Not applies\n" +
"Earthquake and/or volcanic eruption Covered 2% of the sum insured for each damage structure 20%\n" +
"Extra Expenses $41,000.00 Not applies Not applies\n" +
"Robbery with violence $10,000.00 Not applies Not applies\n" +
"Jewerly $3,900.00 Not applies Not applies\n" +
"Electronic Equipment $10,000.00 Not applies Not applies\n" +
"Glasses $10,000.00 Not applies Not applies\n" +
"Tenant $200,000.00 Not applies Not applies\n" +
"Family $200,000.00 Not applies Not applies\n" +
"Family $200,000.00 Not applies Not applies\n" +
"Domestic workers $7,010.00 Not applies Not applies\n" +
"VALUES ADDED, HOME GMX",
);
it("extracts the policy number, insured name, broker, dates, and currency", () => {
const p = parsePolicy(GMX_FULL);
expect(p.provider).toBe("GMX");
expect(p.policyNumber).toBe("007-037-07005947-0000-02");
expect(p.insuredName).toBe("JON ASHLEY STRABALA");
expect(p.additionalInsured).toBe("VIVIAN");
expect(p.agentName).toBe("Jorge Humberto Cuadros");
expect(p.policyFrom?.toISOString().slice(0, 10)).toBe("2026-07-19");
expect(p.policyTo?.toISOString().slice(0, 10)).toBe("2027-07-19");
expect(p.policyDate?.toISOString().slice(0, 10)).toBe("2026-07-23");
expect(p.currency).toBe("USD");
expect(p.zip).toBe("22550");
expect(p.legalAddress).toContain("BONAMPACK");
expect(p.premiumPayment).toBe("CONTADO");
});
it("extracts every coverage row off the second page table", () => {
const p = parsePolicy(GMX_FULL);
const byName = Object.fromEntries(p.coverages.map((c) => [c.risk, c]));
expect(byName.Building?.insuredAmount).toBe(350000);
expect(byName.Contents?.insuredAmount).toBe(60000);
expect(byName["Debris removal Building"]?.insuredAmount).toBe(35000);
expect(byName["Outdoors Constructions"]?.insuredAmount).toBe(10000);
expect(byName["Outdoors Constructions"]?.deductible).toBe("5%");
expect(byName["Outdoors Constructions"]?.lossParticipation).toBe("10%");
// Free-text coverage cells kept verbatim (the policy form surfaces them
// as observations, not as numbers).
expect(byName["Earthquake and/or volcanic eruption"]?.insuredAmount).toBeNull();
expect(byName["Earthquake and/or volcanic eruption"]?.deductible).toContain("2%");
expect(byName["Earthquake and/or volcanic eruption"]?.lossParticipation).toBe("20%");
expect(byName["All Risk"]?.insuredAmount).toBeNull();
expect(p.coverages.length).toBeGreaterThan(10);
});
it("leaves premium fields null on the certificate page and notes it", () => {
const p = parsePolicy(GMX_FULL);
expect(p.netPremium).toBeNull();
expect(p.total).toBeNull();
expect(p.policyFee).toBeNull();
expect(p.notes.join(" ")).toMatch(/prima/i);
});
it("still parses when the broker parens are missing", () => {
const p = parsePolicy(
page(
"Insured JON ASHLEY STRABALA\nBroker Jorge Humberto Cuadros\n" +
"From 19/07/2026\nTo 19/07/2027\nCurrency DOLARES\n" +
"Grupo Mexicano de Seguros",
),
);
expect(p.agentName).toBe("Jorge Humberto Cuadros");
});
it("rejects a page that carries no GMX signal at all", () => {
const p = parsePolicy(page("Random unrelated document with no policy data."));
expect(p.provider).toBe("");
expect(p.notes.join(" ")).toContain("no se reconoció el proveedor");
});
it("captures the deductible / loss-participation columns verbatim as strings", () => {
const p = parsePolicy(GMX_FULL);
const eq = p.coverages.find((c) => c.risk === "Earthquake and/or volcanic eruption");
expect(eq).toBeDefined();
const eqTyped = eq as ParsedCoverage;
expect(eqTyped.deductible).toContain("sum insured");
expect(eqTyped.lossParticipation).toBe("20%");
});
});
@@ -0,0 +1,422 @@
import type { OcrPage } from "../../statements/ocr/ocr.provider";
/**
* What one parsed policy page yields. All fields are nullable because each
* provider prints a different subset (GMX's certificate has no premium
* breakdown, only insured amounts; GMX's receipt page would carry the
* premium), and the matcher + the review queue both work better with
* "field was read" vs "field was not" rather than guessing.
*/
export interface ParsedPolicy {
/** "GMX" today; the dispatcher lives on `detectProvider`. */
provider: string;
policyNumber: string | null;
insuredName: string | null;
additionalInsured: string | null;
/** The "Broker" line on GMX — mapped onto `Policy.agentName`. */
agentName: string | null;
legalAddress: string | null;
zip: string | null;
policyFrom: Date | null;
policyTo: Date | null;
/** Signature/issue date — `Policy.policyDate`. */
policyDate: Date | null;
/** "MXN" | "USD" | …, derived from the printed currency word. */
currency: string | null;
netPremium: number | null;
policyFee: number | null;
brokerFee: number | null;
total: number | null;
/** "CONTADO" / "MENSUAL" / … — premium-payment cadence text. */
premiumPayment: string | null;
/**
* GMX prints per-coverage rows in a table: Building / Contents /
* Earthquake / … with insured amount, deductible, loss participation.
* Preserved verbatim so a missing premium receipt still leaves the
* coverages auditable on the Policy row.
*/
coverages: ParsedCoverage[];
/** Human-readable trail of what was read, surfaced in the review queue. */
notes: string[];
}
export interface ParsedCoverage {
/** "Building", "Contents", "Debris removal Building", "Earthquake…". */
risk: string;
insuredAmount: number | null;
deductible: string | null;
lossParticipation: string | null;
}
// --- shared helpers ---------------------------------------------------------
const DIGIT_CONFUSIONS: Record<string, string> = {
O: "0", o: "0", D: "0", I: "1", l: "1", "|": "1", S: "5", B: "8",
};
/**
* Tesseract confuses these glyphs inside numeric runs with some regularity.
* Same map and same caveat as the statement parser: ONLY apply to fields
* known to be digits, never to free text.
*/
function toDigits(s: string | null | undefined): string {
if (!s) return "";
return s
.split("")
.map((c) => DIGIT_CONFUSIONS[c] ?? c)
.join("")
.replace(/\D/g, "");
}
/**
* Parse a printed amount, treating `,` and `.` by position rather than by
* assumption. Same algorithm as the statement parser — kept here so the
* policy module is self-contained, since importing from `../../statements`
* would couple two unrelated domains through a helper.
*/
function money(s: string | null | undefined): number | null {
if (!s) return null;
const cleaned = s.replace(/[\s$]/g, "");
let m = cleaned.match(/^(\d{1,3}(?:[.,]\d{3})+)([.,]\d{1,2})?$/);
if (m) {
const whole = m[1].replace(/[.,]/g, "");
const cents = m[2] ? m[2].slice(1) : "";
return Number(cents ? `${whole}.${cents.padEnd(2, "0")}` : whole);
}
m = cleaned.match(/^(\d+)[.,](\d{2})$/);
if (m) return Number(`${m[1]}.${m[2]}`);
const n = Number(cleaned.replace(/[,.]/g, ""));
return Number.isFinite(n) ? n : null;
}
function firstMatch(text: string, patterns: RegExp[]): string | null {
for (const p of patterns) {
const m = text.match(p);
if (m?.[1]) return m[1].trim();
}
return null;
}
const MONTHS: Record<string, number> = {
ENE: 0, FEB: 1, MAR: 2, ABR: 3, MAY: 4, JUN: 5,
JUL: 6, AGO: 7, SEP: 8, OCT: 9, NOV: 10, DIC: 11,
};
/**
* DD/MM/YYYY (GMX) and the dash-separated ISO variants. Two-digit years are
* windowed: < 50 → 20YY, ≥ 50 → 19YY, matching what a 1950-2049 window
* expects from a paper document.
*/
function parseDate(raw: string | null | undefined): Date | null {
if (!raw) return null;
const s = raw.trim();
let m = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/);
if (m) return utc(+m[3], +m[2] - 1, +m[1]);
m = s.match(/^(\d{1,2})[-\s/]([A-Z]{3})[-\s/](\d{2,4})$/i);
if (m && MONTHS[m[2].toUpperCase()] !== undefined) {
const yr = +m[3];
const y = m[3].length === 2 ? (yr < 50 ? 2000 + yr : 1900 + yr) : yr;
return utc(y, MONTHS[m[2].toUpperCase()], +m[1]);
}
m = s.match(/^(\d{4})[-/](\d{1,2})[-/](\d{1,2})$/);
if (m) return utc(+m[1], +m[2] - 1, +m[3]);
// "July 23, 2026" — the signature date on the GMX certificate.
m = s.match(/^([A-Za-z]+)\s+(\d{1,2}),\s*(\d{4})$/);
if (m) {
const MONTH_NAMES: Record<string, number> = {
january: 0, february: 1, march: 2, april: 3, may: 4, june: 5,
july: 6, august: 7, september: 8, october: 9, november: 10, december: 11,
};
const mo = MONTH_NAMES[m[1].toLowerCase()];
if (mo !== undefined) return utc(+m[3], mo, +m[2]);
}
return null;
}
function utc(y: number, mo: number, d: number): Date | null {
const dt = new Date(Date.UTC(y, mo, d));
return Number.isNaN(dt.getTime()) ? null : dt;
}
/** Map the printed currency word onto an ISO code. */
function currencyCode(raw: string | null | undefined): string | null {
if (!raw) return null;
const s = raw.trim().toUpperCase();
if (s.startsWith("PESO") || s === "MXN" || s.includes("NACIONAL")) return "MXN";
if (s.startsWith("DOLAR") || s === "USD" || s.includes("DOLLAR")) return "USD";
if (s === "EUR" || s.includes("EURO")) return "EUR";
return null;
}
// --- provider detection -----------------------------------------------------
/**
* Brand first, layout as a fallback. Same ordering rule as the statement
* parser: a brand wordmark is the cheapest, most reliable discriminator, and
* a layout rule that runs first can wrongly claim a page that happens to
* carry the same shape string (the statement parser's lesson with CFE vs
* GAS on "PERIODO FACTURADO").
*/
const BRAND: [string, RegExp][] = [
["GMX", /\bGMX\b|Grupo\s*Mexicano\s*de\s*Seguros|gmx\.com\.mx|JUNTOS\s*EL\s*RIESGO\s*ES\s*MENOR/i],
];
const LAYOUT: [string, RegExp][] = [
["GMX", /Multiple\s*Policy|IMPUESTO\s*PREDIAL[\s\S]{0,80}EN\s*FECHA|Material\s*damages\s*Section/i],
];
export function detectPolicyProvider(text: string): string | null {
for (const group of [BRAND, LAYOUT]) {
for (const [name, pattern] of group) {
if (pattern.test(text)) return name;
}
}
return null;
}
// --- parsers ----------------------------------------------------------------
const PARSERS: Record<string, (page: OcrPage) => ParsedPolicy> = {
GMX: parseGmx,
};
const EMPTY_COVERAGE: ParsedCoverage = {
risk: "",
insuredAmount: null,
deductible: null,
lossParticipation: null,
};
export function parsePolicy(page: OcrPage): ParsedPolicy {
const provider = detectPolicyProvider(page.text);
if (!provider) {
return {
provider: "",
policyNumber: null,
insuredName: null,
additionalInsured: null,
agentName: null,
legalAddress: null,
zip: null,
policyFrom: null,
policyTo: null,
policyDate: null,
currency: null,
netPremium: null,
policyFee: null,
brokerFee: null,
total: null,
premiumPayment: null,
coverages: [],
notes: ["no se reconoció el proveedor"],
};
}
return PARSERS[provider](page);
}
// --- GMX --------------------------------------------------------------------
/**
* GMX policy certificate layout (this is the translation PDF — the Spanish
* version is the canonical source, but every GMX portal download is a
* translation so the parser can rely on these English labels).
*
* Page 1 carries the contract header in a single boxed table:
* Policy | Insured | Additional insured | Legal address | ZIP | Income Tax No.
* Broker | Term | From | To | Currency | Premium payment
* followed by an "Agreed clauses" block, the signature date, and the GMX
* letterhead.
*
* Page 2 carries the per-coverage table (Risk / Insured Amount / Deductible /
* Loss Participation) under "Material damages Section" and "ADDITIONAL RISK".
*
* Premium / total / fees are NOT on the certificate page — they live on
* GMX's separate "recibo" PDF. The parser leaves them null and flags the
* gap in `notes`; the matcher still proposes a Policy update from the
* certificate alone, and the staff confirm step fills premium in by hand
* or after a follow-up receipt upload.
*/
function parseGmx(page: OcrPage): ParsedPolicy {
const text = page.text;
const notes: string[] = [];
// ----- header table (page 1) --------------------------------------------
// The Policy row repeats the number in a long run:
// "Policy 007-037-07005947-0000-02 in accordance with the enclosed clauses…"
// so taking the first token-shaped number is correct; the trailing prose
// never looks like one. The dashes are part of the printed number — keep
// them (don't run toDigits, which would flatten them).
const policyNumber = firstMatch(text, [
/\bPolicy\s+([0-9OIlSBD]{3,4}[-\s][0-9OIlSBD]{3}[-\s][0-9OIlSBD]{8}[-\s][0-9OIlSBD]{4}[-\s][0-9OIlSBD]{2})/i,
/\bPolicy\s+([0-9OIlSBD][0-9OIlSBD\s-]{9,30})/,
]);
// "Insured JON ASHLEY STRABALA" — label, then 1+ whitespace, then the name.
// Names can carry accents (ÁVILA) or apostrophes (O'NEILL); the label is
// always upper-case English on this layout, so case is reliable.
const insuredName = labelValue(text, /^Insured\s+([A-ZÁÉÍÓÚÑ'][A-ZÁÉÍÓÚÑ '\-.]+)$/m);
const additionalInsured = labelValue(text, /^Additional\s+insured\s+([A-ZÁÉÍÓÚÑ '\-.]+)$/m);
// Legal address is a single long line; the parser keeps it whole.
const legalAddress = labelValue(text, /^Legal\s+address\s+(.+)$/m);
const zip = labelValue(text, /^ZIP\s+(\d{4,6})\b/m);
if (!zip && legalAddress) {
// Last resort: zip often appears at the tail of the address run too
// ("…C.P. 22550"). Cheap regex, no false-positive cost on this layout.
const m = legalAddress.match(/\b(\d{5})\b/);
if (m) notes.push(`ZIP leído de la dirección (${m[1]})`);
}
// Broker line on GMX: "(1176) Jorge Humberto Cuadros" — the number is the
// agent code, the name is what lands on `Policy.agentName`. The parens
// are optional: a future layout or scan drop them.
const brokerRaw = labelValue(text, /^Broker\s+(?:\(\d+\)\s*)?(.+)$/m);
const agentName = brokerRaw?.trim() ?? null;
// Term: "12 months" — informational, not a free-standing date. Stored in
// notes; the UI can derive `coveragePeriodDays` from From/To anyway.
const term = firstMatch(text, [/^Term\s+(\d+\s+months?)$/m]);
if (term) notes.push(`vigencia: ${term}`);
const policyFrom = parseDate(
labelValue(text, /^From\s+(\d{1,2}\/\d{1,2}\/\d{4})\b/m),
);
const policyTo = parseDate(
firstMatch(text, [/^To\s+(\d{1,2}\/\d{1,2}\/\d{4})\b/m]),
);
// "at twelve hours (noon) Mexico City time." — kept in notes only.
if (/twelve\s*hours|noon/i.test(text)) notes.push("vencimiento a las 12:00 hora del centro");
// The Currency / Premium payment cells sit next to each other on one
// line; pull them with bounded matches so the trailing label of the
// adjacent cell doesn't swallow the wrong value.
const currency = currencyCode(labelValue(text, /^Currency\s+(\S+?)(?:\s+Premium\s+payment|$)/m));
const premiumPayment = labelValue(text, /Premium\s+payment\s+(\S+)$/m);
// ----- signature date (page 1) -----------------------------------------
// Appears above the signature line on its own: "July 23, 2026".
const dateMatch = text.match(
/\b(January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{1,2},\s*\d{4}\b/,
);
const policyDate = dateMatch ? parseDate(dateMatch[0]) : null;
if (!policyDate) notes.push("no se pudo leer la fecha de firma");
// ----- coverages table (page 2) -----------------------------------------
const coverages = parseGmxCoverages(text, notes);
if (!policyNumber) notes.push("no se pudo leer el número de póliza");
if (!policyFrom || !policyTo) notes.push("no se pudo leer el período de vigencia");
// Premium fields are expected to be missing on the certificate page; flag
// it explicitly so the reviewer knows to look for a separate receipt.
if (!text.match(/Prima\s*neta|net\s*premium/i)) {
notes.push("esta página no trae prima; revisar el recibo de GMX por separado");
}
return {
provider: "GMX",
policyNumber: policyNumber ? policyNumber.replace(/\s+/g, "") : null,
insuredName,
additionalInsured,
agentName,
legalAddress,
zip,
policyFrom,
policyTo,
policyDate,
currency,
netPremium: null,
policyFee: null,
brokerFee: null,
total: null,
premiumPayment,
coverages,
notes,
};
}
/**
* Read the value that follows a `LABEL` on the same line. Used by every
* "Label Value" cell on the GMX header table — matches on the line
* itself rather than across the page, so a label that also appears in body
* text can't accidentally claim a different cell.
*/
function labelValue(text: string, pattern: RegExp): string | null {
const m = text.match(pattern);
if (!m?.[1]) return null;
return m[1].replace(/\s+/g, " ").trim();
}
/**
* Walk the GMX per-coverage table on page 2.
*
* Real sample row (single-line representation of the table after pdftotext
* flattens it; the real layout uses fixed columns):
* "Building $350,000.00 Not applies Not applies"
*
* The four columns are:
* Risk (left), Insured Amount ($ figure OR the word "Covered"),
* Deductible (free text — "Not applies", "5%", "2% of the sum insured…"),
* Loss Participation (same).
*
* "Covered" means the coverage is included with no dollar cap. We record
* the word so the review queue surfaces it instead of inventing a number.
*
* Deductible / Loss Participation are kept as printed strings, not
* converted to numbers — a "20%" loss participation is a different field
* shape from a "$5,000" deductible and the JSON column lets the UI render
* either verbatim.
*
* Multi-line cells (the "Earthquake" row's deductible wraps to three lines
* because the column is narrow) are collapsed by joining consecutive
* non-table-body lines onto the previous row's deductible cell before
* applying the column regex.
*/
function parseGmxCoverages(text: string, notes: string[]): ParsedCoverage[] {
const out: ParsedCoverage[] = [];
// Stop at "VALUES ADDED" — the trailing prose section (homeowner
// services, legal text) is not a coverage table. Re-enter at
// "ADDITIONAL RISK" for the second coverage block on page 2.
const segments = text.split(/VALUES\s*ADDED/i)[0].split(/ADDITIONAL\s*RISK/i);
// `[ \t]` (not `\s`) inside a cell: the deductible/loss-participation
// columns may wrap onto several lines in the raw `pdftotext` output, and
// matching across newlines silently swallows the next row.
const re = /^([A-Za-zÁÉÍÓÚÑ][A-Za-zÁÉÍÓÚÑ /\-.]+?)[ \t]+(\$[\d,.]+|Covered|Not[ \t]+applies)[ \t]+(\S+(?:[ \t]\S+){0,8})[ \t]+(\S+(?:[ \t]\S+){0,8})[ \t]*$/gim;
let m: RegExpExecArray | null;
for (const seg of segments) {
re.lastIndex = 0;
while ((m = re.exec(seg)) !== null) {
const risk = m[1].trim();
const amountCell = m[2].trim();
const deductible = m[3].trim();
const lossParticipation = m[4].trim();
// Skip the "Risk / Insured Amount / Deductible / Loss Participation"
// header row itself, which matches the same regex.
if (/^Risk$/i.test(risk) && /Insured\s*Amount/i.test(amountCell)) continue;
out.push({
risk,
insuredAmount:
amountCell === "Covered" || amountCell === "Not applies"
? null
: money(amountCell),
deductible,
lossParticipation,
});
}
}
if (out.length === 0) notes.push("no se encontraron coberturas en la tabla");
return out;
}
@@ -0,0 +1,101 @@
import { Injectable } from "@nestjs/common";
import { PrismaService } from "../prisma/prisma.service";
import type { ParsedPolicy } from "./parsers/policy-parser";
export interface MatchResult {
policyId: string | null;
customerId: string | null;
/** Why it landed here — shown in the review queue verbatim. */
note: string;
/** True only for an unambiguous hit on `Policy.policyNumber`. */
confident: boolean;
/**
* Every policy that carries the parsed number, with its customer. >1 means
* the policy number is shared across customers and a human must pick.
*/
candidates: { policyId: string; customerId: string; customerName: string; policyNumber: string }[];
}
/**
* Resolves a parsed policy page to an existing Policy (and its customer) the
* office already holds.
*
* **Match on `Policy.policyNumber` alone, never on the printed insured name.**
* The certificate's "Insured" line is the account's registrant, which drifts
* from the current owner — the same problem the statement matcher cites for
* utility bills ("ARNAIZ ROSAS ELSA AURORA" on a CESPT receipt for a
* customer this office holds as "CATT, RANDY"). Names are surfaced for the
* reviewer to sanity-check and never feed matching.
*
* A policy number that matches zero rows means the policy is new: the
* review screen then offers a customer picker and the confirm step creates
* the row. Multiple hits are surfaced rather than auto-picked — duplicate
* policy numbers across customers do occur (same group policy bound by two
* related parties), and picking one arbitrarily would silently book the
* wrong coverage.
*/
@Injectable()
export class PolicyMatcherService {
constructor(private readonly prisma: PrismaService) {}
async match(parsed: ParsedPolicy): Promise<MatchResult> {
if (!parsed.policyNumber) {
return this.unmatched("no se pudo leer el número de póliza");
}
const rows = await this.prisma.policy.findMany({
where: { policyNumber: parsed.policyNumber },
select: {
id: true,
policyNumber: true,
customerId: true,
customer: { select: { name: true } },
},
});
const candidates = rows.map((r) => ({
policyId: r.id,
customerId: r.customerId,
customerName: r.customer.name,
policyNumber: r.policyNumber,
}));
if (rows.length === 0) {
return {
policyId: null,
customerId: null,
note: `no se encontró ninguna póliza con el número ${parsed.policyNumber}`,
confident: false,
candidates: [],
};
}
if (rows.length > 1) {
return {
policyId: null,
customerId: null,
note: `${rows.length} pólizas comparten el número ${parsed.policyNumber}`,
confident: false,
candidates,
};
}
return {
policyId: candidates[0].policyId,
customerId: candidates[0].customerId,
note: `coincidencia exacta por número de póliza ${parsed.policyNumber}`,
confident: true,
candidates,
};
}
private unmatched(note: string): MatchResult {
return {
policyId: null,
customerId: null,
note,
confident: false,
candidates: [],
};
}
}
@@ -0,0 +1,161 @@
import {
Body,
Controller,
Get,
Param,
Patch,
Post,
Query,
Req,
Res,
StreamableFile,
UploadedFiles,
UseGuards,
UseInterceptors,
} from "@nestjs/common";
import { FilesInterceptor } from "@nestjs/platform-express";
import type { Request, Response } from "express";
import { AuthenticatedGuard } from "../auth/authenticated.guard";
import { AbilityGuard } from "../auth/ability.guard";
import { RequireAbility } from "../auth/require-ability.decorator";
import { AuditService } from "../common/audit.service";
import type { UploadedFileLike } from "../storage/upload-file";
import { PolicyOcrService } from "./policy-ocr.service";
import {
ConfirmPolicyBatchDto,
CreatePolicyOcrBatchDto,
ReviewPolicyDocumentDto,
} from "./policy-ocr.dto";
/**
* Insurance OCR intake (policy_ocr_intake).
*
* Mirrors StatementsController shape: one batch = one upload session of
* policy PDFs from a provider portal (GMX today), one document per page.
* Confirming a batch delegates nothing to a separate billing path —
* everything goes through `Policy` (and optionally a Transaction for the
* premium), the same tables the manual `PolicyForm` writes.
*/
@Controller("policy-ocr")
@UseGuards(AuthenticatedGuard, AbilityGuard)
export class PolicyOcrController {
constructor(
private readonly policyOcr: PolicyOcrService,
private readonly audit: AuditService,
) {}
private actingId(req: Request): string {
return (req.user as { id: string } | undefined)?.id ?? "";
}
@Get("status")
async status() {
return {
ocrAvailable: await this.policyOcr.ocrAvailable(),
storageAvailable: this.policyOcr.storageAvailable(),
};
}
@Get("batches")
listBatches(@Query("page") page?: string, @Query("pageSize") pageSize?: string) {
return this.policyOcr.listBatches(
Math.max(1, Number(page) || 1),
Math.min(100, Math.max(1, Number(pageSize) || 25)),
);
}
@Get("batches/:id")
getBatch(@Param("id") id: string) {
return this.policyOcr.getBatch(id);
}
@Get("batches/:id/documents")
listDocuments(@Param("id") id: string) {
return this.policyOcr.listDocuments(id);
}
/**
* The source PDF for a parsed policy document. One PDF = one parsed policy,
* so this returns the entire upload (typically multi-page for insurance
* certificates). The review screen embeds it in an iframe.
*/
@Get("documents/:id/page")
async pageImage(
@Param("id") id: string,
@Res({ passthrough: true }) res: Response,
) {
const { stream, contentType, contentLength } = await this.policyOcr.pageImage(id);
res.set({
// The doc row stores the source PDF, not a rendered page image.
"Content-Type": contentType ?? "application/pdf",
...(contentLength ? { "Content-Length": String(contentLength) } : {}),
});
return new StreamableFile(stream);
}
// --- writes ---------------------------------------------------------------
@Post("batches")
@RequireAbility("policy:ingest")
@UseInterceptors(
FilesInterceptor("files", 25, { limits: { fileSize: 50 * 1024 * 1024 } }),
)
async createBatch(
@UploadedFiles() files: UploadedFileLike[] | undefined,
@Body() _dto: CreatePolicyOcrBatchDto,
@Query("label") label: string | undefined,
@Req() req: Request,
) {
const batch = await this.policyOcr.createBatch(
files ?? [],
this.actingId(req),
label ?? _dto.label,
);
void this.audit.log(this.actingId(req), "policyOcr.batch.create", {
batchId: batch.id,
fileCount: batch.fileCount,
});
return batch;
}
@Patch("documents/:id")
@RequireAbility("policy:ocr-review")
async review(
@Param("id") id: string,
@Body() dto: ReviewPolicyDocumentDto,
@Req() req: Request,
) {
const doc = await this.policyOcr.review(id, dto, this.actingId(req));
void this.audit.log(this.actingId(req), "policyOcr.document.review", {
documentId: id,
status: doc.status,
});
return doc;
}
@Post("documents/:id/reject")
@RequireAbility("policy:ocr-review")
async reject(@Param("id") id: string, @Req() req: Request) {
const doc = await this.policyOcr.reject(id, this.actingId(req));
void this.audit.log(this.actingId(req), "policyOcr.document.reject", {
documentId: id,
});
return doc;
}
@Post("batches/:id/confirm")
@RequireAbility("policy:ocr-review")
async confirm(
@Param("id") id: string,
@Body() dto: ConfirmPolicyBatchDto,
@Req() req: Request,
) {
const result = await this.policyOcr.confirmBatch(id, dto, this.actingId(req));
void this.audit.log(this.actingId(req), "policyOcr.batch.confirm", {
batchId: id,
applied: result.applied,
postedTransactions: result.postedTransactions,
});
return result;
}
}
+85
View File
@@ -0,0 +1,85 @@
import { Type } from "class-transformer";
import {
IsArray,
IsDateString,
IsEnum,
IsNumber,
IsObject,
IsOptional,
IsString,
MinLength,
ValidateNested,
} from "class-validator";
/** One document's confirmed-after-review state. The service reads these
* fields and writes them onto either a matched Policy or a freshly created
* one. Anything null here is not written. */
export class ConfirmPolicyDocumentDto {
@IsString() documentId!: string;
/** Required when creating a new Policy; ignored if `policyId` is set. */
@IsOptional() @IsString() customerId?: string;
/** Set when the document matched an existing Policy. */
@IsOptional() @IsString() policyId?: string;
@IsOptional() @IsString() policyNumber?: string;
@IsOptional() @IsString() insuredName?: string;
@IsOptional() @IsString() additionalInsured?: string;
@IsOptional() @IsString() agentName?: string;
@IsOptional() @IsString() legalAddress?: string;
@IsOptional() @IsString() zip?: string;
@IsOptional() @IsDateString() policyFrom?: string;
@IsOptional() @IsDateString() policyTo?: string;
@IsOptional() @IsDateString() policyDate?: string;
@IsOptional() @IsEnum(["MXN", "USD", "EUR"]) currency?: "MXN" | "USD" | "EUR";
@IsOptional() @IsNumber() netPremium?: number;
@IsOptional() @IsNumber() policyFee?: number;
@IsOptional() @IsNumber() brokerFee?: number;
@IsOptional() @IsNumber() total?: number;
@IsOptional() @IsString() premiumPayment?: string;
/** Coverages parsed off the PDF, passed through verbatim to Policy.coveragesJson. */
@IsOptional() @IsObject() coveragesJson?: unknown;
/** When true, write a Transaction(domain=INSURANCE, amount=-netPremium)
* in addition to creating/updating the Policy. Skipped if netPremium is
* null or zero. */
@IsOptional() postPremium?: boolean;
}
export class ConfirmPolicyBatchDto {
@IsArray()
@ValidateNested({ each: true })
@Type(() => ConfirmPolicyDocumentDto)
documents!: ConfirmPolicyDocumentDto[];
}
/** Staff correction of one document's extracted fields or its match. */
export class ReviewPolicyDocumentDto {
@IsOptional() @IsString() policyNumber?: string;
@IsOptional() @IsString() insuredName?: string;
@IsOptional() @IsString() additionalInsured?: string;
@IsOptional() @IsString() agentName?: string;
@IsOptional() @IsString() legalAddress?: string;
@IsOptional() @IsString() zip?: string;
@IsOptional() @IsDateString() policyFrom?: string;
@IsOptional() @IsDateString() policyTo?: string;
@IsOptional() @IsDateString() policyDate?: string;
@IsOptional() @IsString() currency?: string;
@IsOptional() @IsNumber() netPremium?: number;
@IsOptional() @IsNumber() policyFee?: number;
@IsOptional() @IsNumber() brokerFee?: number;
@IsOptional() @IsNumber() total?: number;
@IsOptional() @IsString() premiumPayment?: string;
@IsOptional() @IsObject() coveragesJson?: unknown;
/** Set by the reviewer when the document matched an existing Policy. */
@IsOptional() @IsString() matchedPolicyId?: string;
/** Set by the reviewer when creating a new Policy. */
@IsOptional() @IsString() matchedCustomerId?: string;
/** Force-confirm a doc even when the matcher left it ambiguous. */
@IsOptional() forceConfirm?: boolean;
}
export class CreatePolicyOcrBatchDto {
@IsOptional() @IsString() @MinLength(1) label?: string;
}
@@ -0,0 +1,18 @@
import { Module } from "@nestjs/common";
import { OcrModule } from "../ocr/ocr.module";
import { PolicyOcrController } from "./policy-ocr.controller";
import { PolicyOcrService } from "./policy-ocr.service";
import { PolicyMatcherService } from "./policy-matcher.service";
/**
* Reuses the OCR seam from OcrModule unchanged: the Tesseract provider is
* bound there and `OcrProvider` is the only thing the parsers touch. This
* module registers its own controller + service + matcher; nothing about
* utility ingestion needs to know about it.
*/
@Module({
imports: [OcrModule],
controllers: [PolicyOcrController],
providers: [PolicyOcrService, PolicyMatcherService],
})
export class PolicyOcrModule {}
@@ -0,0 +1,720 @@
import {
BadRequestException,
Inject,
Injectable,
Logger,
NotFoundException,
} from "@nestjs/common";
import { Currency, Prisma } from "@jorgecuadros/database";
import { PrismaService } from "../prisma/prisma.service";
import { StorageService } from "../storage/storage.service";
import type { UploadedFileLike } from "../storage/upload-file";
import { OCR_PROVIDER, type OcrPage, type OcrProvider } from "../statements/ocr/ocr.provider";
import { parsePolicy } from "./parsers/policy-parser";
import { PolicyMatcherService } from "./policy-matcher.service";
import type {
ConfirmPolicyBatchDto,
ConfirmPolicyDocumentDto,
ReviewPolicyDocumentDto,
} from "./policy-ocr.dto";
/**
* Insurance OCR intake — mirrors the statement pipeline at
* `apps/api/src/statements/statements.service.ts`. Reuses the OCR seam and
* Tesseract binding unchanged; the parsers and matcher are policy-specific.
*
* Why a parallel pipeline rather than a column on StatementDocument: the
* matcher keys on `Policy.policyNumber`, the confirm step writes to a
* different table (`Policy`, not `Transaction`), and the review UI shows
* different fields. Sharing one queue would either bloat the row with null
* columns or force the review screen to branch on a discriminator — both
* worse than a thin second table.
*/
@Injectable()
export class PolicyOcrService {
private readonly logger = new Logger(PolicyOcrService.name);
constructor(
private readonly prisma: PrismaService,
private readonly storage: StorageService,
private readonly matcher: PolicyMatcherService,
@Inject(OCR_PROVIDER) private readonly ocr: OcrProvider,
) {}
ocrAvailable(): Promise<boolean> {
return this.ocr.available();
}
storageAvailable(): boolean {
return this.storage.available;
}
// --- ingest ---------------------------------------------------------------
async createBatch(
files: UploadedFileLike[],
uploadedById: string,
label?: string,
) {
if (!files?.length) throw new BadRequestException("No se recibió ningún archivo.");
if (!(await this.ocr.available())) {
throw new BadRequestException(
"El servidor no tiene OCR instalado; no se pueden leer PDFs de pólizas.",
);
}
if (!this.storage.available) {
throw new BadRequestException(
"El almacenamiento de documentos no está configurado; no se pueden " +
"guardar los PDFs escaneados.",
);
}
const batch = await this.prisma.policyOcrBatch.create({
data: { provider: "GMX", uploadedById, label, fileCount: files.length },
});
const copies = files.map((f) => ({ buffer: f.buffer, name: f.originalname }));
void this.process(batch.id, copies).catch(async (err) => {
this.logger.error(`Policy OCR batch ${batch.id} failed: ${(err as Error).message}`);
await this.prisma.policyOcrBatch.update({
where: { id: batch.id },
data: { status: "FAILED", error: (err as Error).message },
});
});
return batch;
}
/**
* Render → text → parse → match, **one PolicyOcrDocument row per uploaded
* file**. The GMX certificate is a 2-page PDF where page 1 carries the
* contract header and page 2 carries the per-coverage table — both pages
* describe the SAME policy, so the parser concatenates them and the
* matcher runs once. `pageNumber` on the row is repurposed as the file
* ordinal within the batch (1, 2, 3…) — the unique constraint
* `(batchId, pageNumber)` still holds and lets a single batch carry many
* policies.
*
* The doc's `storageKey` is the SOURCE PDF (`policy-ocr/{batchId}/source-N.pdf`)
* rather than a rendered page image, so the review screen can embed the
* exact artifact the office received. The rendered page PNGs are still
* stored under `policy-ocr/{batchId}/page-M.png` for any future re-OCR or
* image-based audit, but they aren't used as `storageKey` for the document.
*/
private async process(
batchId: string,
files: { buffer: Buffer; name?: string }[],
) {
await this.prisma.policyOcrBatch.update({
where: { id: batchId },
data: { status: "PROCESSING" },
});
let fileOrdinal = 0;
let globalPageOrdinal = 0;
for (const file of files) {
fileOrdinal += 1;
const sourceKey = `policy-ocr/${batchId}/source-${fileOrdinal}.pdf`;
await this.storage.put(sourceKey, file.buffer, "application/pdf");
const pages = await this.ocr.renderPages(file.buffer);
const textLayer = await this.ocr.textPages(file.buffer).catch(() => []);
// One OcrPage per rendered page: text-layer wins when present (cheap,
// exact), OCR the rendered image when it isn't. Same precedence rule
// as the statement OCR pipeline.
const perPageOcr: OcrPage[] = [];
for (const [index, image] of pages.entries()) {
globalPageOrdinal += 1;
const pageStorageKey = `policy-ocr/${batchId}/page-${globalPageOrdinal}.png`;
await this.storage.put(pageStorageKey, image, "image/png");
const embedded = textLayer[index] ?? null;
const pageOcr = embedded ?? (await this.ocr.recognize(image));
perPageOcr.push(pageOcr);
}
// Concatenate every page's text with a blank line between pages so the
// parser's anchored regexes (^From$, ^Currency\s+...) still work
// across page boundaries — pdftotext -bbox-layout produces newline-
// separated text per page already, the `\n\n` just preserves a clear
// boundary in ocrRawText for debugging.
const mergedText = perPageOcr.map((p) => p.text).join("\n\n");
const avgConfidence =
perPageOcr.length === 0
? 0
: perPageOcr.reduce((s, p) => s + p.confidence, 0) / perPageOcr.length;
const synthetic: OcrPage = {
text: mergedText,
words: [],
confidence: avgConfidence,
};
try {
const parsed = parsePolicy(synthetic);
if (parsed.provider === "") {
throw new Error("no se reconoció el proveedor");
}
const match = await this.matcher.match(parsed);
const notes = [...parsed.notes, match.note].filter(Boolean);
// Confident when exactly one Policy carries the printed number —
// the only unambiguous hit we trust. A new policy (no match) still
// needs a customer pick, so it stays in review.
const trusted = match.confident && parsed.policyNumber != null;
await this.prisma.policyOcrDocument.create({
data: {
batchId,
pageNumber: fileOrdinal,
storageKey: sourceKey,
status: trusted ? "MATCHED" : "NEEDS_REVIEW",
ocrRawText: mergedText,
ocrConfidence: new Prisma.Decimal(avgConfidence.toFixed(3)),
provider: parsed.provider,
extractedPolicyNumber: parsed.policyNumber,
extractedInsuredName: parsed.insuredName,
extractedAdditionalInsured: parsed.additionalInsured,
extractedAgentName: parsed.agentName,
extractedLegalAddress: parsed.legalAddress,
extractedZip: parsed.zip,
extractedPolicyFrom: parsed.policyFrom,
extractedPolicyTo: parsed.policyTo,
extractedPolicyDate: parsed.policyDate,
extractedCurrency: parsed.currency,
extractedNetPremium:
parsed.netPremium != null ? new Prisma.Decimal(parsed.netPremium) : null,
extractedPolicyFee:
parsed.policyFee != null ? new Prisma.Decimal(parsed.policyFee) : null,
extractedBrokerFee:
parsed.brokerFee != null ? new Prisma.Decimal(parsed.brokerFee) : null,
extractedTotal:
parsed.total != null ? new Prisma.Decimal(parsed.total) : null,
extractedCoveragesJson: parsed.coverages.length
? (parsed.coverages as unknown as Prisma.InputJsonValue)
: Prisma.DbNull,
extractedPremiumPayment: parsed.premiumPayment,
matchedPolicyId: match.policyId,
matchedCustomerId: match.customerId,
matchCandidates: match.candidates.length
? (match.candidates as unknown as Prisma.InputJsonValue)
: Prisma.DbNull,
matchNote: notes.join("; ").slice(0, 190),
},
});
} catch (err) {
// The file as a whole failed to parse (no provider, parse exception).
// One OCR_FAILED row per file is the right granularity — the page
// images are still on disk for a re-run after a parser fix.
await this.prisma.policyOcrDocument.create({
data: {
batchId,
pageNumber: fileOrdinal,
storageKey: sourceKey,
status: "OCR_FAILED",
matchNote: (err as Error).message.slice(0, 190),
},
});
}
}
await this.prisma.policyOcrBatch.update({
where: { id: batchId },
data: { status: "READY_FOR_REVIEW" },
});
}
// --- reads ----------------------------------------------------------------
async listBatches(page: number, pageSize: number) {
const [total, items] = await this.prisma.$transaction([
this.prisma.policyOcrBatch.count(),
this.prisma.policyOcrBatch.findMany({
orderBy: { createdAt: "desc" },
skip: (page - 1) * pageSize,
take: pageSize,
include: {
uploadedBy: { select: { name: true } },
_count: { select: { documents: true } },
},
}),
]);
return { items, total, page, pageSize, pageCount: Math.ceil(total / pageSize) };
}
async getBatch(id: string) {
const batch = await this.prisma.policyOcrBatch.findUnique({
where: { id },
include: { uploadedBy: { select: { name: true } } },
});
if (!batch) throw new NotFoundException("Lote no encontrado.");
const counts = await this.prisma.policyOcrDocument.groupBy({
by: ["status"],
where: { batchId: id },
_count: { _all: true },
});
return {
...batch,
byStatus: Object.fromEntries(counts.map((c) => [c.status, c._count._all])),
};
}
async listDocuments(batchId: string) {
return this.prisma.policyOcrDocument.findMany({
where: { batchId },
orderBy: { pageNumber: "asc" },
include: {
matchedCustomer: { select: { id: true, name: true } },
matchedPolicy: {
select: {
id: true,
policyNumber: true,
customerId: true,
customer: { select: { name: true } },
},
},
},
});
}
/**
* The source PDF for the document, so the review screen can show the
* exact artifact the office uploaded (the browser's PDF viewer handles
* scrolling, zoom, and selection natively). The rendered page PNGs
* remain on disk under `policy-ocr/{batchId}/page-N.png` for any
* future re-OCR, but the doc row points here at the source.
*/
async pageImage(documentId: string) {
const doc = await this.prisma.policyOcrDocument.findUnique({
where: { id: documentId },
select: { storageKey: true },
});
if (!doc) throw new NotFoundException("Documento no encontrado.");
return this.storage.getStream(doc.storageKey);
}
// --- review ---------------------------------------------------------------
async review(id: string, dto: ReviewPolicyDocumentDto, reviewedById: string) {
const doc = await this.prisma.policyOcrDocument.findUnique({ where: { id } });
if (!doc) throw new NotFoundException("Documento no encontrado.");
if (doc.status === "POSTED") {
throw new BadRequestException("Este documento ya fue aplicado.");
}
// Trusting a customer-supplied pair (policyId, customerId) without
// cross-check is how a document lands on the wrong customer's ledger;
// pin them here from the DB.
let matchedPolicyId = dto.matchedPolicyId ?? doc.matchedPolicyId;
let matchedCustomerId = doc.matchedCustomerId;
if (matchedPolicyId) {
const p = await this.prisma.policy.findUnique({
where: { id: matchedPolicyId },
select: { customerId: true },
});
if (!p) throw new BadRequestException("Póliza no encontrada.");
matchedCustomerId = p.customerId;
} else if (dto.matchedCustomerId) {
const c = await this.prisma.customer.findUnique({
where: { id: dto.matchedCustomerId },
select: { id: true },
});
if (!c) throw new BadRequestException("Cliente no encontrado.");
matchedCustomerId = c.id;
}
return this.prisma.policyOcrDocument.update({
where: { id },
data: {
extractedPolicyNumber: dto.policyNumber ?? undefined,
extractedInsuredName: dto.insuredName ?? undefined,
extractedAdditionalInsured: dto.additionalInsured ?? undefined,
extractedAgentName: dto.agentName ?? undefined,
extractedLegalAddress: dto.legalAddress ?? undefined,
extractedZip: dto.zip ?? undefined,
extractedPolicyFrom: dto.policyFrom ? new Date(dto.policyFrom) : undefined,
extractedPolicyTo: dto.policyTo ? new Date(dto.policyTo) : undefined,
extractedPolicyDate: dto.policyDate ? new Date(dto.policyDate) : undefined,
extractedCurrency: dto.currency ?? undefined,
extractedNetPremium:
dto.netPremium != null ? new Prisma.Decimal(dto.netPremium) : undefined,
extractedPolicyFee:
dto.policyFee != null ? new Prisma.Decimal(dto.policyFee) : undefined,
extractedBrokerFee:
dto.brokerFee != null ? new Prisma.Decimal(dto.brokerFee) : undefined,
extractedTotal:
dto.total != null ? new Prisma.Decimal(dto.total) : undefined,
extractedCoveragesJson: dto.coveragesJson
? (dto.coveragesJson as Prisma.InputJsonValue)
: undefined,
extractedPremiumPayment: dto.premiumPayment ?? undefined,
matchedPolicyId,
matchedCustomerId,
status: dto.forceConfirm ? "CONFIRMED" : "MATCHED",
reviewedById,
reviewedAt: new Date(),
},
});
}
async reject(id: string, reviewedById: string) {
const doc = await this.prisma.policyOcrDocument.findUnique({ where: { id } });
if (!doc) throw new NotFoundException("Documento no encontrado.");
if (doc.status === "POSTED") {
throw new BadRequestException("Este documento ya fue aplicado.");
}
return this.prisma.policyOcrDocument.update({
where: { id },
data: { status: "REJECTED", reviewedById, reviewedAt: new Date() },
});
}
// --- confirm --------------------------------------------------------------
/**
* Apply every confirmed document: create or update the Policy, attach the
* source PDF as a PolicyDocument, and (when staff asked + premium parses)
* write a Transaction row. Each step is guarded by status checks so a
* double-confirm cannot re-apply a document.
*/
async confirmBatch(batchId: string, dto: ConfirmPolicyBatchDto, reviewedById: string) {
const batch = await this.prisma.policyOcrBatch.findUnique({ where: { id: batchId } });
if (!batch) throw new NotFoundException("Lote no encontrado.");
const results: { documentId: string; policyId: string; postedTransactionId: string | null }[] = [];
for (const item of dto.documents) {
const doc = await this.prisma.policyOcrDocument.findUnique({
where: { id: item.documentId },
});
if (!doc) {
throw new BadRequestException(`Documento ${item.documentId} no encontrado.`);
}
if (doc.status === "POSTED") {
throw new BadRequestException(
`El documento página ${doc.pageNumber} ya fue aplicado.`,
);
}
if (!item.policyId && !item.customerId) {
throw new BadRequestException(
`Documento página ${doc.pageNumber}: falta póliza destino o cliente.`,
);
}
// 1. Resolve target Policy (create or update). Field selection: every
// non-null `extracted*` on the doc (post-review) is written. Null is
// preserved — never overwrite an existing Policy's `netPremium` with
// null because the certificate page didn't carry one.
let policyId = item.policyId ?? null;
if (policyId) {
const updateData = buildPolicyUpdateFromDoc(item, doc);
await this.prisma.policy.update({
where: { id: policyId },
data: updateData,
});
} else {
// Create under the picked customer. `policyNumber` is the only field
// that must be present.
if (!item.policyNumber && !doc.extractedPolicyNumber) {
throw new BadRequestException(
`Documento página ${doc.pageNumber}: falta número de póliza.`,
);
}
const createData = buildPolicyCreateFromDoc(item, doc, item.customerId!);
const created = await this.prisma.policy.create({
data: createData,
});
policyId = created.id;
}
// 2. Attach the source PDF as a PolicyDocument. `doc.storageKey`
// already points at the exact upload (`policy-ocr/{batchId}/source-N.pdf`)
// so the attach is just a stream copy into the policy's namespace —
// the previous per-page "which file did this page come from" walk is
// gone because one PDF = one doc now.
await this.attachSourcePdf(doc.storageKey, policyId);
// 3. Optionally post the premium to the ledger. Only when staff
// explicitly asked (`postPremium` true) and netPremium parses — without
// that gate a missing premium would silently book $0.
let postedTransactionId: string | null = null;
const premium =
item.netPremium != null
? item.netPremium
: doc.extractedNetPremium != null
? Number(doc.extractedNetPremium)
: null;
if (item.postPremium && premium && premium > 0) {
const tx = await this.prisma.transaction.create({
data: {
customerId: (await this.policyCustomerId(policyId))!,
domain: "INSURANCE",
amount: new Prisma.Decimal(-Math.abs(premium)),
transactionDate: doc.extractedPolicyDate ?? doc.extractedPolicyFrom ?? new Date(),
currency: (item.currency ??
doc.extractedCurrency ??
"MXN") as Currency,
reference: item.policyNumber ?? doc.extractedPolicyNumber ?? null,
period: null,
captureSource: "OCR",
captureRef: doc.id,
message: `Prima de póliza ${item.policyNumber ?? doc.extractedPolicyNumber ?? ""}`,
},
});
postedTransactionId = tx.id;
}
await this.prisma.policyOcrDocument.update({
where: { id: doc.id },
data: {
status: "POSTED",
matchedPolicyId: policyId,
reviewedById,
reviewedAt: new Date(),
createdPolicyId: item.policyId ? null : policyId,
postedTransactionId,
},
});
results.push({
documentId: doc.id,
policyId,
postedTransactionId,
});
}
await this.closeIfDone(batchId);
return {
applied: results.length,
policies: results.map((r) => r.policyId),
postedTransactions: results.filter((r) => r.postedTransactionId).length,
};
}
/**
* Stream the source PDF (`sourceKey`, set by `process` on the doc row)
* into the policy's storage namespace and create a `PolicyDocument`
* pointer. Trivial now that the doc row holds the exact source key —
* the old per-page "which file did this page come from" walk is gone.
*/
private async attachSourcePdf(sourceKey: string, policyId: string): Promise<void> {
const got = await this.storage.getStream(sourceKey);
const chunks: Buffer[] = [];
for await (const c of got.stream) chunks.push(c as Buffer);
const buf = Buffer.concat(chunks);
const newKey = `policy/${policyId}/${Date.now()}-${crypto.randomUUID()}.pdf`;
await this.storage.put(newKey, buf, "application/pdf");
await this.prisma.policyDocument.create({
data: {
policyId,
documentType: "GMX_POLICY",
storageKey: newKey,
},
});
}
private async policyCustomerId(policyId: string): Promise<string | null> {
const p = await this.prisma.policy.findUnique({
where: { id: policyId },
select: { customerId: true },
});
return p?.customerId ?? null;
}
private async closeIfDone(batchId: string) {
const open = await this.prisma.policyOcrDocument.count({
where: {
batchId,
status: { in: ["PENDING_OCR", "NEEDS_REVIEW", "MATCHED", "CONFIRMED"] },
},
});
if (open === 0) {
await this.prisma.policyOcrBatch.update({
where: { id: batchId },
data: { status: "COMPLETED", completedAt: new Date() },
});
}
}
}
/** Map a (post-review) doc + final confirmed fields onto a `Policy.update`
* payload. Every field that is null in both inputs is omitted so we never
* write null over a value the Policy already carries (the GMX certificate
* has no premium — we must not blank the existing Policy.netPremium). */
function buildPolicyUpdateFromDoc(
item: ConfirmPolicyDocumentDto,
doc: {
extractedPolicyNumber: string | null;
extractedInsuredName: string | null;
extractedAdditionalInsured: string | null;
extractedAgentName: string | null;
extractedLegalAddress: string | null;
extractedZip: string | null;
extractedPolicyFrom: Date | null;
extractedPolicyTo: Date | null;
extractedPolicyDate: Date | null;
extractedCurrency: string | null;
extractedNetPremium: Prisma.Decimal | null;
extractedPolicyFee: Prisma.Decimal | null;
extractedBrokerFee: Prisma.Decimal | null;
extractedTotal: Prisma.Decimal | null;
extractedCoveragesJson: Prisma.JsonValue | null;
extractedPremiumPayment: string | null;
},
): Prisma.PolicyUpdateInput {
const numOrUndef = (a: number | undefined, b: Prisma.Decimal | null): Prisma.Decimal | undefined => {
if (a != null) return new Prisma.Decimal(a);
if (b != null) return b;
return undefined;
};
const dateOrUndef = (a: string | undefined, b: Date | null): Date | undefined => {
if (a) return new Date(a);
if (b) return b;
return undefined;
};
const strOrUndef = (a: string | undefined, b: string | null): string | undefined => {
if (a != null && a !== "") return a;
if (b != null && b !== "") return b;
return undefined;
};
return {
policyNumber: strOrUndef(item.policyNumber, doc.extractedPolicyNumber),
agentName: strOrUndef(item.agentName, doc.extractedAgentName),
policyFrom: dateOrUndef(item.policyFrom, doc.extractedPolicyFrom),
policyTo: dateOrUndef(item.policyTo, doc.extractedPolicyTo),
policyDate: dateOrUndef(item.policyDate, doc.extractedPolicyDate),
currency: strOrUndef(item.currency, doc.extractedCurrency) as Currency | undefined,
netPremium: numOrUndef(item.netPremium, doc.extractedNetPremium),
policyFee: numOrUndef(item.policyFee, doc.extractedPolicyFee),
brokerFee: numOrUndef(item.brokerFee, doc.extractedBrokerFee),
total: numOrUndef(item.total, doc.extractedTotal),
// coveragesJson / observations: freeform, keep the GMX data when present.
coveragesJson:
item.coveragesJson !== undefined
? (item.coveragesJson as Prisma.InputJsonValue)
: doc.extractedCoveragesJson != null
? (doc.extractedCoveragesJson as Prisma.InputJsonValue)
: undefined,
// Premium payment cadence ("CONTADO") and insured-name fields land in
// `observations` so the PolicyForm's edits stay the source of truth for
// structured fields. The reviewer can move them by hand if needed.
observations: joinObservations(
doc.extractedInsuredName,
doc.extractedAdditionalInsured,
doc.extractedLegalAddress,
doc.extractedZip,
doc.extractedPremiumPayment,
item,
),
};
}
/** Same shape as `buildPolicyUpdateFromDoc`, but for `Policy.create`. The
* `customerId` is supplied separately and `policyNumber` is required (a
* Policy without a number can't be re-matched by the OCR pipeline). */
function buildPolicyCreateFromDoc(
item: ConfirmPolicyDocumentDto,
doc: {
extractedPolicyNumber: string | null;
extractedInsuredName: string | null;
extractedAdditionalInsured: string | null;
extractedAgentName: string | null;
extractedLegalAddress: string | null;
extractedZip: string | null;
extractedPolicyFrom: Date | null;
extractedPolicyTo: Date | null;
extractedPolicyDate: Date | null;
extractedCurrency: string | null;
extractedNetPremium: Prisma.Decimal | null;
extractedPolicyFee: Prisma.Decimal | null;
extractedBrokerFee: Prisma.Decimal | null;
extractedTotal: Prisma.Decimal | null;
extractedCoveragesJson: Prisma.JsonValue | null;
extractedPremiumPayment: string | null;
},
customerId: string,
): Prisma.PolicyUncheckedCreateInput {
const numOrUndef = (a: number | undefined, b: Prisma.Decimal | null): Prisma.Decimal | undefined => {
if (a != null) return new Prisma.Decimal(a);
if (b != null) return b;
return undefined;
};
const dateOrUndef = (a: string | undefined, b: Date | null): Date | undefined => {
if (a) return new Date(a);
if (b) return b;
return undefined;
};
const strOrUndef = (a: string | undefined, b: string | null): string | undefined => {
if (a != null && a !== "") return a;
if (b != null && b !== "") return b;
return undefined;
};
const policyNumber =
strOrUndef(item.policyNumber, doc.extractedPolicyNumber);
if (!policyNumber) {
// Caller already guards this; the throw is a type-narrowing aid.
throw new Error("policyNumber required for create");
}
return {
policyNumber,
customerId,
agentName: strOrUndef(item.agentName, doc.extractedAgentName),
policyFrom: dateOrUndef(item.policyFrom, doc.extractedPolicyFrom),
policyTo: dateOrUndef(item.policyTo, doc.extractedPolicyTo),
policyDate: dateOrUndef(item.policyDate, doc.extractedPolicyDate),
currency: strOrUndef(item.currency, doc.extractedCurrency) as Currency | undefined,
netPremium: numOrUndef(item.netPremium, doc.extractedNetPremium),
policyFee: numOrUndef(item.policyFee, doc.extractedPolicyFee),
brokerFee: numOrUndef(item.brokerFee, doc.extractedBrokerFee),
total: numOrUndef(item.total, doc.extractedTotal),
coveragesJson:
item.coveragesJson !== undefined
? (item.coveragesJson as Prisma.InputJsonValue)
: doc.extractedCoveragesJson != null
? (doc.extractedCoveragesJson as Prisma.InputJsonValue)
: undefined,
observations: joinObservations(
doc.extractedInsuredName,
doc.extractedAdditionalInsured,
doc.extractedLegalAddress,
doc.extractedZip,
doc.extractedPremiumPayment,
item,
),
};
}
function joinObservations(
insured: string | null,
additional: string | null,
address: string | null,
zip: string | null,
premiumPayment: string | null,
item: ConfirmPolicyDocumentDto,
): string | undefined {
const lines: string[] = [];
const insuredName = strOrUndefDb(item.insuredName, insured);
if (insuredName) lines.push(`Asegurado: ${insuredName}`);
const additionalInsured = strOrUndefDb(item.additionalInsured, additional);
if (additionalInsured) lines.push(`Asegurado adicional: ${additionalInsured}`);
const legalAddress = strOrUndefDb(item.legalAddress, address);
if (legalAddress) lines.push(`Dirección: ${legalAddress}`);
const zipVal = strOrUndefDb(item.zip, zip);
if (zipVal) lines.push(`C.P.: ${zipVal}`);
const cadence = strOrUndefDb(item.premiumPayment, premiumPayment);
if (cadence) lines.push(`Pago de prima: ${cadence}`);
return lines.length ? lines.join("\n") : undefined;
}
function strOrUndefDb(a: string | undefined, b: string | null): string | undefined {
if (a != null && a !== "") return a;
if (b != null && b !== "") return b;
return undefined;
}
@@ -0,0 +1,71 @@
/**
* The OCR seam. Everything above this interface works in terms of page text and
* word boxes, so the concrete engine is swappable without touching the parsers,
* the matcher, or the schema.
*
* The shipped implementation is self-hosted Tesseract (see tesseract.provider).
* That choice is evidence-based rather than assumed: run against 46 pages of
* real scanned CFE, CESPT and Telnor statements, it identified the provider on
* 46/46 and extracted a usable account reference on 43/46, and on a later
* corpus of 19 scanned municipal predial receipts it read the provider on
* 19/19 and an identifier on 18/19 — well past the bar for a queue whose whole
* point is that a human confirms every row. A
* managed document-extraction API (Textract, Document Intelligence, Document
* AI) fits behind this same interface if per-page accuracy ever proves
* insufficient, with no schema change — but at 300+ pages/month/company it
* would carry a real recurring cost for accuracy that is not currently the
* bottleneck.
*/
/** One OCR'd word, with where it sits on the page. */
export interface OcrWord {
text: string;
/** Pixel box in the rendered page image. */
left: number;
top: number;
width: number;
height: number;
/** Engine confidence for this word, 0..1. */
confidence: number;
}
export interface OcrPage {
/** Full page text, reading order, newline-separated. */
text: string;
/**
* Word boxes. Needed because several of the real layouts are *tables* — the
* CESPT "RECIBO" prints `No. DE CUENTA` as a column header with the value in
* the row beneath it, which line-oriented text cannot associate. Parsers fall
* back to geometry for exactly those fields.
*/
words: OcrWord[];
/** Mean word confidence across the page, 0..1. */
confidence: number;
}
export interface OcrProvider {
/** True when the engine is actually usable in this deployment. */
available(): Promise<boolean>;
/** Split a PDF into one rendered page image per page. */
renderPages(pdf: Buffer): Promise<Buffer[]>;
/** OCR a single rendered page image. */
recognize(pageImage: Buffer): Promise<OcrPage>;
/**
* Read a PDF's own text layer, one entry per page, `null` where the page has
* none worth using.
*
* Not every statement is a scan. The gas company e-mails born-digital CFDI
* invoices whose text is already exact and already positioned — running those
* through a rasteriser and a character recogniser can only lose information
* (one sample turned `MEDIDOR: VM01014426` into `ar (LTR): 014420`) while
* costing about a minute of CPU per page for the privilege. Where the layer
* exists it is strictly better input for the same parsers, so it is tried
* first and OCR remains the fallback for genuine scans.
*
* Positions are reported in the same pixel space `recognize` uses, so the
* geometric helpers in the parsers work unchanged on either source.
*/
textPages(pdf: Buffer): Promise<(OcrPage | null)[]>;
}
export const OCR_PROVIDER = Symbol("OCR_PROVIDER");
@@ -0,0 +1,67 @@
import { parseBboxLayout } from "./tesseract.provider";
/**
* Shaped like real `pdftotext -bbox-layout` output: the gas invoice lays its
* header out as two columns of independent text flows, so poppler puts a label
* and the value printed beside it in *different* `<line>` elements. Trusting
* that grouping is what left `PERIODO FACTURADO` with no value next to it and
* every period field empty on a batch whose text was perfectly readable.
*/
function word(x: number, y: number, text: string): string {
return `<word xMin="${x}" yMin="${y}" xMax="${x + 20}" yMax="${y + 8}">${text}</word>`;
}
function doc(...lines: string[]): string {
return `<doc><page width="612" height="792">${lines
.map((l) => `<flow><block><line>${l}</line></block></flow>`)
.join("")}</page></doc>`;
}
/** Enough words on the page to clear the "is this a real text layer" floor. */
function padding(): string {
return Array.from({ length: 50 }, (_, i) => word(10, 400 + i * 10, `w${i}`)).join("");
}
describe("parseBboxLayout", () => {
it("rejoins a label with the value printed beside it in another flow", () => {
const [page] = parseBboxLayout(
doc(
word(20, 100, "PERIODO") + word(45, 100, "FACTURADO:"),
word(300, 100.4, "20260630-20260630"),
padding(),
),
1,
);
expect(page).not.toBeNull();
expect(page!.text).toContain("PERIODO FACTURADO: 20260630-20260630");
});
it("keeps genuinely separate lines apart", () => {
const [page] = parseBboxLayout(
doc(word(20, 100, "Cuenta:") + word(80, 100, "0900003463"), word(20, 130, "Nombre:"), padding()),
1,
);
expect(page!.text.split("\n")).toContain("Cuenta: 0900003463");
expect(page!.text.split("\n")).toContain("Nombre:");
});
it("scales point coordinates into the render's pixel space", () => {
// Word boxes have to land in the same coordinate space tesseract reports,
// or the geometric helpers the parsers share silently stop finding values.
const [page] = parseBboxLayout(doc(word(72, 144, "X") + padding()), 300 / 72);
const x = page!.words.find((w) => w.text === "X")!;
expect(x.left).toBeCloseTo(300);
expect(x.top).toBeCloseTo(600);
});
it("reports no text layer for a scan carrying a few stray glyphs", () => {
expect(parseBboxLayout(doc(word(10, 10, "3") + word(40, 10, "of") + word(60, 10, "5")), 1)).toEqual([
null,
]);
});
it("decodes the entities poppler escapes", () => {
const [page] = parseBboxLayout(doc(word(10, 10, "A&amp;B") + padding()), 1);
expect(page!.text).toContain("A&B");
});
});
@@ -0,0 +1,350 @@
import { Injectable, Logger, ServiceUnavailableException } from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { execFile } from "node:child_process";
import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { promisify } from "node:util";
import type { OcrPage, OcrProvider, OcrWord } from "./ocr.provider";
const run = promisify(execFile);
/**
* Self-hosted OCR: `pdftoppm` (poppler) to rasterise, `tesseract` to read.
*
* Both are external binaries rather than a native npm addon, which keeps the
* pnpm workspace free of a compiled dependency and makes the alpine runtime
* image a two-package change (see docker/api.Dockerfile). Like StorageService,
* a missing binary degrades rather than crashes the API: the module reports
* itself unavailable and statement ingest returns 503, while every other
* feature keeps working.
*
* The settings below are not arbitrary — they were measured against the real
* scanned samples:
* - 300 DPI grayscale. The source scans are phone photos of paper at ~5MB a
* page; below 300 the small print (RMU, clave catastral) stops resolving,
* above it costs time for no additional fields.
* - `--psm 6` ("assume a single uniform block of text"). The default page
* segmentation splits these dense forms into columns and interleaves them,
* which destroys the label-then-value adjacency every parser depends on.
* - Spanish traineddata, with a graceful fall back to English if the language
* pack is absent — an accented label reads worse but the digits, which are
* what actually gets matched, are unaffected.
*/
@Injectable()
export class TesseractOcrProvider implements OcrProvider {
private readonly logger = new Logger(TesseractOcrProvider.name);
private readonly dpi: number;
private readonly lang: string;
private probe: Promise<boolean> | null = null;
constructor(config: ConfigService) {
this.dpi = Number(config.get("OCR_DPI") ?? 300);
this.lang = config.get<string>("OCR_LANG") ?? "spa";
}
/** Cached — the binaries do not appear or vanish while the process runs. */
available(): Promise<boolean> {
if (!this.probe) {
this.probe = (async () => {
try {
await Promise.all([
run("tesseract", ["--version"]),
run("pdftoppm", ["-v"]),
]);
return true;
} catch {
this.logger.warn(
"OCR unavailable: `tesseract` and/or `pdftoppm` not found on PATH. " +
"Statement ingest is disabled; every other feature is unaffected.",
);
return false;
}
})();
}
return this.probe;
}
private async require(): Promise<void> {
if (!(await this.available())) {
throw new ServiceUnavailableException(
"El servicio de OCR no está disponible en este servidor.",
);
}
}
private async scratch<T>(fn: (dir: string) => Promise<T>): Promise<T> {
const dir = await mkdtemp(join(tmpdir(), "stmt-ocr-"));
try {
return await fn(dir);
} finally {
await rm(dir, { recursive: true, force: true });
}
}
async renderPages(pdf: Buffer): Promise<Buffer[]> {
await this.require();
return this.scratch(async (dir) => {
const src = join(dir, "in.pdf");
await writeFile(src, pdf);
// -gray: these are grayscale scans already; colour triples the bytes
// handed to tesseract for no gain in character recognition.
await run("pdftoppm", [
"-r",
String(this.dpi),
"-gray",
"-png",
src,
join(dir, "page"),
]);
const files = (await readdir(dir))
.filter((f) => f.startsWith("page") && f.endsWith(".png"))
// pdftoppm zero-pads its page numbers, so lexical order is page order.
.sort();
return Promise.all(files.map((f) => readFile(join(dir, f))));
});
}
/**
* `pdftotext -bbox-layout` — the same poppler package `pdftoppm` comes from,
* so this costs no extra dependency in the runtime image.
*
* A page is only accepted when it carries a real text layer. Scanned PDFs
* frequently contain a handful of stray glyphs (a scanner watermark, a page
* number stamped by the MFP), and treating those as the page's text would
* hand every parser an almost-empty string and silently take OCR out of the
* loop — so a floor of MIN_TEXT_WORDS words has to be present before the
* layer is believed.
*/
async textPages(pdf: Buffer): Promise<(OcrPage | null)[]> {
await this.require();
return this.scratch(async (dir) => {
const src = join(dir, "in.pdf");
await writeFile(src, pdf);
const out = join(dir, "out.html");
try {
await run("pdftotext", ["-bbox-layout", src, out]);
} catch (err) {
this.logger.warn(
`pdftotext failed; falling back to OCR for this file: ${(err as Error).message}`,
);
return [];
}
// Points to pixels at the render DPI, so word boxes from either source
// land in one coordinate space and `valueUnder`'s thresholds hold.
return parseBboxLayout(await readFile(out, "utf8"), this.dpi / 72);
});
}
async recognize(pageImage: Buffer): Promise<OcrPage> {
await this.require();
return this.scratch(async (dir) => {
const img = join(dir, "page.png");
await writeFile(img, pageImage);
// One tesseract invocation produces both outputs; TSV carries the word
// boxes and per-word confidence, and its text can be reassembled into
// reading order, so there is no need to run the engine twice.
const out = join(dir, "out");
try {
await run("tesseract", [img, out, "-l", this.lang, "--psm", "6", "tsv"]);
} catch (err) {
if (this.lang !== "eng") {
this.logger.warn(
`Tesseract failed with lang "${this.lang}", retrying with "eng": ${
(err as Error).message
}`,
);
await run("tesseract", [img, out, "-l", "eng", "--psm", "6", "tsv"]);
} else {
throw err;
}
}
const tsv = await readFile(`${out}.tsv`, "utf8");
return parseTsv(tsv);
});
}
}
/**
* Below this many words a "text layer" is scanner debris, not a document.
* The real born-digital samples carry 400+ words a page; the scanned ones
* carry none at all, so the exact threshold is not delicate.
*/
const MIN_TEXT_WORDS = 40;
const ENTITIES: Record<string, string> = {
amp: "&",
lt: "<",
gt: ">",
quot: '"',
apos: "'",
};
function decodeEntities(s: string): string {
return s.replace(/&(#x?[0-9a-fA-F]+|[a-z]+);/g, (whole, body: string) => {
if (body[0] === "#") {
const code =
body[1] === "x" || body[1] === "X"
? parseInt(body.slice(2), 16)
: parseInt(body.slice(1), 10);
return Number.isFinite(code) ? String.fromCodePoint(code) : whole;
}
return ENTITIES[body] ?? whole;
});
}
/**
* Turn `pdftotext -bbox-layout`'s XHTML into one OcrPage per PDF page.
*
* Parsed with regexes rather than an XML library on purpose: the output is
* machine-generated by poppler with a fixed element shape (`page` > `flow` >
* `block` > `line` > `word`), and the alternative is a parser dependency in
* the API for one file format read in one place. Only `page` and `word` are
* consulted — see below for why poppler's own `line` grouping is discarded.
*
* `confidence` is 1 for every word: these are the document's own characters,
* not a recognition guess.
*/
export function parseBboxLayout(xhtml: string, scale: number): (OcrPage | null)[] {
const pages: (OcrPage | null)[] = [];
for (const pageMatch of xhtml.matchAll(/<page\b[^>]*>([\s\S]*?)<\/page>/g)) {
const words: OcrWord[] = [];
for (const w of pageMatch[1].matchAll(
/<word\s+xMin="([\d.eE+-]+)"\s+yMin="([\d.eE+-]+)"\s+xMax="([\d.eE+-]+)"\s+yMax="([\d.eE+-]+)"\s*>([\s\S]*?)<\/word>/g,
)) {
const text = decodeEntities(w[5]).trim();
if (!text) continue;
const left = Number(w[1]) * scale;
const top = Number(w[2]) * scale;
words.push({
text,
left,
top,
width: Number(w[3]) * scale - left,
height: Number(w[4]) * scale - top,
confidence: 1,
});
}
pages.push(
words.length >= MIN_TEXT_WORDS
? { text: toVisualRows(words), words, confidence: 1 }
: null,
);
}
return pages;
}
/**
* Reassemble words into the rows a reader sees, left to right.
*
* Poppler's own `<line>` grouping cannot be used for this. It groups by text
* flow, and these invoices lay their fields out as two columns of independent
* flows — so `PERIODO FACTURADO:` and the `20260630-20260630` printed beside
* it end up in different `<line>` elements, and every label-then-value pattern
* in the parsers misses a value that is plainly there on the page. Regrouping
* by vertical position restores the adjacency, and matches what tesseract
* hands back for the scanned version of the same layout.
*
* Rows are cut when a word's vertical centre leaves the band established by
* the row's first word, which tolerates the sub-pixel baseline differences
* between fonts on one line without merging two genuinely separate lines.
*/
function toVisualRows(words: OcrWord[]): string {
const centre = (w: OcrWord) => w.top + w.height / 2;
const sorted = [...words].sort((a, b) => centre(a) - centre(b) || a.left - b.left);
const rows: OcrWord[][] = [];
let current: OcrWord[] = [];
let band = 0;
for (const w of sorted) {
if (!current.length) {
current = [w];
band = centre(w);
continue;
}
// Half the word's own height: tall headings and body text both sit within
// their own line's band, and neither reaches into the next one.
if (Math.abs(centre(w) - band) <= Math.max(w.height, current[0].height) / 2) {
current.push(w);
} else {
rows.push(current);
current = [w];
band = centre(w);
}
}
if (current.length) rows.push(current);
return rows
.map((r) =>
[...r]
.sort((a, b) => a.left - b.left)
.map((w) => w.text)
.join(" "),
)
.join("\n");
}
/**
* Turn tesseract's TSV into words plus reassembled text.
*
* Columns are: level, page_num, block_num, par_num, line_num, word_num, left,
* top, width, height, conf, text. Rows with level < 5 are structural (page,
* block, paragraph, line) and carry no text; only level 5 is a word. A conf of
* -1 marks a structural row, so those are dropped rather than averaged in —
* including them would drag every page's confidence toward zero.
*/
export function parseTsv(tsv: string): OcrPage {
const lines = tsv.split("\n");
const header = lines[0]?.split("\t") ?? [];
const col = (name: string) => header.indexOf(name);
const iLeft = col("left");
const iTop = col("top");
const iWidth = col("width");
const iHeight = col("height");
const iConf = col("conf");
const iText = col("text");
const iLine = col("line_num");
const iBlock = col("block_num");
const words: OcrWord[] = [];
// Keyed by block+line so the reassembled text preserves the engine's own
// reading order instead of sorting words by raw y, which interleaves columns.
const byLine = new Map<string, string[]>();
for (let i = 1; i < lines.length; i++) {
const f = lines[i].split("\t");
if (f.length <= iText) continue;
const text = f[iText]?.trim();
if (!text) continue;
const confidence = Number(f[iConf]);
if (!Number.isFinite(confidence) || confidence < 0) continue;
words.push({
text,
left: Number(f[iLeft]) || 0,
top: Number(f[iTop]) || 0,
width: Number(f[iWidth]) || 0,
height: Number(f[iHeight]) || 0,
confidence: confidence / 100,
});
const key = `${f[iBlock]}:${f[iLine]}`;
const bucket = byLine.get(key);
if (bucket) bucket.push(text);
else byLine.set(key, [text]);
}
const text = [...byLine.values()].map((w) => w.join(" ")).join("\n");
const confidence = words.length
? words.reduce((sum, w) => sum + w.confidence, 0) / words.length
: 0;
return { text, words, confidence };
}
@@ -0,0 +1,270 @@
import type { OcrPage } from "../ocr/ocr.provider";
import {
detectProvider,
normalizeCadastralKey,
normalizeZofematKey,
parseStatement,
} from "./statement-parser";
/**
* Every string in this file is a verbatim excerpt of what the OCR engine
* actually returned for a real receipt — misreads, dropped spaces, mangled
* accents and all. That is the point: these are the specific ways these five
* layouts have been observed to fail, and the assertions pin down what the
* parser is supposed to do about each one. Inventing clean input here would
* test nothing, because clean input was never the problem.
*/
function page(text: string): OcrPage {
return { text, words: [], confidence: 0.9 };
}
describe("detectProvider", () => {
it("reads a Rosarito predial receipt as predial, not as a water bill", () => {
// "Clave Catastral" is also a CESPT structural marker, so a predial page
// whose header OCR'd badly must still not be claimed by the CESPT rule.
expect(
detectProvider(
"e | Clave Catastral. KP-128-105 IMPUESTO PREDIAL ea rita\n" +
"TASA | VALOR FISCAL | BIMESTRES | INCISO. | IMPUESTO",
),
).toBe("PREDIAL ROSARITO");
});
it("keeps telling the three municipalities apart by their RFC", () => {
expect(detectProvider("R.F.C. ATB-541201-KK2")).toBe("PREDIAL TIJUANA");
expect(detectProvider("R.F.C. AMP-981201-HJ4")).toBe("PREDIAL ROSARITO");
expect(detectProvider("MEN-540301-9J5")).toBe("PREDIAL ENSENADA");
});
it("does not let the CFE rule claim a gas bill over 'PERIODO FACTURADO'", () => {
expect(
detectProvider("Orden de Facturación: 000009801640\nPERIODO FACTURADO: 20260630-20260630"),
).toBe("GAS TIJUANA");
});
});
describe("normalizeCadastralKey", () => {
it("keeps a letter in the third position instead of digitising it", () => {
// `MMB01041` is a real key on file; mapping its B to 8 produced a key that
// matches no property at all.
expect(normalizeCadastralKey("MM-B01-041", [])).toBe("MMB01041");
});
it("repairs the spurious I tesseract inserts into the prefix", () => {
expect(normalizeCadastralKey("MIM-200-010", [])).toBe("MM200010");
});
it("digitises confusable glyphs from position four onward", () => {
expect(normalizeCadastralKey("KP-1O8-O45", [])).toBe("KP108045");
});
it("flags a prefix it had to truncate", () => {
const notes: string[] = [];
expect(normalizeCadastralKey("KPX-128-106", notes)).toBe("KP128106");
expect(notes).toHaveLength(1);
});
});
describe("parsePredialTijuana", () => {
const TIJUANA = page(
"Hats | AYUNTAMIENTO DE TIJUANA, BC $2,613.00 23/01/2026\n" +
"y) TELEFONO: 973-7000 R.F.C. ATB-541201-KK2\n" +
"ER AÑO VALOR FISCAL TASA IMPUESTO |CONCEPTO IMPORTE\n" +
"ED ca 2026 1,207,15778 246 2,969.61 1102 - IMPUESTO PREDIAL 2,969.61\n" +
"55164964310126000002613000054192\n" +
"se 0 O (54427 [a] | TOTALAPAGAR: 2,613.00\n" +
"Dc 1097 : FECHA VENCE : 31/ENE/2026",
);
it("splits the payment barcode into account, deadline and amount", () => {
const p = parseStatement(TIJUANA);
expect(p.provider).toBe("PREDIAL TIJUANA");
expect(p.serviceKind).toBe("PROPERTY_TAX");
expect(p.accountRef).toBe("55164964");
expect(p.amount).toBe(2613);
expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-01-31");
expect(p.period).toBe("2026");
});
it("reads the printed total even when the space in the label is lost", () => {
// The real page OCR'd the label as "TOTALAPAGAR:", and it is that reading
// that cross-checks the barcode's amount.
expect(parseStatement(TIJUANA).crossChecked).toBe(true);
});
it("refuses to trust a barcode the printed total contradicts", () => {
const p = parseStatement(
page(
"R.F.C. ATB-541201-KK2\n" +
"55164964310126000002613000054192\n" +
"TOTAL A PAGAR: 9,613.00\nFECHA VENCE : 31/ENE/2026",
),
);
expect(p.crossChecked).toBe(false);
expect(p.notes.join(" ")).toContain("no coincide");
});
});
describe("parsePredialRosarito", () => {
it("takes the rounded Total, not the Sub Total printed above it", () => {
const p = parseStatement(
page(
"AYUNTAMIENTO MUNICIPAL DE PLAYAS DE ROSARITO, B.C.\n" +
"Ce Clave Catastral: + JR-400-008 7 | IMPUESTO PREDIAL\n" +
"SUPERFICIE: 228.31 ZONA 30025 “Redondeo IT049 -$0.39 Sub Total $5,409.39\n" +
"¿XTEMPORANEO DESPUES DE: 31/01/2026 Elaboro: MGLG\n" +
"Total | $5,409.00\n" +
"| Periodo por Pagar: 2026/1 2026/6",
),
);
expect(p.cadastralKey).toBe("JR400008");
expect(p.amount).toBe(5409);
expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-01-31");
expect(p.period).toBe("2026");
});
it("is not fooled by the unspaced 'SubTotal' spelling", () => {
// This exact page read $9,624.85 off a receipt for $9,625.00 while the
// lookbehind still assumed a space.
const p = parseStatement(
page(
"AMP-981201-HJ4 IMPUESTO PREDIAL\n" +
"SUPERFICIE. 367.62 ZONA:30151 | Redondco 17049 $0.15 SubTotal $9,624.85\n" +
": Total | $9,625.00",
),
);
expect(p.amount).toBe(9625);
});
});
describe("parsePredialEnsenada", () => {
const totals = (tail: string) =>
page(
"IMPRESION MAQUINA REGISTRADORA ez | MUNICIPIO DE ENSENADA\n" +
"+7] DATOS. DEL.CAUSANTE alta A pe CLAVE MM-200-010 2 CUENTA\n" +
`ES g € S| TOTALES 12,744.47 0.00 0.00 324.56 0.00 13,069.03 ${tail} |`,
);
it("reads the paid total off the TOTALES row however the label OCR'd", () => {
expect(parseStatement(totals("TOTA LA A $5,797.00")).amount).toBe(5797);
expect(parseStatement(totals("orAL: M7 z] $14,414.00")).amount).toBe(14414);
expect(parseStatement(totals("| TOTAL: = $6 246.00")).amount).toBe(6246);
});
it("reports no amount rather than one whose $ was misread as an 8", () => {
// `TOTAL: A 82,203.00` is a $2,203.00 receipt. Posting $82,203 would look
// entirely ordinary in the ledger, so this page must go to review instead.
const p = parseStatement(totals("TOTAL: A 82,203.00"));
expect(p.amount).toBeNull();
expect(p.notes.join(" ")).toContain("capturarlo a mano");
});
it("never falls back to the assessed total on the same row", () => {
expect(parseStatement(totals("yo: se TE= 58/4690]")).amount).toBeNull();
});
});
describe("parseGas", () => {
const gas = (...cuentas: string[]) =>
page(
"GTI4608032K2 COMPAÑIA DE GAS DE TIJUANA\n" +
"Fecha de Vencimiento: 2026/08/08\n" +
cuentas.map((c) => `Cuenta: ${c}`).join("\n") +
"\nPERIODO FACTURADO: 20260630-20260630\nTOTAL A PAGAR: $275.82",
);
it("strips the printed leading zero to the stored account number", () => {
const p = parseStatement(gas("0900003463", "0900003463", "0900003463"));
expect(p.serviceKind).toBe("GAS");
expect(p.accountRef).toBe("900003463");
expect(p.amount).toBe(275.82);
expect(p.dueDate?.toISOString().slice(0, 10)).toBe("2026-08-08");
expect(p.period).toBe("2026-06");
expect(p.crossChecked).toBe(true);
});
it("takes the majority reading but still sends a disagreement to review", () => {
const p = parseStatement(gas("0900003463", "0900003463", "0900003468"));
expect(p.accountRef).toBe("900003463");
expect(p.crossChecked).toBe(false);
});
it("claims no cross-check from a single printing", () => {
expect(parseStatement(gas("0900003463")).crossChecked).toBeNull();
});
});
describe("parseZonaFederal", () => {
/**
* The Tijuana zona federal receipt, trimmed to the rows the parser reads.
* Verbatim from page 7 of the August 2026 batch, including the two ways the
* heading OCR'd: the clave line is struck through by the office's own
* highlighter, which is what cost two of eight pages their concession clave.
*/
const zf = (clave: string, body = "") =>
page(
"ESIZ <pYl Av. Independencia y Esq. Paseo del CentenaxiaiiArlhnto de Tijuana, B.C.\n" +
"Teléfono: 9737000 R.F.C. ATB-541201-BK2 0070000146 12:54 PM\n" +
"Zona Federal Marítimo Terrestre\n" +
`${clave} Nombre: DENNIS JOHN SEIN Concesión:\n` +
"Periodo Construcción Tasa Ornato Tasa Impuesto Actualiza. Recargo Multa Importe\n" +
"2026-2 / 2026-2 316.40 35.00 0.00 12.11 1,845.66 0.00 27.13 1,000.00 2,872.79\n" +
"SubTotal 1,845.66 0.00 27.13 1,000.00 2,872.79\n" +
"Concepto: Derechos de ocupación de Zona Federal Marítimo Terrestre\n" +
body,
);
it("is not claimed by the predial parser that shares its RFC and header", () => {
// Tijuana bills predial and zona federal from the same treasury, so
// "Ayuntamiento de Tijuana" and ATB-541201 identify neither on their own.
expect(detectProvider("R.F.C. ATB-541201-BK2\nZona Federal Marítimo Terrestre")).toBe(
"ZONA FEDERAL TIJUANA",
);
expect(parseStatement(zf("Clave: 14-D -014")).serviceKind).toBe("FEDERAL_ZONE");
});
it("still recognises the layout when the heading itself did not survive OCR", () => {
// Real: page 1 came back as "Zona Ledera) Maritimo Terrestre".
expect(
detectProvider("Zona Ledera) Maritimo Terrestre\nClave EJ -012% Nombre: STEFAN"),
).toBe("ZONA FEDERAL TIJUANA");
});
it("reads the clave through the loose spacing the receipt prints", () => {
expect(parseStatement(zf("Clave: 14-D -014")).accountRef).toBe("14D014");
expect(parseStatement(zf("Clave: 14-A-119")).accountRef).toBe("14A119");
});
it("keeps the letter instead of digitising it", () => {
// toDigits maps D to 0 and B to 8; a real 14-D -014 must not become 140014.
expect(normalizeZofematKey("14-D -014")).toBe("14D014");
expect(normalizeZofematKey("12-B -013")).toBe("12B013");
});
it("takes the payable amount from the SubTotal row, rounded to whole pesos", () => {
// The municipality rounds and prints the difference as "Ajuste Ley Hacienda
// Mpal"; 2,872.79 is charged as $2,873.00.
expect(parseStatement(zf("Clave: 14-D -014")).amount).toBe(2873);
});
it("prefers the printed total and cross-checks it against the subtotal", () => {
const p = parseStatement(zf("Clave: 14-D -014", "Total a pagar $2,873.00"));
expect(p.amount).toBe(2873);
expect(p.crossChecked).toBe(true);
});
it("sends a printed total that contradicts the subtotal to review", () => {
const p = parseStatement(zf("Clave: 14-D -014", "Total a pagar $2,973.00"));
expect(p.crossChecked).toBe(false);
});
it("translates the printed bimester into the ledger's own vocabulary", () => {
expect(parseStatement(zf("Clave: 14-D -014")).period).toBe("MAR/APR");
});
it("leaves the clave blank rather than guessing when the marker ate it", () => {
const p = parseStatement(zf("Clave EJ -012%"));
expect(p.accountRef).toBeNull();
expect(p.notes.join(" ")).toContain("clave");
});
});
@@ -0,0 +1,872 @@
import type { ServiceKind } from "@jorgecuadros/database";
import type { OcrPage, OcrWord } from "../ocr/ocr.provider";
/**
* What one parsed statement page yields. `accountRef` is already normalised to
* the form the migrated `PropertyService` columns hold, so the matcher compares
* like with like and never has to know about provider-specific formatting.
*/
export interface ParsedStatement {
/**
* "CFE" | "CESPT" | "TELNOR" | "GAS TIJUANA" | "PREDIAL TIJUANA" |
* "PREDIAL ROSARITO" | "PREDIAL ENSENADA" | "ZONA FEDERAL TIJUANA", or null
* when no parser claimed the page.
*/
provider: string | null;
serviceKind: ServiceKind | null;
accountRef: string | null;
/** Clave catastral, when printed — a second key to match on. */
cadastralKey: string | null;
amount: number | null;
dueDate: Date | null;
period: string | null;
/**
* Independent corroboration of `accountRef`. CFE and Telnor both print a
* payment barcode that repeats the account number (and the amount), so when
* the barcode and the label agree the extraction is near-certainly right;
* when they disagree, or only one is present, the page is worth a human
* glance. Null when the layout has no second source.
*/
crossChecked: boolean | null;
/** Human-readable trail of what was read, surfaced in the review queue. */
notes: string[];
}
// --- shared helpers ---------------------------------------------------------
/**
* Tesseract confuses these glyphs inside numeric runs with some regularity —
* a real clave catastral `KB078025` came back as `KBO78025`. Applied ONLY to
* fields known to be digits, never to free text, where it would corrupt words.
*/
const DIGIT_CONFUSIONS: Record<string, string> = {
O: "0",
o: "0",
D: "0",
I: "1",
l: "1",
"|": "1",
S: "5",
B: "8",
};
export function toDigits(s: string | null | undefined): string {
if (!s) return "";
return s
.split("")
.map((c) => DIGIT_CONFUSIONS[c] ?? c)
.join("")
.replace(/\D/g, "");
}
/**
* Parse a printed amount, treating `,` and `.` by position rather than by
* assumption. A real Telnor bill OCR'd as "$ 649,00" — blindly stripping commas
* as thousands separators turned $649.00 into $64,900, a hundredfold error that
* would post silently. Two trailing digits after a single separator are always
* cents here; a separator followed by three digits is a thousands group.
*/
function money(s: string | null | undefined): number | null {
if (!s) return null;
const cleaned = s.replace(/[\s$]/g, "");
// 1.234,56 or 1,234.56 — grouped thousands plus optional cents.
let m = cleaned.match(/^(\d{1,3}(?:[.,]\d{3})+)([.,]\d{1,2})?$/);
if (m) {
const whole = m[1].replace(/[.,]/g, "");
const cents = m[2] ? m[2].slice(1) : "";
return Number(cents ? `${whole}.${cents.padEnd(2, "0")}` : whole);
}
// 649,00 / 649.00 — a single separator with exactly two digits after it.
m = cleaned.match(/^(\d+)[.,](\d{2})$/);
if (m) return Number(`${m[1]}.${m[2]}`);
const n = Number(cleaned.replace(/[,.]/g, ""));
return Number.isFinite(n) ? n : null;
}
function firstMatch(text: string, patterns: RegExp[]): string | null {
for (const p of patterns) {
const m = text.match(p);
if (m?.[1]) return m[1].trim();
}
return null;
}
/** Every capture of `pattern` across the page, in order. */
function allMatches(text: string, pattern: RegExp): string[] {
const out: string[] = [];
const re = new RegExp(pattern.source, pattern.flags.includes("g") ? pattern.flags : `${pattern.flags}g`);
for (const m of text.matchAll(re)) {
if (m[1]) out.push(m[1].trim());
}
return out;
}
const MONTHS: Record<string, number> = {
ENE: 0, FEB: 1, MAR: 2, ABR: 3, MAY: 4, JUN: 5,
JUL: 6, AGO: 7, SEP: 8, OCT: 9, NOV: 10, DIC: 11,
};
/** Parses the three date shapes these statements actually print. */
export function parseDate(raw: string | null | undefined): Date | null {
if (!raw) return null;
const s = raw.trim().toUpperCase();
// 16/07/2026
let m = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/);
if (m) return utc(+m[3], +m[2] - 1, +m[1]);
// 22-JUL-2026 / 22 JUN 26 / 31/ENE/2026 (Tijuana predial)
m = s.match(/^(\d{1,2})[-\s/]([A-Z]{3})[A-Z]*[-\s/](\d{2,4})$/);
if (m && MONTHS[m[2]] !== undefined) {
const y = m[3].length === 2 ? 2000 + +m[3] : +m[3];
return utc(y, MONTHS[m[2]], +m[1]);
}
// 2026-07-22 (already normalised, e.g. decoded from a barcode) and the
// 2026/08/08 the gas bill prints — same field order, different separator.
m = s.match(/^(\d{4})[-/](\d{2})[-/](\d{2})$/);
if (m) return utc(+m[1], +m[2] - 1, +m[3]);
return null;
}
function utc(y: number, mo: number, d: number): Date | null {
const dt = new Date(Date.UTC(y, mo, d));
return Number.isNaN(dt.getTime()) ? null : dt;
}
/**
* Read the value printed *underneath* a column header.
*
* The CESPT "RECIBO" is a table: `No. DE CUENTA` is a header cell and its value
* sits in the row below it, so no amount of label-adjacent regex on line text
* can associate the two. This walks the word boxes instead — find the header
* word, then take the nearest word below it whose horizontal centre falls
* within the column.
*/
export function valueUnder(
page: OcrPage,
header: RegExp,
opts: { maxDy?: number; tolerance?: number; match?: RegExp } = {},
): string | null {
const { maxDy = 300, tolerance = 200, match } = opts;
const centre = (w: OcrWord) => ({
x: w.left + w.width / 2,
y: w.top + w.height / 2,
});
for (const h of page.words.filter((w) => header.test(w.text))) {
const hc = centre(h);
const below = page.words
.filter((w) => {
const c = centre(w);
return c.y > hc.y && c.y <= hc.y + maxDy && Math.abs(c.x - hc.x) <= tolerance;
})
.sort((a, b) => centre(a).y - centre(b).y);
for (const w of below) {
if (!match || match.test(w.text)) return w.text;
}
}
return null;
}
// --- provider detection -----------------------------------------------------
/**
* Brand wordmarks first, page structure only as a fallback — and the two passes
* must not be interleaved. Scanned logos OCR badly (one CESPT header came back
* as "E BAJA ES PAGO / EALIFORNIA", with neither "CESPT" nor "COMISIÓN ESTATAL"
* readable), so the structural pass is what rescues those pages. But a Telnor
* bill contains the words "Pagar antes de", which a CFE structural rule
* evaluated first will happily claim — running all brand checks before any
* structural check is what keeps that from happening.
*/
const BRAND: [string, RegExp][] = [
["CFE", /comisi[oó]n federal de electricidad|CFE.?contigo|Suministrador de Servicios/i],
["CESPT", /CESPT|COMISI[OÓ]N ESTATAL DE SERVICIOS/i],
["TELNOR", /TELNOR|TELEFONOS DEL NOROESTE/i],
["GAS TIJUANA", /COMPA[ÑN][IÍ]?A\s*DE\s*GAS\s*DE\s*TIJUANA|bajagas/i],
// Ahead of the predial rules on purpose. Tijuana's zona federal receipt is
// issued by the same treasury and carries the same header — "Ayuntamiento de
// Tijuana", the same address, the same `ATB-541201` RFC — so every predial
// discriminator matches it too, and whichever rule is asked first wins the
// page. What only the zona federal layout says is "Marítimo Terrestre", which
// survived OCR on all eight sample pages even where the heading above it came
// back as "Zona Ledera) Maritimo Terrestre" and the printed concession clave
// was lost under a highlighter mark.
["ZONA FEDERAL TIJUANA", /ZOFEMAT|Mar[ií]timo\s*Terrestre|ocupaci[oó]n\s*de\s*Zona\s*Federal/i],
// The municipal RFCs are the single most reliable discriminator on a predial
// receipt: they are printed in a clean monospaced run on every layout, they
// never change, and they say which of the three city treasuries issued the
// page — which the wordmarks alone do not, since a Tijuana receipt also
// carries "PLAYAS DE TIJUANA" and a Rosarito one "TIJUANA ENSENADA".
["PREDIAL TIJUANA", /AYUNTAMIENTO\s*DE\s*TIJUANA|ATB.?541201/i],
["PREDIAL ROSARITO", /AYUNTAMIENTO\s*MUNICIPAL\s*DE\s*PLAYAS\s*DE\s*ROSARITO|AMP.?981201|rosarito\.gob/i],
["PREDIAL ENSENADA", /MUNICIPIO\s*DE\s*ENSENADA|MEN.?540301/i],
];
/**
* The predial rules come first because a Rosarito receipt prints "Clave
* Catastral" as a boxed label — the very string the CESPT structural rule
* looks for — so a page whose municipal header failed to OCR would otherwise
* be claimed as a water bill and matched against the wrong column entirely.
* "IMPUESTO PREDIAL" appears on all three municipal layouts and on none of the
* utility ones, so it is the safe first question to ask.
*/
const LAYOUT: [string, RegExp][] = [
// Same reasoning as the brand pass, one rule earlier: the concept line
// "Derechos de ocupación de Zona Federal Marítimo Terrestre" is printed on
// the stub of every zona federal page and on no other layout, and it read
// cleanly on 8 of 8 samples — including the two whose heading did not.
["ZONA FEDERAL TIJUANA", /Derechos\s*de\s*ocupaci[oó]n/i],
["PREDIAL TIJUANA", /IMPUESTO\s*PREDIAL[\s\S]*?(?:CERTIFICACION\s*DE\s*CAJA|PASEO\s*DEL\s*CENTENARIO|PAGA\s*TU\s*PREDIAL)/i],
["PREDIAL ENSENADA", /(?:IMPUESTO\s*PREDIAL[\s\S]*?TRANSPENINSULAR)|(?:IMPRESION\s*MAQUINA\s*REGISTRADORA)/i],
["PREDIAL ROSARITO", /IMPUESTO\s*PREDIAL/i],
["GAS TIJUANA", /Orden\s*de\s*Facturaci[oó]n|FACTOR\s*DE\s*PRESI[OÓ]N|GAS\s*LP/i],
["CFE", /NO\.?\s*DE\s*SERVICIO|L[IÍ]MITE\s*DE\s*PAGO|PERIODO\s*FACTURADO/i],
["CESPT", /SALDO\s+CORRIENTE|CLAVE\s*CATASTRAL|No\.?\s*DE\s*CUENTA/i],
["TELNOR", /Mes\s*de\s*Facturaci[oó]n|Pagar\s*antes\s*de/i],
];
export function detectProvider(text: string): string | null {
for (const group of [BRAND, LAYOUT]) {
for (const [name, pattern] of group) {
if (pattern.test(text)) return name;
}
}
return null;
}
// --- CFE (electric) ---------------------------------------------------------
function parseCfe(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
// The payment barcode line repeats the service number, the due date (YYMMDD)
// and the amount in one fixed-width run, and reads far more reliably than the
// label: on one sample the label came back as "0059603001917" (a digit too
// many) while its barcode gave the correct "005960300191". So the barcode
// wins, and the label becomes the cross-check rather than the source.
const barcode = text.match(/\b01\s+([0-9OIlSBD]{12})\s+([0-9OIlSBD]{6})\s+([0-9OIlSBD]{9})\b/);
const label = firstMatch(text, [/NO\.?\s*DE\s*SERVICIO\s*[:;.]?\s*([0-9OIlSBD]{10,14})/i]);
let accountRef: string | null = null;
let amount: number | null = null;
let dueDate: Date | null = null;
let crossChecked: boolean | null = null;
if (barcode) {
// Leading zeros are print padding: DATMEX.rpu holds the bare 10 digits.
accountRef = toDigits(barcode[1]).replace(/^0+/, "");
amount = Number(toDigits(barcode[3]));
const d = toDigits(barcode[2]);
dueDate = parseDate(`20${d.slice(0, 2)}-${d.slice(2, 4)}-${d.slice(4, 6)}`);
notes.push("importe y vencimiento leídos del código de barras");
if (label) {
crossChecked = toDigits(label).replace(/^0+/, "") === accountRef;
if (!crossChecked) {
notes.push(
`el número impreso (${toDigits(label).replace(/^0+/, "")}) no coincide con el código de barras`,
);
}
}
} else if (label) {
accountRef = toDigits(label).replace(/^0+/, "");
notes.push("sin código de barras legible; número tomado de la etiqueta");
}
if (amount == null) {
amount = money(firstMatch(text, [/TOTAL\s*A\s*PAGAR\s*[:;.]?\s*\$?\s*([\d,]+\.?\d*)/i]));
}
if (!dueDate) {
dueDate = parseDate(
firstMatch(text, [/L[IÍ]MITE\s*DE\s*PAGO\s*[:;.]?\s*(\d{1,2}\s+\w{3}\s+\d{2,4})/i]),
);
}
return {
provider: "CFE",
serviceKind: "ELECTRIC",
accountRef: accountRef || null,
cadastralKey: null,
amount,
dueDate,
period: firstMatch(text, [
/PERIODO\s*FACTURADO\s*[:;.]?\s*(\d{1,2}\s+\w{3}\s+\d{2}\s*-\s*\d{1,2}\s+\w{3}\s+\d{2})/i,
]),
crossChecked,
notes,
};
}
// --- CESPT (water) ----------------------------------------------------------
/**
* Two different layouts arrive under the same brand:
* - the line-oriented "COMPROBANTE DE PAGO" (`Cuenta : 7604192`), and
* - the tabular "RECIBO", where `No. DE CUENTA` is a column header.
* Line patterns are tried first; anything they miss falls through to the
* geometric read, which is what the tabular layout needs.
*/
function parseCespt(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
let account = firstMatch(text, [/Cuenta\s*[:;.]?\s*([0-9OIlSBD]{5,9})/i]);
if (!account) {
account = valueUnder(page, /^CUENTA$/i, { match: /^[0-9OIlSBD]{5,9}$/ });
if (account) notes.push("número de cuenta leído de la columna del recibo");
}
let clave = firstMatch(text, [/Cve\.?\s*Cat\.?\s*[:;.]?\s*([A-Z]{2}\s?[0-9OIlSBD]{6})/i]);
if (!clave) {
clave = valueUnder(page, /^CATASTRAL$/i, { match: /^[A-Z]{2}[0-9OIlSBD]{6}$/i });
if (clave) notes.push("clave catastral leída de la columna del recibo");
}
let due = firstMatch(text, [/Fecha\s*Venc\s*[:;.]?\s*(\d{2}\/\d{2}\/\d{4})/i]);
if (!due) due = valueUnder(page, /^VENCIMIENTO$/i, { match: /^\d{2}\/\d{2}\/\d{4}$/ });
const amount = money(
firstMatch(text, [
/TOTAL\s*[:;.]?\s*\$?\s*([\d,]+\.\d{2})/i,
/SALDO\s+CORRIENTE[^\n]*?([\d,]+\.\d{2})/i,
]),
);
// Leading zeros are print padding here too: the RECIBO prints `0457341` for
// what DATMEX.agua holds as `457341`.
const accountRef = account ? toDigits(account).replace(/^0+/, "") : null;
const cadastralKey = clave
? clave.replace(/\s/g, "").slice(0, 2).toUpperCase() +
toDigits(clave.replace(/\s/g, "").slice(2))
: null;
return {
provider: "CESPT",
serviceKind: "WATER",
accountRef: accountRef || null,
cadastralKey: cadastralKey || null,
amount,
dueDate: parseDate(due),
period: null,
crossChecked: null,
notes,
};
}
// --- TELNOR (telephone) -----------------------------------------------------
function parseTelnor(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
const label = firstMatch(text, [
/Tel[eé]fono\s*[:;.]?\s*([0-9OIlSBD]{3}\s?[0-9OIlSBD]{3}\s?[0-9OIlSBD]{4})/i,
]);
// The payment stub prints phone (10 digits) + amount in cents (9) + a check
// digit: `6646093444 000099900 7` for a $999.00 bill. Reading the amount as
// 10 digits swallows the check digit and inflates the figure 100-fold.
const barcode = text.match(/\b(\d{10})(\d{9})\d\b/);
let accountRef: string | null = null;
let crossChecked: boolean | null = null;
// The bill prints the number with its 664 Tijuana LADA; DATMEX stores the
// bare local 7 digits, so the LADA is dropped rather than the stored value
// being padded — padding would guess at an area code for the 500+ existing
// rows that never recorded one.
if (label) accountRef = toDigits(label).slice(-7);
if (barcode) {
const fromBarcode = barcode[1].slice(-7);
if (accountRef) {
crossChecked = fromBarcode === accountRef;
if (!crossChecked) notes.push("el teléfono impreso no coincide con el código de barras");
} else {
accountRef = fromBarcode;
notes.push("teléfono leído del código de barras");
}
}
let amount = money(firstMatch(text, [/Total\s*a\s*Pagar\s*[:;.]?\s*\$?\s*([\d,]+\.?\d{0,2})/i]));
if (amount == null && barcode) {
amount = Number(barcode[2]) / 100;
notes.push("importe leído del código de barras");
}
return {
provider: "TELNOR",
serviceKind: "TELEPHONE",
accountRef: accountRef || null,
cadastralKey: null,
amount,
dueDate: parseDate(
firstMatch(text, [/Pagar\s*antes\s*de\s*[:;.]?\s*(\d{2}-\w{3}-\d{4})/i]),
),
period: firstMatch(text, [/Mes\s*de\s*Facturaci[oó]n\s*[:;.]?\s*(\w+)/i]),
crossChecked,
notes,
};
}
// --- GAS (Compañía de Gas de Tijuana / bajagas) ------------------------------
/**
* These arrive as born-digital CFDI PDFs rather than scans, so the text layer
* (see `TesseractOcrProvider.textPages`) usually reads them exactly and the
* patterns below only have to be tolerant enough for the scanned case.
*
* The account number is printed three times — supply address, fiscal data, and
* the payment stub at the foot — which is a free cross-check: three readings
* that agree are near-certainly right, and any disagreement means one of them
* was misread and the page deserves a human glance.
*
* `Cuenta` is what the matcher compares, not `Contrato`. The migration
* recovered gas references out of `PropertyService.notes` into `meterNumber`
* and what sat there is the 9-digit account (`900003463`), printed here with a
* leading zero as `0900003463`.
*/
function parseGas(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
const seen = allMatches(text, /Cuenta\s*[:;.]?\s*([0-9OIlSBD]{6,12})/i).map((s) =>
toDigits(s).replace(/^0+/, ""),
);
const distinct = [...new Set(seen.filter(Boolean))];
let accountRef: string | null = null;
let crossChecked: boolean | null = null;
if (distinct.length === 1) {
accountRef = distinct[0];
if (seen.length > 1) crossChecked = true;
} else if (distinct.length > 1) {
// Majority wins — the stub and the two address blocks print the same
// number, so a single divergent reading is the misread one. It still goes
// to review: `crossChecked: false` is what keeps the batch from
// auto-matching a number one of three readings disagreed with.
const tally = new Map<string, number>();
for (const s of seen) tally.set(s, (tally.get(s) ?? 0) + 1);
accountRef = [...tally.entries()].sort((a, b) => b[1] - a[1])[0][0];
crossChecked = false;
notes.push(`el número de cuenta se leyó de ${distinct.length} formas distintas (${distinct.join(", ")})`);
}
const amount = money(
firstMatch(text, [
/TOTAL\s*A\s*PAGAR\s*[:;.]?\s*\$\s*([\d,]+\.\d{2})/i,
/Total\s*a\s*pagar\s*[:;.]?\s*\$\s*([\d,]+\.\d{2})/i,
]),
);
// `20260630-20260630` — the range the bill was cut for. Both ends are the
// same reading date on every sample, so the period is reported as the ISO
// month rather than a range no ledger row would ever be searched by.
const facturado = firstMatch(text, [/PERIODO\s*FACTURADO\s*[:;.]?\s*(\d{8})\s*-\s*\d{8}/i]);
const period = facturado ? `${facturado.slice(0, 4)}-${facturado.slice(4, 6)}` : null;
return {
provider: "GAS TIJUANA",
serviceKind: "GAS",
accountRef: accountRef || null,
cadastralKey: null,
amount,
dueDate: parseDate(
firstMatch(text, [/Fecha\s*de\s*Vencimiento\s*[:;.]?\s*(\d{4}\s*\/\s*\d{2}\s*\/\s*\d{2})/i])?.replace(
/\s/g,
"",
),
),
period,
crossChecked,
notes,
};
}
// --- PREDIAL (municipal property tax) ---------------------------------------
/**
* Normalise a printed clave catastral to the eight-character form
* `Property.cadastralKey` holds. The municipalities print it grouped
* (`KP-128-106`, `MM-B01-041`); the stored value drops the separators
* (`KP128106`, `MMB01041`).
*
* The shape is *not* two letters and six digits, which is the assumption that
* has to be resisted here. Across the 932 distinct claves on file, characters
* four through eight are digits without exception, but the third is a digit in
* 917 of them and one of `A`, `B`, `H`, `T` in the other fifteen. Running the
* whole tail through `toDigits` — which maps `B` to `8` — is what turned a real
* `MMB01041` into a nonexistent `MM801041`, so only positions four onward get
* that treatment and a letter in the third position is kept as printed.
*
* That leaves a genuine ambiguity at that one position: a `B` there might be a
* misread `8`, and 34 stored claves do carry an `8` there against six with a
* `B`. It is left as read rather than guessed, because a page that fails to
* match lands in the review queue where a human fixes it in seconds, while a
* page that matches the wrong property posts a charge to the wrong customer.
*
* The two-letter prefix is the other fragile part. Tesseract inserts a spurious
* `I` into letter pairs with some regularity — a real `MM-200-010` came back as
* `MIM-200-010` — so a run longer than two letters has its `I`/`L` dropped
* first, which recovers exactly that case. Anything still not two letters is
* truncated and flagged, because a wrong prefix silently matches the wrong
* property or, more often, nothing at all.
*/
export function normalizeCadastralKey(
raw: string,
notes: string[],
): string | null {
const m = raw.match(/^([A-Za-z|]{2,5})[-\s]?([A-Za-z0-9|]{3})[-\s]?([0-9OIlSBD]{3})$/);
if (!m) return null;
let letters = m[1].toUpperCase().replace(/[^A-Z]/g, "");
if (letters.length > 2) {
const stripped = letters.replace(/[IL]/g, "");
if (stripped.length === 2) {
letters = stripped;
} else {
letters = letters.slice(0, 2);
notes.push(`la clave catastral se leyó como "${m[1]}"; se tomó "${letters}"`);
}
}
if (letters.length !== 2) return null;
const third = m[2][0].toUpperCase();
const tail =
(/[A-Z]/.test(third) ? third : toDigits(third)) +
toDigits(m[2].slice(1)) +
toDigits(m[3]);
return tail.length === 6 ? letters + tail : null;
}
/** The grouped clave as printed, anchored to its label when one survived OCR. */
const GROUPED_CLAVE = "[A-Z|]{2,5}-[A-Z0-9OIlSBD]{3}-[0-9OIlSBD]{3}";
function findCadastralKey(text: string, notes: string[]): string | null {
const labelled = firstMatch(text, [
new RegExp(`Clave\\s*Catastral\\s*[^A-Z0-9]{0,8}(${GROUPED_CLAVE})`, "i"),
new RegExp(`CLAVE\\s*[^A-Z0-9]{0,8}(${GROUPED_CLAVE})`, "i"),
]);
if (labelled) return normalizeCadastralKey(labelled, notes);
// Ensenada's label ("CLAVE") lands inside a table header that OCRs into
// noise more often than not, so the bare grouped shape is accepted as a
// fallback. It is distinctive enough — two letters and two three-character
// groups joined by hyphens appears nowhere else on these pages.
const bare = firstMatch(text, [new RegExp(`\\b(${GROUPED_CLAVE})\\b`)]);
return bare ? normalizeCadastralKey(bare, notes) : null;
}
/**
* Tijuana: a "CERTIFICACIÓN DE CAJA" whose payment barcode is one 32-digit run
* of `account(8) + due date(DDMMYY) + amount(9) + folio(9)`, verified against
* all five sample pages. Municipal totals are whole pesos (the receipt itself
* carries a "Redondeo" line), so the barcode amount needs no decimal point.
*
* No clave catastral is printed anywhere on this layout — the 8-digit
* municipal account is the only identifier, and it is not a number the legacy
* database ever held. Until a reviewer confirms one, every Tijuana page lands
* in review; confirming teaches the matcher (see `learnAccountRefs`) so the
* same property matches itself next year.
*/
function parsePredialTijuana(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
const barcode = text.match(/(?<![0-9OIlSBD])([0-9OIlSBD]{32})(?![0-9OIlSBD])/);
const printedTotal = money(
firstMatch(text, [/TOTAL\s*A?\s*PAGAR\s*[:;.]?\s*\$?\s*([\d,]+\.?\d{0,2})/i]),
);
let accountRef: string | null = null;
let amount: number | null = printedTotal;
let dueDate: Date | null = null;
let crossChecked: boolean | null = null;
if (barcode) {
const run = toDigits(barcode[1]);
const d = run.slice(8, 14);
const fromBarcode = Number(run.slice(14, 23));
accountRef = run.slice(0, 8);
dueDate = parseDate(`20${d.slice(4, 6)}-${d.slice(2, 4)}-${d.slice(0, 2)}`);
notes.push("cuenta, importe y vencimiento leídos del código de barras");
if (printedTotal != null) {
// Guarding the money, not the account number: the printed total is the
// figure a human would key, so when the two disagree one of them is a
// misread peso amount and nothing should post unreviewed.
crossChecked = Math.abs(printedTotal - fromBarcode) < 0.5;
if (!crossChecked) {
notes.push(
`el total impreso (${printedTotal}) no coincide con el código de barras (${fromBarcode})`,
);
}
}
if (amount == null) amount = fromBarcode;
}
if (!dueDate) {
dueDate = parseDate(
firstMatch(text, [/FECHA\s*VENCE\s*[:;.]?\s*(\d{1,2}\/\w{3}\/\d{4})/i]),
);
}
return {
provider: "PREDIAL TIJUANA",
serviceKind: "PROPERTY_TAX",
accountRef: accountRef || null,
cadastralKey: null,
amount,
dueDate,
// The fiscal year, which is what the legacy ledger's `period` holds for
// predial ("2026" is its single most common value). It is read from the
// assessment table's year column, and failing that from the deadline: a
// predial bill for year N falls due on 31 January of year N.
period:
firstMatch(text, [/VALOR\s*FISCAL[\s\S]{0,160}?\b(20\d{2})\b/i]) ??
(dueDate ? String(dueDate.getUTCFullYear()) : null),
crossChecked,
notes,
};
}
/**
* Rosarito: a wide "CERTIFICACIÓN DE CAJA" keyed by clave catastral, with no
* account number of its own — the clave is the identifier, which is exactly
* what `Property.cadastralKey` holds, so these match on the first pass.
*
* The total is read with a negative lookbehind on "Sub": the receipt prints
* `Sub Total $5,409.39` (before the peso rounding) directly above
* `Total $5,409.00`, and taking the first "Total" on the page books 39 cents
* that the municipality did not charge. The lookbehind allows zero spaces
* because the label prints both ways — `Sub Total` on one sample and
* `SubTotal` on the next, and the tight one is what slipped past a fixed
* `Sub\s` and read $9,624.85 off a receipt for $9,625.00.
*/
function parsePredialRosarito(page: OcrPage): ParsedStatement {
const notes: string[] = [];
const text = page.text;
return {
provider: "PREDIAL ROSARITO",
serviceKind: "PROPERTY_TAX",
accountRef: null,
cadastralKey: findCadastralKey(text, notes),
amount: money(firstMatch(text, [/(?<!Sub\s{0,3})Total\s*[|:;.]?\s*\$\s*([\d,]+\.\d{2})/i])),
// "EXTEMPORANEO DESPUES DE: 31/01/2026" — the leading E is regularly eaten
// by the box rule printed over it, so the anchor starts at "XTEMPORANEO".
dueDate: parseDate(
firstMatch(text, [/XTEMPOR[AÁ]NEO\s*DESPU[EÉ]S\s*DE\s*[:;.]?\s*(\d{2}\/\d{2}\/\d{4})/i]),
),
period: firstMatch(text, [/Periodo\s*por\s*Pagar\s*[:;.]?\s*(20\d{2})/i]),
crossChecked: null,
notes,
};
}
/**
* Ensenada: a dot-matrix "IMPRESION MAQUINA REGISTRADORA" statement, by some
* distance the worst-scanning of the three. Matching is by clave catastral.
*
* The amount is read positionally rather than by label, because the label does
* not survive: across five real pages the same word came back as `TOTAL:`,
* `TOTA LA A` and `orAL:`. What is stable is the row — the summary line that
* starts `TOTALES` carries the assessed figures across it and the amount
* actually paid last, at the right margin.
*
* That last figure must carry a literal `$`. On a real sample the paid total
* printed as `TOTAL: A $2,203.00` and OCR'd as `TOTAL: A 82,203.00` — the
* dollar sign read as an 8, a mistake that would post a $2,203 charge as
* $82,203 and look entirely ordinary in the ledger. Requiring the `$` costs
* that page its amount and sends it to review, which is the only acceptable
* failure here. The unprefixed figures earlier on the row are deliberately not
* a fallback: they are the tax assessed before the early-payment discount, not
* what was paid.
*/
function parsePredialEnsenada(page: OcrPage): ParsedStatement {
const notes: string[] = [];
const text = page.text;
const totalsRow = text.split("\n").find((l) => /TOTALES/i.test(l)) ?? "";
const figures = allMatches(totalsRow, /\$\s*(\d[\d,.\s]*\.\d{2})/);
const amount = figures.length ? money(figures[figures.length - 1]) : null;
if (amount == null) {
notes.push("no se pudo leer el importe con certeza; capturarlo a mano");
}
return {
provider: "PREDIAL ENSENADA",
serviceKind: "PROPERTY_TAX",
accountRef: null,
cadastralKey: findCadastralKey(text, notes),
amount,
// This layout prints no payment deadline at all — it is a receipt for a
// payment already made at the municipal window.
dueDate: null,
period: firstMatch(text, [/A[ÑN]O\s*[\s\S]{0,60}?\b(20\d{2})\b/i]),
crossChecked: null,
notes,
};
}
// --- ZONA FEDERAL (ZOFEMAT, Tijuana) ----------------------------------------
/**
* Normalise the concession clave the zona federal receipt is keyed by.
*
* It is printed grouped and loosely spaced — `12-T -012`, `14-A-119`,
* `14-K -031` — and is a different shape from the cadastral key entirely: two
* digits, one letter, three digits. The letter is kept as printed rather than
* digitised, for the same reason `normalizeCadastralKey` keeps its third
* character: `toDigits` maps `B` to `8` and `D` to `0`, and a real `14-D -014`
* run through it becomes `140014`, which is not a clave at all.
*
* Stored without separators, because nothing on file holds this value yet (see
* `parseZonaFederal`) so the canonical form is ours to pick, and a bare run
* cannot be broken by the hyphen the scan renders as a dash, a minus or
* nothing.
*/
export function normalizeZofematKey(raw: string): string | null {
const m = raw.match(/^([0-9OIlSBD]{2})\s*-\s*([A-Za-z])\s*-?\s*([0-9OIlSBD]{3})$/);
if (!m) return null;
const zone = toDigits(m[1]);
const lot = toDigits(m[3]);
if (zone.length !== 2 || lot.length !== 3) return null;
return `${zone}${m[2].toUpperCase()}${lot}`;
}
/**
* The bimester the receipt prints as `2026-2 / 2026-2`, rendered in the
* vocabulary the ledger already speaks.
*
* All 258 legacy FEDERAL ZONE transactions carry a period of `JAN/FEB`,
* `MAR/APR`, `MAY/JUN` or `NOV/DEC`, and their payment dates confirm the
* ordering — JAN/FEB was paid in March, MAR/APR in May, MAY/JUN in July,
* NOV/DEC in January, i.e. always the month after the bimester closes. The
* receipts agree: the two `2026-3` samples fall due 17/07/2026 with no
* surcharge, which is bimester three, May and June. Writing `2026-3` instead
* would leave the OCR-posted rows unsearchable alongside every hand-keyed one.
*/
const BIMESTERS = ["JAN/FEB", "MAR/APR", "MAY/JUN", "JUL/AUG", "SEP/OCT", "NOV/DEC"];
/**
* Tijuana's "Zona Federal Marítimo Terrestre" — the federal maritime-zone
* occupancy fee, billed by the municipality for beachfront lots.
*
* Nothing on file identifies these. `PropertyService.accountNumber` for
* FEDERAL_ZONE holds DATMEX.zfed, which is not a reference at all but an
* amount: its 77 values include `246.06`, `2369.09`, `22653.94` and a negative
* `-1679`, and the concession claves these receipts are keyed by appear nowhere
* in the database. So the clave goes to `meterNumber` (see `scopedRefField`),
* every page starts cold, and the first confirm teaches the match — the same
* arrangement Tijuana predial needed, for the same reason.
*
* The amount is taken from the SubTotal row rather than the "Total a pagar"
* box, which is printed on a grey fill and OCR'd on only 1 of 8 sample pages
* while the SubTotal row read on 8 of 8. The two differ by design: the
* municipality rounds to whole pesos and prints the difference on its own
* "Ajuste Ley Hacienda Mpal" line — `-$0.05` against a 591.05 subtotal, `$0.21`
* against 2,872.79 — so the payable figure is the rounded subtotal, and where
* the printed box did read, it agreed.
*/
function parseZonaFederal(page: OcrPage): ParsedStatement {
const text = page.text;
const notes: string[] = [];
// Printed twice, once on the receipt and once on the stub below it, which is
// a free second reading: on one sample the heading was struck through by the
// office's own highlighter and only the stub survived.
const claves = [
...new Set(
allMatches(text, /Clave\s*[:;.]?\s*([0-9OIlSBD]{2}\s*-\s*[A-Za-z]\s*-?\s*[0-9OIlSBD]{3})/i)
.map(normalizeZofematKey)
.filter((k): k is string => k != null),
),
];
const accountRef = claves[0] ?? null;
let crossChecked: boolean | null = null;
const subtotalRow = text.split("\n").find((l) => /SubTotal/i.test(l)) ?? "";
const figures = allMatches(subtotalRow, /(\d[\d,]*\.\d{2})/);
// Impuesto, Actualización, Recargo, Multa, Importe — the payable one is last.
const importe = figures.length ? money(figures[figures.length - 1]) : null;
const rounded = importe != null ? Math.round(importe) : null;
const printed = money(
firstMatch(text, [/Total\s*a\s*pagar\s*[:;.]?\s*\$?\s*([\d,]+\.\d{2})/i]),
);
if (printed != null && rounded != null) {
crossChecked = Math.abs(printed - rounded) < 0.5;
if (!crossChecked) {
notes.push(
`el total impreso (${printed}) no coincide con el subtotal redondeado (${rounded})`,
);
}
} else if (rounded != null) {
notes.push("importe tomado del subtotal, redondeado al peso");
} else if (printed == null) {
notes.push("no se pudo leer el importe con certeza; capturarlo a mano");
}
// A clave read two different ways means one of the two readings is wrong and
// there is no third to break the tie, so the page goes to a human even if the
// money cross-checked.
if (claves.length > 1) {
crossChecked = false;
notes.push(`la clave se leyó de ${claves.length} formas distintas (${claves.join(", ")})`);
}
if (!accountRef) notes.push("no se pudo leer la clave de la concesión");
const bimester = text.match(/\b(20\d{2})\s*-\s*([1-6])\s*\/\s*20\d{2}\s*-\s*[1-6]/);
return {
provider: "ZONA FEDERAL TIJUANA",
serviceKind: "FEDERAL_ZONE",
accountRef,
cadastralKey: null,
amount: printed ?? rounded,
dueDate: parseDate(
firstMatch(text, [/Vencimiento\s*[:;.]?\s*(\d{2}\/\d{2}\/\d{4})/i]),
),
period: bimester ? BIMESTERS[+bimester[2] - 1] : null,
crossChecked,
notes,
};
}
const PARSERS: Record<string, (page: OcrPage) => ParsedStatement> = {
CFE: parseCfe,
CESPT: parseCespt,
TELNOR: parseTelnor,
"GAS TIJUANA": parseGas,
"PREDIAL TIJUANA": parsePredialTijuana,
"PREDIAL ROSARITO": parsePredialRosarito,
"PREDIAL ENSENADA": parsePredialEnsenada,
"ZONA FEDERAL TIJUANA": parseZonaFederal,
};
const EMPTY: ParsedStatement = {
provider: null,
serviceKind: null,
accountRef: null,
cadastralKey: null,
amount: null,
dueDate: null,
period: null,
crossChecked: null,
notes: [],
};
/** Detect the provider and run its parser. */
export function parseStatement(page: OcrPage): ParsedStatement {
const provider = detectProvider(page.text);
if (!provider) return { ...EMPTY, notes: ["no se reconoció el proveedor"] };
return PARSERS[provider](page);
}
@@ -0,0 +1,236 @@
import { Injectable } from "@nestjs/common";
import type { ServiceKind } from "@jorgecuadros/database";
import { PrismaService } from "../prisma/prisma.service";
import type { ParsedStatement } from "./parsers/statement-parser";
export interface MatchResult {
propertyServiceId: string | null;
customerId: string | null;
/** Why it landed here — shown in the review queue verbatim. */
note: string;
/** True only for an unambiguous hit on the scoped field. */
confident: boolean;
/** Populated when more than one service claims the same number. */
candidates: { propertyServiceId: string; customerId: string; customerName: string }[];
}
/**
* Resolves a parsed statement to the customer who should be billed for it.
*
* Two rules govern everything here.
*
* **Match on one scoped field, never fuzzily across all identifiers.** Each
* service kind has exactly one column its statements print, and only that
* column is consulted. A blanket search over accountNumber/meterNumber/route
* would let a water account number collide with an unrelated phone number, and
* the resulting mis-post would look perfectly ordinary in the ledger.
*
* **Never match on the customer name.** The name on a utility bill is the
* account's registrant, which drifts from the current owner and is often years
* stale — one sample CESPT receipt is printed to "ARNAIZ ROSAS ELSA AURORA"
* for an account this office holds under "CATT, RANDY", who is not the same
* person. Names are displayed for the reviewer to sanity-check, and are never
* an input to matching.
*/
/**
* Which `PropertyService` column a given kind's statements actually print.
*
* Exported because the same answer governs three places that must agree: the
* lookup here, the blank-service fill on review, and the write-back on confirm.
* When they disagree, a reference gets learned into a column nothing searches,
* and the same page returns to the review queue every month forever.
*
* `meterNumber` is doing double duty for the three kinds whose printed
* reference DATMEX never held in `accountNumber`:
* - GAS, where the number lived in free-text notes,
* - PROPERTY_TAX, where `accountNumber` holds DATMEX.predial — a 3-4 digit
* office file number that is neither unique nor printed on any statement.
* The Tijuana municipal receipt prints an 8-digit account and no clave
* catastral at all, so it needs a column of its own; overwriting the legacy
* predial numbers to make room would destroy the only link back to the
* original records, and
* - FEDERAL_ZONE, where `accountNumber` holds DATMEX.zfed, which is not a
* reference of any kind but a peso amount: 3 of its 77 values carry cents
* (`246.06`, `2369.09`, `22653.94`) and one is negative. Searching it for
* the concession clave the receipt prints would never hit, and — worse —
* because every row already has a value, the `[field]: null` guards in
* `learnAccountRefs` and the blank-service fill would never fire either, so
* the same page would return to the review queue every bimester forever.
*/
export function scopedRefField(
kind: ServiceKind,
): "accountNumber" | "meterNumber" | null {
switch (kind) {
case "ELECTRIC": // CFE "NO. DE SERVICIO" -> DATMEX.rpu
case "WATER": // CESPT "Cuenta" / "No. DE CUENTA" -> DATMEX.agua
case "TELEPHONE": // Telnor "Teléfono" (LADA stripped) -> DATMEX.telefono
case "CABLE":
return "accountNumber";
case "GAS": // bajagas "Cuenta" -> recovered from notes into meterNumber
case "PROPERTY_TAX": // Tijuana's 8-digit municipal account
case "FEDERAL_ZONE": // ZOFEMAT concession clave, e.g. `12T012`
return "meterNumber";
default:
return null;
}
}
@Injectable()
export class StatementMatcherService {
constructor(private readonly prisma: PrismaService) {}
async match(parsed: ParsedStatement, expectedKind: ServiceKind): Promise<MatchResult> {
const kind = parsed.serviceKind ?? expectedKind;
// The uploader labels a batch with one service kind. If the parser reads a
// page as a different provider, that is a mis-sorted page, not a match —
// posting it would book a phone bill as a water charge.
if (parsed.serviceKind && parsed.serviceKind !== expectedKind) {
return this.unmatched(
`la página parece de ${parsed.provider} (${parsed.serviceKind}) pero el lote es de ${expectedKind}`,
);
}
const field = scopedRefField(kind);
if (field && parsed.accountRef) {
const hit = await this.byServiceField(kind, field, parsed.accountRef);
if (hit) return hit;
}
// The clave catastral is printed on CESPT bills as well as predial ones, so
// it rescues a page whose account number did not OCR — which happened on
// real samples, where the clave read cleanly and the account number did
// not. On the Rosarito and Ensenada predial layouts it is not a rescue at
// all but the only identifier the receipt carries, so a unique hit there is
// as good as any account-number match and is treated as one.
if (parsed.cadastralKey) {
const primary = kind === "PROPERTY_TAX" && !parsed.accountRef;
const hit = await this.byCadastralKey(kind, parsed.cadastralKey, primary);
if (hit) return hit;
}
if (!field && !parsed.cadastralKey) {
return this.unmatched(`no hay campo de búsqueda definido para ${kind}`);
}
if (!parsed.accountRef && !parsed.cadastralKey) {
return this.unmatched(
kind === "PROPERTY_TAX"
? "no se leyó ni la clave catastral ni la cuenta municipal"
: "no se pudo leer la referencia de la cuenta",
);
}
return this.unmatched(
parsed.accountRef
? `no se encontró ningún servicio de ${kind} con la referencia ${parsed.accountRef}`
: `no se encontró ninguna propiedad con la clave catastral ${parsed.cadastralKey}`,
);
}
private async byServiceField(
kind: ServiceKind,
field: "accountNumber" | "meterNumber",
ref: string,
): Promise<MatchResult | null> {
const rows = await this.prisma.propertyService.findMany({
where: { kind, [field]: ref },
select: {
id: true,
property: {
select: { customerId: true, customer: { select: { name: true } } },
},
},
});
if (rows.length === 0) return null;
const candidates = rows.map((r) => ({
propertyServiceId: r.id,
customerId: r.property.customerId,
customerName: r.property.customer.name,
}));
// Duplicate account numbers do occur in the legacy data (the office's own
// DUPLICADOS report existed for a reason), so every candidate is surfaced
// for the reviewer to choose rather than one being picked arbitrarily.
if (rows.length > 1) {
return {
propertyServiceId: null,
customerId: null,
note: `${rows.length} servicios comparten la referencia ${ref}`,
confident: false,
candidates,
};
}
return {
propertyServiceId: candidates[0].propertyServiceId,
customerId: candidates[0].customerId,
note: `coincidencia exacta por ${field === "accountNumber" ? "número de cuenta" : "medidor"} ${ref}`,
confident: true,
candidates,
};
}
private async byCadastralKey(
kind: ServiceKind,
key: string,
/** True when the clave is the identifier the statement was issued against. */
primary: boolean,
): Promise<MatchResult | null> {
const props = await this.prisma.property.findMany({
where: { cadastralKey: key },
select: {
customerId: true,
customer: { select: { name: true } },
services: { where: { kind }, select: { id: true } },
},
});
if (props.length === 0) return null;
const candidates = props.flatMap((p) =>
(p.services.length ? p.services.map((s) => s.id) : [null]).map((sid) => ({
propertyServiceId: sid as string,
customerId: p.customerId,
customerName: p.customer.name,
})),
);
if (candidates.length > 1) {
return {
propertyServiceId: null,
customerId: null,
note: `${candidates.length} propiedades comparten la clave catastral ${key}`,
confident: false,
candidates,
};
}
// When the clave is the *secondary* key — a utility bill that also happens
// to print it — the page is left for review, because the clave was not the
// number the statement was issued against and confirming is what teaches
// the matcher the account number for next month. When it is the primary key
// (Rosarito and Ensenada predial, which print nothing else), a unique hit
// is a real match and there is no second number to learn.
return {
propertyServiceId: candidates[0].propertyServiceId ?? null,
customerId: candidates[0].customerId,
note: primary
? `coincidencia exacta por clave catastral ${key}`
: `identificado por clave catastral ${key}; confirme para registrar también el número de cuenta`,
// A clave with no service row of the right kind behind it still needs a
// human: there is nothing to attach the posting to.
confident: primary && candidates[0].propertyServiceId != null,
candidates,
};
}
private unmatched(note: string): MatchResult {
return {
propertyServiceId: null,
customerId: null,
note,
confident: false,
candidates: [],
};
}
}
+52
View File
@@ -0,0 +1,52 @@
import {
IsBoolean,
IsEnum,
IsInt,
IsNumber,
IsOptional,
IsString,
MinLength,
} from "class-validator";
import { Currency, ServiceKind, StatementDocumentStatus } from "@jorgecuadros/database";
export class CreateStatementBatchDto {
@IsEnum(ServiceKind) serviceKind!: ServiceKind;
@IsOptional() @IsString() label?: string;
}
/** Staff correction of one document's extracted fields or its match. */
export class ReviewDocumentDto {
@IsOptional() @IsString() accountRef?: string;
@IsOptional() @IsNumber() amount?: number;
@IsOptional() @IsString() period?: string;
@IsOptional() @IsString() dueDate?: string;
@IsOptional() @IsString() matchedPropertyServiceId?: string;
@IsOptional() @IsString() matchedCustomerId?: string;
// Restricted to the review-reachable states: a client cannot declare a
// document POSTED, because only a successful ledger write may do that.
@IsOptional()
@IsEnum(StatementDocumentStatus)
status?: Extract<StatementDocumentStatus, "MATCHED" | "NEEDS_REVIEW" | "CONFIRMED">;
}
/**
* Post a batch's confirmed documents. The check-level fields are shared by
* every line, exactly as on the manual batch-capture screen — an OCR batch is
* still "these receipts, paid by this check".
*/
export class ConfirmBatchDto {
@IsString() @MinLength(1) checkNumber!: string;
@IsString() @MinLength(1) transactionDate!: string;
@IsOptional() @IsEnum(Currency) currency?: Currency;
/** Overrides the concept derived from the batch's service kind. */
@IsOptional() @IsString() typeId?: string;
/** Post as outstanding (sin fondos) — captured but not yet funded. */
@IsOptional() @IsBoolean() outstanding?: boolean;
/** Also post documents a reviewer explicitly marked CONFIRMED. */
@IsOptional() @IsBoolean() includeReviewed?: boolean;
}
export class ListBatchesQuery {
@IsOptional() @IsInt() page?: number;
@IsOptional() @IsInt() pageSize?: number;
}
@@ -0,0 +1,162 @@
import {
Body,
Controller,
Get,
Param,
Patch,
Post,
Query,
Req,
Res,
StreamableFile,
UploadedFiles,
UseGuards,
UseInterceptors,
} from "@nestjs/common";
import { FilesInterceptor } from "@nestjs/platform-express";
import type { ServiceKind, StatementDocumentStatus } from "@jorgecuadros/database";
import type { Request, Response } from "express";
import { AuthenticatedGuard } from "../auth/authenticated.guard";
import { AbilityGuard } from "../auth/ability.guard";
import { RequireAbility } from "../auth/require-ability.decorator";
import { AuditService } from "../common/audit.service";
import type { UploadedFileLike } from "../storage/upload-file";
import { StatementsService } from "./statements.service";
import { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto";
/**
* Statement OCR intake (RECEIPT_CAPTURE_SPEC §2).
*
* Nothing here writes to the ledger directly — confirming a batch delegates to
* BillingService, so an OCR-captured charge is indistinguishable from a
* hand-keyed one except for its `captureSource`.
*/
@Controller("statements")
@UseGuards(AuthenticatedGuard, AbilityGuard)
export class StatementsController {
constructor(
private readonly statements: StatementsService,
private readonly audit: AuditService,
) {}
private actingId(req: Request): string {
return (req.user as { id: string } | undefined)?.id ?? "";
}
/**
* Whether this deployment can ingest scans at all — the UI hides automatic
* capture without it. Both halves are needed: OCR to read the page, object
* storage to keep it.
*/
@Get("status")
async status() {
return {
ocrAvailable: await this.statements.ocrAvailable(),
storageAvailable: this.statements.storageAvailable(),
};
}
@Get("batches")
listBatches(@Query("page") page?: string, @Query("pageSize") pageSize?: string) {
return this.statements.listBatches(
Math.max(1, Number(page) || 1),
Math.min(100, Math.max(1, Number(pageSize) || 25)),
);
}
@Get("batches/:id")
getBatch(@Param("id") id: string) {
return this.statements.getBatch(id);
}
@Get("batches/:id/documents")
listDocuments(@Param("id") id: string, @Query("status") status?: string) {
return this.statements.listDocuments(
id,
(status || undefined) as StatementDocumentStatus | undefined,
);
}
/** The rendered page, so a reviewer can compare it against what was read. */
@Get("documents/:id/page")
async pageImage(@Param("id") id: string, @Res({ passthrough: true }) res: Response) {
const { stream, contentType, contentLength } = await this.statements.pageImage(id);
res.set({
"Content-Type": contentType ?? "image/png",
...(contentLength ? { "Content-Length": String(contentLength) } : {}),
});
return new StreamableFile(stream);
}
// --- writes ---------------------------------------------------------------
@Post("batches")
@RequireAbility("statement:ingest")
@UseInterceptors(
// A month of one company's statements is a handful of multi-page scans;
// 25 files at 50MB covers that with room to spare.
FilesInterceptor("files", 25, { limits: { fileSize: 50 * 1024 * 1024 } }),
)
async createBatch(
@UploadedFiles() files: UploadedFileLike[] | undefined,
@Query("serviceKind") serviceKind: ServiceKind,
@Query("label") label: string | undefined,
@Req() req: Request,
) {
const batch = await this.statements.createBatch(
files ?? [],
serviceKind,
this.actingId(req),
label,
);
void this.audit.log(this.actingId(req), "statement.batch.create", {
batchId: batch.id,
serviceKind,
fileCount: batch.fileCount,
});
return batch;
}
@Patch("documents/:id")
@RequireAbility("statement:review")
async review(
@Param("id") id: string,
@Body() dto: ReviewDocumentDto,
@Req() req: Request,
) {
const doc = await this.statements.review(id, dto, this.actingId(req));
void this.audit.log(this.actingId(req), "statement.document.review", {
documentId: id,
status: doc.status,
});
return doc;
}
@Post("documents/:id/reject")
@RequireAbility("statement:review")
async reject(@Param("id") id: string, @Req() req: Request) {
const doc = await this.statements.reject(id, this.actingId(req));
void this.audit.log(this.actingId(req), "statement.document.reject", {
documentId: id,
});
return doc;
}
/** Post every matched document in the batch, against one check. */
@Post("batches/:id/confirm")
@RequireAbility("statement:review")
async confirm(
@Param("id") id: string,
@Body() dto: ConfirmBatchDto,
@Req() req: Request,
) {
const result = await this.statements.confirmBatch(id, dto, this.actingId(req));
void this.audit.log(this.actingId(req), "statement.batch.confirm", {
batchId: id,
posted: result.posted,
total: result.total,
checkNumber: dto.checkNumber,
});
return result;
}
}
@@ -0,0 +1,18 @@
import { Module } from "@nestjs/common";
import { BillingModule } from "../billing/billing.module";
import { OcrModule } from "../ocr/ocr.module";
import { StatementsController } from "./statements.controller";
import { StatementsService } from "./statements.service";
import { StatementMatcherService } from "./statement-matcher.service";
/**
* The concrete OCR engine is bound in OcrModule (see apps/api/src/ocr/) —
* everything downstream depends on the OcrProvider interface, so swapping
* Tesseract for a managed extraction API is a one-line change there.
*/
@Module({
imports: [BillingModule, OcrModule],
controllers: [StatementsController],
providers: [StatementsService, StatementMatcherService],
})
export class StatementsModule {}
@@ -0,0 +1,480 @@
import {
BadRequestException,
Inject,
Injectable,
Logger,
NotFoundException,
} from "@nestjs/common";
import {
Prisma,
type ServiceKind,
type StatementDocumentStatus,
} from "@jorgecuadros/database";
import { PrismaService } from "../prisma/prisma.service";
import { StorageService } from "../storage/storage.service";
import { BillingService } from "../billing/billing.service";
import type { UploadedFileLike } from "../storage/upload-file";
import { OCR_PROVIDER, type OcrProvider } from "./ocr/ocr.provider";
import { parseStatement } from "./parsers/statement-parser";
import { StatementMatcherService, scopedRefField } from "./statement-matcher.service";
import type { ConfirmBatchDto, ReviewDocumentDto } from "./statement.dto";
/**
* Default ledger concept per service kind. The names are the legacy
* `TYPE OF TRX` values already in `type_transactions`, resolved by name once
* per confirm rather than hard-coded as ids, which differ per environment.
*/
const CONCEPT_BY_KIND: Partial<Record<ServiceKind, string>> = {
ELECTRIC: "ELECTRIC",
WATER: "WATER",
TELEPHONE: "TELEPHONE",
GAS: "GAS BUTANO",
PROPERTY_TAX: "PROPERTY TAXES",
FEDERAL_ZONE: "FEDERAL ZONE",
CABLE: "CABLE",
};
/** Statuses a document can still be worked on from. */
const OPEN: StatementDocumentStatus[] = ["NEEDS_REVIEW", "MATCHED", "CONFIRMED"];
@Injectable()
export class StatementsService {
private readonly logger = new Logger(StatementsService.name);
constructor(
private readonly prisma: PrismaService,
private readonly storage: StorageService,
private readonly billing: BillingService,
private readonly matcher: StatementMatcherService,
@Inject(OCR_PROVIDER) private readonly ocr: OcrProvider,
) {}
ocrAvailable(): Promise<boolean> {
return this.ocr.available();
}
/** Scans are stored as blobs, so no object storage means no intake. */
storageAvailable(): boolean {
return this.storage.available;
}
// --- ingest ---------------------------------------------------------------
/**
* Accept a batch of scanned PDFs and start processing.
*
* Processing is kicked off but deliberately not awaited: 300 pages of OCR is
* minutes of CPU, far past any sane HTTP timeout. The caller gets the batch
* id immediately and polls its status, which is also what lets the review
* queue show partial progress.
*/
async createBatch(
files: UploadedFileLike[],
serviceKind: ServiceKind,
uploadedById: string,
label?: string,
) {
if (!files?.length) throw new BadRequestException("No se recibió ningún archivo.");
if (!(await this.ocr.available())) {
throw new BadRequestException(
"El servidor no tiene OCR instalado; no se pueden procesar recibos.",
);
}
// Checked here rather than at the first `put`, which would only surface as
// a FAILED batch minutes later.
if (!this.storage.available) {
throw new BadRequestException(
"El almacenamiento de documentos no está configurado; no se pueden " +
"guardar los recibos escaneados.",
);
}
const batch = await this.prisma.statementBatch.create({
data: { serviceKind, uploadedById, label, fileCount: files.length },
});
// Buffers are held for the background pass; the request's own copies would
// otherwise be garbage once the response is sent.
const copies = files.map((f) => ({ buffer: f.buffer, name: f.originalname }));
void this.process(batch.id, copies, serviceKind).catch(async (err) => {
this.logger.error(`Batch ${batch.id} failed: ${(err as Error).message}`);
await this.prisma.statementBatch.update({
where: { id: batch.id },
data: { status: "FAILED", error: (err as Error).message },
});
});
return batch;
}
/** Render → OCR → parse → match, one document row per page. */
private async process(
batchId: string,
files: { buffer: Buffer; name?: string }[],
serviceKind: ServiceKind,
) {
await this.prisma.statementBatch.update({
where: { id: batchId },
data: { status: "PROCESSING" },
});
let pageNumber = 0;
for (const file of files) {
// The source PDF is kept as well as the page images: it is the artifact
// the office actually received, and the only way to re-run a corrected
// parser over the original later.
const sourceKey = `statement/${batchId}/source-${pageNumber + 1}.pdf`;
await this.storage.put(sourceKey, file.buffer, "application/pdf");
const pages = await this.ocr.renderPages(file.buffer);
// Page images are still rendered and stored for every file, text layer or
// not: the review screen shows the reviewer the page, and "what the
// parser read" is only checkable against a picture of the paper.
const textLayer = await this.ocr.textPages(file.buffer).catch(() => []);
for (const [index, image] of pages.entries()) {
pageNumber += 1;
const storageKey = `statement/${batchId}/page-${pageNumber}.png`;
await this.storage.put(storageKey, image, "image/png");
try {
const embedded = textLayer[index] ?? null;
const ocr = embedded ?? (await this.ocr.recognize(image));
const parsed = parseStatement(ocr);
if (embedded) {
parsed.notes.unshift("texto leído del PDF original, sin OCR");
}
const match = await this.matcher.match(parsed, serviceKind);
const notes = [...parsed.notes, match.note].filter(Boolean);
// A confident field match is only trusted when nothing contradicts
// it: a barcode that disagrees with the printed number means one of
// the two was misread, and which one is a judgement call.
const trusted = match.confident && parsed.crossChecked !== false;
await this.prisma.statementDocument.create({
data: {
batchId,
pageNumber,
storageKey,
status: trusted ? "MATCHED" : "NEEDS_REVIEW",
ocrRawText: ocr.text,
ocrConfidence: new Prisma.Decimal(ocr.confidence.toFixed(3)),
provider: parsed.provider,
extractedAccountRef: parsed.accountRef,
extractedAmount:
parsed.amount != null ? new Prisma.Decimal(parsed.amount) : null,
extractedPeriod: parsed.period,
extractedDueDate: parsed.dueDate,
extractedCadastralKey: parsed.cadastralKey,
matchedPropertyServiceId: match.propertyServiceId,
matchedCustomerId: match.customerId,
matchNote: notes.join("; ").slice(0, 190),
},
});
} catch (err) {
// One unreadable page must not abandon the other 299.
await this.prisma.statementDocument.create({
data: {
batchId,
pageNumber,
storageKey,
status: "OCR_FAILED",
matchNote: (err as Error).message.slice(0, 190),
},
});
}
}
}
await this.prisma.statementBatch.update({
where: { id: batchId },
data: { status: "READY_FOR_REVIEW" },
});
}
// --- reads ----------------------------------------------------------------
async listBatches(page: number, pageSize: number) {
const [total, items] = await this.prisma.$transaction([
this.prisma.statementBatch.count(),
this.prisma.statementBatch.findMany({
orderBy: { createdAt: "desc" },
skip: (page - 1) * pageSize,
take: pageSize,
include: {
uploadedBy: { select: { name: true } },
_count: { select: { documents: true } },
},
}),
]);
return { items, total, page, pageSize, pageCount: Math.ceil(total / pageSize) };
}
async getBatch(id: string) {
const batch = await this.prisma.statementBatch.findUnique({
where: { id },
include: { uploadedBy: { select: { name: true } } },
});
if (!batch) throw new NotFoundException("Lote no encontrado.");
const counts = await this.prisma.statementDocument.groupBy({
by: ["status"],
where: { batchId: id },
_count: { _all: true },
});
const totals = await this.prisma.statementDocument.aggregate({
where: { batchId: id, status: { in: OPEN } },
_sum: { extractedAmount: true },
});
return {
...batch,
byStatus: Object.fromEntries(counts.map((c) => [c.status, c._count._all])),
pendingTotal: totals._sum.extractedAmount?.toFixed(2) ?? "0.00",
};
}
async listDocuments(batchId: string, status?: StatementDocumentStatus) {
return this.prisma.statementDocument.findMany({
where: { batchId, ...(status ? { status } : {}) },
orderBy: { pageNumber: "asc" },
include: {
matchedCustomer: { select: { id: true, name: true } },
matchedPropertyService: {
select: {
id: true,
kind: true,
accountNumber: true,
meterNumber: true,
property: { select: { id: true, addressLine1: true } },
},
},
},
});
}
/** The rendered page image, so a reviewer can read what the parser read. */
async pageImage(documentId: string) {
const doc = await this.prisma.statementDocument.findUnique({
where: { id: documentId },
select: { storageKey: true },
});
if (!doc) throw new NotFoundException("Documento no encontrado.");
return this.storage.getStream(doc.storageKey);
}
// --- review ---------------------------------------------------------------
/** Staff correction of an extracted field or of the match itself. */
async review(id: string, dto: ReviewDocumentDto, reviewedById: string) {
const doc = await this.prisma.statementDocument.findUnique({ where: { id } });
if (!doc) throw new NotFoundException("Documento no encontrado.");
if (doc.status === "POSTED") {
throw new BadRequestException("Este documento ya fue registrado.");
}
// Changing the service implies its owner; deriving the customer here rather
// than trusting a client-supplied pair is what stops a page being posted to
// one customer's ledger against another customer's service.
let matchedCustomerId = doc.matchedCustomerId;
let matchedPropertyServiceId = dto.matchedPropertyServiceId ?? undefined;
if (dto.matchedPropertyServiceId) {
const svc = await this.prisma.propertyService.findUnique({
where: { id: dto.matchedPropertyServiceId },
select: { property: { select: { customerId: true } } },
});
if (!svc) throw new BadRequestException("Servicio no encontrado.");
matchedCustomerId = svc.property.customerId;
} else if (dto.matchedCustomerId) {
matchedCustomerId = dto.matchedCustomerId;
// A reviewer picks a *customer*, not one of their service rows. Without
// a service the posting still works, but the confirmed reference has
// nowhere to be written back, so the same account would land in review
// again next month — which is exactly the behaviour that is supposed to
// make gas (whose numbers the migration never populated) a one-time cost.
// So: if the batch's service kind resolves to exactly one of that
// customer's services that has no reference yet, attach it. Exactly one
// — with two candidates there is no way to tell which meter or line the
// bill belongs to, and guessing would write a real number onto the wrong
// service.
const batch = await this.prisma.statementBatch.findUnique({
where: { id: doc.batchId },
select: { serviceKind: true },
});
const field = batch && scopedRefField(batch.serviceKind);
if (batch && field) {
const blank = await this.prisma.propertyService.findMany({
where: {
kind: batch.serviceKind,
[field]: null,
property: { customerId: matchedCustomerId },
},
select: { id: true },
take: 2,
});
if (blank.length === 1) matchedPropertyServiceId = blank[0].id;
}
}
return this.prisma.statementDocument.update({
where: { id },
data: {
extractedAccountRef: dto.accountRef ?? undefined,
extractedAmount:
dto.amount != null ? new Prisma.Decimal(dto.amount) : undefined,
extractedPeriod: dto.period ?? undefined,
extractedDueDate: dto.dueDate ? new Date(dto.dueDate) : undefined,
matchedPropertyServiceId,
matchedCustomerId,
status: dto.status ?? "MATCHED",
reviewedById,
reviewedAt: new Date(),
},
});
}
async reject(id: string, reviewedById: string) {
const doc = await this.prisma.statementDocument.findUnique({ where: { id } });
if (!doc) throw new NotFoundException("Documento no encontrado.");
if (doc.status === "POSTED") {
throw new BadRequestException("Este documento ya fue registrado.");
}
return this.prisma.statementDocument.update({
where: { id },
data: { status: "REJECTED", reviewedById, reviewedAt: new Date() },
});
}
// --- posting --------------------------------------------------------------
/**
* Post every confirmable document in a batch to the ledger.
*
* This goes through `BillingService.createBatch` — the same method the manual
* "Editor" screen uses — rather than writing `Transaction` rows directly, so
* OCR-sourced and hand-keyed receipts share one write path, one validation
* path and one audit trail. `source: "OCR"` and a per-line `captureRef` of
* the document id give the duplicate-post guard something to key on, so a
* batch confirmed twice cannot double-charge anyone.
*/
async confirmBatch(batchId: string, dto: ConfirmBatchDto, reviewedById: string) {
const batch = await this.prisma.statementBatch.findUnique({
where: { id: batchId },
});
if (!batch) throw new NotFoundException("Lote no encontrado.");
const docs = await this.prisma.statementDocument.findMany({
where: {
batchId,
status: { in: dto.includeReviewed ? ["MATCHED", "CONFIRMED"] : ["MATCHED"] },
matchedCustomerId: { not: null },
},
orderBy: { pageNumber: "asc" },
});
if (!docs.length) {
throw new BadRequestException("No hay documentos listos para registrar.");
}
const missing = docs.filter((d) => d.extractedAmount == null);
if (missing.length) {
throw new BadRequestException(
`Falta el importe en ${missing.length} documento(s): página(s) ` +
missing.map((d) => d.pageNumber).join(", "),
);
}
const typeId = dto.typeId ?? (await this.conceptFor(batch.serviceKind));
const result = await this.billing.createBatch(
{
domain: "UTILITY",
transactionDate: dto.transactionDate,
checkNumber: dto.checkNumber,
currency: dto.currency ?? "MXN",
typeId,
lines: docs.map((d) => ({
customerId: d.matchedCustomerId!,
// Charges are negative in this ledger: a negative amount is what the
// customer owes. The parser reads the printed (positive) figure, so
// the sign is applied here, at the single point where a statement
// becomes a ledger row.
amount: -Math.abs(Number(d.extractedAmount)),
reference: d.extractedAccountRef ?? undefined,
period: d.extractedPeriod ?? undefined,
outstanding: dto.outstanding ?? false,
})),
},
{ source: "OCR", refs: docs.map((d) => d.id) },
);
// `items[i]` is positionally parallel to `lines[i]` (seam guarantee 1), so
// the created rows zip straight back onto the documents that produced them.
await this.prisma.$transaction(
docs.map((d, i) =>
this.prisma.statementDocument.update({
where: { id: d.id },
data: {
status: "POSTED",
postedTransactionId: result.items[i].id,
reviewedById,
reviewedAt: new Date(),
},
}),
),
);
// Teach the matcher. When a document was matched by clave catastral or by
// hand because the scoped field was blank, writing the reference back means
// next month's statement for the same account matches on its own — this is
// what turns gas (whose numbers the migration never populated) from a
// permanent review queue into a one-time cost.
await this.learnAccountRefs(docs, batch.serviceKind);
await this.closeIfDone(batchId);
return { posted: result.count, total: result.total, checkNumber: dto.checkNumber };
}
/** Write a confirmed reference onto a service that had none. */
private async learnAccountRefs(
docs: { matchedPropertyServiceId: string | null; extractedAccountRef: string | null }[],
kind: ServiceKind,
) {
const field = scopedRefField(kind);
if (!field) return;
for (const d of docs) {
if (!d.matchedPropertyServiceId || !d.extractedAccountRef) continue;
await this.prisma.propertyService.updateMany({
// Only fills a hole — never overwrites a number already on file, which
// would let one misread page rewrite good reference data.
where: { id: d.matchedPropertyServiceId, [field]: null },
data: { [field]: d.extractedAccountRef },
});
}
}
private async closeIfDone(batchId: string) {
const open = await this.prisma.statementDocument.count({
where: { batchId, status: { in: OPEN } },
});
if (open === 0) {
await this.prisma.statementBatch.update({
where: { id: batchId },
data: { status: "COMPLETED", completedAt: new Date() },
});
}
}
private async conceptFor(kind: ServiceKind): Promise<string | undefined> {
const name = CONCEPT_BY_KIND[kind];
if (!name) return undefined;
const row = await this.prisma.typeTransaction.findFirst({
where: { nameEn: name },
select: { id: true },
});
return row?.id;
}
}
+10
View File
@@ -73,6 +73,16 @@ export class StorageService implements OnModuleInit {
}
}
/**
* Whether the deployment has object storage at all. Callers use this to
* refuse work up front instead of failing halfway through — a recibo batch
* that dies on its first `put` leaves a FAILED batch and no explanation the
* office can act on.
*/
get available(): boolean {
return this.client !== null;
}
private require(): S3Client {
if (!this.client) {
throw new ServiceUnavailableException(
+4
View File
@@ -0,0 +1,4 @@
{
"extends": "./tsconfig.json",
"exclude": ["node_modules", "dist", "**/*.spec.ts"]
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@jorgecuadros/web",
"version": "1.0.0",
"version": "1.0.6",
"private": true,
"scripts": {
"dev": "next dev -p 4500",
+3 -549
View File
@@ -1,557 +1,11 @@
"use client";
import { useEffect, useMemo, useState } from "react";
import Link from "next/link";
import { AppShell } from "@/components/AppShell";
import { CustomerPicker } from "@/components/CustomerPicker";
import { createMovementBatch, getBillingFacets, getByCheck } from "@/lib/api";
import { useCan } from "@/lib/abilities";
import { formatMoney, formatNumber, txTypeLabel } from "@/lib/labels";
import type {
BatchCreateInput,
BillingFacets,
ByCheckResponse,
Currency,
LedgerCurrency,
TransactionDomain,
} from "@/lib/types";
/**
* Batch capture by check — the "Editor" screen from the legacy system
* (docs/RECEIPT_CAPTURE_SPEC.md §1.2).
*
* Staff key many customers' receipts against ONE physical check before cutting
* it, then check that the captured total matches the check's amount. That
* reconciliation is the whole point, so the running total is the most prominent
* thing on the page and an optional "importe del cheque" field turns it into a
* live difference.
*
* No batch entity is persisted: `checkNumber` is a plain column, and grouping
* by it answers every by-check question (see the "Reporte por cheque" report).
*/
const DOMAINS: { key: TransactionDomain; label: string }[] = [
{ key: "UTILITY", label: "Servicios" },
{ key: "INSURANCE", label: "Seguros" },
{ key: "TRUST", label: "Fideicomiso" },
];
interface Line {
/** Local row key — lines have no server identity until the batch posts. */
key: number;
customerId: string;
customerName: string;
amount: string;
reference: string;
period: string;
outstanding: boolean;
}
function blankLine(key: number): Line {
return {
key,
customerId: "",
customerName: "",
amount: "",
reference: "",
period: "",
outstanding: false,
};
}
import { Captura } from "@/components/Captura";
/** Daily capture, opened on the manual (key-by-hand) mode. */
export default function BatchCapturePage() {
return (
<AppShell>
<BatchCapture />
<Captura initialMode="manual" />
</AppShell>
);
}
function BatchCapture() {
const canCapture = useCan("ledger:create");
const [facets, setFacets] = useState<BillingFacets | null>(null);
// Check-level fields — shared by every line.
const [domain, setDomain] = useState<TransactionDomain>("UTILITY");
const [currency, setCurrency] = useState<LedgerCurrency>("MXN");
const [typeId, setTypeId] = useState("");
const [checkNumber, setCheckNumber] = useState("");
const [transactionDate, setTransactionDate] = useState(
new Date().toISOString().slice(0, 10),
);
/** The physical check's amount, for reconciliation only — never submitted. */
const [checkAmount, setCheckAmount] = useState("");
const [lines, setLines] = useState<Line[]>([blankLine(1), blankLine(2), blankLine(3)]);
const [nextKey, setNextKey] = useState(4);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const [posted, setPosted] = useState<ByCheckResponse | null>(null);
useEffect(() => {
getBillingFacets().then(setFacets).catch(() => setFacets(null));
}, []);
const filled = lines.filter(
(l) => l.customerId && l.amount.trim() !== "" && Number.isFinite(Number(l.amount)),
);
// Charges are captured as positive numbers and signed on submit, matching
// MovementForm — staff type what's on the bill, not a negative.
const total = useMemo(
() =>
filled
.filter((l) => !l.outstanding)
.reduce((sum, l) => sum + Math.abs(Number(l.amount)), 0),
[filled],
);
const outstandingTotal = useMemo(
() =>
filled
.filter((l) => l.outstanding)
.reduce((sum, l) => sum + Math.abs(Number(l.amount)), 0),
[filled],
);
const checkAmt = Number(checkAmount);
const hasCheckAmt = checkAmount.trim() !== "" && Number.isFinite(checkAmt);
const diff = hasCheckAmt ? checkAmt - total : 0;
const reconciled = hasCheckAmt && Math.abs(diff) < 0.005;
function update(key: number, patch: Partial<Line>) {
setLines((ls) => ls.map((l) => (l.key === key ? { ...l, ...patch } : l)));
}
function addLine() {
setLines((ls) => [...ls, blankLine(nextKey)]);
setNextKey((k) => k + 1);
}
function removeLine(key: number) {
setLines((ls) => (ls.length === 1 ? ls : ls.filter((l) => l.key !== key)));
}
async function submit(e: React.FormEvent) {
e.preventDefault();
if (!checkNumber.trim()) {
setError("Indica el número de cheque.");
return;
}
if (filled.length === 0) {
setError("Captura al menos una línea con cliente y monto.");
return;
}
const dupes = filled
.map((l) => l.customerId)
.filter((id, i, arr) => arr.indexOf(id) !== i);
if (dupes.length) {
const names = filled
.filter((l) => dupes.includes(l.customerId))
.map((l) => l.customerName);
if (
!window.confirm(
`Hay más de una línea para el mismo cliente (${[...new Set(names)].join(
", ",
)}). ¿Continuar?`,
)
)
return;
}
const payload: BatchCreateInput = {
domain,
transactionDate,
checkNumber: checkNumber.trim(),
currency: currency as Currency,
typeId: typeId || undefined,
lines: filled.map((l) => ({
customerId: l.customerId,
// Every line of a check batch is a charge the office paid out.
amount: -Math.abs(Number(l.amount)),
reference: l.reference.trim() || undefined,
period: l.period.trim() || undefined,
outstanding: l.outstanding || undefined,
})),
};
setSaving(true);
setError(null);
try {
await createMovementBatch(payload);
// Re-read through the by-check view so the confirmation shows what's
// actually stored (including anything captured against this check
// earlier), not just what this request sent.
setPosted(await getByCheck(payload.checkNumber));
} catch (e2) {
setError((e2 as Error)?.message ?? "No se pudo guardar el lote.");
} finally {
setSaving(false);
}
}
function reset() {
setPosted(null);
setLines([blankLine(nextKey), blankLine(nextKey + 1), blankLine(nextKey + 2)]);
setNextKey((k) => k + 3);
setCheckNumber("");
setCheckAmount("");
}
if (!canCapture) {
return (
<div className="state-box state-error">
No tienes permiso para capturar movimientos.
</div>
);
}
if (posted) {
return (
<>
<div className="page-head">
<div>
<h1 className="page-title">Lote capturado</h1>
<p className="eyebrow">
Cheque {posted.checkNumber} · {formatNumber(posted.count)}{" "}
{posted.count === 1 ? "movimiento" : "movimientos"}
</p>
</div>
<div style={{ display: "flex", gap: 10 }}>
<button type="button" className="btn btn-primary" onClick={reset}>
Capturar otro cheque
</button>
<Link href="/estado-cuenta" className="btn btn-outline">
Volver a estado de cuenta
</Link>
</div>
</div>
<div className="filtered-totals" style={{ marginBottom: 16 }}>
{posted.totals.map((t) => (
<div className="filtered-total" key={t.currency}>
<span className="filtered-total-cur">{t.currency}</span>
<span className="filtered-total-net">
Total del cheque <strong>{formatMoney(t.total, t.currency)}</strong>
</span>
<span>{formatNumber(t.count)} movimientos</span>
</div>
))}
{posted.outstandingCount > 0 && (
<div className="filtered-total">
<span>
{formatNumber(posted.outstandingCount)} sin fondos (no suman al
total)
</span>
</div>
)}
</div>
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th>Cliente</th>
<th>Referencia</th>
<th>Periodo</th>
<th>Estado</th>
<th className="num">Monto</th>
</tr>
</thead>
<tbody>
{posted.items.map((i) => (
<tr key={i.id}>
<td>
<Link
href={`/estado-cuenta/${i.customerId}`}
className="inline-link"
>
{i.customerName}
</Link>
</td>
<td>{i.reference || "—"}</td>
<td>{i.period || "—"}</td>
<td>{i.outstanding ? "Sin fondos" : "Pagado"}</td>
<td className="num">
<span className="tx-amount neg">
{formatMoney(i.amount, i.currency)}
</span>
</td>
</tr>
))}
</tbody>
</table>
</div>
<p className="muted" style={{ marginTop: 14 }}>
Para imprimir la conciliación, usa el reporte{" "}
<Link
href={`/reportes/cheque-count?checkNumber=${encodeURIComponent(
posted.checkNumber,
)}`}
className="inline-link"
>
Reporte por cheque
</Link>
.
</p>
</>
);
}
return (
<>
<div className="page-head">
<div>
<h1 className="page-title">Captura por cheque</h1>
<p className="eyebrow">
Captura los recibos de varios clientes contra un mismo cheque y
concilia el total antes de guardar.
</p>
</div>
<Link href="/estado-cuenta" className="btn btn-outline">
Cancelar
</Link>
</div>
{error && <div className="state-box state-error">{error}</div>}
<form onSubmit={submit}>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<h2 className="section-title" style={{ marginBottom: 14 }}>
Datos del cheque
</h2>
<div className="form-grid">
<label className="field">
<span className="field-label">Número de cheque *</span>
<input
className="input"
value={checkNumber}
onChange={(e) => setCheckNumber(e.target.value)}
required
/>
</label>
<label className="field">
<span className="field-label">Fecha *</span>
<input
className="input"
type="date"
required
value={transactionDate}
onChange={(e) => setTransactionDate(e.target.value)}
/>
</label>
<label className="field">
<span className="field-label">Línea de negocio *</span>
<select
className="select"
value={domain}
onChange={(e) => setDomain(e.target.value as TransactionDomain)}
>
{DOMAINS.map((d) => (
<option key={d.key} value={d.key}>
{d.label}
</option>
))}
</select>
</label>
<label className="field">
<span className="field-label">Moneda *</span>
<select
className="select"
value={currency}
onChange={(e) => setCurrency(e.target.value as LedgerCurrency)}
>
<option value="MXN">Pesos (MXN)</option>
<option value="USD">Dólares (USD)</option>
</select>
</label>
<label className="field">
<span className="field-label">Concepto</span>
<select
className="select"
value={typeId}
onChange={(e) => setTypeId(e.target.value)}
>
<option value="">(sin concepto)</option>
{facets?.types.map((t) => (
<option key={t.id} value={t.id}>
{txTypeLabel({ nameEn: t.name })}
</option>
))}
</select>
</label>
<label className="field">
<span className="field-label">Importe del cheque</span>
<input
className="input"
type="number"
step="0.01"
min="0"
value={checkAmount}
onChange={(e) => setCheckAmount(e.target.value)}
placeholder="Para conciliar"
/>
</label>
</div>
</div>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: 14,
}}
>
<h2 className="section-title" style={{ margin: 0 }}>
Recibos ({formatNumber(filled.length)})
</h2>
<button type="button" className="btn btn-outline" onClick={addLine}>
Agregar línea
</button>
</div>
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th style={{ minWidth: 240 }}>Cliente *</th>
<th style={{ minWidth: 120 }}>Referencia</th>
<th style={{ minWidth: 100 }}>Periodo</th>
<th style={{ minWidth: 110 }} className="num">
Monto *
</th>
<th style={{ whiteSpace: "nowrap" }}>Sin fondos</th>
<th style={{ width: 1 }} />
</tr>
</thead>
<tbody>
{lines.map((l) => (
<tr key={l.key}>
<td>
<CustomerPicker
value={l.customerId}
valueName={l.customerId ? l.customerName : undefined}
onPick={(id, name) =>
update(l.key, { customerId: id, customerName: name })
}
/>
</td>
<td>
<input
className="input"
value={l.reference}
onChange={(e) =>
update(l.key, { reference: e.target.value })
}
/>
</td>
<td>
<input
className="input"
value={l.period}
onChange={(e) => update(l.key, { period: e.target.value })}
placeholder="2026-07"
/>
</td>
<td>
<input
className="input num"
type="number"
step="0.01"
min="0"
value={l.amount}
onChange={(e) => update(l.key, { amount: e.target.value })}
placeholder="0.00"
/>
</td>
<td style={{ textAlign: "center" }}>
<input
type="checkbox"
checked={l.outstanding}
onChange={(e) =>
update(l.key, { outstanding: e.target.checked })
}
aria-label="Sin fondos"
/>
</td>
<td>
<button
type="button"
className="btn btn-ghost"
style={{ padding: "4px 10px", fontSize: 12 }}
onClick={() => removeLine(l.key)}
disabled={lines.length === 1}
>
Quitar
</button>
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<h2 className="section-title" style={{ marginBottom: 14 }}>
Conciliación
</h2>
<div className="filtered-totals">
<div className="filtered-total">
<span className="filtered-total-cur">{currency}</span>
<span className="filtered-total-net">
Capturado <strong>{formatMoney(String(-total), currency)}</strong>
</span>
<span>{formatNumber(filled.filter((l) => !l.outstanding).length)} recibos</span>
</div>
{outstandingTotal > 0 && (
<div className="filtered-total">
<span>
Sin fondos{" "}
<strong>{formatMoney(String(-outstandingTotal), currency)}</strong>{" "}
(no suma al cheque)
</span>
</div>
)}
{hasCheckAmt && (
<div className="filtered-total">
<span className="filtered-total-net">
{reconciled ? (
<strong className="tx-amount pos">Cuadra con el cheque</strong>
) : (
<>
Diferencia{" "}
<strong className="tx-amount neg">
{formatMoney(String(diff), currency)}
</strong>
</>
)}
</span>
</div>
)}
</div>
</div>
<div className="form-actions">
<button
type="submit"
className="btn btn-primary"
disabled={saving || filled.length === 0}
>
{saving
? "Guardando…"
: `Capturar ${formatNumber(filled.length)} ${
filled.length === 1 ? "recibo" : "recibos"
}`}
</button>
<Link href="/estado-cuenta" className="btn btn-outline">
Cancelar
</Link>
</div>
</form>
</>
);
}
+45
View File
@@ -180,6 +180,11 @@ button {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
cursor: help;
/* Full 40-char commit hashes, and two of them when the tiers disagree. Hex
offers no break opportunity, so without this the line pushes the page into
horizontal scroll on a phone. */
min-width: 0;
overflow-wrap: anywhere;
}
.shell-footer-warn {
color: var(--negative);
@@ -832,6 +837,46 @@ button {
display: inline-block;
}
/* Upload progress (Operaciones ingest) */
.upload-progress {
display: flex;
flex-direction: column;
gap: 6px;
padding: 4px 0 8px;
}
.progress-track {
position: relative;
overflow: hidden;
height: 8px;
border-radius: 999px;
background: var(--paper-2);
}
.progress-fill {
height: 100%;
border-radius: 999px;
background: var(--brand-500);
transition: width 0.2s linear;
}
.progress-indeterminate .progress-fill {
width: 40% !important;
animation: progress-slide 1.2s var(--ease-out-quart) infinite;
}
@keyframes progress-slide {
0% {
transform: translateX(-100%);
}
100% {
transform: translateX(250%);
}
}
.upload-progress-stats {
display: flex;
flex-wrap: wrap;
gap: 12px;
font-size: 12px;
color: var(--muted);
}
@keyframes shimmer {
0% {
background-position: -420px 0;
+72 -3
View File
@@ -1,6 +1,6 @@
"use client";
import { useCallback, useEffect, useRef, useState } from "react";
import { Fragment, useCallback, useEffect, useRef, useState } from "react";
import { AppShell } from "@/components/AppShell";
import { useCan } from "@/lib/abilities";
import {
@@ -20,6 +20,7 @@ import {
startOpsJob,
uploadIngest,
} from "@/lib/api";
import type { UploadProgress } from "@/lib/api";
import type {
BackupFile,
IngestFile,
@@ -56,6 +57,7 @@ function Operaciones() {
const [confirm, setConfirm] = useState<ConfirmState>(null);
const [confirmText, setConfirmText] = useState("");
const [uploading, setUploading] = useState<string | null>(null);
const [progress, setProgress] = useState<UploadProgress | null>(null);
const [starting, setStarting] = useState(false);
const fileInputs = useRef<Record<string, HTMLInputElement | null>>({});
@@ -119,14 +121,16 @@ function Operaciones() {
setError(null);
setNotice(null);
setUploading(name);
setProgress(null);
try {
await uploadIngest(name, file);
await uploadIngest(name, file, setProgress);
setNotice(`${name} cargado.`);
refreshLists();
} catch (e) {
setError((e as Error)?.message ?? "No se pudo cargar el archivo.");
} finally {
setUploading(null);
setProgress(null);
const input = fileInputs.current[name];
if (input) input.value = "";
}
@@ -243,7 +247,8 @@ function Operaciones() {
</thead>
<tbody>
{(ingest ?? []).map((f) => (
<tr key={f.name}>
<Fragment key={f.name}>
<tr>
<td className="mono">{f.name}</td>
<td>
<span className={`badge ${f.present ? "badge-positive" : "badge-negative"}`}>
@@ -274,6 +279,7 @@ function Operaciones() {
<button
className="btn btn-ghost"
type="button"
disabled={uploading === f.name}
onClick={() => handleDeleteIngest(f.name)}
>
Eliminar
@@ -282,6 +288,14 @@ function Operaciones() {
</div>
</td>
</tr>
{uploading === f.name && (
<tr>
<td colSpan={5}>
<UploadProgressBar progress={progress} />
</td>
</tr>
)}
</Fragment>
))}
</tbody>
</table>
@@ -496,6 +510,61 @@ function Operaciones() {
);
}
/** "1:05" / "0:09" — remaining time, coarse on purpose. */
function formatEta(seconds: number): string {
const s = Math.max(0, Math.round(seconds));
if (s >= 3600) {
const h = Math.floor(s / 3600);
const m = Math.round((s % 3600) / 60);
return `${h} h ${m} min`;
}
return `${Math.floor(s / 60)}:${String(s % 60).padStart(2, "0")}`;
}
/**
* Live upload readout. The bar tracks bytes handed to the network; once those
* are all sent the server still has to write the file, so the tail of the
* upload reads "Procesando…" instead of sitting at 100%.
*/
function UploadProgressBar({ progress }: { progress: UploadProgress | null }) {
const pct = progress?.fraction != null ? Math.round(progress.fraction * 100) : null;
return (
<div className="upload-progress">
<div
className={`progress-track${pct === null ? " progress-indeterminate" : ""}`}
role="progressbar"
aria-valuemin={0}
aria-valuemax={100}
aria-valuenow={pct ?? undefined}
>
<div className="progress-fill" style={{ width: `${pct ?? 100}%` }} />
</div>
<div className="upload-progress-stats mono">
{progress === null ? (
"Preparando…"
) : progress.finishing ? (
`Procesando en el servidor… (${formatBytes(progress.total)} enviados)`
) : (
<>
{pct !== null && <strong>{pct}%</strong>}
{progress.total > 0 && (
<span>
{formatBytes(progress.loaded)} / {formatBytes(progress.total)}
</span>
)}
{progress.bytesPerSecond > 0 && (
<span>{formatBytes(progress.bytesPerSecond)}/s</span>
)}
{progress.secondsRemaining !== null && progress.bytesPerSecond > 0 && (
<span>faltan {formatEta(progress.secondsRemaining)}</span>
)}
</>
)}
</div>
</div>
);
}
function OpTile({
title,
desc,
@@ -0,0 +1,16 @@
"use client";
import { AppShell } from "@/components/AppShell";
import { PolicyOcrReview } from "@/components/PolicyOcrReview";
export default function PolicyOcrBatchPage({
params,
}: {
params: { id: string };
}) {
return (
<AppShell>
<PolicyOcrReview id={params.id} />
</AppShell>
);
}
+21
View File
@@ -0,0 +1,21 @@
"use client";
import { AppShell } from "@/components/AppShell";
import { PolicyCaptura } from "@/components/PolicyCaptura";
/**
* OCR mode of the policy intake screen. Drops the GMX PDF, walks through
* per-page review, confirms. Same wrapper as `/polizas/nuevo` (manual)
* with `initialMode="auto"`, so the tab strip is identical and swapping
* modes doesn't drop state.
*
* Sister route `/polizas/captura/[id]` is the batch review screen once a
* batch is uploaded.
*/
export default function CapturaOcrPage() {
return (
<AppShell>
<PolicyCaptura initialMode="auto" />
</AppShell>
);
}
+9 -28
View File
@@ -1,41 +1,22 @@
"use client";
import { Suspense } from "react";
import Link from "next/link";
import { useSearchParams } from "next/navigation";
import { AppShell } from "@/components/AppShell";
import { PolicyForm } from "@/components/PolicyForm";
import { useCan } from "@/lib/abilities";
import { PolicyCaptura } from "@/components/PolicyCaptura";
/**
* Manual mode of the policy intake screen. Shares the tab wrapper with
* `/polizas/captura` (OCR mode) so staff can swap between the two without
* losing their place. Customer picker comes from the `?customerId=`
* / `?customerName=` query string — used by `/clientes/[id]` when staff
* creates a policy from a customer detail page.
*/
export default function NuevaPolizaPage() {
return (
<AppShell>
<Suspense fallback={null}>
<NuevaPoliza />
<PolicyCaptura initialMode="manual" />
</Suspense>
</AppShell>
);
}
function NuevaPoliza() {
const allowed = useCan("policy:create");
const params = useSearchParams();
const customerId = params.get("customerId") ?? undefined;
const customerName = params.get("customerName") ?? undefined;
return (
<>
<div className="page-head">
<Link href="/polizas" className="back-link"> Pólizas</Link>
<h1 className="page-title">Nueva póliza</h1>
</div>
{allowed ? (
<PolicyForm fixedCustomerId={customerId} fixedCustomerName={customerName} />
) : (
<div className="state-box state-error">
No tiene permisos para crear pólizas.
</div>
)}
</>
);
}
+6
View File
@@ -57,6 +57,7 @@ export default function PolizasPage() {
function PolizasBrowser() {
const canCreate = useCan("policy:create");
const canIngest = useCan("policy:ingest");
const [stats, setStats] = useState<PolicyStats | null>(null);
const [facets, setFacets] = useState<PolicyFacets | null>(null);
@@ -135,6 +136,11 @@ function PolizasBrowser() {
{ slug: "vigente", label: "Por vencer (Incen.)", params: { typeName: "INCEN" } },
]}
/>
{canIngest && (
<Link href="/polizas/captura" className="btn btn-outline">
+ Captura OCR
</Link>
)}
{canCreate && (
<Link href="/polizas/nuevo" className="btn btn-primary">+ Nueva póliza</Link>
)}
+525
View File
@@ -0,0 +1,525 @@
"use client";
import { useCallback, useEffect, useMemo, useState } from "react";
import Link from "next/link";
import { AppShell } from "@/components/AppShell";
import { CustomerPicker } from "@/components/CustomerPicker";
import {
confirmStatementBatch,
getStatementBatch,
listStatementDocuments,
rejectStatementDocument,
reviewStatementDocument,
statementPageUrl,
} from "@/lib/api";
import { useCan } from "@/lib/abilities";
import { formatDate, formatMoney, serviceKindLabel } from "@/lib/labels";
import type {
ConfirmBatchInput,
StatementBatchDetail,
StatementDocument,
StatementDocumentStatus,
} from "@/lib/types";
/**
* Review queue for one batch of scanned bills.
*
* The reviewer's job is to answer one question per page — "is this the right
* customer for this amount?" — so the page image sits next to the extracted
* fields and every row can be corrected in place. Rows the matcher is sure
* about are pre-approved and can be posted in bulk; everything else is listed
* first, because that is the work.
*/
const STATUS_LABEL: Record<StatementDocumentStatus, string> = {
PENDING_OCR: "En proceso",
OCR_FAILED: "No se pudo leer",
NEEDS_REVIEW: "Requiere revisión",
MATCHED: "Identificado",
CONFIRMED: "Confirmado",
POSTED: "Registrado",
REJECTED: "Descartado",
};
/** Rows still needing a decision, listed before the settled ones. */
const OPEN_FIRST: StatementDocumentStatus[] = [
"NEEDS_REVIEW",
"OCR_FAILED",
"MATCHED",
"CONFIRMED",
"POSTED",
"REJECTED",
"PENDING_OCR",
];
export default function RecibosBatchPage({ params }: { params: { id: string } }) {
return (
<AppShell>
<BatchReview id={params.id} />
</AppShell>
);
}
function BatchReview({ id }: { id: string }) {
const canReview = useCan("statement:review");
const [batch, setBatch] = useState<StatementBatchDetail | null>(null);
const [docs, setDocs] = useState<StatementDocument[]>([]);
const [error, setError] = useState<string | null>(null);
const [loading, setLoading] = useState(true);
const load = useCallback(async () => {
try {
const [b, d] = await Promise.all([
getStatementBatch(id),
listStatementDocuments(id),
]);
setBatch(b);
setDocs(d);
setError(null);
} catch (e) {
setError((e as Error)?.message ?? "No se pudo cargar el lote.");
} finally {
setLoading(false);
}
}, [id]);
useEffect(() => {
void load();
}, [load]);
const processing = batch?.status === "PROCESSING" || batch?.status === "UPLOADED";
useEffect(() => {
if (!processing) return;
const t = setInterval(() => void load(), 4000);
return () => clearInterval(t);
}, [processing, load]);
const sorted = useMemo(
() =>
[...docs].sort(
(a, b) =>
OPEN_FIRST.indexOf(a.status) - OPEN_FIRST.indexOf(b.status) ||
a.pageNumber - b.pageNumber,
),
[docs],
);
const readyCount = docs.filter(
(d) => d.status === "MATCHED" && d.matchedCustomer,
).length;
if (loading) return <div className="state-box">Cargando</div>;
if (!batch) return <div className="state-box state-error">{error ?? "No encontrado."}</div>;
return (
<div className="stack">
<header className="page-head">
<div>
<h1 className="page-title">
Recibos {serviceKindLabel(batch.serviceKind)}
{batch.label ? ` · ${batch.label}` : ""}
</h1>
<p className="page-sub">
{formatDate(batch.createdAt)} · {docs.length} página(s) ·{" "}
{STATUS_LABEL_BATCH[batch.status] ?? batch.status}
</p>
</div>
<Link className="btn btn-ghost" href="/recibos">
Volver a captura
</Link>
</header>
{error && <div className="state-box state-error">{error}</div>}
{processing && (
<ProcessingBanner docsLength={docs.length} pendingOcr={batch.byStatus.PENDING_OCR ?? 0} />
)}
<SummaryCard batch={batch} readyCount={readyCount} />
{canReview && readyCount > 0 && (
<ConfirmCard
batchId={id}
readyCount={readyCount}
onDone={load}
setError={setError}
/>
)}
<section className="stack">
{sorted.map((doc) => (
<DocumentRow
key={doc.id}
doc={doc}
canReview={canReview}
onChange={load}
/>
))}
</section>
</div>
);
}
const STATUS_LABEL_BATCH: Record<string, string> = {
UPLOADED: "Recibido",
PROCESSING: "Procesando",
READY_FOR_REVIEW: "Listo para revisar",
COMPLETED: "Registrado",
FAILED: "Falló",
};
/**
* Live readout while OCR is running. The backend tells us how many pages are
* still PENDING_OCR, so we can show real progress instead of "loading…". When
* the docs list hasn't caught up to the upload yet (total === 0) we fall back
* to the indeterminate bar.
*/
function ProcessingBanner({
docsLength,
pendingOcr,
}: {
docsLength: number;
pendingOcr: number;
}) {
const done = Math.max(docsLength - pendingOcr, 0);
const pct =
docsLength > 0 ? Math.min(100, Math.round((done / docsLength) * 100)) : null;
return (
<div className="card" style={{ padding: 16 }}>
<div className="upload-progress" style={{ padding: 0 }}>
<div
className={`progress-track${pct === null ? " progress-indeterminate" : ""}`}
role="progressbar"
aria-valuemin={0}
aria-valuemax={100}
aria-valuenow={pct ?? undefined}
>
<div className="progress-fill" style={{ width: `${pct ?? 100}%` }} />
</div>
<div className="upload-progress-stats">
{pct === null ? (
<span>Leyendo los recibos</span>
) : (
<>
<strong>{pct}%</strong>
<span>
{done} de {docsLength} página(s) leídas
</span>
{pendingOcr > 0 && <span>{pendingOcr} en cola</span>}
</>
)}
<span style={{ marginLeft: "auto" }}>
Esta pantalla se actualiza sola.
</span>
</div>
</div>
</div>
);
}
function SummaryCard({
batch,
readyCount,
}: {
batch: StatementBatchDetail;
readyCount: number;
}) {
const entries = Object.entries(batch.byStatus) as [StatementDocumentStatus, number][];
return (
<section className="card" style={{ padding: 16 }}>
<div className="inline-form" style={{ flexWrap: "wrap", gap: 20 }}>
{entries.map(([status, count]) => (
<div key={status}>
<div className="page-sub">{STATUS_LABEL[status] ?? status}</div>
<div style={{ fontSize: "1.4rem", fontWeight: 600 }}>{count}</div>
</div>
))}
<div>
<div className="page-sub">Importe pendiente</div>
<div style={{ fontSize: "1.4rem", fontWeight: 600 }}>
{formatMoney(batch.pendingTotal, "MXN")}
</div>
</div>
<div>
<div className="page-sub">Listos para registrar</div>
<div style={{ fontSize: "1.4rem", fontWeight: 600 }}>{readyCount}</div>
</div>
</div>
</section>
);
}
/**
* Posting is by check, exactly as on the manual capture screen — an OCR batch
* is still "these bills, paid with this check", so the same fields are asked
* for and the same ledger path is used.
*/
function ConfirmCard({
batchId,
readyCount,
onDone,
setError,
}: {
batchId: string;
readyCount: number;
onDone: () => void;
setError: (m: string | null) => void;
}) {
const [checkNumber, setCheckNumber] = useState("");
const [transactionDate, setTransactionDate] = useState(
new Date().toISOString().slice(0, 10),
);
const [outstanding, setOutstanding] = useState(false);
const [includeReviewed, setIncludeReviewed] = useState(true);
const [busy, setBusy] = useState(false);
const [result, setResult] = useState<string | null>(null);
async function submit() {
if (!checkNumber.trim()) return;
setBusy(true);
setError(null);
try {
const input: ConfirmBatchInput = {
checkNumber: checkNumber.trim(),
transactionDate,
outstanding,
includeReviewed,
};
const r = await confirmStatementBatch(batchId, input);
setResult(
`Se registraron ${r.posted} movimiento(s) por ${formatMoney(r.total, "MXN")} con el cheque ${r.checkNumber}.`,
);
setCheckNumber("");
onDone();
} catch (e) {
setError((e as Error)?.message ?? "No se pudo registrar el lote.");
} finally {
setBusy(false);
}
}
return (
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Registrar {readyCount} recibo(s)
</h2>
<div className="inline-form" style={{ flexWrap: "wrap", gap: 12 }}>
<label>
<span className="page-sub">Cheque</span>
<input
className="input"
value={checkNumber}
onChange={(e) => setCheckNumber(e.target.value)}
placeholder="Número de cheque"
/>
</label>
<label>
<span className="page-sub">Fecha</span>
<input
type="date"
className="input"
value={transactionDate}
onChange={(e) => setTransactionDate(e.target.value)}
/>
</label>
<label className="check">
<input
type="checkbox"
checked={outstanding}
onChange={(e) => setOutstanding(e.target.checked)}
/>{" "}
Sin fondos (queda pendiente)
</label>
<label className="check">
<input
type="checkbox"
checked={includeReviewed}
onChange={(e) => setIncludeReviewed(e.target.checked)}
/>{" "}
Incluir los confirmados a mano
</label>
<button
type="button"
className="btn btn-primary"
disabled={!checkNumber.trim() || busy}
onClick={submit}
>
{busy ? "Registrando…" : "Registrar"}
</button>
</div>
{result && (
<div className="state-box" style={{ marginTop: 12 }}>
{result}
</div>
)}
<p className="page-sub" style={{ marginTop: 12 }}>
Se registran como cargos del cliente, por la misma vía que la captura
manual. Un lote registrado dos veces no duplica cobros.
</p>
</section>
);
}
function DocumentRow({
doc,
canReview,
onChange,
}: {
doc: StatementDocument;
canReview: boolean;
onChange: () => void;
}) {
const [open, setOpen] = useState(
doc.status === "NEEDS_REVIEW" || doc.status === "OCR_FAILED",
);
const [amount, setAmount] = useState(doc.extractedAmount ?? "");
const [accountRef, setAccountRef] = useState(doc.extractedAccountRef ?? "");
const [customerId, setCustomerId] = useState(doc.matchedCustomer?.id ?? "");
const [customerName, setCustomerName] = useState(doc.matchedCustomer?.name ?? "");
const [busy, setBusy] = useState(false);
const [err, setErr] = useState<string | null>(null);
const settled = doc.status === "POSTED" || doc.status === "REJECTED";
async function save(status: "MATCHED" | "CONFIRMED") {
setBusy(true);
setErr(null);
try {
await reviewStatementDocument(doc.id, {
accountRef: accountRef.trim() || undefined,
amount: amount ? Number(amount) : undefined,
matchedCustomerId: customerId || undefined,
status,
});
onChange();
} catch (e) {
setErr((e as Error)?.message ?? "No se pudo guardar.");
} finally {
setBusy(false);
}
}
async function reject() {
setBusy(true);
setErr(null);
try {
await rejectStatementDocument(doc.id);
onChange();
} catch (e) {
setErr((e as Error)?.message ?? "No se pudo descartar.");
} finally {
setBusy(false);
}
}
return (
<div className="card" style={{ padding: 16 }}>
<div
className="inline-form"
style={{ justifyContent: "space-between", flexWrap: "wrap", gap: 12 }}
>
<div>
<strong>Página {doc.pageNumber}</strong>{" "}
<span className="tag">{STATUS_LABEL[doc.status] ?? doc.status}</span>{" "}
{doc.provider && <span className="page-sub">{doc.provider}</span>}
<div className="page-sub" style={{ marginTop: 4 }}>
{doc.matchedCustomer ? (
<Link href={`/clientes/${doc.matchedCustomer.id}`}>
{doc.matchedCustomer.name}
</Link>
) : (
"Sin cliente asignado"
)}
{doc.extractedAccountRef && ` · cuenta ${doc.extractedAccountRef}`}
{doc.extractedCadastralKey && ` · clave ${doc.extractedCadastralKey}`}
</div>
{doc.matchNote && (
<div className="page-sub" style={{ marginTop: 4 }}>
{doc.matchNote}
</div>
)}
</div>
<div className="inline-form" style={{ gap: 8 }}>
<strong>
{doc.extractedAmount
? formatMoney(doc.extractedAmount, "MXN")
: "sin importe"}
</strong>
<button
type="button"
className="btn btn-ghost"
onClick={() => setOpen((v) => !v)}
>
{open ? "Ocultar" : "Ver recibo"}
</button>
</div>
</div>
{open && (
<div style={{ marginTop: 12, display: "grid", gap: 16 }}>
{/* The scan itself — the reviewer's source of truth, not the OCR. */}
<img
src={statementPageUrl(doc.id)}
alt={`Recibo página ${doc.pageNumber}`}
style={{
maxWidth: "100%",
border: "1px solid var(--border, #ddd)",
borderRadius: 6,
}}
/>
{canReview && !settled && (
<div className="inline-form" style={{ flexWrap: "wrap", gap: 12 }}>
<label>
<span className="page-sub">Cuenta</span>
<input
className="input"
value={accountRef}
onChange={(e) => setAccountRef(e.target.value)}
/>
</label>
<label>
<span className="page-sub">Importe</span>
<input
className="input"
inputMode="decimal"
value={amount}
onChange={(e) => setAmount(e.target.value)}
/>
</label>
<div style={{ minWidth: 260 }}>
<span className="page-sub">Cliente</span>
<CustomerPicker
value={customerId}
valueName={customerName}
onPick={(cid, name) => {
setCustomerId(cid);
setCustomerName(name);
}}
/>
</div>
<button
type="button"
className="btn btn-primary"
disabled={busy || !customerId}
onClick={() => save("MATCHED")}
>
Guardar
</button>
<button
type="button"
className="btn btn-ghost"
disabled={busy}
onClick={reject}
>
Descartar
</button>
</div>
)}
{err && <div className="state-box state-error">{err}</div>}
</div>
)}
</div>
);
}
+15
View File
@@ -0,0 +1,15 @@
import { AppShell } from "@/components/AppShell";
import { Captura } from "@/components/Captura";
/**
* Same capture screen as `/estado-cuenta/lote`, opened on the automatic
* (scanned recibos + OCR) mode. Kept as its own route so links from a batch
* review page and older bookmarks land on the right tab.
*/
export default function RecibosPage() {
return (
<AppShell>
<Captura initialMode="auto" />
</AppShell>
);
}
+30 -8
View File
@@ -4,7 +4,7 @@ import { useEffect, useRef, useState, type ReactNode } from "react";
import { usePathname, useRouter } from "next/navigation";
import Link from "next/link";
import { getApiVersion, logout, me, updateUiScale, type ServiceVersion } from "@/lib/api";
import { shortSha, webBuildInfo } from "@/lib/build-info";
import { webBuildInfo } from "@/lib/build-info";
import { AuthContext, can } from "@/lib/abilities";
import { ROLE_LABEL } from "@/lib/labels";
import {
@@ -24,7 +24,15 @@ import type { AuthUser, Ability } from "@/lib/types";
* abilities. Used by every authenticated page.
*/
type NavLink = { href: string; label: string; ability?: Ability; exact?: boolean };
type NavLink = {
href: string;
label: string;
ability?: Ability;
exact?: boolean;
/** Extra path prefixes that belong to this entry (e.g. a second route into
* the same screen), so they highlight it instead of nothing. */
aliases?: string[];
};
type NavEntry =
| ({ kind: "link" } & NavLink)
| { kind: "group"; label: string; items: NavLink[] };
@@ -45,8 +53,16 @@ const NAV: NavEntry[] = [
label: "Cobranza",
items: [
// Daily data-entry screen (the legacy "Editor"). Hidden from VIEWER, who
// can't capture anyway — the page itself also refuses.
{ href: "/estado-cuenta/lote", label: "Captura", ability: "ledger:create" },
// can't capture anyway — the page itself also refuses. Both capture modes
// live behind this one entry: keying receipts by hand, and scanning a
// stack of bills for OCR (the `/recibos` route opens the same screen on
// its automatic tab).
{
href: "/estado-cuenta/lote",
label: "Captura",
ability: "ledger:create",
aliases: ["/recibos"],
},
{ href: "/estado-cuenta", label: "Estado de cuenta" },
{ href: "/banco", label: "Chequera" },
],
@@ -98,9 +114,11 @@ function activeHref(pathname: string | null): string | null {
if (!pathname) return null;
let best: string | null = null;
for (const item of NAV_LINKS) {
const under = (href: string) =>
pathname === href || pathname.startsWith(`${href}/`);
const match = item.exact
? pathname === item.href
: pathname === item.href || pathname.startsWith(`${item.href}/`);
: under(item.href) || (item.aliases?.some(under) ?? false);
if (match && (best === null || item.href.length > best.length)) {
best = item.href;
}
@@ -217,12 +235,16 @@ function BuildFooter() {
return (
<footer className="shell-footer">
<span>Jorge Cuadros &amp; Asociados</span>
{/* The FULL 40-char commit, not an abbreviation: this line exists to be
pasted into `git show` or compared against a registry tag, and a
7-char prefix makes both a manual step. It is what GIT_SHA already
carries — build.yml bakes in `github.sha` whole. */}
<span
className="shell-footer-build"
title={`web ${web.version} (${shortSha(web.gitSha)}) — ${web.buildDate}`}
title={`web ${web.version} (${web.gitSha}) — ${web.buildDate}`}
>
v{web.version} · {shortSha(web.gitSha)}
{mismatch && api ? ` · API ${shortSha(api.gitSha)}` : ""}
v{web.version} · {web.gitSha}
{mismatch && api ? ` · API ${api.gitSha}` : ""}
</span>
{mismatch && (
<span className="shell-footer-warn" role="status">
+90
View File
@@ -0,0 +1,90 @@
"use client";
import { useState } from "react";
import Link from "next/link";
import { ManualCheckCapture } from "@/components/ManualCheckCapture";
import { StatementIntake } from "@/components/StatementIntake";
import { useCan } from "@/lib/abilities";
/**
* The daily capture screen (the legacy "Editor"), with two ways in:
*
* - **manual** — key each customer's receipt against one check by hand.
* - **auto** — scan the stack of paper bills and let OCR propose customer and
* amount for every page, which a human still confirms.
*
* Both end in the same place: charges on the customer's ledger, posted against
* one check. They are modes of one screen rather than two menu entries because
* it is one job — staff pick the mode by what's on the desk that morning, a
* stack of bills or a keyboard.
*
* `/estado-cuenta/lote` opens on manual, `/recibos` opens on auto; both render
* this component, so an old bookmark still lands on the right tab.
*/
export type CaptureMode = "manual" | "auto";
const MODE_HINT: Record<CaptureMode, string> = {
manual:
"Captura los recibos de varios clientes contra un mismo cheque y concilia el total antes de guardar.",
auto: "Escanea los recibos del mes y el sistema propone cliente e importe para cada página. Nada se registra sin tu confirmación.",
};
export function Captura({ initialMode = "manual" }: { initialMode?: CaptureMode }) {
const canCapture = useCan("ledger:create");
const canIngest = useCan("statement:ingest");
// Gating is cosmetic (the API enforces every write), but a user who only has
// one of the two abilities should land on the mode they can actually use.
const modes: { key: CaptureMode; label: string }[] = [
...(canCapture ? [{ key: "manual" as const, label: "Captura manual" }] : []),
...(canIngest
? [{ key: "auto" as const, label: "Captura automática (OCR)" }]
: []),
];
const [mode, setMode] = useState<CaptureMode>(
modes.some((m) => m.key === initialMode) ? initialMode : (modes[0]?.key ?? "manual"),
);
if (modes.length === 0) {
return (
<div className="state-box state-error">
No tienes permiso para capturar movimientos.
</div>
);
}
return (
<>
<div className="page-head">
<div>
<h1 className="page-title">Captura</h1>
<p className="eyebrow">{MODE_HINT[mode]}</p>
</div>
<Link href="/estado-cuenta" className="btn btn-outline">
Volver a estado de cuenta
</Link>
</div>
{modes.length > 1 && (
<div className="seg" role="tablist" style={{ marginBottom: 16 }}>
{modes.map((m) => (
<button
key={m.key}
type="button"
role="tab"
aria-selected={mode === m.key}
className={`seg-btn ${mode === m.key ? "active" : ""}`}
onClick={() => setMode(m.key)}
>
{m.label}
</button>
))}
</div>
)}
{mode === "manual" ? <ManualCheckCapture /> : <StatementIntake />}
</>
);
}
@@ -0,0 +1,526 @@
"use client";
import { useEffect, useMemo, useState } from "react";
import Link from "next/link";
import { CustomerPicker } from "@/components/CustomerPicker";
import { createMovementBatch, getBillingFacets, getByCheck } from "@/lib/api";
import { formatMoney, formatNumber, txTypeLabel } from "@/lib/labels";
import type {
BatchCreateInput,
BillingFacets,
ByCheckResponse,
Currency,
LedgerCurrency,
TransactionDomain,
} from "@/lib/types";
/**
* Batch capture by check — the "Editor" screen from the legacy system
* (docs/RECEIPT_CAPTURE_SPEC.md §1.2), and the manual half of the Captura
* screen (see `Captura`).
*
* Staff key many customers' receipts against ONE physical check before cutting
* it, then check that the captured total matches the check's amount. That
* reconciliation is the whole point, so the running total is the most prominent
* thing on the page and an optional "importe del cheque" field turns it into a
* live difference.
*
* No batch entity is persisted: `checkNumber` is a plain column, and grouping
* by it answers every by-check question (see the "Reporte por cheque" report).
*/
const DOMAINS: { key: TransactionDomain; label: string }[] = [
{ key: "UTILITY", label: "Servicios" },
{ key: "INSURANCE", label: "Seguros" },
{ key: "TRUST", label: "Fideicomiso" },
];
interface Line {
/** Local row key — lines have no server identity until the batch posts. */
key: number;
customerId: string;
customerName: string;
amount: string;
reference: string;
period: string;
outstanding: boolean;
}
function blankLine(key: number): Line {
return {
key,
customerId: "",
customerName: "",
amount: "",
reference: "",
period: "",
outstanding: false,
};
}
export function ManualCheckCapture() {
const [facets, setFacets] = useState<BillingFacets | null>(null);
// Check-level fields — shared by every line.
const [domain, setDomain] = useState<TransactionDomain>("UTILITY");
const [currency, setCurrency] = useState<LedgerCurrency>("MXN");
const [typeId, setTypeId] = useState("");
const [checkNumber, setCheckNumber] = useState("");
const [transactionDate, setTransactionDate] = useState(
new Date().toISOString().slice(0, 10),
);
/** The physical check's amount, for reconciliation only — never submitted. */
const [checkAmount, setCheckAmount] = useState("");
const [lines, setLines] = useState<Line[]>([blankLine(1), blankLine(2), blankLine(3)]);
const [nextKey, setNextKey] = useState(4);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const [posted, setPosted] = useState<ByCheckResponse | null>(null);
useEffect(() => {
getBillingFacets().then(setFacets).catch(() => setFacets(null));
}, []);
const filled = lines.filter(
(l) => l.customerId && l.amount.trim() !== "" && Number.isFinite(Number(l.amount)),
);
// Charges are captured as positive numbers and signed on submit, matching
// MovementForm — staff type what's on the bill, not a negative.
const total = useMemo(
() =>
filled
.filter((l) => !l.outstanding)
.reduce((sum, l) => sum + Math.abs(Number(l.amount)), 0),
[filled],
);
const outstandingTotal = useMemo(
() =>
filled
.filter((l) => l.outstanding)
.reduce((sum, l) => sum + Math.abs(Number(l.amount)), 0),
[filled],
);
const checkAmt = Number(checkAmount);
const hasCheckAmt = checkAmount.trim() !== "" && Number.isFinite(checkAmt);
const diff = hasCheckAmt ? checkAmt - total : 0;
const reconciled = hasCheckAmt && Math.abs(diff) < 0.005;
function update(key: number, patch: Partial<Line>) {
setLines((ls) => ls.map((l) => (l.key === key ? { ...l, ...patch } : l)));
}
function addLine() {
setLines((ls) => [...ls, blankLine(nextKey)]);
setNextKey((k) => k + 1);
}
function removeLine(key: number) {
setLines((ls) => (ls.length === 1 ? ls : ls.filter((l) => l.key !== key)));
}
async function submit(e: React.FormEvent) {
e.preventDefault();
if (!checkNumber.trim()) {
setError("Indica el número de cheque.");
return;
}
if (filled.length === 0) {
setError("Captura al menos una línea con cliente y monto.");
return;
}
const dupes = filled
.map((l) => l.customerId)
.filter((id, i, arr) => arr.indexOf(id) !== i);
if (dupes.length) {
const names = filled
.filter((l) => dupes.includes(l.customerId))
.map((l) => l.customerName);
if (
!window.confirm(
`Hay más de una línea para el mismo cliente (${[...new Set(names)].join(
", ",
)}). ¿Continuar?`,
)
)
return;
}
const payload: BatchCreateInput = {
domain,
transactionDate,
checkNumber: checkNumber.trim(),
currency: currency as Currency,
typeId: typeId || undefined,
lines: filled.map((l) => ({
customerId: l.customerId,
// Every line of a check batch is a charge the office paid out.
amount: -Math.abs(Number(l.amount)),
reference: l.reference.trim() || undefined,
period: l.period.trim() || undefined,
outstanding: l.outstanding || undefined,
})),
};
setSaving(true);
setError(null);
try {
await createMovementBatch(payload);
// Re-read through the by-check view so the confirmation shows what's
// actually stored (including anything captured against this check
// earlier), not just what this request sent.
setPosted(await getByCheck(payload.checkNumber));
} catch (e2) {
setError((e2 as Error)?.message ?? "No se pudo guardar el lote.");
} finally {
setSaving(false);
}
}
function reset() {
setPosted(null);
setLines([blankLine(nextKey), blankLine(nextKey + 1), blankLine(nextKey + 2)]);
setNextKey((k) => k + 3);
setCheckNumber("");
setCheckAmount("");
}
if (posted) {
return (
<>
<div className="page-head">
<div>
<h2 className="page-title">Lote capturado</h2>
<p className="eyebrow">
Cheque {posted.checkNumber} · {formatNumber(posted.count)}{" "}
{posted.count === 1 ? "movimiento" : "movimientos"}
</p>
</div>
<div style={{ display: "flex", gap: 10 }}>
<button type="button" className="btn btn-primary" onClick={reset}>
Capturar otro cheque
</button>
<Link href="/estado-cuenta" className="btn btn-outline">
Volver a estado de cuenta
</Link>
</div>
</div>
<div className="filtered-totals" style={{ marginBottom: 16 }}>
{posted.totals.map((t) => (
<div className="filtered-total" key={t.currency}>
<span className="filtered-total-cur">{t.currency}</span>
<span className="filtered-total-net">
Total del cheque <strong>{formatMoney(t.total, t.currency)}</strong>
</span>
<span>{formatNumber(t.count)} movimientos</span>
</div>
))}
{posted.outstandingCount > 0 && (
<div className="filtered-total">
<span>
{formatNumber(posted.outstandingCount)} sin fondos (no suman al
total)
</span>
</div>
)}
</div>
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th>Cliente</th>
<th>Referencia</th>
<th>Periodo</th>
<th>Estado</th>
<th className="num">Monto</th>
</tr>
</thead>
<tbody>
{posted.items.map((i) => (
<tr key={i.id}>
<td>
<Link
href={`/estado-cuenta/${i.customerId}`}
className="inline-link"
>
{i.customerName}
</Link>
</td>
<td>{i.reference || "—"}</td>
<td>{i.period || "—"}</td>
<td>{i.outstanding ? "Sin fondos" : "Pagado"}</td>
<td className="num">
<span className="tx-amount neg">
{formatMoney(i.amount, i.currency)}
</span>
</td>
</tr>
))}
</tbody>
</table>
</div>
<p className="muted" style={{ marginTop: 14 }}>
Para imprimir la conciliación, usa el reporte{" "}
<Link
href={`/reportes/cheque-count?checkNumber=${encodeURIComponent(
posted.checkNumber,
)}`}
className="inline-link"
>
Reporte por cheque
</Link>
.
</p>
</>
);
}
return (
<>
{error && <div className="state-box state-error">{error}</div>}
<form onSubmit={submit}>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<h2 className="section-title" style={{ marginBottom: 14 }}>
Datos del cheque
</h2>
<div className="form-grid">
<label className="field">
<span className="field-label">Número de cheque *</span>
<input
className="input"
value={checkNumber}
onChange={(e) => setCheckNumber(e.target.value)}
required
/>
</label>
<label className="field">
<span className="field-label">Fecha *</span>
<input
className="input"
type="date"
required
value={transactionDate}
onChange={(e) => setTransactionDate(e.target.value)}
/>
</label>
<label className="field">
<span className="field-label">Línea de negocio *</span>
<select
className="select"
value={domain}
onChange={(e) => setDomain(e.target.value as TransactionDomain)}
>
{DOMAINS.map((d) => (
<option key={d.key} value={d.key}>
{d.label}
</option>
))}
</select>
</label>
<label className="field">
<span className="field-label">Moneda *</span>
<select
className="select"
value={currency}
onChange={(e) => setCurrency(e.target.value as LedgerCurrency)}
>
<option value="MXN">Pesos (MXN)</option>
<option value="USD">Dólares (USD)</option>
</select>
</label>
<label className="field">
<span className="field-label">Concepto</span>
<select
className="select"
value={typeId}
onChange={(e) => setTypeId(e.target.value)}
>
<option value="">(sin concepto)</option>
{facets?.types.map((t) => (
<option key={t.id} value={t.id}>
{txTypeLabel({ nameEn: t.name })}
</option>
))}
</select>
</label>
<label className="field">
<span className="field-label">Importe del cheque</span>
<input
className="input"
type="number"
step="0.01"
min="0"
value={checkAmount}
onChange={(e) => setCheckAmount(e.target.value)}
placeholder="Para conciliar"
/>
</label>
</div>
</div>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: 14,
}}
>
<h2 className="section-title" style={{ margin: 0 }}>
Recibos ({formatNumber(filled.length)})
</h2>
<button type="button" className="btn btn-outline" onClick={addLine}>
Agregar línea
</button>
</div>
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th style={{ minWidth: 240 }}>Cliente *</th>
<th style={{ minWidth: 120 }}>Referencia</th>
<th style={{ minWidth: 100 }}>Periodo</th>
<th style={{ minWidth: 110 }} className="num">
Monto *
</th>
<th style={{ whiteSpace: "nowrap" }}>Sin fondos</th>
<th style={{ width: 1 }} />
</tr>
</thead>
<tbody>
{lines.map((l) => (
<tr key={l.key}>
<td>
<CustomerPicker
value={l.customerId}
valueName={l.customerId ? l.customerName : undefined}
onPick={(id, name) =>
update(l.key, { customerId: id, customerName: name })
}
/>
</td>
<td>
<input
className="input"
value={l.reference}
onChange={(e) =>
update(l.key, { reference: e.target.value })
}
/>
</td>
<td>
<input
className="input"
value={l.period}
onChange={(e) => update(l.key, { period: e.target.value })}
placeholder="2026-07"
/>
</td>
<td>
<input
className="input num"
type="number"
step="0.01"
min="0"
value={l.amount}
onChange={(e) => update(l.key, { amount: e.target.value })}
placeholder="0.00"
/>
</td>
<td style={{ textAlign: "center" }}>
<input
type="checkbox"
checked={l.outstanding}
onChange={(e) =>
update(l.key, { outstanding: e.target.checked })
}
aria-label="Sin fondos"
/>
</td>
<td>
<button
type="button"
className="btn btn-ghost"
style={{ padding: "4px 10px", fontSize: 12 }}
onClick={() => removeLine(l.key)}
disabled={lines.length === 1}
>
Quitar
</button>
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
<div className="card" style={{ padding: 20, marginBottom: 16 }}>
<h2 className="section-title" style={{ marginBottom: 14 }}>
Conciliación
</h2>
<div className="filtered-totals">
<div className="filtered-total">
<span className="filtered-total-cur">{currency}</span>
<span className="filtered-total-net">
Capturado <strong>{formatMoney(String(-total), currency)}</strong>
</span>
<span>{formatNumber(filled.filter((l) => !l.outstanding).length)} recibos</span>
</div>
{outstandingTotal > 0 && (
<div className="filtered-total">
<span>
Sin fondos{" "}
<strong>{formatMoney(String(-outstandingTotal), currency)}</strong>{" "}
(no suma al cheque)
</span>
</div>
)}
{hasCheckAmt && (
<div className="filtered-total">
<span className="filtered-total-net">
{reconciled ? (
<strong className="tx-amount pos">Cuadra con el cheque</strong>
) : (
<>
Diferencia{" "}
<strong className="tx-amount neg">
{formatMoney(String(diff), currency)}
</strong>
</>
)}
</span>
</div>
)}
</div>
</div>
<div className="form-actions">
<button
type="submit"
className="btn btn-primary"
disabled={saving || filled.length === 0}
>
{saving
? "Guardando…"
: `Capturar ${formatNumber(filled.length)} ${
filled.length === 1 ? "recibo" : "recibos"
}`}
</button>
<Link href="/estado-cuenta" className="btn btn-outline">
Cancelar
</Link>
</div>
</form>
</>
);
}
+124
View File
@@ -0,0 +1,124 @@
"use client";
import { useState } from "react";
import Link from "next/link";
import { useSearchParams } from "next/navigation";
import { PolicyForm } from "@/components/PolicyForm";
import { PolicyOcrIntake } from "@/components/PolicyOcrIntake";
import { useCan } from "@/lib/abilities";
/**
* Policy intake — mirror of `Captura.tsx` (statement OCR side): one screen,
* two ways in:
*
* - **manual** — `PolicyForm` keys every field by hand.
* - **auto** — `PolicyOcrIntake` uploads a GMX PDF, OCR proposes the
* policy, a human still confirms.
*
* Both end at the same place (a `Policy` row on a customer's file) so they
* live as two modes of one screen rather than two menu entries — exactly the
* same shape Captura uses for `ManualCheckCapture` vs `StatementIntake`.
*
* `/polizas/nuevo` opens manual, `/polizas/captura` opens auto; both render
* this component so the tab toggle works either way and an old bookmark
* still lands on the right tab.
*/
export type PolicyCaptureMode = "manual" | "auto";
const MODE_HINT: Record<PolicyCaptureMode, string> = {
manual:
"Captura cada campo a mano. Use esta opción cuando la póliza llega en papel, en un correo sin PDF legible, o cuando hay que revisar cada dato.",
auto: "Suelte el PDF descargado del portal de GMX y el sistema propondrá los campos. Nada se registra sin tu confirmación.",
};
export function PolicyCaptura({ initialMode = "manual" }: { initialMode?: PolicyCaptureMode }) {
const canCreate = useCan("policy:create");
const canIngest = useCan("policy:ingest");
// `/clientes/[id]` deep-links into /polizas/nuevo with the customer
// pre-picked so staff can fill the rest without retyping. The OCR pane
// ignores these — there's no customer to lock in until the batch is
// confirmed.
const params = useSearchParams();
const fixedCustomerId = params.get("customerId") ?? undefined;
const fixedCustomerName = params.get("customerName") ?? undefined;
// One user can land on either mode. The tab strip only renders when both
// abilities are held — a STAFF with only policy:ingest (no create) still
// sees the screen but only the OCR tab is offered.
const modes: { key: PolicyCaptureMode; label: string }[] = [
...(canCreate ? [{ key: "manual" as const, label: "Captura manual" }] : []),
...(canIngest ? [{ key: "auto" as const, label: "Captura automática (OCR)" }] : []),
];
const [mode, setMode] = useState<PolicyCaptureMode>(
modes.some((m) => m.key === initialMode) ? initialMode : (modes[0]?.key ?? "manual"),
);
if (modes.length === 0) {
return (
<div className="state-box state-error">
No tienes permiso para crear ni capturar pólizas.
</div>
);
}
return (
<>
<div className="page-head">
<Link href="/polizas" className="back-link"> Pólizas</Link>
<h1 className="page-title">Nueva póliza</h1>
<p className="eyebrow">{MODE_HINT[mode]}</p>
</div>
{modes.length > 1 && (
<div className="seg" role="tablist" style={{ marginBottom: 16 }}>
{modes.map((m) => (
<button
key={m.key}
type="button"
role="tab"
aria-selected={mode === m.key}
className={`seg-btn ${mode === m.key ? "active" : ""}`}
onClick={() => setMode(m.key)}
>
{m.label}
</button>
))}
</div>
)}
{mode === "manual" ? (
<ManualPane
fixedCustomerId={fixedCustomerId}
fixedCustomerName={fixedCustomerName}
/>
) : (
<PolicyOcrIntake />
)}
</>
);
}
function ManualPane({
fixedCustomerId,
fixedCustomerName,
}: {
fixedCustomerId?: string;
fixedCustomerName?: string;
}) {
const allowed = useCan("policy:create");
if (!allowed) {
return (
<div className="state-box state-error">
No tiene permisos para crear pólizas.
</div>
);
}
return (
<PolicyForm
fixedCustomerId={fixedCustomerId}
fixedCustomerName={fixedCustomerName}
/>
);
}
+232
View File
@@ -0,0 +1,232 @@
"use client";
import { useCallback, useEffect, useState } from "react";
import Link from "next/link";
import {
getPolicyOcrStatus,
listPolicyOcrBatches,
uploadPolicyOcrBatch,
} from "@/lib/api";
import { useCan } from "@/lib/abilities";
import { formatDate } from "@/lib/labels";
import type { PolicyOcrBatch, PolicyOcrBatchStatus } from "@/lib/types";
/**
* Insurance OCR intake — mirror of StatementIntake, scoped to the insurance
* side. Today the only provider is GMX; the parser dispatches on a brand
* wordmark (`Grupo Mexicano de Seguros` / `gmx.com.mx` / the GMX letterhead)
* and a new portal only needs a new BRAND entry plus a parser file.
*
* Lives inside the `Pólizas` page rather than a top-level route because it
* is one mode of one job (staff uploading whatever PDFs the office has on
* hand that day, mixed service vs insurance), and the matching/review queue
* already keys on the policyNumber → existing Policy transition that the
* rest of /polizas owns.
*/
const STATUS_LABEL: Record<PolicyOcrBatchStatus, string> = {
UPLOADED: "Recibido",
PROCESSING: "Procesando…",
READY_FOR_REVIEW: "Listo para revisar",
COMPLETED: "Aplicado",
FAILED: "Falló",
};
export function PolicyOcrIntake() {
const canIngest = useCan("policy:ingest");
const [batches, setBatches] = useState<PolicyOcrBatch[]>([]);
const [ocrAvailable, setOcrAvailable] = useState<boolean | null>(null);
const [storageAvailable, setStorageAvailable] = useState<boolean | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const load = useCallback(async () => {
try {
const [list, status] = await Promise.all([
listPolicyOcrBatches(),
getPolicyOcrStatus(),
]);
setBatches(list.items);
setOcrAvailable(status.ocrAvailable);
setStorageAvailable(status.storageAvailable);
setError(null);
} catch (e) {
setError((e as Error)?.message ?? "No se pudieron cargar los lotes.");
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
void load();
}, [load]);
const working = batches.some(
(b) => b.status === "PROCESSING" || b.status === "UPLOADED",
);
useEffect(() => {
if (!working) return;
const t = setInterval(() => void load(), 4000);
return () => clearInterval(t);
}, [working, load]);
const ready = ocrAvailable === true && storageAvailable === true;
return (
<div className="stack">
{ocrAvailable === false && (
<div className="state-box state-error">
Este servidor no tiene OCR instalado, así que no se pueden leer PDFs
de pólizas escaneados. La captura manual sigue funcionando.
</div>
)}
{storageAvailable === false && (
<div className="state-box state-error">
Este servidor no tiene configurado el almacenamiento de documentos, así
que no hay dónde guardar los PDFs. Mientras tanto, capture las
pólizas a mano.
</div>
)}
{canIngest && ready && <UploadCard onDone={load} />}
{error && <div className="state-box state-error">{error}</div>}
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Lotes
</h2>
{loading ? (
<div className="state-box">Cargando</div>
) : batches.length === 0 ? (
<div className="state-box">
Todavía no hay lotes de pólizas. Descargue el certificado del portal
de GMX y suéltelo arriba.
</div>
) : (
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th>Fecha</th>
<th>Aseguradora</th>
<th>Referencia</th>
<th>Estado</th>
<th className="num">Páginas</th>
<th>Subido por</th>
<th />
</tr>
</thead>
<tbody>
{batches.map((b) => (
<tr key={b.id}>
<td style={{ whiteSpace: "nowrap" }}>{formatDate(b.createdAt)}</td>
<td>{b.provider}</td>
<td>{b.label || "—"}</td>
<td>
<StatusTag status={b.status} />
{b.error && (
<div className="page-sub" style={{ marginTop: 4 }}>
{b.error}
</div>
)}
</td>
<td className="num">{b._count?.documents ?? 0}</td>
<td>{b.uploadedBy?.name ?? "—"}</td>
<td>
<Link
className="btn btn-ghost"
href={`/polizas/captura/${b.id}`}
>
Revisar
</Link>
</td>
</tr>
))}
</tbody>
</table>
</div>
)}
</section>
</div>
);
}
function StatusTag({ status }: { status: PolicyOcrBatchStatus }) {
return <span className="tag">{STATUS_LABEL[status] ?? status}</span>;
}
function UploadCard({ onDone }: { onDone: () => void }) {
const [files, setFiles] = useState<File[]>([]);
const [label, setLabel] = useState("");
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
async function submit() {
if (!files.length) return;
setBusy(true);
setError(null);
try {
await uploadPolicyOcrBatch(files, label.trim() || undefined);
setFiles([]);
setLabel("");
onDone();
} catch (e) {
setError((e as Error)?.message ?? "No se pudo subir el lote.");
} finally {
setBusy(false);
}
}
return (
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Subir PDFs de pólizas (GMX)
</h2>
<div className="inline-form" style={{ flexWrap: "wrap", gap: 12 }}>
<label>
<span className="page-sub">Referencia (opcional)</span>
<input
className="input"
placeholder="ej. GMX julio 2026"
value={label}
onChange={(e) => setLabel(e.target.value)}
/>
</label>
<label>
<span className="page-sub">Archivos PDF</span>
<input
type="file"
className="input"
accept="application/pdf"
multiple
onChange={(e) => setFiles(Array.from(e.target.files ?? []))}
/>
</label>
<button
type="button"
className="btn btn-primary"
disabled={!files.length || busy}
onClick={submit}
>
{busy ? "Subiendo…" : `Procesar ${files.length || ""}`.trim()}
</button>
</div>
{error && (
<div className="state-box state-error" style={{ marginTop: 12 }}>
{error}
</div>
)}
<p className="page-sub" style={{ marginTop: 12 }}>
Un lote puede traer varios PDFs. Cada página se procesa por separado; el
sistema busca una póliza existente por número y, si no la encuentra,
propone crear una nueva bajo el cliente que se elija en la revisión.
</p>
</section>
);
}
+578
View File
@@ -0,0 +1,578 @@
"use client";
import { useCallback, useEffect, useMemo, useState } from "react";
import Link from "next/link";
import { CustomerPicker } from "@/components/CustomerPicker";
import {
confirmPolicyOcrBatch,
getPolicyOcrBatch,
listCustomers,
listPolicyOcrDocuments,
policyOcrDocumentUrl,
rejectPolicyOcrDocument,
reviewPolicyOcrDocument,
} from "@/lib/api";
import { useCan } from "@/lib/abilities";
import { formatDate, formatMoney } from "@/lib/labels";
import type {
CustomerListItem,
PolicyOcrBatchDetail,
PolicyOcrConfirmDocument,
PolicyOcrCoverage,
PolicyOcrDocument,
PolicyOcrReviewInput,
} from "@/lib/types";
const STATUS_LABEL: Record<string, string> = {
PENDING_OCR: "Pendiente",
OCR_FAILED: "Falló OCR",
NEEDS_REVIEW: "Para revisar",
MATCHED: "Listo",
CONFIRMED: "Confirmado",
POSTED: "Aplicado",
REJECTED: "Rechazado",
};
const OPEN_FIRST = [
"NEEDS_REVIEW",
"MATCHED",
"CONFIRMED",
"PENDING_OCR",
"OCR_FAILED",
"REJECTED",
"POSTED",
];
type EditMap = Record<string, PolicyOcrConfirmDocument | undefined>;
export function PolicyOcrReview({ id }: { id: string }) {
const canReview = useCan("policy:ocr-review");
const [batch, setBatch] = useState<PolicyOcrBatchDetail | null>(null);
const [docs, setDocs] = useState<PolicyOcrDocument[]>([]);
const [edits, setEdits] = useState<EditMap>({});
const [customerIndex, setCustomerIndex] = useState<Record<string, CustomerListItem>>({});
const [error, setError] = useState<string | null>(null);
const [loading, setLoading] = useState(true);
const [submitting, setSubmitting] = useState(false);
const load = useCallback(async () => {
try {
const [b, d, c] = await Promise.all([
getPolicyOcrBatch(id),
listPolicyOcrDocuments(id),
canReview ? listCustomers({ pageSize: 200 }).then((r) => r.items) : Promise.resolve([]),
]);
setBatch(b);
setDocs(d);
setCustomerIndex(Object.fromEntries(c.map((x) => [x.id, x])));
setError(null);
} catch (e) {
setError((e as Error)?.message ?? "No se pudo cargar el lote.");
} finally {
setLoading(false);
}
}, [id, canReview]);
useEffect(() => {
void load();
}, [load]);
const processing = batch?.status === "PROCESSING" || batch?.status === "UPLOADED";
useEffect(() => {
if (!processing) return;
const t = setInterval(() => void load(), 4000);
return () => clearInterval(t);
}, [processing, load]);
const sorted = useMemo(
() =>
[...docs].sort(
(a, b) =>
OPEN_FIRST.indexOf(a.status) - OPEN_FIRST.indexOf(b.status) ||
a.pageNumber - b.pageNumber,
),
[docs],
);
const readyCount = Object.values(edits).filter(Boolean).length;
function setEdit(docId: string, edit: PolicyOcrConfirmDocument) {
setEdits((prev) => ({ ...prev, [docId]: edit }));
}
async function onConfirm() {
if (!batch) return;
const payload: PolicyOcrConfirmDocument[] = [];
for (const d of docs) {
const edit = edits[d.id];
if (!edit) continue;
if (!edit.policyId && !edit.customerId) {
setError(`Página ${d.pageNumber}: falta cliente o póliza destino.`);
return;
}
payload.push(edit);
}
if (!payload.length) {
setError("No hay documentos revisados. Guarde cada página antes de aplicar.");
return;
}
setSubmitting(true);
setError(null);
try {
await confirmPolicyOcrBatch(batch.id, { documents: payload });
setEdits({});
await load();
} catch (e) {
setError((e as Error)?.message ?? "No se pudo aplicar el lote.");
} finally {
setSubmitting(false);
}
}
if (loading) return <div className="state-box">Cargando</div>;
if (!batch) return <div className="state-box state-error">{error ?? "No encontrado."}</div>;
return (
<div className="stack">
<header className="page-head">
<div>
<h1 className="page-title">
Pólizas {batch.provider}
{batch.label ? ` · ${batch.label}` : ""}
</h1>
<p className="page-sub">
{formatDate(batch.createdAt)} · {docs.length} página(s) ·{" "}
{STATUS_LABEL[batch.status] ?? batch.status}
</p>
</div>
<Link className="btn btn-ghost" href="/polizas">
Volver a pólizas
</Link>
</header>
{processing && <div className="state-box">Procesando</div>}
{error && <div className="state-box state-error">{error}</div>}
{canReview && readyCount > 0 && (
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Aplicar lote
</h2>
<p className="page-sub" style={{ marginBottom: 12 }}>
{readyCount} página(s) revisada(s). Se creará o actualizará la póliza
y, si marcó la casilla, se registrará la prima en el estado de
cuenta.
</p>
<button
type="button"
className="btn btn-primary"
disabled={submitting}
onClick={onConfirm}
>
{submitting ? "Aplicando…" : "Aplicar"}
</button>
</section>
)}
<section className="stack">
{sorted.map((doc) => (
<DocumentRow
key={doc.id}
doc={doc}
customerIndex={customerIndex}
canReview={canReview}
onSave={async (edit) => {
await reviewPolicyOcrDocument(doc.id, edit.reviewInput);
setEdit(doc.id, edit.confirmInput);
await load();
}}
onReject={async () => {
await rejectPolicyOcrDocument(doc.id);
setEdits((prev) => {
const { [doc.id]: _, ...rest } = prev;
return rest;
});
await load();
}}
/>
))}
</section>
</div>
);
}
interface RowSaved {
reviewInput: PolicyOcrReviewInput;
confirmInput: PolicyOcrConfirmDocument;
}
interface DocumentRowProps {
doc: PolicyOcrDocument;
customerIndex: Record<string, CustomerListItem>;
canReview: boolean;
onSave: (saved: RowSaved) => Promise<void>;
onReject: () => Promise<void>;
}
function DocumentRow({ doc, customerIndex, canReview, onSave, onReject }: DocumentRowProps) {
const [v, setV] = useState({
policyNumber: doc.extractedPolicyNumber ?? "",
insuredName: doc.extractedInsuredName ?? "",
additionalInsured: doc.extractedAdditionalInsured ?? "",
agentName: doc.extractedAgentName ?? "",
legalAddress: doc.extractedLegalAddress ?? "",
zip: doc.extractedZip ?? "",
policyFrom: doc.extractedPolicyFrom?.slice(0, 10) ?? "",
policyTo: doc.extractedPolicyTo?.slice(0, 10) ?? "",
policyDate: doc.extractedPolicyDate?.slice(0, 10) ?? "",
currency: doc.extractedCurrency ?? "USD",
netPremium: doc.extractedNetPremium ?? "",
total: doc.extractedTotal ?? "",
premiumPayment: doc.extractedPremiumPayment ?? "",
postPremium: doc.extractedNetPremium != null && Number(doc.extractedNetPremium) > 0,
});
const [customerId, setCustomerId] = useState(
doc.matchedCustomer?.id ?? doc.matchedPolicy?.customerId ?? "",
);
const [customerName, setCustomerName] = useState(
doc.matchedCustomer?.name ?? doc.matchedPolicy?.customer.name ?? "",
);
const [policyId, setPolicyId] = useState(doc.matchedPolicy?.id ?? "");
const [busy, setBusy] = useState(false);
const [err, setErr] = useState<string | null>(null);
function set<K extends keyof typeof v>(k: K, val: (typeof v)[K]) {
setV((p) => ({ ...p, [k]: val }));
}
async function save() {
setBusy(true);
setErr(null);
try {
const numOrUndef = (s: string) => (s.trim() === "" ? undefined : Number(s));
const trimOrUndef = (s: string) => (s.trim() === "" ? undefined : s.trim());
const currency = v.currency || undefined;
const reviewInput: PolicyOcrReviewInput = {
policyNumber: trimOrUndef(v.policyNumber),
insuredName: trimOrUndef(v.insuredName),
additionalInsured: trimOrUndef(v.additionalInsured),
agentName: trimOrUndef(v.agentName),
legalAddress: trimOrUndef(v.legalAddress),
zip: trimOrUndef(v.zip),
policyFrom: v.policyFrom || undefined,
policyTo: v.policyTo || undefined,
policyDate: v.policyDate || undefined,
currency,
netPremium: numOrUndef(v.netPremium),
total: numOrUndef(v.total),
premiumPayment: trimOrUndef(v.premiumPayment),
matchedPolicyId: policyId || undefined,
matchedCustomerId: !policyId && customerId ? customerId : undefined,
forceConfirm: true,
};
const confirmInput: PolicyOcrConfirmDocument = {
documentId: doc.id,
policyId: policyId || undefined,
customerId: !policyId && customerId ? customerId : undefined,
policyNumber: reviewInput.policyNumber,
insuredName: reviewInput.insuredName,
additionalInsured: reviewInput.additionalInsured,
agentName: reviewInput.agentName,
legalAddress: reviewInput.legalAddress,
zip: reviewInput.zip,
policyFrom: reviewInput.policyFrom,
policyTo: reviewInput.policyTo,
policyDate: reviewInput.policyDate,
currency: (currency as "MXN" | "USD" | "EUR" | undefined) ?? undefined,
netPremium: reviewInput.netPremium,
total: reviewInput.total,
premiumPayment: reviewInput.premiumPayment,
coveragesJson: (doc.extractedCoveragesJson ?? undefined) as
| PolicyOcrCoverage[]
| undefined,
postPremium: v.postPremium,
};
await onSave({ reviewInput, confirmInput });
} catch (e) {
setErr((e as Error)?.message ?? "No se pudo guardar.");
} finally {
setBusy(false);
}
}
const locked = doc.status === "POSTED" || doc.status === "REJECTED";
const matchedExisting = !!doc.matchedPolicy;
const candidates = doc.matchCandidates ?? [];
return (
<article className="card" style={{ padding: 16 }}>
<header className="row" style={{ gap: 12, alignItems: "center" }}>
<span className="tag">{STATUS_LABEL[doc.status] ?? doc.status}</span>
<span className="page-sub">Página {doc.pageNumber}</span>
{doc.extractedPolicyNumber && (
<strong style={{ marginLeft: 8 }}>{doc.extractedPolicyNumber}</strong>
)}
{doc.extractedInsuredName && (
<span className="page-sub">· {doc.extractedInsuredName}</span>
)}
</header>
<div className="doc-detail">
{/*
* Embed the source PDF the office uploaded. One PDF = one parsed
* policy, so the browser's PDF viewer handles multi-page navigation
* natively; we don't need to render individual pages on the server.
*/}
<iframe
src={policyOcrDocumentUrl(doc.id)}
title={`Póliza ${doc.extractedPolicyNumber ?? doc.pageNumber}`}
style={{
width: "100%",
height: 720,
border: "1px solid var(--border, #ddd)",
borderRadius: 6,
background: "#fff",
}}
/>
<div className="stack" style={{ flex: 1, minWidth: 0 }}>
{doc.matchNote && <p className="page-sub">{doc.matchNote}</p>}
{matchedExisting ? (
<div className="state-box">
Coincide con la póliza{" "}
<strong>{doc.matchedPolicy?.policyNumber}</strong> del cliente{" "}
<strong>{doc.matchedPolicy?.customer.name}</strong>.
</div>
) : candidates.length > 1 ? (
<div className="state-box state-warn">
{candidates.length} pólizas comparten este número. Elija
manualmente abajo.
</div>
) : (
<div className="state-box">
No se encontró una póliza con este número. Se creará una nueva
bajo el cliente que elija abajo.
</div>
)}
<fieldset className="form-grid" disabled={locked || !canReview}>
<Field label="Número de póliza">
<input
className="input"
value={v.policyNumber}
onChange={(e) => set("policyNumber", e.target.value)}
/>
</Field>
<Field label="Asegurado">
<input
className="input"
value={v.insuredName}
onChange={(e) => set("insuredName", e.target.value)}
/>
</Field>
<Field label="Asegurado adicional">
<input
className="input"
value={v.additionalInsured}
onChange={(e) => set("additionalInsured", e.target.value)}
/>
</Field>
<Field label="Agente">
<input
className="input"
value={v.agentName}
onChange={(e) => set("agentName", e.target.value)}
/>
</Field>
<Field label="Desde">
<input
className="input"
type="date"
value={v.policyFrom}
onChange={(e) => set("policyFrom", e.target.value)}
/>
</Field>
<Field label="Hasta">
<input
className="input"
type="date"
value={v.policyTo}
onChange={(e) => set("policyTo", e.target.value)}
/>
</Field>
<Field label="Fecha de firma">
<input
className="input"
type="date"
value={v.policyDate}
onChange={(e) => set("policyDate", e.target.value)}
/>
</Field>
<Field label="Moneda">
<select
className="input select"
value={v.currency}
onChange={(e) => set("currency", e.target.value)}
>
<option value="MXN">MXN</option>
<option value="USD">USD</option>
<option value="EUR">EUR</option>
</select>
</Field>
<Field label="Prima neta">
<input
className="input"
type="number"
step="0.01"
value={v.netPremium}
onChange={(e) => set("netPremium", e.target.value)}
/>
</Field>
<Field label="Total">
<input
className="input"
type="number"
step="0.01"
value={v.total}
onChange={(e) => set("total", e.target.value)}
/>
</Field>
<Field label="Pago de prima">
<input
className="input"
value={v.premiumPayment}
onChange={(e) => set("premiumPayment", e.target.value)}
/>
</Field>
<Field label="Dirección">
<input
className="input"
value={v.legalAddress}
onChange={(e) => set("legalAddress", e.target.value)}
/>
</Field>
<Field label="C.P.">
<input
className="input"
value={v.zip}
onChange={(e) => set("zip", e.target.value)}
/>
</Field>
</fieldset>
{doc.extractedCoveragesJson && doc.extractedCoveragesJson.length > 0 && (
<details>
<summary>
Coberturas ({doc.extractedCoveragesJson.length}) ·{" "}
{formatMoney(
doc.extractedCoveragesJson
.map((c) => Number(c.insuredAmount ?? 0))
.reduce((a, b) => a + b, 0)
.toString(),
v.currency,
)}
</summary>
<table className="tx-table" style={{ marginTop: 8 }}>
<thead>
<tr>
<th>Riesgo</th>
<th className="num">Suma</th>
<th>Deducible</th>
<th>Participación</th>
</tr>
</thead>
<tbody>
{doc.extractedCoveragesJson.map((c, i) => (
<tr key={i}>
<td>{c.risk}</td>
<td className="num">
{formatMoney(c.insuredAmount?.toString() ?? null, v.currency)}
</td>
<td>{c.deductible ?? "—"}</td>
<td>{c.lossParticipation ?? "—"}</td>
</tr>
))}
</tbody>
</table>
</details>
)}
{candidates.length > 1 && (
<Field label="Póliza destino">
<select
className="input select"
value={policyId}
onChange={(e) => {
setPolicyId(e.target.value);
const found = candidates.find((c) => c.policyId === e.target.value);
if (found) {
setCustomerId(found.customerId);
setCustomerName(customerIndex[found.customerId]?.name ?? found.customerName);
}
}}
>
<option value=""> elegir póliza </option>
{candidates.map((c) => (
<option key={c.policyId} value={c.policyId}>
{c.policyNumber} · {c.customerName}
</option>
))}
</select>
</Field>
)}
{!policyId && (
<Field label={matchedExisting ? "Cliente" : "Cliente (póliza nueva)"}>
<CustomerPicker
value={customerId}
valueName={customerName}
onPick={(id, name) => {
setCustomerId(id);
setCustomerName(name);
}}
/>
</Field>
)}
<label className="field">
<input
type="checkbox"
checked={v.postPremium}
onChange={(e) => set("postPremium", e.target.checked)}
disabled={!v.netPremium || Number(v.netPremium) <= 0}
/>{" "}
Registrar prima en el estado de cuenta
</label>
{err && <div className="state-box state-error">{err}</div>}
{!locked && canReview && (
<div className="row" style={{ gap: 8 }}>
<button type="button" className="btn btn-primary" disabled={busy} onClick={save}>
{busy ? "Guardando…" : "Guardar revisión"}
</button>
<button
type="button"
className="btn btn-ghost"
onClick={() => void onReject()}
>
Rechazar
</button>
</div>
)}
</div>
</div>
</article>
);
}
function Field({ label, children }: { label: string; children: React.ReactNode }) {
return (
<label className="field">
<span className="field-label">{label}</span>
{children}
</label>
);
}
+279
View File
@@ -0,0 +1,279 @@
"use client";
import { useCallback, useEffect, useState } from "react";
import Link from "next/link";
import {
getStatementStatus,
listStatementBatches,
uploadStatementBatch,
} from "@/lib/api";
import { useCan } from "@/lib/abilities";
import { formatDate, SERVICE_KIND_LABELS, serviceKindLabel } from "@/lib/labels";
import type { ServiceKind, StatementBatch, StatementBatchStatus } from "@/lib/types";
/**
* Automatic capture — statement OCR intake (docs/RECEIPT_CAPTURE_SPEC.md §2).
*
* Each utility company mails 300+ paper bills a month, one per customer, which
* staff otherwise key in by hand on the manual tab of the same "Captura"
* screen. Here they scan the stack instead, and the machine proposes customer +
* amount for every page; a human still confirms before anything reaches the
* ledger. Same daily job, same ledger path — only the input differs, which is
* why it lives as a mode of Captura rather than a screen of its own.
*
* One batch = one service kind, because the matcher is scoped per kind: a
* water account number and a phone number are compared against different
* columns, and mixing them in one upload is how a bill gets posted to the
* wrong customer.
*/
/** The kinds the parsers actually recognise today. */
const SUPPORTED: ServiceKind[] = [
"ELECTRIC",
"WATER",
"TELEPHONE",
"GAS",
"PROPERTY_TAX",
"FEDERAL_ZONE",
];
/** Uploadable, but every page will land in review until a parser learns it. */
const OTHER_KINDS: ServiceKind[] = ["CABLE"];
const STATUS_LABEL: Record<StatementBatchStatus, string> = {
UPLOADED: "Recibido",
PROCESSING: "Procesando…",
READY_FOR_REVIEW: "Listo para revisar",
COMPLETED: "Registrado",
FAILED: "Falló",
};
export function StatementIntake() {
const canIngest = useCan("statement:ingest");
const [batches, setBatches] = useState<StatementBatch[]>([]);
const [ocrAvailable, setOcrAvailable] = useState<boolean | null>(null);
const [storageAvailable, setStorageAvailable] = useState<boolean | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const load = useCallback(async () => {
try {
const [list, status] = await Promise.all([
listStatementBatches(),
getStatementStatus(),
]);
setBatches(list.items);
setOcrAvailable(status.ocrAvailable);
setStorageAvailable(status.storageAvailable);
setError(null);
} catch (e) {
setError((e as Error)?.message ?? "No se pudieron cargar los lotes.");
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
void load();
}, [load]);
// A batch of 300 pages takes minutes to OCR, so the list refreshes itself
// while anything is still working rather than making staff reload.
const working = batches.some(
(b) => b.status === "PROCESSING" || b.status === "UPLOADED",
);
useEffect(() => {
if (!working) return;
const t = setInterval(() => void load(), 4000);
return () => clearInterval(t);
}, [working, load]);
const ready = ocrAvailable === true && storageAvailable === true;
return (
<div className="stack">
{ocrAvailable === false && (
<div className="state-box state-error">
Este servidor no tiene OCR instalado, así que no se pueden leer recibos
escaneados. Usa la captura manual; el resto del sistema funciona con
normalidad.
</div>
)}
{storageAvailable === false && (
<div className="state-box state-error">
Este servidor no tiene configurado el almacenamiento de documentos, así
que no hay dónde guardar los recibos escaneados. Usa la captura manual
mientras se configura.
</div>
)}
{canIngest && ready && <UploadCard onDone={load} />}
{error && <div className="state-box state-error">{error}</div>}
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Lotes
</h2>
{loading ? (
<div className="state-box">Cargando</div>
) : batches.length === 0 ? (
<div className="state-box">Todavía no hay lotes de recibos.</div>
) : (
<div className="tx-scroll">
<table className="tx-table">
<thead>
<tr>
<th>Fecha</th>
<th>Servicio</th>
<th>Referencia</th>
<th>Estado</th>
<th className="num">Páginas</th>
<th>Subido por</th>
<th />
</tr>
</thead>
<tbody>
{batches.map((b) => (
<tr key={b.id}>
<td style={{ whiteSpace: "nowrap" }}>{formatDate(b.createdAt)}</td>
<td>{serviceKindLabel(b.serviceKind)}</td>
<td>{b.label || "—"}</td>
<td>
<StatusTag status={b.status} />
{b.error && (
<div className="page-sub" style={{ marginTop: 4 }}>
{b.error}
</div>
)}
</td>
<td className="num">{b._count?.documents ?? 0}</td>
<td>{b.uploadedBy?.name ?? "—"}</td>
<td>
<Link className="btn btn-ghost" href={`/recibos/${b.id}`}>
Revisar
</Link>
</td>
</tr>
))}
</tbody>
</table>
</div>
)}
</section>
</div>
);
}
function StatusTag({ status }: { status: StatementBatchStatus }) {
return <span className="tag">{STATUS_LABEL[status] ?? status}</span>;
}
function UploadCard({ onDone }: { onDone: () => void }) {
const [files, setFiles] = useState<File[]>([]);
const [serviceKind, setServiceKind] = useState<ServiceKind>("ELECTRIC");
const [label, setLabel] = useState("");
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
async function submit() {
if (!files.length) return;
setBusy(true);
setError(null);
try {
await uploadStatementBatch(files, serviceKind, label.trim() || undefined);
setFiles([]);
setLabel("");
onDone();
} catch (e) {
setError((e as Error)?.message ?? "No se pudo subir el lote.");
} finally {
setBusy(false);
}
}
const unsupported = !SUPPORTED.includes(serviceKind);
return (
<section className="card" style={{ padding: 16 }}>
<h2 className="section-title" style={{ marginTop: 0 }}>
Subir recibos escaneados
</h2>
<div className="inline-form" style={{ flexWrap: "wrap", gap: 12 }}>
<label>
<span className="page-sub">Servicio</span>
<select
className="input"
value={serviceKind}
onChange={(e) => setServiceKind(e.target.value as ServiceKind)}
>
<optgroup label="Con lectura automática">
{SUPPORTED.map((k) => (
<option key={k} value={k}>
{SERVICE_KIND_LABELS[k] ?? k}
</option>
))}
</optgroup>
<optgroup label="Sin lectura automática (revisión manual)">
{OTHER_KINDS.map((k) => (
<option key={k} value={k}>
{SERVICE_KIND_LABELS[k] ?? k}
</option>
))}
</optgroup>
</select>
</label>
<label>
<span className="page-sub">Referencia (opcional)</span>
<input
className="input"
placeholder="ej. CFE julio 2026"
value={label}
onChange={(e) => setLabel(e.target.value)}
/>
</label>
<label>
<span className="page-sub">Archivos PDF</span>
<input
type="file"
className="input"
accept="application/pdf"
multiple
onChange={(e) => setFiles(Array.from(e.target.files ?? []))}
/>
</label>
<button
type="button"
className="btn btn-primary"
disabled={!files.length || busy}
onClick={submit}
>
{busy ? "Subiendo…" : `Procesar ${files.length || ""}`.trim()}
</button>
</div>
{error && (
<div className="state-box state-error" style={{ marginTop: 12 }}>
{error}
</div>
)}
{unsupported && (
<div className="state-box" style={{ marginTop: 12 }}>
Todavía no hay lectura automática para{" "}
{SERVICE_KIND_LABELS[serviceKind] ?? serviceKind}: cada página quedará
para revisión manual. Al confirmarlas se guarda el número de cuenta,
así que los recibos del mes siguiente se reconocerán solos.
</div>
)}
<p className="page-sub" style={{ marginTop: 12 }}>
Un lote es de un solo servicio. Cada página del PDF se trata como un
recibo distinto, salvo que el proveedor imprima varias hojas por cliente.
</p>
</section>
);
}
+296 -15
View File
@@ -16,6 +16,8 @@ import type {
BankStats,
BankSummary,
BatchCreateInput,
ConfirmBatchInput,
ConfirmBatchResult,
BatchCreateResponse,
BillingFacets,
BillingStats,
@@ -27,6 +29,11 @@ import type {
CreateMovementInput,
UpdateBankAccountInput,
ResolveOutstandingInput,
ReviewDocumentInput,
StatementBatch,
StatementBatchDetail,
StatementDocument,
StatementDocumentStatus,
CustomerDetail,
CustomerInput,
CustomerListResponse,
@@ -42,6 +49,12 @@ import type {
PolicySort,
PolicyStats,
PolicyStatus,
PolicyOcrBatch,
PolicyOcrBatchDetail,
PolicyOcrDocument,
PolicyOcrReviewInput,
PolicyOcrConfirmInput,
PolicyOcrConfirmResult,
LookupsResponse,
OpsJob,
OpsJobKind,
@@ -796,38 +809,117 @@ export function listIngest(): Promise<IngestFile[]> {
return apiFetch<IngestFile[]>("/ops/ingest");
}
/** Live upload stats reported to `uploadFile`'s `onProgress` callback. */
export type UploadProgress = {
loaded: number;
/** 0 when the browser can't compute the request size. */
total: number;
/** 0..1, or null when `total` is unknown. */
fraction: number | null;
/** Smoothed transfer rate. */
bytesPerSecond: number;
/** null until a rate and a total are both known. */
secondsRemaining: number | null;
/** True once the bytes are sent and we're waiting on the server's reply. */
finishing: boolean;
};
/**
* Multipart upload — not JSON, so it bypasses apiFetch's Content-Type. `path`
* is API-relative (may include a query string); `filename` overrides the part
* name sent to the server.
* name sent to the server. Uses XHR rather than fetch because fetch has no way
* to report request-body progress.
*/
export async function uploadFile(
export function uploadFile(
path: string,
file: File,
filename?: string,
onProgress?: (p: UploadProgress) => void,
): Promise<unknown> {
const body = new FormData();
body.append("file", file, filename ?? file.name);
const res = await fetch(`${API_ORIGIN}${path}`, {
method: "POST",
credentials: "include",
body,
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("POST", `${API_ORIGIN}${path}`);
xhr.withCredentials = true;
if (onProgress) {
// Exponentially smoothed rate — raw per-chunk deltas jump around too
// much to read.
let lastAt = performance.now();
let lastLoaded = 0;
let rate = 0;
xhr.upload.onprogress = (e) => {
const now = performance.now();
const dt = (now - lastAt) / 1000;
if (dt >= 0.15) {
const sample = (e.loaded - lastLoaded) / dt;
rate = rate === 0 ? sample : rate * 0.7 + sample * 0.3;
lastAt = now;
lastLoaded = e.loaded;
}
const total = e.lengthComputable ? e.total : 0;
onProgress({
loaded: e.loaded,
total,
fraction: total ? e.loaded / total : null,
bytesPerSecond: rate,
secondsRemaining:
total && rate > 0 ? (total - e.loaded) / rate : null,
finishing: false,
});
if (!res.ok) {
let message = `Error ${res.status}`;
};
// Bytes are out the door; the server still has to write the file.
xhr.upload.onload = () => {
onProgress({
loaded: file.size,
total: file.size,
fraction: 1,
bytesPerSecond: rate,
secondsRemaining: 0,
finishing: true,
});
};
}
xhr.onload = () => {
let parsed: unknown;
try {
const b = await res.json();
if (b?.message) message = b.message;
parsed = xhr.responseText ? JSON.parse(xhr.responseText) : undefined;
} catch {
/* ignore */
parsed = undefined;
}
throw new ApiError(res.status, message);
if (xhr.status >= 200 && xhr.status < 300) {
resolve(parsed);
return;
}
return res.status === 204 ? undefined : res.json().catch(() => undefined);
const message =
(parsed as { message?: string } | undefined)?.message ??
`Error ${xhr.status}`;
reject(new ApiError(xhr.status, message));
};
xhr.onerror = () =>
reject(new ApiError(0, "Fallo de red durante la carga."));
xhr.onabort = () => reject(new ApiError(0, "Carga cancelada."));
xhr.ontimeout = () => reject(new ApiError(0, "Tiempo de carga agotado."));
xhr.send(body);
});
}
export function uploadIngest(name: string, file: File): Promise<unknown> {
return uploadFile(`/ops/ingest/${encodeURIComponent(name)}`, file, name);
export function uploadIngest(
name: string,
file: File,
onProgress?: (p: UploadProgress) => void,
): Promise<unknown> {
return uploadFile(
`/ops/ingest/${encodeURIComponent(name)}`,
file,
name,
onProgress,
);
}
export function deleteIngest(name: string): Promise<unknown> {
@@ -895,3 +987,192 @@ export function reportDownloadUrl(
const tail = qs.toString();
return `${API_ORIGIN}/reports/${slug}/${format}${tail ? `?${tail}` : ""}`;
}
/* ------------------------------------- Statement OCR intake (recibos) */
/**
* Whether this deployment can ingest scans — automatic capture is hidden
* without it. OCR reads the page, object storage keeps it; both are required.
*/
export function getStatementStatus(): Promise<{
ocrAvailable: boolean;
storageAvailable: boolean;
}> {
return apiFetch("/statements/status");
}
export function listStatementBatches(
page = 1,
pageSize = 25,
): Promise<{
items: StatementBatch[];
total: number;
page: number;
pageSize: number;
pageCount: number;
}> {
return apiFetch(`/statements/batches?page=${page}&pageSize=${pageSize}`);
}
export function getStatementBatch(id: string): Promise<StatementBatchDetail> {
return apiFetch(`/statements/batches/${id}`);
}
export function listStatementDocuments(
batchId: string,
status?: StatementDocumentStatus,
): Promise<StatementDocument[]> {
const q = status ? `?status=${status}` : "";
return apiFetch(`/statements/batches/${batchId}/documents${q}`);
}
/** Multi-file upload — one batch is usually several multi-page scans. */
export async function uploadStatementBatch(
files: File[],
serviceKind: ServiceKind,
label?: string,
): Promise<StatementBatch> {
const body = new FormData();
for (const f of files) body.append("files", f, f.name);
const qs = new URLSearchParams({ serviceKind });
if (label) qs.set("label", label);
const res = await fetch(`${API_ORIGIN}/statements/batches?${qs}`, {
method: "POST",
credentials: "include",
body,
});
if (!res.ok) {
let message = `Error ${res.status}`;
try {
const b = await res.json();
if (b?.message) message = b.message;
} catch {
/* non-JSON error body */
}
throw new Error(message);
}
return res.json();
}
export function reviewStatementDocument(
id: string,
input: ReviewDocumentInput,
): Promise<StatementDocument> {
return apiFetch(`/statements/documents/${id}`, {
method: "PATCH",
body: JSON.stringify(input),
});
}
export function rejectStatementDocument(id: string): Promise<StatementDocument> {
return apiFetch(`/statements/documents/${id}/reject`, { method: "POST" });
}
export function confirmStatementBatch(
batchId: string,
input: ConfirmBatchInput,
): Promise<ConfirmBatchResult> {
return apiFetch(`/statements/batches/${batchId}/confirm`, {
method: "POST",
body: JSON.stringify(input),
});
}
/** The rendered page image. A plain <img src> — the cookie rides along. */
export function statementPageUrl(documentId: string): string {
return `${API_ORIGIN}/statements/documents/${documentId}/page`;
}
/* ----------------------------------------------------- Policy OCR (GMX) */
export function getPolicyOcrStatus(): Promise<{
ocrAvailable: boolean;
storageAvailable: boolean;
}> {
return apiFetch("/policy-ocr/status");
}
export function listPolicyOcrBatches(
page = 1,
pageSize = 25,
): Promise<{
items: PolicyOcrBatch[];
total: number;
page: number;
pageSize: number;
pageCount: number;
}> {
return apiFetch(`/policy-ocr/batches?page=${page}&pageSize=${pageSize}`);
}
export function getPolicyOcrBatch(id: string): Promise<PolicyOcrBatchDetail> {
return apiFetch(`/policy-ocr/batches/${id}`);
}
export function listPolicyOcrDocuments(batchId: string): Promise<PolicyOcrDocument[]> {
return apiFetch(`/policy-ocr/batches/${batchId}/documents`);
}
export async function uploadPolicyOcrBatch(
files: File[],
label?: string,
): Promise<PolicyOcrBatch> {
const body = new FormData();
for (const f of files) body.append("files", f, f.name);
const qs = new URLSearchParams();
if (label) qs.set("label", label);
const res = await fetch(
`${API_ORIGIN}/policy-ocr/batches${qs.toString() ? `?${qs}` : ""}`,
{
method: "POST",
credentials: "include",
body,
},
);
if (!res.ok) {
let message = `Error ${res.status}`;
try {
const b = await res.json();
if (b?.message) message = b.message;
} catch {
/* non-JSON error body */
}
throw new Error(message);
}
return res.json();
}
export function reviewPolicyOcrDocument(
id: string,
input: PolicyOcrReviewInput,
): Promise<PolicyOcrDocument> {
return apiFetch(`/policy-ocr/documents/${id}`, {
method: "PATCH",
body: JSON.stringify(input),
});
}
export function rejectPolicyOcrDocument(id: string): Promise<PolicyOcrDocument> {
return apiFetch(`/policy-ocr/documents/${id}/reject`, { method: "POST" });
}
export function confirmPolicyOcrBatch(
batchId: string,
input: PolicyOcrConfirmInput,
): Promise<PolicyOcrConfirmResult> {
return apiFetch(`/policy-ocr/batches/${batchId}/confirm`, {
method: "POST",
body: JSON.stringify(input),
});
}
/**
* URL for the source PDF of a parsed policy document. The endpoint returns
* the original upload (one PDF = one parsed policy), not a rendered page
* image, so the review screen embeds it in an iframe.
*/
export function policyOcrDocumentUrl(documentId: string): string {
return `${API_ORIGIN}/policy-ocr/documents/${documentId}/page`;
}
-5
View File
@@ -35,8 +35,3 @@ export function webBuildInfo(): BuildInfo {
const injected = (window as { __APP_BUILD__?: BuildInfo }).__APP_BUILD__;
return injected ?? UNKNOWN_BUILD;
}
/** First 7 chars, the length git itself abbreviates to. */
export function shortSha(sha: string): string {
return sha === "unknown" ? sha : sha.slice(0, 7);
}
+2
View File
@@ -45,6 +45,7 @@ export const SERVICE_KIND_LABELS: Record<string, string> = {
PROPERTY_TAX: "Predial",
FEDERAL_ZONE: "Zona Federal",
ALARM: "Alarma",
TELEPHONE: "Teléfono",
OTHER: "Otro",
};
@@ -57,6 +58,7 @@ export const SERVICE_KIND_GLYPH: Record<string, string> = {
WATER: "≈",
ELECTRIC: "⚡",
GAS: "◐",
TELEPHONE: "☎",
CABLE: "▤",
PROPERTY_TAX: "⌂",
FEDERAL_ZONE: "⇲",
+228
View File
@@ -12,6 +12,8 @@ export type Ability =
| "policy:create"
| "policy:update"
| "policy:delete"
| "policy:ingest"
| "policy:ocr-review"
| "property:create"
| "property:update"
| "property:delete"
@@ -20,6 +22,8 @@ export type Ability =
| "bank:create"
| "bank:void"
| "bank:manage-accounts"
| "statement:ingest"
| "statement:review"
| "lookup:manage"
| "user:manage"
| "db:manage";
@@ -134,6 +138,7 @@ export type ServiceKind =
| "PROPERTY_TAX"
| "FEDERAL_ZONE"
| "ALARM"
| "TELEPHONE"
| "OTHER"
| string;
@@ -1204,3 +1209,226 @@ export interface ReportRunResult {
export interface ReportCatalog {
items: ReportDef[];
}
/* ------------------------------------- Statement OCR intake (recibos) */
export type StatementBatchStatus =
| "UPLOADED"
| "PROCESSING"
| "READY_FOR_REVIEW"
| "COMPLETED"
| "FAILED";
export type StatementDocumentStatus =
| "PENDING_OCR"
| "OCR_FAILED"
| "NEEDS_REVIEW"
| "MATCHED"
| "CONFIRMED"
| "POSTED"
| "REJECTED";
export interface StatementBatch {
id: string;
serviceKind: ServiceKind;
status: StatementBatchStatus;
label: string | null;
fileCount: number;
error: string | null;
createdAt: string;
completedAt: string | null;
uploadedBy?: { name: string };
_count?: { documents: number };
}
export interface StatementBatchDetail extends StatementBatch {
byStatus: Partial<Record<StatementDocumentStatus, number>>;
/** Sum of the amounts still awaiting posting. */
pendingTotal: string;
}
export interface StatementDocument {
id: string;
pageNumber: number;
status: StatementDocumentStatus;
provider: string | null;
ocrConfidence: string | null;
extractedAccountRef: string | null;
extractedAmount: string | null;
extractedPeriod: string | null;
extractedDueDate: string | null;
extractedCadastralKey: string | null;
matchNote: string | null;
matchedCustomer: { id: string; name: string } | null;
matchedPropertyService: {
id: string;
kind: ServiceKind;
accountNumber: string | null;
meterNumber: string | null;
property: { id: string; addressLine1: string | null };
} | null;
postedTransactionId: string | null;
}
export interface ReviewDocumentInput {
accountRef?: string;
amount?: number;
period?: string;
dueDate?: string;
matchedPropertyServiceId?: string;
matchedCustomerId?: string;
status?: "MATCHED" | "NEEDS_REVIEW" | "CONFIRMED";
}
/** Check-level fields shared by every line posted from a batch. */
export interface ConfirmBatchInput {
checkNumber: string;
transactionDate: string;
currency?: Currency;
typeId?: string;
outstanding?: boolean;
includeReviewed?: boolean;
}
export interface ConfirmBatchResult {
posted: number;
total: string;
checkNumber: string;
}
/* ------------------------------------------ Policy OCR intake (GMX) */
export type PolicyOcrBatchStatus =
| "UPLOADED"
| "PROCESSING"
| "READY_FOR_REVIEW"
| "COMPLETED"
| "FAILED";
export type PolicyOcrDocumentStatus =
| "PENDING_OCR"
| "OCR_FAILED"
| "NEEDS_REVIEW"
| "MATCHED"
| "CONFIRMED"
| "POSTED"
| "REJECTED";
export interface PolicyOcrBatch {
id: string;
provider: string;
status: PolicyOcrBatchStatus;
label: string | null;
fileCount: number;
error: string | null;
createdAt: string;
completedAt: string | null;
uploadedBy?: { name: string };
_count?: { documents: number };
}
export interface PolicyOcrBatchDetail extends PolicyOcrBatch {
byStatus: Partial<Record<PolicyOcrDocumentStatus, number>>;
}
export interface PolicyOcrCoverage {
risk: string;
insuredAmount: number | null;
deductible: string | null;
lossParticipation: string | null;
}
export interface PolicyOcrMatchCandidate {
policyId: string;
customerId: string;
customerName: string;
policyNumber: string;
}
export interface PolicyOcrDocument {
id: string;
pageNumber: number;
status: PolicyOcrDocumentStatus;
provider: string | null;
ocrConfidence: string | null;
extractedPolicyNumber: string | null;
extractedInsuredName: string | null;
extractedAdditionalInsured: string | null;
extractedAgentName: string | null;
extractedLegalAddress: string | null;
extractedZip: string | null;
extractedPolicyFrom: string | null;
extractedPolicyTo: string | null;
extractedPolicyDate: string | null;
extractedCurrency: string | null;
extractedNetPremium: string | null;
extractedPolicyFee: string | null;
extractedBrokerFee: string | null;
extractedTotal: string | null;
extractedCoveragesJson: PolicyOcrCoverage[] | null;
extractedPremiumPayment: string | null;
matchedPolicy: {
id: string;
policyNumber: string | null;
customerId: string;
customer: { name: string };
} | null;
matchedCustomer: { id: string; name: string } | null;
matchCandidates: PolicyOcrMatchCandidate[] | null;
matchNote: string | null;
}
export interface PolicyOcrReviewInput {
policyNumber?: string;
insuredName?: string;
additionalInsured?: string;
agentName?: string;
legalAddress?: string;
zip?: string;
policyFrom?: string;
policyTo?: string;
policyDate?: string;
currency?: string;
netPremium?: number;
policyFee?: number;
brokerFee?: number;
total?: number;
premiumPayment?: string;
coveragesJson?: PolicyOcrCoverage[];
matchedPolicyId?: string;
matchedCustomerId?: string;
forceConfirm?: boolean;
}
export interface PolicyOcrConfirmDocument {
documentId: string;
policyId?: string;
customerId?: string;
policyNumber?: string;
insuredName?: string;
additionalInsured?: string;
agentName?: string;
legalAddress?: string;
zip?: string;
policyFrom?: string;
policyTo?: string;
policyDate?: string;
currency?: string;
netPremium?: number;
policyFee?: number;
brokerFee?: number;
total?: number;
premiumPayment?: string;
coveragesJson?: PolicyOcrCoverage[];
postPremium?: boolean;
}
export interface PolicyOcrConfirmInput {
documents: PolicyOcrConfirmDocument[];
}
export interface PolicyOcrConfirmResult {
applied: number;
policies: string[];
postedTransactions: number;
}
+18 -1
View File
@@ -46,7 +46,24 @@ ENV NODE_ENV=production
# ERROR 1045: Plugin caching_sha2_password could not be loaded
# That breaks the pre-migrate deploy backup AND the whole "Operaciones" admin
# panel (backup, restore, sync, re-import all shell out to these binaries).
RUN apk add --no-cache python3 mdbtools mysql-client mariadb-connector-c openssl \
#
# mdbtools-utils, NOT mdbtools. Alpine splits the project: `mdbtools` is the
# shared library only, and the command-line tools migration/extract.py shells out
# to (`mdb-tables`, `mdb-export`) are in the -utils subpackage. Installing the
# wrong one builds fine and fails at run time — the re-import in the "Operaciones"
# panel dies with:
# RuntimeError: mdbtools not found on PATH (need mdb-tables and mdb-export)
#
# tesseract-ocr + tesseract-ocr-data-spa + poppler-utils drive the statement
# OCR intake (RECEIPT_CAPTURE_SPEC §2): poppler's `pdftoppm` rasterises each
# scanned page and tesseract reads it, with the Spanish traineddata for the
# accented labels on CFE/CESPT/Telnor bills. These are external binaries rather
# than a native npm addon so the pnpm workspace stays free of a compiled
# dependency. If they are absent the API still boots — the statements module
# reports itself unavailable and only that feature is disabled — but statement
# ingest is the point of shipping them.
RUN apk add --no-cache python3 mdbtools-utils mysql-client mariadb-connector-c openssl \
tesseract-ocr tesseract-ocr-data-spa poppler-utils \
&& apk add --no-cache --virtual .pybuild python3-dev build-base \
&& rm -rf /var/cache/apk/*
+28 -5
View File
@@ -5,13 +5,20 @@ rollbacks possible.
## The short version
Dispatch **Cut release** from the Actions tab and pick `patch`, `minor` or
`major` (or `explicit` plus a number). It stamps every `package.json`, commits
`chore(release): vX.Y.Z`, tags, and pushes both refs in one go. It refuses a
version that already exists as a tag, and refuses a no-op bump.
The equivalent by hand, if you would rather cut it locally:
```bash
pnpm version:set 1.2.0 # stamp every package.json
git commit -am "chore(release): v1.2.0"
git tag v1.2.0 && git push origin master v1.2.0
```
That push triggers `.gitea/workflows/build.yml`, which builds **both** images in
Either way that push triggers `.gitea/workflows/build.yml`, which builds **both** images in
one matrix run and publishes:
| tag pushed | image tags produced |
@@ -28,6 +35,13 @@ Then dispatch a deploy from the Actions tab:
> `{{version}}` strips it. Git tag `v1.2.0`, dispatch `1.2.0`. Dispatching
> `v1.2.0` deploys nothing that exists.
**Cut release needs a `RELEASE_TOKEN` secret** — a Gitea PAT with
`write:repository`. It does not use the built-in Actions token on purpose:
whether a push made with that token re-triggers `build.yml` depends on the Gitea
version, and a release that quietly publishes no images is worse than one that
fails outright. If the build somehow does not start, `build.yml` has
`workflow_dispatch` — run it against the new tag by hand.
Because api and web are built from one matrix run, they cannot drift at build
time. They *can* drift at deploy time if a stack is applied with only one image
moved — the web footer shows both versions and flags a mismatch, and the deploy
@@ -280,10 +294,19 @@ a warning, which is what local development wants.
Two more things the panel's dumps now do, for the same reasons the pre-migrate
backup does them (see `deploy/scripts/pre-migrate-backup.mjs`):
- **`--set-gtid-purged=OFF`.** galactus is the replication *source* with GTID
on, so without this every dump embeds `SET @@GLOBAL.GTID_PURGED` and cannot be
restored onto the server it came from — which is precisely what the restore
screen exists to do.
- **`--set-gtid-purged=OFF`, but only when the dumper supports it.** galactus is
the replication *source* with GTID on, so on a MySQL client this flag is what
keeps every dump from embedding `SET @@GLOBAL.GTID_PURGED` and becoming
unrestorable onto the server it came from — which is precisely what the
restore screen exists to do. The panel, however, dumps from *inside the API
container*, where Alpine's `mysql-client` is MariaDB's: there `mysqldump` is a
shim over `mariadb-dump`, the flag does not exist, and passing it failed every
backup with `mysqldump: unknown variable 'set-gtid-purged=OFF'`. So the panel
probes `mysqldump --help` and passes the flag only if it is advertised,
invoking `mariadb-dump` directly otherwise (MariaDB writes no GTID state
unless asked with `--gtid`, so there is nothing to suppress). The pre-migrate
backup keeps the flag unconditionally — it runs in a real `mysql:8.4` image,
not in the API container.
- **`set -o pipefail` and a `CREATE TABLE` count.** `mysqldump | gzip` reports
gzip's exit status, and a `mysqldump` that dies on its first statement still
produces a ~372-byte perfectly valid archive that passes `gzip -t`. Without
+192 -12
View File
@@ -131,6 +131,179 @@ single-movement form.
## 2. PDF / OCR auto-capture
> **BUILT — 2026-08-01.** Implemented and verified end to end against real
> scanned statements. `apps/api/src/statements/` holds the module: a swappable
> `OcrProvider` seam with a self-hosted Tesseract implementation, per-provider
> parsers for CFE / CESPT / Telnor / gas / predial, a scoped matcher, and a
> review queue that
> posts through `BillingService.createBatch` with `source: "OCR"`. Web:
> the "Captura automática (OCR)" tab of the Captura screen (upload + batch
> list) and `/recibos/:id` (review queue with the page image beside the
> extracted fields). New abilities `statement:ingest` / `statement:review`, both
> STAFF.
>
> Auto-capture is a *mode of* §1.2's capture screen, not a separate menu entry:
> it is the same daily job with a scanner instead of a keyboard, and both modes
> post through the same ledger path. `/estado-cuenta/lote` opens the manual tab,
> `/recibos` the automatic one; both render `components/Captura.tsx`.
>
> Requires object storage (`S3_ENDPOINT` + credentials): the scans are kept as
> blobs. `GET /statements/status` reports `ocrAvailable` and `storageAvailable`,
> and the upload card hides itself unless both hold.
>
> **Measured, not assumed.** Ten real scans (46 pages of CFE, CESPT and Telnor
> bills) drove every decision below. Against them the shipped parser identifies
> the provider on **46/46**, reads an account reference on **43/46**, an amount
> on **42/46**, and a due date on **44/46**. Matched against the dev database
> that is **39/46 (85%) exact auto-match, 40/46 (87%) identified**. The
> remainder are legitimate review cases: one account number shared by two
> services, three phone numbers not yet on file, one clave not in the book, and
> one page too poorly scanned to read.
>
> **The OCR-provider question is closed: self-hosted Tesseract.** It clears the
> bar for a queue where a human confirms every row, and at 300+ pages/month/
> company a per-page API would carry real recurring cost for accuracy that is
> not the bottleneck. `OcrProvider` keeps a managed API (Textract, Document
> Intelligence, Document AI) a one-line swap in `statements.module.ts` with no
> schema change.
>
> **Four things the samples proved that this spec had wrong or unknown:**
>
> 1. **Clave catastral ≠ predial — gap 2 below is resolved.** `DATMEX.clave` is
> 934 rows of `[A-Z]{2}[0-9]{6}` (`MM000012`, `KH220204`), the exact format
> printed as `Cve. Cat.` / `CLAVE CATASTRAL` on real CESPT bills
> (`KB078025`, `KA903009`). `DATMEX.predial` — what
> `PROPERTY_TAX.accountNumber` actually holds — is 1135 numeric rows with
> only **663 distinct values**, so it is not a per-property key at all and
> appears on no statement. The clave was never migrated; it now lives on
> `Property.cadastralKey` (property-level, because two different services
> both print it) and is the matcher's secondary key. Predial is left
> untouched. Predial statements match on the clave alone.
> 2. **Gas is not a dead end — gap 3 below was wrong.** `DATMEX.gas` has 334
> filled rows, of which **160 are real numeric account numbers**
> (`900004807`); the other 174 are tank descriptors (`ESTACIONARIO`,
> `CILINDRO`). All 334 went to `notes`. The 160 are recovered into
> `GAS.meterNumber`; only the descriptor rows start cold.
> 3. **Phone is one line per property, not three.** Of 1518 properties, 534
> have `phone1`, 18 have `phone2` and exactly **1** has `phone3`. The
> secondaries are alternate contacts, so `TELEPHONE` backfills from `phone1`
> only rather than fanning out. This answers the open question below.
> 4. **Statements arrive bundled, and their printed names are stale.** One PDF
> holds many customers, one per page (Telnor's own `Pág 3 de 6` refers to
> its internal pagination, not the office's scan). And the name on a utility
> bill is the account registrant, not the current owner: a CESPT receipt for
> account `5365218` prints `ARNAIZ ROSAS ELSA AURORA` where the office's
> book — corroborated by the clave — has `CATT, RANDY`. **The matcher never
> reads the name.**
>
> **Two OCR traps worth keeping in mind if the parsers are ever extended:**
> scanned logos read badly (a CESPT header came back as `E BAJA ES PAGO /
> EALIFORNIA`), so provider detection falls back to layout anchors — but only
> after *every* brand check has run, since a Telnor page contains words a CFE
> structural rule would otherwise claim. And amounts must be parsed by
> separator position: a real Telnor bill OCR'd as `$ 649,00`, which naive
> comma-stripping turns into $64,900.
>
> **Not covered:** handwritten folder numbers. Staff pencil a customer number on
> each bill (`9`, `405`, `406`); Tesseract read `405` as `205`. Handwriting is
> a review hint at best and is deliberately not an input to matching.
> **EXTENDED — gas and predial, 2026-08-01.** A second corpus (14 documents,
> 29 pages: five municipal predial batches and ten gas invoices) added four
> parsers — `GAS TIJUANA` plus one per municipality, because Tijuana, Rosarito
> and Ensenada issue three completely different documents. End to end against
> the dev database that is **21/29 auto-matched, 22/29 identified**, with the
> provider read on 29/29 and an amount on 26/29.
>
> The eight review cases are all legitimate: five Tijuana pages whose municipal
> account is not yet on file (see below), one clave not in the book, one page
> too poorly scanned to read a clave at all, and one gas account shared by two
> services. Excluding the structural Tijuana case, that is 21/24.
>
> **Five things this corpus proved:**
>
> 1. **Not every statement is a scan.** The gas company sends born-digital CFDI
> invoices whose text layer is exact. Rasterising and re-recognising those
> can only lose information — one sample turned `MEDIDOR: VM01014426` into
> `ar (LTR): 014420` — so `OcrProvider.textPages` reads the embedded layer
> first (`pdftotext -bbox-layout`, same poppler package as `pdftoppm`) and
> OCR stays the fallback for real scans. Page images are still rendered and
> stored either way, because the reviewer needs to see the paper.
> 2. **The clave catastral is not two letters and six digits.** Positions four
> through eight are digits in all 932 stored claves, but the third is a
> letter in fifteen of them (`MMB01041`, `CGH52121`). Digitising the whole
> tail maps that `B` to an `8` and produces a key matching no property.
> 3. **Tijuana predial prints no clave catastral at all.** Its only identifier
> is an 8-digit municipal account, carried in a 32-digit payment barcode
> (`account(8) + DDMMYY + amount(9) + folio(9)`) that the legacy database
> never held. It goes in `PROPERTY_TAX.meterNumber` — the same column gas
> uses, and for the same reason: `accountNumber` holds `DATMEX.predial`,
> which is not a per-property key and overwriting it would destroy the only
> link back to the original records. So Tijuana pages start cold and are
> taught by the first confirm, exactly like gas.
> 4. **On Rosarito and Ensenada the clave is the primary key, not a fallback.**
> Those receipts print nothing else, so a unique clave hit there is a real
> match and auto-matches; on a utility bill that merely happens to print one
> it stays a review hint, as before.
> 5. **A misread `$` is the dangerous failure, not a missing one.** An Ensenada
> receipt for `$2,203.00` OCR'd as `82,203.00` — the dollar sign read as an
> 8, which would post a charge 37× too large and look entirely ordinary in
> the ledger. Every predial amount therefore requires a literal `$`, and a
> page that cannot produce one reports no amount and goes to review. Two of
> the 29 pages take that path, which is the correct outcome for both.
>
> Regression cover for all of the above lives in
> `statement-parser.spec.ts` and `tesseract.provider.spec.ts`; every fixture in
> them is a verbatim OCR excerpt from a real receipt.
> **EXTENDED — zona federal, 2026-08-01.** A third corpus (one document, 8
> pages of Tijuana "Zona Federal Marítimo Terrestre" receipts — the federal
> maritime-zone occupancy fee billed on beachfront lots) added the
> `ZONA FEDERAL TIJUANA` parser. Provider read on 8/8, amount on 8/8 (all
> eight verified against the paper), concession clave on 6/8, period on 8/8,
> payment deadline on 2/8. Nothing auto-matched, and nothing could have — see
> point 2.
>
> **Four things this corpus proved:**
>
> 1. **Tijuana bills predial and zona federal from the same treasury.** Same
> "Ayuntamiento de Tijuana" header, same Paseo del Centenario address, same
> `ATB-541201` RFC — every discriminator the predial parser uses matches a
> zona federal page too, so whichever rule is asked first wins. The words
> only this layout prints are `Marítimo Terrestre`, so its brand rule is
> asked ahead of all three predial ones.
> 2. **`FEDERAL_ZONE.accountNumber` is an amount, not a reference.** It holds
> `DATMEX.zfed`, whose 77 values include `246.06`, `2369.09`, `22653.94` and
> a negative `-1679`; the concession claves the receipts are keyed by
> (`12-T -012`, `14-D -014`) appear nowhere in the database. Matching on that
> column could never hit — and because every row already has a value, the
> `[field]: null` guards on learning and on the blank-service fill would
> never fire either, so every page would return to review every bimester
> forever. The clave moves to `meterNumber`, joining gas and Tijuana predial,
> and the first confirm teaches the match. This is the same trap as
> `policies.total` and `PROPERTY_TAX.accountNumber`: a legacy column whose
> name promises an identifier and whose contents are something else.
> 3. **The payable figure is not the printed subtotal.** The municipality rounds
> to whole pesos and prints the difference as its own `Ajuste Ley Hacienda
> Mpal` line — `-$0.05` against a 591.05 subtotal, `$0.21` against 2,872.79.
> The "Total a pagar" box that carries the rounded figure sits on a grey fill
> and OCR'd on 1 of 8 pages; the SubTotal row read on 8 of 8. So the amount
> is the rounded subtotal, cross-checked against the printed box wherever it
> survives (it agreed).
> 4. **The office's own highlighter is an OCR failure mode.** Both pages that
> lost their clave lost it to a marker stroke drawn across the `Clave:` line
> — not to scan quality, which was otherwise fine. The clave is printed twice
> (receipt and stub), which rescued a third page whose heading was struck
> but whose stub was not; where both copies are struck, the page reports no
> clave and goes to review rather than guessing.
>
> **Not attempted:** deriving the payment deadline from the bimester. It is the
> 17th of the month after the bimester closes on a current bill, but four of
> these eight are late — they carry a $1,000 `Multa` — and print a
> recalculated deadline a month out. A derived date would be wrong on exactly
> the pages a human most wants to look at, so an unreadable deadline stays
> null.
### Motivation (from the meeting)
Each utility company (CFE, water, phone, gas...) sends 300+ individual
@@ -254,8 +427,8 @@ Per the meeting notes' own field list:
| Agua — Número de cuenta | `WATER` | `accountNumber` | `AGUA` | ✅ populated today |
| Zona Fed — Número de Zona Federal | `FEDERAL_ZONE` | `accountNumber` | `ZFED` | ✅ populated today |
| Tel — Número de teléfono | `TELEPHONE` *(new)* | `accountNumber` | `Property.phone1/2/3` (currently on `Property`, not `PropertyService`) | ⚠️ schema gap — see below |
| Impuesto — Clave Catastral | `PROPERTY_TAX` | `accountNumber` | migrated from `PREDIAL`, **not** `CLAVE` | ⚠️ needs verification — see below |
| Gas — Número de medidor | `GAS` | `meterNumber` | not populated — folded into free-text `notes` today | ⚠️ data gap — see below |
| Impuesto — Clave Catastral | `PROPERTY_TAX` | `Property.cadastralKey`, plus `meterNumber` for Tijuana's municipal account | `CLAVE`; `PREDIAL` is left on `accountNumber` and never matched against | ✅ built — see the 2026-08-01 extension note |
| Gas — Número de medidor | `GAS` | `meterNumber` | not populated — folded into free-text `notes` today | ✅ 160/334 recovered from `notes` |
Confidence rule of thumb once a field is confirmed populated, tune after
seeing real statements:
@@ -810,16 +983,23 @@ action (`customer:purge`) taken well after release — not bundled into
## Open questions to take back to Jorge (collected)
- OCR provider/budget for §2 (self-hosted vs. managed API, given 300+
pages/month/company).
- Whether source PDFs arrive pre-split per customer or as one bundled file
needing page-range detection (§2).
- Whether "Clave Catastral" and the already-migrated `PREDIAL`-sourced
`PROPERTY_TAX.accountNumber` are the same number — blocks OCR matching
for predial statements specifically until confirmed (§2).
- Whether phone billing is really one service per phone number on file, or
one per property regardless of how many numbers are recorded — decides
how the new `TELEPHONE` service kind gets backfilled (§2).
- ~~OCR provider/budget for §2~~ — **CLOSED**: self-hosted Tesseract, chosen on
measured accuracy against real scans (see §2's BUILT note). No per-page cost.
- ~~Whether source PDFs arrive pre-split per customer or bundled~~ —
**CLOSED**: bundled, one customer per page. Split per page.
- ~~Whether "Clave Catastral" and the `PREDIAL`-sourced
`PROPERTY_TAX.accountNumber` are the same number~~**CLOSED**: they are
different. `clave` is the cadastral key and is now on
`Property.cadastralKey`; `predial` is not unique and is not printed on
statements.
- ~~Whether phone billing is one service per number or one per property~~ —
**CLOSED**: effectively one (534 / 18 / 1 across phone1/2/3), backfilled
from `phone1`.
- **Still open (§2):** whether the CFE amount staff should owe is the rounded
headline (`$268`, what the barcode encodes and what is paid at the window) or
the exact `Total` in the breakdown (`$268.88`). The parser currently takes
the barcode figure, which matches what the office actually pays; worth one
confirmation from Jorge.
- The actual bank name/currency/details for the Seguros USD account, and
whether any historical Seguros bank data exists to migrate (§3).
- Whether `BankAccount.businessLine` should be enforced or a soft hint
@@ -0,0 +1,155 @@
"""
Closes the three data gaps the statement-OCR matcher depends on
(docs/RECEIPT_CAPTURE_SPEC.md §2, "Matching logic").
OCR matching is only as good as the field it matches against, and a
field-by-field check of real scanned statements against what
`transform_properties.py` actually loaded turned up three mismatches. This
script fixes them on an existing database; `transform_properties.py` has been
updated in the same commit so a full re-migration produces them directly.
1. CLAVE CATASTRAL — printed on both the CESPT water bill ("Cve. Cat.:
KB078025") and the predial statement, and held in `DATMEX.clave` (934
rows, format `[A-Z]{2}[0-9]{6}`). It was never migrated. What
`PROPERTY_TAX.accountNumber` carries instead is `DATMEX.predial`, a
different, purely numeric column that is *not* unique — 663 distinct
values across 1135 filled rows — and appears on no statement. So predial
is left exactly where it is, and the clave lands on `Property` (it is a
property-level key, which is why two different services both print it).
2. GAS — `GAS.meterNumber` is empty for all 334 rows because the transform
put `DATMEX.gas` into `notes`. That column is mixed: 160 rows hold a real
numeric account/meter number, the remaining 174 hold a tank descriptor
("ESTACIONARIO", "CILINDRO"). The numeric ones are recoverable now; the
descriptors legitimately have no number, so those statements still start
cold and get their number from the first human confirmation.
3. TELEPHONE — no such `ServiceKind` existed, so a Telnor bill had nothing to
match against. One service row is created per property with a `phone1`.
Only phone1: 534 properties have one, 18 have a phone2 and exactly 1 has a
phone3, so the secondaries are alternate contacts rather than separately
billed lines.
Idempotent — re-running updates nothing it has already done, and it never
overwrites a value a human has since corrected.
./.venv/bin/python backfill_statement_match_fields.py --env dev
"""
from __future__ import annotations
import uuid
from pathlib import Path
import pandas as pd
from dbenv import connect
from sync import parse_mode
STG = Path(__file__).parent / "output" / "stg_utilities"
NULL = ""
def s(v):
if v is None or pd.isna(v):
return None
v = str(v).strip()
return None if v in ("", NULL, "0") else v
def main():
# `--sync` is accepted and ignored — the script is idempotent either way.
env, _sync_mode = parse_mode()
conn = connect(env)
c = conn.cursor()
print(f"[statement-match-fields] target env: {env}")
# --- 1. clave catastral -> properties.cadastralKey ----------------------
# Joined on provenance, the same key transform_properties.py writes, so a
# property that was re-created by a later sync still lines up.
dm = pd.read_parquet(STG / "datmex.parquet")
claves = []
for _, row in dm.iterrows():
clave = s(row["clave"])
if clave:
claves.append((clave, str(int(row["_row_num"]))))
updated = 0
for clave, legacy_id in claves:
c.execute(
"UPDATE properties SET cadastralKey = %s "
"WHERE legacySourceTable = 'DATMEX' AND legacyId = %s AND cadastralKey IS NULL",
(clave, legacy_id),
)
updated += c.rowcount
print(f" cadastralKey: set on {updated} propert(ies) ({len(claves)} in source)")
# --- 2. gas account numbers out of notes -> GAS.meterNumber -------------
# REGEXP rather than a Python loop: the value is already sitting in the
# notes column, so this is one pass over 334 rows inside the database.
c.execute(
"UPDATE property_services SET meterNumber = notes "
"WHERE kind = 'GAS' AND meterNumber IS NULL "
"AND notes REGEXP '^[0-9]{5,}$'"
)
print(f" GAS.meterNumber: recovered {c.rowcount} account number(s) from notes")
# --- 3. TELEPHONE service rows ------------------------------------------
# Digits only, matching how the transform now writes them: a scanned Telnor
# bill prints "664 609 3444" and reduces to the stored local 6093444 once
# the LADA is stripped, which is the matcher's job, not this script's.
c.execute(
"SELECT p.id, p.phone1 FROM properties p "
"WHERE p.phone1 IS NOT NULL AND p.phone1 <> '' "
"AND NOT EXISTS (SELECT 1 FROM property_services ps "
" WHERE ps.propertyId = p.id AND ps.kind = 'TELEPHONE')"
)
rows = c.fetchall()
made = []
for pid, phone in rows:
digits = "".join(ch for ch in str(phone) if ch.isdigit())
if digits:
made.append((str(uuid.uuid4()), pid, digits))
if made:
c.executemany(
"INSERT INTO property_services "
"(id, propertyId, kind, accountNumber, active, notes) "
"VALUES (%s, %s, 'TELEPHONE', %s, 1, 'from DATMEX.telefono')",
made,
)
print(f" TELEPHONE: created {len(made)} service row(s)")
conn.commit()
# --- validation ---------------------------------------------------------
c.execute("SELECT COUNT(*) FROM properties WHERE cadastralKey IS NOT NULL")
n_clave = c.fetchone()[0]
c.execute(
"SELECT COUNT(*) FROM property_services WHERE kind='GAS' AND meterNumber IS NOT NULL"
)
n_gas = c.fetchone()[0]
c.execute("SELECT COUNT(*) FROM property_services WHERE kind='TELEPHONE'")
n_tel = c.fetchone()[0]
# A clave that is not unique would silently make the secondary match key
# ambiguous, which is worse than not having one — surface it rather than
# letting the matcher discover it a statement at a time.
c.execute(
"SELECT COUNT(*) FROM (SELECT cadastralKey FROM properties "
"WHERE cadastralKey IS NOT NULL GROUP BY cadastralKey HAVING COUNT(*) > 1) d"
)
dupe_claves = c.fetchone()[0]
print("=== Statement match fields ready ===")
print(f" properties with cadastralKey : {n_clave}")
print(f" GAS services with meterNumber: {n_gas}")
print(f" TELEPHONE services : {n_tel}")
print(f" duplicated cadastralKey values: {dupe_claves}"
+ (" (matcher treats these as ambiguous)" if dupe_claves else ""))
assert n_clave > 0 and n_tel > 0, "backfill produced nothing — check staging output"
print(" validation: OK")
conn.close()
if __name__ == "__main__":
main()
+9 -5
View File
@@ -33,7 +33,7 @@ from pathlib import Path
import boto3
from botocore.config import Config
from dbenv import connect, load_env
from dbenv import connect, require, setting
from extract import sanitize_column_name as san
csv.field_size_limit(300_000_000)
@@ -102,12 +102,16 @@ def main():
only = set(x.strip() for x in args.tables.split(",") if x.strip())
sources = [s for s in SOURCES if not only or s["key"] in only]
env = load_env(args.env)
# Process environment first, deploy/.env.<env> second, with the same
# credential aliases the API uses — the "Operaciones" re-import runs this
# inside the API container, which has S3_ENDPOINT / MINIO_ROOT_* injected
# and no deploy/ directory at all.
s3 = boto3.client(
"s3", endpoint_url=env["S3_ENDPOINT"],
aws_access_key_id=env["MINIO_ROOT_USER"], aws_secret_access_key=env["MINIO_ROOT_PASSWORD"],
"s3", endpoint_url=require(args.env, "S3_ENDPOINT"),
aws_access_key_id=require(args.env, "S3_ACCESS_KEY", "MINIO_ROOT_USER"),
aws_secret_access_key=require(args.env, "S3_SECRET_KEY", "MINIO_ROOT_PASSWORD"),
config=Config(signature_version="s3v4"), region_name="us-east-1")
bucket = env["S3_BUCKET"]
bucket = setting(args.env, "S3_BUCKET") or "jorgecuadros-documents"
conn = connect(args.env)
cur = conn.cursor()
+31 -7
View File
@@ -32,29 +32,53 @@ REPO = Path(__file__).resolve().parents[1]
def load_env(env: str) -> dict:
"""deploy/.env.<env> parsed to a dict, or {} when the file is absent.
Absent is normal, not an error: the API container runs these scripts with
DATABASE_URL / S3_* injected as real environment variables and ships no
deploy/ directory. Use `setting()` / `require()` rather than this — they
layer the process environment on top, which is what actually resolves."""
f = REPO / "deploy" / f".env.{env}"
if not f.exists():
raise SystemExit(
f"missing {f} — deploy the '{env}' DB stack and write its .env first "
f"(see dbenv.py header)."
)
return {}
out = {}
for line in f.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
out[k] = v
if "DATABASE_URL" not in out:
raise SystemExit(f"{f} has no DATABASE_URL")
return out
def setting(env: str, *keys: str):
"""First non-empty value for `keys`, process environment first, then
deploy/.env.<env>. Several keys = fallback aliases (S3_ACCESS_KEY then
MINIO_ROOT_USER, as apps/api/src/storage/storage.service.ts does)."""
fromfile = load_env(env)
for k in keys:
v = os.environ.get(k) or fromfile.get(k)
if v:
return v
return None
def require(env: str, *keys: str) -> str:
v = setting(env, *keys)
if not v:
raise SystemExit(
f"missing {' / '.join(keys)} — set it in the environment, or deploy the "
f"'{env}' stack and write {REPO / 'deploy' / f'.env.{env}'} "
f"(see dbenv.py header)."
)
return v
def database_url(env: str) -> str:
"""Target DB URL. A DATABASE_URL in the process environment wins over
deploy/.env.<env> — this is how the API container (which has its own
DATABASE_URL and no deploy/.env files) drives a re-import against its own
database."""
return os.environ.get("DATABASE_URL") or load_env(env)["DATABASE_URL"]
return require(env, "DATABASE_URL")
def connect(env: str):
+12
View File
@@ -41,6 +41,12 @@ PY = sys.executable # the venv python running this orchestrator
STEPS = [
"transform_customers.py",
"transform_properties.py",
# Statement-OCR match fields. transform_properties.py now produces these
# directly, so on a full rebuild this is a no-op that re-asserts they are
# there; on a database predating the OCR module it is what fills them in.
# Must follow transform_properties.py, which truncates both tables it
# touches.
"backfill_statement_match_fields.py",
"transform_policies.py",
"transform_transactions.py",
"prune_empty_customers.py",
@@ -54,6 +60,12 @@ STEPS = [
SYNC_STEPS = [
"transform_customers.py",
"transform_properties.py",
# Statement-OCR match fields. transform_properties.py now produces these
# directly, so on a full rebuild this is a no-op that re-asserts they are
# there; on a database predating the OCR module it is what fills them in.
# Must follow transform_properties.py, which truncates both tables it
# touches.
"backfill_statement_match_fields.py",
"transform_policies.py",
"transform_transactions.py",
# Manual-safe prune: drops legacy-owned empties that the customer upsert
+25 -5
View File
@@ -147,7 +147,7 @@ def main():
props.append((
pid, cust_id, s(row["direccion"]), ", ".join(addr2_parts) or None,
s(row["telefono"]), s(row["telefono2"]), s(row["telefono3"]),
s(row["zona"]), "DATMEX", legacy_id,
s(row["zona"]), s(row["clave"]), "DATMEX", legacy_id,
))
@@ -178,9 +178,16 @@ def main():
svc("ELECTRIC", account=s(row[rc]),
notes=s(row["luz_tipo"]) if rc == "rpu" else None,
active=flag("electric"))
# GAS
# GAS — the column mixes two things: an account/meter number for 160 of
# the 334 filled rows, and a tank descriptor ("ESTACIONARIO",
# "CILINDRO") for the rest. Only the numeric form can be matched
# against a scanned gas statement, so it is promoted to meterNumber;
# the descriptor stays a note, as before.
if s(row["gas"]) or flag("gas1", False):
svc("GAS", due=s(row["gas_vence"]), notes=s(row["gas"]), active=flag("gas1"))
gas_val = s(row["gas"])
gas_meter = gas_val if gas_val and gas_val.isdigit() and len(gas_val) >= 5 else None
svc("GAS", meter=gas_meter, due=s(row["gas_vence"]),
notes=s(row["gas"]), active=flag("gas1"))
# CABLE
if s(row["cable_num"]) or (s_keep0(row["cable_sky"]) or "0") in _TRUE:
svc("CABLE", account=s(row["cable_num"]), route=s(row["cia_cable"]),
@@ -194,6 +201,19 @@ def main():
if s(row["zfed"]) or flag("federalzone", False):
svc("FEDERAL_ZONE", account=s(row["zfed"]), notes=s(row["zfed_t"]),
active=flag("federalzone"))
# TELEPHONE — DATMEX never had a phone *service*, only the contact
# numbers unpivoted into Property.phone1/2/3 above, even though the
# legacy ledger billed phone as its own transaction type. A Telnor bill
# can only be matched against a service row, so the primary number
# becomes one. Only phone1: of 1518 properties, 534 have phone1, 18
# phone2 and exactly 1 phone3 — the secondaries are alternate contacts,
# not additional billed lines. Stored as the bare local number, which is
# how DATMEX holds it and what a printed bill reduces to once the 664
# Tijuana LADA is stripped.
tel = s(row["telefono"])
if tel:
svc("TELEPHONE", account="".join(ch for ch in tel if ch.isdigit()) or None,
notes="from DATMEX.telefono")
# ALARM
if s(row["alarm_system"]):
svc("ALARM", notes=s(row["alarm_system"]))
@@ -218,7 +238,7 @@ def main():
cur.execute("DELETE ps FROM property_services ps JOIN properties p ON p.id=ps.propertyId WHERE p.legacyId IS NOT NULL")
cur.execute("DELETE ta FROM trust_accounts ta JOIN properties p ON p.id=ta.propertyId WHERE p.legacyId IS NOT NULL")
cur.executemany(
"INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),archivedAt=NULL", props)
"INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) ON DUPLICATE KEY UPDATE customerId=VALUES(customerId),addressLine1=VALUES(addressLine1),addressLine2=VALUES(addressLine2),phone1=VALUES(phone1),phone2=VALUES(phone2),phone3=VALUES(phone3),zone=VALUES(zone),cadastralKey=VALUES(cadastralKey),archivedAt=NULL", props)
delete_missing(cur, "properties", ("legacySourceTable", "legacyId"), prop_keys, "WHERE legacyId IS NOT NULL")
cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services)
cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts)
@@ -227,7 +247,7 @@ def main():
for t in ("property_services", "service_documents", "trust_accounts", "properties"):
cur.execute(f"TRUNCATE TABLE {t}")
cur.execute("SET FOREIGN_KEY_CHECKS=1")
cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props)
cur.executemany("INSERT INTO properties (id,customerId,addressLine1,addressLine2,phone1,phone2,phone3,zone,cadastralKey,legacySourceTable,legacyId) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", props)
cur.executemany("INSERT INTO property_services (id,propertyId,kind,accountNumber,meterNumber,route,dueDay,active,notes) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)", services)
cur.executemany("INSERT INTO trust_accounts (id,propertyId,bankName,trustNumber,bankFee,dueDate1,dueDate2) VALUES (%s,%s,%s,%s,%s,%s,%s)", trusts)
conn.commit()
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "jorgecuadros-platform",
"version": "1.0.0",
"version": "1.0.6",
"private": true,
"workspaces": [
"apps/*",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@jorgecuadros/database",
"version": "1.0.0",
"version": "1.0.6",
"private": true,
"main": "generated/client/index.js",
"types": "generated/client/index.d.ts",
@@ -0,0 +1,79 @@
-- AlterTable
ALTER TABLE `properties` ADD COLUMN `cadastralKey` VARCHAR(191) NULL;
-- AlterTable
ALTER TABLE `property_services` MODIFY `kind` ENUM('WATER', 'ELECTRIC', 'GAS', 'CABLE', 'PROPERTY_TAX', 'FEDERAL_ZONE', 'ALARM', 'TELEPHONE', 'OTHER') NOT NULL;
-- CreateTable
CREATE TABLE `statement_batches` (
`id` VARCHAR(191) NOT NULL,
`serviceKind` ENUM('WATER', 'ELECTRIC', 'GAS', 'CABLE', 'PROPERTY_TAX', 'FEDERAL_ZONE', 'ALARM', 'TELEPHONE', 'OTHER') NOT NULL,
`status` ENUM('UPLOADED', 'PROCESSING', 'READY_FOR_REVIEW', 'COMPLETED', 'FAILED') NOT NULL DEFAULT 'UPLOADED',
`uploadedById` VARCHAR(191) NOT NULL,
`label` VARCHAR(191) NULL,
`fileCount` INTEGER NOT NULL DEFAULT 0,
`error` TEXT NULL,
`createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`completedAt` DATETIME(3) NULL,
INDEX `statement_batches_status_createdAt_idx`(`status`, `createdAt`),
PRIMARY KEY (`id`)
) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- CreateTable
CREATE TABLE `statement_documents` (
`id` VARCHAR(191) NOT NULL,
`batchId` VARCHAR(191) NOT NULL,
`pageNumber` INTEGER NOT NULL,
`storageKey` VARCHAR(191) NOT NULL,
`status` ENUM('PENDING_OCR', 'OCR_FAILED', 'NEEDS_REVIEW', 'MATCHED', 'CONFIRMED', 'POSTED', 'REJECTED') NOT NULL DEFAULT 'PENDING_OCR',
`ocrRawText` TEXT NULL,
`ocrConfidence` DECIMAL(4, 3) NULL,
`provider` VARCHAR(191) NULL,
`extractedAccountRef` VARCHAR(191) NULL,
`extractedAmount` DECIMAL(12, 2) NULL,
`extractedPeriod` VARCHAR(191) NULL,
`extractedDueDate` DATETIME(3) NULL,
`extractedCadastralKey` VARCHAR(191) NULL,
`matchedPropertyServiceId` VARCHAR(191) NULL,
`matchedCustomerId` VARCHAR(191) NULL,
`matchNote` VARCHAR(191) NULL,
`reviewedById` VARCHAR(191) NULL,
`reviewedAt` DATETIME(3) NULL,
`postedTransactionId` VARCHAR(191) NULL,
`createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE INDEX `statement_documents_postedTransactionId_key`(`postedTransactionId`),
INDEX `statement_documents_status_idx`(`status`),
INDEX `statement_documents_matchedCustomerId_idx`(`matchedCustomerId`),
UNIQUE INDEX `statement_documents_batchId_pageNumber_key`(`batchId`, `pageNumber`),
PRIMARY KEY (`id`)
) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- CreateIndex
CREATE INDEX `properties_cadastralKey_idx` ON `properties`(`cadastralKey`);
-- CreateIndex
CREATE INDEX `property_services_kind_accountNumber_idx` ON `property_services`(`kind`, `accountNumber`);
-- CreateIndex
CREATE INDEX `property_services_kind_meterNumber_idx` ON `property_services`(`kind`, `meterNumber`);
-- AddForeignKey
ALTER TABLE `statement_batches` ADD CONSTRAINT `statement_batches_uploadedById_fkey` FOREIGN KEY (`uploadedById`) REFERENCES `users`(`id`) ON DELETE RESTRICT ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_batchId_fkey` FOREIGN KEY (`batchId`) REFERENCES `statement_batches`(`id`) ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_matchedPropertyServiceId_fkey` FOREIGN KEY (`matchedPropertyServiceId`) REFERENCES `property_services`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_matchedCustomerId_fkey` FOREIGN KEY (`matchedCustomerId`) REFERENCES `customers`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_reviewedById_fkey` FOREIGN KEY (`reviewedById`) REFERENCES `users`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `statement_documents` ADD CONSTRAINT `statement_documents_postedTransactionId_fkey` FOREIGN KEY (`postedTransactionId`) REFERENCES `transactions`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
@@ -0,0 +1,80 @@
-- CreateTable
CREATE TABLE `policy_ocr_batches` (
`id` VARCHAR(191) NOT NULL,
`provider` VARCHAR(191) NOT NULL DEFAULT 'GMX',
`status` ENUM('UPLOADED', 'PROCESSING', 'READY_FOR_REVIEW', 'COMPLETED', 'FAILED') NOT NULL DEFAULT 'UPLOADED',
`uploadedById` VARCHAR(191) NOT NULL,
`label` VARCHAR(191) NULL,
`fileCount` INTEGER NOT NULL DEFAULT 0,
`error` TEXT NULL,
`createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`completedAt` DATETIME(3) NULL,
INDEX `policy_ocr_batches_status_createdAt_idx`(`status`, `createdAt`),
PRIMARY KEY (`id`)
) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- CreateTable
CREATE TABLE `policy_ocr_documents` (
`id` VARCHAR(191) NOT NULL,
`batchId` VARCHAR(191) NOT NULL,
`pageNumber` INTEGER NOT NULL,
`storageKey` VARCHAR(191) NOT NULL,
`status` ENUM('PENDING_OCR', 'OCR_FAILED', 'NEEDS_REVIEW', 'MATCHED', 'CONFIRMED', 'POSTED', 'REJECTED') NOT NULL DEFAULT 'PENDING_OCR',
`ocrRawText` TEXT NULL,
`ocrConfidence` DECIMAL(4, 3) NULL,
`provider` VARCHAR(191) NULL,
`extractedPolicyNumber` VARCHAR(191) NULL,
`extractedInsuredName` VARCHAR(191) NULL,
`extractedAdditionalInsured` VARCHAR(191) NULL,
`extractedAgentName` VARCHAR(191) NULL,
`extractedLegalAddress` TEXT NULL,
`extractedZip` VARCHAR(191) NULL,
`extractedPolicyFrom` DATETIME(3) NULL,
`extractedPolicyTo` DATETIME(3) NULL,
`extractedPolicyDate` DATETIME(3) NULL,
`extractedCurrency` VARCHAR(191) NULL,
`extractedNetPremium` DECIMAL(12, 2) NULL,
`extractedPolicyFee` DECIMAL(12, 2) NULL,
`extractedBrokerFee` DECIMAL(12, 2) NULL,
`extractedTotal` DECIMAL(12, 2) NULL,
`extractedCoveragesJson` JSON NULL,
`extractedPremiumPayment` VARCHAR(191) NULL,
`matchedPolicyId` VARCHAR(191) NULL,
`matchedCustomerId` VARCHAR(191) NULL,
`matchCandidates` JSON NULL,
`matchNote` VARCHAR(191) NULL,
`reviewedById` VARCHAR(191) NULL,
`reviewedAt` DATETIME(3) NULL,
`createdPolicyId` VARCHAR(191) NULL,
`postedTransactionId` VARCHAR(191) NULL,
`createdAt` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE INDEX `policy_ocr_documents_createdPolicyId_key`(`createdPolicyId`),
UNIQUE INDEX `policy_ocr_documents_postedTransactionId_key`(`postedTransactionId`),
INDEX `policy_ocr_documents_status_idx`(`status`),
INDEX `policy_ocr_documents_matchedCustomerId_idx`(`matchedCustomerId`),
UNIQUE INDEX `policy_ocr_documents_batchId_pageNumber_key`(`batchId`, `pageNumber`),
PRIMARY KEY (`id`)
) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- AddForeignKey
ALTER TABLE `policy_ocr_batches` ADD CONSTRAINT `policy_ocr_batches_uploadedById_fkey` FOREIGN KEY (`uploadedById`) REFERENCES `users`(`id`) ON DELETE RESTRICT ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_batchId_fkey` FOREIGN KEY (`batchId`) REFERENCES `policy_ocr_batches`(`id`) ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_matchedPolicyId_fkey` FOREIGN KEY (`matchedPolicyId`) REFERENCES `policies`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_matchedCustomerId_fkey` FOREIGN KEY (`matchedCustomerId`) REFERENCES `customers`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_reviewedById_fkey` FOREIGN KEY (`reviewedById`) REFERENCES `users`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_createdPolicyId_fkey` FOREIGN KEY (`createdPolicyId`) REFERENCES `policies`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE `policy_ocr_documents` ADD CONSTRAINT `policy_ocr_documents_postedTransactionId_fkey` FOREIGN KEY (`postedTransactionId`) REFERENCES `transactions`(`id`) ON DELETE SET NULL ON UPDATE CASCADE;
+262 -1
View File
@@ -53,6 +53,12 @@ enum ServiceKind {
PROPERTY_TAX
FEDERAL_ZONE
ALARM
/// Telephone was never unpivoted out of DATMEX — the numbers sat on
/// `Property.phone1/2/3` as contact fields even though the legacy ledger
/// billed phone as its own transaction type. OCR matching needs a real
/// service row to match a Telnor bill against, so it becomes one; see
/// `migration/backfill_statement_match_fields.py`.
TELEPHONE
OTHER
}
@@ -115,6 +121,9 @@ model Customer {
vehicles Vehicle[]
transactions Transaction[]
statementDocuments StatementDocument[]
policyOcrDocuments PolicyOcrDocument[] @relation("PolicyOcrDocumentCustomer")
@@map("customers")
}
@@ -202,6 +211,9 @@ model Policy {
properties Property[]
renewalNotices RenewalNotice[]
ocrMatchedDocuments PolicyOcrDocument[] @relation("PolicyOcrDocumentPolicy")
ocrCreatedDocuments PolicyOcrDocument[] @relation("PolicyOcrDocumentCreatedPolicy")
@@unique([legacySourceDb, legacySourceTable, legacyId])
@@index([policyNumber])
@@map("policies")
@@ -355,6 +367,115 @@ model PolicyDocument {
@@map("policy_documents")
}
/// Insurance OCR intake (mirrors statement_batches / statement_documents for
/// the utility side). One upload session of policy PDFs from a provider
/// portal (GMX, etc.) — the parser proposes policyNumber → existing Policy
/// (or "new, pick customer"), staff confirms, and the system attaches the
/// source PDF and optionally writes a premium Transaction.
model PolicyOcrBatch {
id String @id @default(uuid())
/// Which insurance provider portal the batch came from. "GMX" today;
/// future providers (AXA, GNP, …) extend the parser, not this table.
provider String @default("GMX")
status PolicyOcrBatchStatus @default(UPLOADED)
uploadedById String
uploadedBy User @relation("PolicyOcrBatchUploader", fields: [uploadedById], references: [id])
label String?
fileCount Int @default(0)
/// Set when the pipeline fails as a whole (bad PDF, OCR binaries missing).
error String? @db.Text
createdAt DateTime @default(now())
completedAt DateTime?
documents PolicyOcrDocument[]
@@index([status, createdAt])
@@map("policy_ocr_batches")
}
enum PolicyOcrBatchStatus {
UPLOADED
PROCESSING
READY_FOR_REVIEW
COMPLETED
FAILED
}
/// One parsed policy page — one Policy → one Customer (after staff confirms).
model PolicyOcrDocument {
id String @id @default(uuid())
batchId String
batch PolicyOcrBatch @relation(fields: [batchId], references: [id], onDelete: Cascade)
pageNumber Int
/// The rendered page image in object storage. Source PDF kept too on the
/// batch (statement pattern) so re-running a corrected parser is possible.
storageKey String
status PolicyOcrDocumentStatus @default(PENDING_OCR)
ocrRawText String? @db.Text
ocrConfidence Decimal? @db.Decimal(4, 3)
/// Which parser claimed the page ("GMX" today).
provider String?
// Extracted header fields, all staff-editable in review.
extractedPolicyNumber String?
extractedInsuredName String?
extractedAdditionalInsured String?
extractedAgentName String?
extractedLegalAddress String? @db.Text
extractedZip String?
extractedPolicyFrom DateTime?
extractedPolicyTo DateTime?
extractedPolicyDate DateTime?
extractedCurrency String?
extractedNetPremium Decimal? @db.Decimal(12, 2)
extractedPolicyFee Decimal? @db.Decimal(12, 2)
extractedBrokerFee Decimal? @db.Decimal(12, 2)
extractedTotal Decimal? @db.Decimal(12, 2)
/// Per-coverage rows from the GMX "Material damages" / "Additional risk"
/// tables — preserved verbatim so a missing premium receipt still leaves
/// the coverages auditable.
extractedCoveragesJson Json?
extractedPremiumPayment String?
// Match by `Policy.policyNumber` → existing Policy / Customer.
matchedPolicyId String?
matchedPolicy Policy? @relation("PolicyOcrDocumentPolicy", fields: [matchedPolicyId], references: [id])
matchedCustomerId String?
matchedCustomer Customer? @relation("PolicyOcrDocumentCustomer", fields: [matchedCustomerId], references: [id])
/// All policies carrying the same number, with their customer. One is
/// normal; >1 means the policy number is shared across customers and a
/// human must pick.
matchCandidates Json?
matchNote String?
reviewedById String?
reviewedBy User? @relation("PolicyOcrDocumentReviewer", fields: [reviewedById], references: [id])
reviewedAt DateTime?
createdPolicyId String? @unique
createdPolicy Policy? @relation("PolicyOcrDocumentCreatedPolicy", fields: [createdPolicyId], references: [id])
postedTransactionId String? @unique
postedTransaction Transaction? @relation("PolicyOcrDocumentTransaction", fields: [postedTransactionId], references: [id])
createdAt DateTime @default(now())
@@unique([batchId, pageNumber])
@@index([status])
@@index([matchedCustomerId])
@@map("policy_ocr_documents")
}
enum PolicyOcrDocumentStatus {
PENDING_OCR
OCR_FAILED
NEEDS_REVIEW
MATCHED
CONFIRMED
POSTED
REJECTED
}
// ---------------------------------------------------------------------------
// Utilities domain
// ---------------------------------------------------------------------------
@@ -373,6 +494,14 @@ model Property {
phone2 String?
phone3 String?
zone String?
/// Clave catastral (DATMEX.clave) — the cadastral key, format `KA903009`.
/// Property-level, not per-service: it is printed on both the CESPT water
/// bill and the predial statement, which is exactly why it is a useful
/// secondary match key when a bill's account number does not OCR cleanly.
/// Distinct from the numeric DATMEX.predial that `PROPERTY_TAX.accountNumber`
/// carries — that column is not unique (663 distinct across 1135 rows) and
/// is not what any statement prints.
cadastralKey String?
// Soft-delete marker (see Customer.archivedAt).
archivedAt DateTime?
legacySourceTable String?
@@ -384,6 +513,7 @@ model Property {
trustAccount TrustAccount?
@@unique([legacySourceTable, legacyId])
@@index([cadastralKey])
@@map("properties")
}
@@ -401,6 +531,13 @@ model PropertyService {
active Boolean @default(true)
notes String? @db.Text
statementDocuments StatementDocument[]
// The OCR matcher looks a service up by (kind, accountNumber) — always
// scoped to one kind, never fuzzily across every identifier column, so a
// water account number cannot collide with an unrelated phone number.
@@index([kind, accountNumber])
@@index([kind, meterNumber])
@@map("property_services")
}
@@ -415,6 +552,120 @@ model ServiceDocument {
@@map("service_documents")
}
// ---------------------------------------------------------------------------
// Statement OCR intake (RECEIPT_CAPTURE_SPEC §2)
//
// Each utility company mails 300+ paper statements a month, one per customer,
// which staff key in by hand. These two tables are the intake side of removing
// that: a batch of scanned PDFs is split per page, OCR'd, matched to a
// PropertyService by its scoped account number, and queued for review. Nothing
// here writes to the ledger — confirming a document posts it through
// `BillingService.createBatch`, the same path hand-keyed batches take.
//
// Everything ingested is a CHARGE (a bill awaiting payment), never a proof of
// payment: the office scans what it must pay, and settles it by check through
// the existing capture flow.
// ---------------------------------------------------------------------------
enum StatementBatchStatus {
UPLOADED
PROCESSING
READY_FOR_REVIEW
COMPLETED
FAILED
}
enum StatementDocumentStatus {
PENDING_OCR
OCR_FAILED
/// No confident match, or the extraction itself was low-confidence.
NEEDS_REVIEW
/// Confident auto-match, awaiting a human confirm.
MATCHED
/// Staff confirmed; not yet posted.
CONFIRMED
POSTED
/// Duplicate, unreadable, or wrong batch.
REJECTED
}
/// One upload session — e.g. "October CFE statements".
model StatementBatch {
id String @id @default(uuid())
/// What kind of service every statement in this batch bills. The parser
/// still detects the provider per page and flags any page that disagrees,
/// rather than trusting the uploader's label.
serviceKind ServiceKind
status StatementBatchStatus @default(UPLOADED)
uploadedById String
uploadedBy User @relation("StatementBatchUploader", fields: [uploadedById], references: [id])
label String?
fileCount Int @default(0)
/// Set when the pipeline fails as a whole (bad PDF, OCR binaries missing).
error String? @db.Text
createdAt DateTime @default(now())
completedAt DateTime?
documents StatementDocument[]
@@index([status, createdAt])
@@map("statement_batches")
}
/// One statement — one customer, one period — after splitting the batch.
model StatementDocument {
id String @id @default(uuid())
batchId String
batch StatementBatch @relation(fields: [batchId], references: [id], onDelete: Cascade)
/// 1-based page of the source PDF this was split from.
pageNumber Int
/// The rendered page image in object storage. The source PDF is kept too, so
/// a reviewer can always see exactly what the parser read.
storageKey String
status StatementDocumentStatus @default(PENDING_OCR)
/// Raw OCR text, kept even after a manual correction so a mismatch between
/// what the machine read and what staff entered stays auditable.
ocrRawText String? @db.Text
/// Mean per-word confidence reported by the OCR engine, 0..1.
ocrConfidence Decimal? @db.Decimal(4, 3)
/// Which parser claimed the page ("CFE", "CESPT", "TELNOR").
provider String?
// Extracted, then staff-corrected in place. `extractedAccountRef` is already
// normalised for matching (CFE leading zeros stripped, Telnor LADA removed).
extractedAccountRef String?
extractedAmount Decimal? @db.Decimal(12, 2)
extractedPeriod String?
extractedDueDate DateTime?
/// Clave catastral when the statement prints one — a second key to match on
/// when the account number is unreadable.
extractedCadastralKey String?
matchedPropertyServiceId String?
matchedPropertyService PropertyService? @relation(fields: [matchedPropertyServiceId], references: [id])
matchedCustomerId String?
matchedCustomer Customer? @relation(fields: [matchedCustomerId], references: [id])
/// Why this landed where it did — "exact account match", "no candidate",
/// "2 candidates". Shown in the review queue so staff can trust or distrust
/// the suggestion without opening the image.
matchNote String?
reviewedById String?
reviewedBy User? @relation("StatementDocumentReviewer", fields: [reviewedById], references: [id])
reviewedAt DateTime?
postedTransactionId String? @unique
postedTransaction Transaction? @relation(fields: [postedTransactionId], references: [id])
createdAt DateTime @default(now())
@@unique([batchId, pageNumber])
@@index([status])
@@index([matchedCustomerId])
@@map("statement_documents")
}
/// From TRUSTVENCE.
model TrustAccount {
id String @id @default(uuid())
@@ -483,13 +734,17 @@ model Transaction {
legacyId String?
createdAt DateTime @default(now())
/// Set only on OCR-posted rows — the statement page this came from.
statementDocument StatementDocument?
policyOcrDocument PolicyOcrDocument? @relation("PolicyOcrDocumentTransaction")
@@unique([legacySourceDb, legacySourceTable, legacyId])
@@index([customerId, transactionDate])
// By-check reconciliation (billing.byCheck / the cheque-count report) looks
// rows up by check number alone — the legacy EDITA CHEQUE COUNT lookup.
@@index([checkNumber])
// Drives the duplicate-post guard in BillingService.createBatch.
@@index([captureRef])
@@unique([legacySourceDb, legacySourceTable, legacyId])
@@map("transactions")
}
@@ -600,6 +855,12 @@ model User {
updatedAt DateTime @updatedAt
activityLogs ActivityLog[]
statementBatches StatementBatch[] @relation("StatementBatchUploader")
statementsReviewed StatementDocument[] @relation("StatementDocumentReviewer")
policyOcrBatches PolicyOcrBatch[] @relation("PolicyOcrBatchUploader")
policyOcrReviewed PolicyOcrDocument[] @relation("PolicyOcrDocumentReviewer")
@@map("users")
}