feat(statements): OCR intake for scanned utility bills
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -53,6 +53,12 @@ enum ServiceKind {
|
||||
PROPERTY_TAX
|
||||
FEDERAL_ZONE
|
||||
ALARM
|
||||
/// Telephone was never unpivoted out of DATMEX — the numbers sat on
|
||||
/// `Property.phone1/2/3` as contact fields even though the legacy ledger
|
||||
/// billed phone as its own transaction type. OCR matching needs a real
|
||||
/// service row to match a Telnor bill against, so it becomes one; see
|
||||
/// `migration/backfill_statement_match_fields.py`.
|
||||
TELEPHONE
|
||||
OTHER
|
||||
}
|
||||
|
||||
@@ -115,6 +121,8 @@ model Customer {
|
||||
vehicles Vehicle[]
|
||||
transactions Transaction[]
|
||||
|
||||
statementDocuments StatementDocument[]
|
||||
|
||||
@@map("customers")
|
||||
}
|
||||
|
||||
@@ -373,6 +381,14 @@ model Property {
|
||||
phone2 String?
|
||||
phone3 String?
|
||||
zone String?
|
||||
/// Clave catastral (DATMEX.clave) — the cadastral key, format `KA903009`.
|
||||
/// Property-level, not per-service: it is printed on both the CESPT water
|
||||
/// bill and the predial statement, which is exactly why it is a useful
|
||||
/// secondary match key when a bill's account number does not OCR cleanly.
|
||||
/// Distinct from the numeric DATMEX.predial that `PROPERTY_TAX.accountNumber`
|
||||
/// carries — that column is not unique (663 distinct across 1135 rows) and
|
||||
/// is not what any statement prints.
|
||||
cadastralKey String?
|
||||
// Soft-delete marker (see Customer.archivedAt).
|
||||
archivedAt DateTime?
|
||||
legacySourceTable String?
|
||||
@@ -383,6 +399,8 @@ model Property {
|
||||
documents ServiceDocument[]
|
||||
trustAccount TrustAccount?
|
||||
|
||||
@@index([cadastralKey])
|
||||
|
||||
@@unique([legacySourceTable, legacyId])
|
||||
@@map("properties")
|
||||
}
|
||||
@@ -401,6 +419,13 @@ model PropertyService {
|
||||
active Boolean @default(true)
|
||||
notes String? @db.Text
|
||||
|
||||
statementDocuments StatementDocument[]
|
||||
|
||||
// The OCR matcher looks a service up by (kind, accountNumber) — always
|
||||
// scoped to one kind, never fuzzily across every identifier column, so a
|
||||
// water account number cannot collide with an unrelated phone number.
|
||||
@@index([kind, accountNumber])
|
||||
@@index([kind, meterNumber])
|
||||
@@map("property_services")
|
||||
}
|
||||
|
||||
@@ -415,6 +440,120 @@ model ServiceDocument {
|
||||
@@map("service_documents")
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Statement OCR intake (RECEIPT_CAPTURE_SPEC §2)
|
||||
//
|
||||
// Each utility company mails 300+ paper statements a month, one per customer,
|
||||
// which staff key in by hand. These two tables are the intake side of removing
|
||||
// that: a batch of scanned PDFs is split per page, OCR'd, matched to a
|
||||
// PropertyService by its scoped account number, and queued for review. Nothing
|
||||
// here writes to the ledger — confirming a document posts it through
|
||||
// `BillingService.createBatch`, the same path hand-keyed batches take.
|
||||
//
|
||||
// Everything ingested is a CHARGE (a bill awaiting payment), never a proof of
|
||||
// payment: the office scans what it must pay, and settles it by check through
|
||||
// the existing capture flow.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
enum StatementBatchStatus {
|
||||
UPLOADED
|
||||
PROCESSING
|
||||
READY_FOR_REVIEW
|
||||
COMPLETED
|
||||
FAILED
|
||||
}
|
||||
|
||||
enum StatementDocumentStatus {
|
||||
PENDING_OCR
|
||||
OCR_FAILED
|
||||
/// No confident match, or the extraction itself was low-confidence.
|
||||
NEEDS_REVIEW
|
||||
/// Confident auto-match, awaiting a human confirm.
|
||||
MATCHED
|
||||
/// Staff confirmed; not yet posted.
|
||||
CONFIRMED
|
||||
POSTED
|
||||
/// Duplicate, unreadable, or wrong batch.
|
||||
REJECTED
|
||||
}
|
||||
|
||||
/// One upload session — e.g. "October CFE statements".
|
||||
model StatementBatch {
|
||||
id String @id @default(uuid())
|
||||
/// What kind of service every statement in this batch bills. The parser
|
||||
/// still detects the provider per page and flags any page that disagrees,
|
||||
/// rather than trusting the uploader's label.
|
||||
serviceKind ServiceKind
|
||||
status StatementBatchStatus @default(UPLOADED)
|
||||
uploadedById String
|
||||
uploadedBy User @relation("StatementBatchUploader", fields: [uploadedById], references: [id])
|
||||
label String?
|
||||
fileCount Int @default(0)
|
||||
/// Set when the pipeline fails as a whole (bad PDF, OCR binaries missing).
|
||||
error String? @db.Text
|
||||
createdAt DateTime @default(now())
|
||||
completedAt DateTime?
|
||||
|
||||
documents StatementDocument[]
|
||||
|
||||
@@index([status, createdAt])
|
||||
@@map("statement_batches")
|
||||
}
|
||||
|
||||
/// One statement — one customer, one period — after splitting the batch.
|
||||
model StatementDocument {
|
||||
id String @id @default(uuid())
|
||||
batchId String
|
||||
batch StatementBatch @relation(fields: [batchId], references: [id], onDelete: Cascade)
|
||||
/// 1-based page of the source PDF this was split from.
|
||||
pageNumber Int
|
||||
/// The rendered page image in object storage. The source PDF is kept too, so
|
||||
/// a reviewer can always see exactly what the parser read.
|
||||
storageKey String
|
||||
status StatementDocumentStatus @default(PENDING_OCR)
|
||||
|
||||
/// Raw OCR text, kept even after a manual correction so a mismatch between
|
||||
/// what the machine read and what staff entered stays auditable.
|
||||
ocrRawText String? @db.Text
|
||||
/// Mean per-word confidence reported by the OCR engine, 0..1.
|
||||
ocrConfidence Decimal? @db.Decimal(4, 3)
|
||||
/// Which parser claimed the page ("CFE", "CESPT", "TELNOR").
|
||||
provider String?
|
||||
|
||||
// Extracted, then staff-corrected in place. `extractedAccountRef` is already
|
||||
// normalised for matching (CFE leading zeros stripped, Telnor LADA removed).
|
||||
extractedAccountRef String?
|
||||
extractedAmount Decimal? @db.Decimal(12, 2)
|
||||
extractedPeriod String?
|
||||
extractedDueDate DateTime?
|
||||
/// Clave catastral when the statement prints one — a second key to match on
|
||||
/// when the account number is unreadable.
|
||||
extractedCadastralKey String?
|
||||
|
||||
matchedPropertyServiceId String?
|
||||
matchedPropertyService PropertyService? @relation(fields: [matchedPropertyServiceId], references: [id])
|
||||
matchedCustomerId String?
|
||||
matchedCustomer Customer? @relation(fields: [matchedCustomerId], references: [id])
|
||||
/// Why this landed where it did — "exact account match", "no candidate",
|
||||
/// "2 candidates". Shown in the review queue so staff can trust or distrust
|
||||
/// the suggestion without opening the image.
|
||||
matchNote String?
|
||||
|
||||
reviewedById String?
|
||||
reviewedBy User? @relation("StatementDocumentReviewer", fields: [reviewedById], references: [id])
|
||||
reviewedAt DateTime?
|
||||
|
||||
postedTransactionId String? @unique
|
||||
postedTransaction Transaction? @relation(fields: [postedTransactionId], references: [id])
|
||||
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@unique([batchId, pageNumber])
|
||||
@@index([status])
|
||||
@@index([matchedCustomerId])
|
||||
@@map("statement_documents")
|
||||
}
|
||||
|
||||
/// From TRUSTVENCE.
|
||||
model TrustAccount {
|
||||
id String @id @default(uuid())
|
||||
@@ -483,6 +622,9 @@ model Transaction {
|
||||
legacyId String?
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
/// Set only on OCR-posted rows — the statement page this came from.
|
||||
statementDocument StatementDocument?
|
||||
|
||||
@@index([customerId, transactionDate])
|
||||
// By-check reconciliation (billing.byCheck / the cheque-count report) looks
|
||||
// rows up by check number alone — the legacy EDITA CHEQUE COUNT lookup.
|
||||
@@ -600,6 +742,9 @@ model User {
|
||||
updatedAt DateTime @updatedAt
|
||||
activityLogs ActivityLog[]
|
||||
|
||||
statementBatches StatementBatch[] @relation("StatementBatchUploader")
|
||||
statementsReviewed StatementDocument[] @relation("StatementDocumentReviewer")
|
||||
|
||||
@@map("users")
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user