Staff key 300+ utility statements per company per month by hand. This adds
the ingest -> split -> OCR -> match -> review pipeline that proposes customer
and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the
existing BillingService.createBatch seam with source=OCR and a per-document
captureRef so machine and hand capture share one write path and audit trail.
Everything was designed against 10 real scanned statements (46 pages of CFE,
CESPT and Telnor bills) rather than from the sample-free spec. The scans have
no text layer at all — they are camera images — so OCR is mandatory, and they
arrive bundled one customer per page. Measured on those pages the parser
identifies the provider 46/46 and reads an account reference 43/46; against
the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with
the rest genuine review cases. That closes the OCR-provider question in favour
of self-hosted Tesseract: it clears the bar for a queue where a human confirms
every row, and OcrProvider keeps a managed API a one-line swap.
The samples corrected three things the spec had wrong or unknown:
- Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and
predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds,
has 663 distinct values across 1135 rows and appears on no statement. The
clave now lives on Property.cadastralKey as the matcher's secondary key;
predial is left untouched. This had been blocking predial matching.
- Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers
(the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber.
- Phone is one billed line per property (534/18/1 across phone1/2/3), so the
new TELEPHONE ServiceKind backfills from phone1 only, not three rows.
Matching is scoped to one column per service kind and never reads the customer
name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this
office holds under CATT, RANDY, because the printed name is the registrant,
not the current owner. Where a provider prints a payment barcode it beats the
printed label (one CFE label OCR'd a digit too many while its barcode was
correct) and the two cross-check, with disagreement forcing review.
Confirming a document whose service had no reference writes it back, so gas
and any other cold start is a one-time cost rather than a permanent queue.
Verified end to end against the live dev API and MinIO: real scans uploaded
over HTTP, matched, confirmed against a check, and the resulting rows checked
in MySQL (negative amounts, captureSource=OCR, concept derived from the batch
kind, captureRef linking back to each page). Re-confirming a posted batch is
refused. Test data was removed afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The office keeps more than one operating account (Utilities banks in MXN,
Seguros in USD), but bank_transactions was a single implicit MXN register by
design. Adds Bank/BankAccount and makes every read and write in the module
scoped to exactly one account.
Schema:
- Bank / BankAccount. Currency is fixed per account and BankTransaction has
no currency column of its own — a movement inherits its account's, the way
a real bank account doesn't mix currencies.
- BankTransaction.bankAccountId, required. A movement with no known account
isn't reconcilable against a statement.
- @@index([bankAccountId, transactionDate]): every read now filters by
account and orders/groups by date.
Migration:
- backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)"
and backfills all 22,669 existing rows onto it, then promotes the column to
NOT NULL and attaches the FK. Standalone because prisma db push cannot add
a required column to a populated table. Idempotent; re-running once a second
account exists does not re-point rows.
- run_all.py runs it (both modes) before transform_bank.py, which now resolves
the account by label and fails fast if it is missing.
API:
- ?bankAccountId= required on list/stats/facets/summary — not optional with an
"all accounts" default, since summing an MXN and a USD register repeats the
currency-collapsing mistake the billing module exists to prevent. Missing is
400, unknown is 404.
- facets() had no account clause at all and summary() has two raw-SQL rollups;
all three are now parameterised. Scoping only one of summary's queries would
leave the year list and its drill-down describing different books.
- New bank/accounts + bank/banks sub-resource under a MANAGER
bank:manage-accounts ability. currency is absent from the update DTO: booked
movements are denominated in it, so editing would re-denominate history.
Capture into a closed account is rejected.
Web:
- /banco gains an account picker (remembered per browser) and reads every
figure in the selected account's currency; the "single currency (MXN)"
doc-comment and the hardcoded MXN formatting are gone.
- New /banco/cuentas for banks and accounts. Accounts are closed, never
deleted — the FK is required, so deleting one would destroy its register.
- /inicio's chequera card names the account it is reading instead of implying
a single register.
Verified against dev + browser: a second USD account showed full read/write
isolation from the MXN register, whose totals were unchanged (22,669
movements, net 1,014,266.97).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implements docs/RECEIPT_CAPTURE_SPEC.md §1, the legacy "Editor"
replacement, on top of the single-movement capture from plan step 6.
No new abilities: batching and resolving are both capturing.
- outstanding (legacy NOPAGO): capture flag, ?outstanding= filter, and
POST /billing/:id/resolve-outstanding (gated ledger:create, not
ledger:void — resolving completes a capture rather than reversing
one). Outstanding rows are excluded from every balance aggregate,
matching the legacy SALDOS ULTIMO 0 query's HAVING NOPAGO = 0, but
still count in the movement browser's filtered totals.
- POST /billing/batch: many customers' receipts against one check, in
one $transaction. Deliberately not a persisted batch entity —
checkNumber is already a column and grouping by it answers every
legacy by-check query.
- GET /billing/by-check + a cheque-count report, replacing REPORTE
CHEQUE COUNT / REPORTE POR CHEQUE / EDITA CHEQUE ALF|COUNT|NUM. Print,
PDF, CSV and XLSX come free from the existing /reportes/:slug machinery.
- Web: /estado-cuenta/lote (the Editor screen, with live reconciliation
against the physical check amount), an "Estado de pago" filter, a
"sin fondos" row tag and a Resolver dialog, plus a top-level "Captura"
nav entry.
Integration seam for the OCR auto-capture module (spec §2), which is
required to post through createBatch rather than writing Transaction
rows itself: items[i] maps to lines[i] so postedTransactionId can be
zipped back on; opts.refs[i] stamps captureRef with a duplicate-post
guard that a voided row deliberately does not block; opts.source is
service-level only, so an HTTP client cannot label hand-keyed rows as
machine-captured. captureSource/captureRef are nullable so the 40,136
migrated rows stay NULL rather than being mislabelled.
Fixes two pre-existing bugs found while building this:
- statement() filtered legacySourceTable with `notIn`, which compiles to
SQL NOT IN — and `NULL NOT IN (...)` is NULL, so every app-captured
movement was invisible on the customer statement (438 rows in the
movement browser vs 392 on the statement) while showing everywhere
else. This would have made the whole capture feature look broken.
- The balances count query omitted the void filter its own page query
applied, so the total disagreed with the rows.
Nav highlighting now resolves by longest match; the previous
first-startsWith logic lit up both the parent and any nested entry.
Verified end-to-end against the dev DB, API and browser; all test rows
removed afterwards. Also corrects RESUME.md, which documented the dev
ports as :3001/:3000 — they are :4501/:4500, from the env files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Points PLAN.md at docs/RECEIPT_CAPTURE_SPEC.md and surfaces its open
design questions (OCR provider, Seguros bank details, clave catastral
vs. predial, recycling triggers) separately from the existing ops-only
open items list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Working-tree checkpoint of in-progress work carried across prior
sessions on the feat/crud-rbac branch, committed so it lands on the
remote alongside the CI changes.
- Operaciones admin panel: apps/api/src/ops (ingest upload, backup /
restore / re-import jobs) wired into app.module + RBAC abilities, and
the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR
/ BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups.
- migration/sync.py plus transform_*.py / run_all / config / dbenv /
blob_extract adjustments for the additive sync path.
- crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals.
- schema.prisma + PLAN/RESUME doc updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
with the corrected de-dup rule, plus a box explaining why the original
verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
summed.
- Build sequencing step 6 marked done.
RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
"not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
worth knowing — run_all.py vs single transforms, `next build` clobbering a
running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
many sessions ago. Renumbered, with an explicit "next" block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 5