rmancinasandClaude Opus 5
4d5008b545
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
111 lines
5.6 KiB
Docker
111 lines
5.6 KiB
Docker
FROM node:20-alpine AS base
|
|
WORKDIR /repo
|
|
# Pin pnpm 9 to match pnpm-lock.yaml (lockfileVersion 9.0). pnpm 9 runs
|
|
# dependency build scripts automatically (the v10 build-allowlist gating does
|
|
# not apply), so argon2's native addon + prisma engines build without extra
|
|
# approval config.
|
|
RUN corepack enable && corepack prepare pnpm@9.15.9 --activate
|
|
|
|
FROM base AS deps
|
|
# argon2's native addon has no musl prebuild -> compiles from source here.
|
|
# openssl so `prisma generate` in the build stage sees the same platform the
|
|
# runtime stage does (see the binaryTargets note in schema.prisma).
|
|
RUN apk add --no-cache python3 make g++ openssl
|
|
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
|
|
COPY apps/api/package.json apps/api/package.json
|
|
COPY apps/web/package.json apps/web/package.json
|
|
COPY packages/database/package.json packages/database/package.json
|
|
# node-linker=hoisted flattens the store into a single npm-style /repo/node_modules
|
|
# so the runtime stage can copy one tree (pnpm's default symlinked layout would
|
|
# break across COPY stages).
|
|
RUN pnpm install --frozen-lockfile --config.node-linker=hoisted
|
|
|
|
FROM deps AS build
|
|
COPY packages/database packages/database
|
|
COPY apps/api apps/api
|
|
RUN pnpm --filter @jorgecuadros/database generate
|
|
RUN pnpm --filter @jorgecuadros/api build
|
|
|
|
FROM node:20-alpine AS runtime
|
|
WORKDIR /repo
|
|
ENV NODE_ENV=production
|
|
|
|
# DB-ops toolchain baked in so the "Operaciones" admin panel can run backups
|
|
# (mysqldump), restores (mysql), and the re-import pipeline (python + mdbtools)
|
|
# from inside the API container. Build deps are installed in a throwaway virtual
|
|
# package so pandas/pyarrow build on musl, then dropped from the final layer.
|
|
# openssl is NOT optional: Prisma's query engine resolves its binary target at
|
|
# runtime (linux-musl-openssl-3.0.x) and aborts with "Please manually install
|
|
# OpenSSL" without it. Node bundles its own OpenSSL, so nothing else in this
|
|
# image pulls the system package in.
|
|
# mariadb-connector-c is REQUIRED, not incidental. Alpine's `mysql-client` is
|
|
# MariaDB's client, and it ships with an EMPTY /usr/lib/mariadb/plugin — so it
|
|
# cannot perform caching_sha2_password, which is MySQL 8.4's default and
|
|
# effectively only auth method. Without this package every mysqldump/mysql call
|
|
# from the container dies with:
|
|
# ERROR 1045: Plugin caching_sha2_password could not be loaded
|
|
# That breaks the pre-migrate deploy backup AND the whole "Operaciones" admin
|
|
# panel (backup, restore, sync, re-import all shell out to these binaries).
|
|
#
|
|
# tesseract-ocr + tesseract-ocr-data-spa + poppler-utils drive the statement
|
|
# OCR intake (RECEIPT_CAPTURE_SPEC §2): poppler's `pdftoppm` rasterises each
|
|
# scanned page and tesseract reads it, with the Spanish traineddata for the
|
|
# accented labels on CFE/CESPT/Telnor bills. These are external binaries rather
|
|
# than a native npm addon so the pnpm workspace stays free of a compiled
|
|
# dependency. If they are absent the API still boots — the statements module
|
|
# reports itself unavailable and only that feature is disabled — but statement
|
|
# ingest is the point of shipping them.
|
|
RUN apk add --no-cache python3 mdbtools mysql-client mariadb-connector-c openssl \
|
|
tesseract-ocr tesseract-ocr-data-spa poppler-utils \
|
|
&& apk add --no-cache --virtual .pybuild python3-dev build-base \
|
|
&& rm -rf /var/cache/apk/*
|
|
|
|
COPY --from=build /repo/node_modules node_modules
|
|
COPY --from=build /repo/packages/database packages/database
|
|
COPY --from=build /repo/apps/api/dist apps/api/dist
|
|
COPY --from=build /repo/apps/api/package.json apps/api/package.json
|
|
# Operational scripts, run on demand — never automatically. seed-user.mjs is the
|
|
# only way to create the first sign-in account on a fresh database, and without
|
|
# it in the image that had to be done from a developer's machine against a
|
|
# production DATABASE_URL. Run it with:
|
|
# docker exec <api> node apps/api/scripts/seed-user.mjs
|
|
# honouring SEED_EMAIL / SEED_PASSWORD / SEED_NAME. It upserts, so re-running is
|
|
# safe — but note it RESETS the password of an existing account.
|
|
COPY --from=build /repo/apps/api/scripts apps/api/scripts
|
|
# node-linker=hoisted flattens EXTERNAL deps into /repo/node_modules, but the
|
|
# workspace dependency is still linked per-package:
|
|
# apps/api/node_modules/@jorgecuadros/database -> ../../../../packages/database
|
|
# Copying only /repo/node_modules therefore drops it and the API dies at boot
|
|
# with "Cannot find module '@jorgecuadros/database'". Copy just the scope dir —
|
|
# the rest of apps/api/node_modules is devDependencies (typescript) we don't
|
|
# want in the runtime layer. The relative link resolves because packages/database
|
|
# is copied to the same place above.
|
|
COPY --from=build /repo/apps/api/node_modules/@jorgecuadros apps/api/node_modules/@jorgecuadros
|
|
|
|
# Migration scripts + their own Python venv (ops.service.ts prefers this venv).
|
|
COPY migration migration
|
|
RUN python3 -m venv migration/.venv \
|
|
&& migration/.venv/bin/pip install --no-cache-dir -r migration/requirements.txt \
|
|
&& apk del .pybuild
|
|
|
|
# Ingest (uploaded Access files) and backups live on mounted volumes.
|
|
ENV MIGRATION_DIR=/repo/migration \
|
|
INGEST_DIR=/data/ingest \
|
|
BACKUP_DIR=/data/backups \
|
|
MIGRATION_ENV=dev
|
|
RUN mkdir -p /data/ingest /data/backups
|
|
|
|
# Build/version metadata baked in at image build time (see .gitea/workflows/build.yml).
|
|
# APP_VERSION is the metadata-action primary tag (semver tag, branch, or sha);
|
|
# GIT_SHA/BUILD_DATE pin the exact commit + build instant. Exposed as ENV so a
|
|
# running container can self-report what is deployed (e.g. a /version endpoint).
|
|
ARG APP_VERSION=dev
|
|
ARG GIT_SHA=unknown
|
|
ARG BUILD_DATE=unknown
|
|
ENV APP_VERSION=$APP_VERSION \
|
|
GIT_SHA=$GIT_SHA \
|
|
BUILD_DATE=$BUILD_DATE
|
|
|
|
EXPOSE 3001
|
|
CMD ["node", "apps/api/dist/main.js"]
|