d173c9e9a0d7873a362f7cfccf0e6a1db64c822b
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  rmancinasandClaude Opus 5
|
ed19f51a52 |
fix(ops): re-stage before an additive sync
SYNC ran `run_all.py --sync` without `--stage`, so it depended on staged
Parquet under migration/output. That directory is part of the image, not a
volume, so any redeploy wiped it and the job died on the first transform:
FileNotFoundError: '/repo/migration/output/stg_utilities/datgral.parquet'
Re-staging is also what makes the job's own label true — without it a sync
would replay whatever upload staged last, not the files currently in the
ingest folder.
Staging now counts as a numbered step when it runs, so the Operaciones
progress bar moves during the slowest phase instead of sitting empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
rmancinasandClaude Opus 5
|
66d0d071b0 |
feat(ops): show step progress for reimport and sync jobs
A REIMPORT takes ~110 seconds and, until now, showed only a scrolling log — there was no way to tell "halfway" from "wedged", which mattered the day one actually did wedge. run_all.py emits "[paso i/N] name" before each step and the API derives progress from the job log. Emitting the marker from the Python rather than having the UI count STEPS itself means the step count is stated in exactly one place; adding a step cannot desync the display. Progress is derived, not stored, for the same reason: the log is already the record of what happened, and a separate counter could contradict it, which is precisely the confusion a progress display exists to remove. While RUNNING, step i is IN PROGRESS rather than finished, so only i-1 count as done. Counting i would show 100% while the final step was still working — and the final step (blob_extract) is the slowest, so the bar would sit at "100%" for the longest stretch of the job. BACKUP and RESTORE are a single mysqldump with no steps and deliberately render no bar; a fabricated percentage would be worse than none. The safety backup that precedes a REIMPORT is likewise named explicitly instead of showing 0%, which reads as stuck. Pinned by job-progress.spec.ts, including the literal line run_all.py emits, so a change to the Python format fails a test rather than silently blanking the panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 5
|
4d5008b545 |
feat(statements): OCR intake for scanned utility bills
Staff key 300+ utility statements per company per month by hand. This adds the ingest -> split -> OCR -> match -> review pipeline that proposes customer and amount per page instead (RECEIPT_CAPTURE_SPEC §2), posting through the existing BillingService.createBatch seam with source=OCR and a per-document captureRef so machine and hand capture share one write path and audit trail. Everything was designed against 10 real scanned statements (46 pages of CFE, CESPT and Telnor bills) rather than from the sample-free spec. The scans have no text layer at all — they are camera images — so OCR is mandatory, and they arrive bundled one customer per page. Measured on those pages the parser identifies the provider 46/46 and reads an account reference 43/46; against the dev database that is 39/46 (85%) exact auto-match, 40/46 identified, with the rest genuine review cases. That closes the OCR-provider question in favour of self-hosted Tesseract: it clears the bar for a queue where a human confirms every row, and OcrProvider keeps a managed API a one-line swap. The samples corrected three things the spec had wrong or unknown: - Clave catastral is NOT predial. DATMEX.clave (934 rows) is what CESPT and predial bills print; DATMEX.predial, which PROPERTY_TAX.accountNumber holds, has 663 distinct values across 1135 rows and appears on no statement. The clave now lives on Property.cadastralKey as the matcher's secondary key; predial is left untouched. This had been blocking predial matching. - Gas was recoverable: 160 of 334 DATMEX.gas values are real account numbers (the rest are ESTACIONARIO/CILINDRO descriptors), now in GAS.meterNumber. - Phone is one billed line per property (534/18/1 across phone1/2/3), so the new TELEPHONE ServiceKind backfills from phone1 only, not three rows. Matching is scoped to one column per service kind and never reads the customer name — a CESPT receipt prints ARNAIZ ROSAS ELSA AURORA for an account this office holds under CATT, RANDY, because the printed name is the registrant, not the current owner. Where a provider prints a payment barcode it beats the printed label (one CFE label OCR'd a digit too many while its barcode was correct) and the two cross-check, with disagreement forcing review. Confirming a document whose service had no reference writes it back, so gas and any other cold start is a one-time cost rather than a permanent queue. Verified end to end against the live dev API and MinIO: real scans uploaded over HTTP, matched, confirmed against a check, and the resulting rows checked in MySQL (negative amounts, captureSource=OCR, concept derived from the batch kind, captureRef linking back to each page). Re-confirming a posted batch is refused. Test data was removed afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 5
|
9ba5d2d09a |
feat(bank): multi-bank chequera — required bankAccountId, per-account scoping
The office keeps more than one operating account (Utilities banks in MXN, Seguros in USD), but bank_transactions was a single implicit MXN register by design. Adds Bank/BankAccount and makes every read and write in the module scoped to exactly one account. Schema: - Bank / BankAccount. Currency is fixed per account and BankTransaction has no currency column of its own — a movement inherits its account's, the way a real bank account doesn't mix currencies. - BankTransaction.bankAccountId, required. A movement with no known account isn't reconcilable against a statement. - @@index([bankAccountId, transactionDate]): every read now filters by account and orders/groups by date. Migration: - backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)" and backfills all 22,669 existing rows onto it, then promotes the column to NOT NULL and attaches the FK. Standalone because prisma db push cannot add a required column to a populated table. Idempotent; re-running once a second account exists does not re-point rows. - run_all.py runs it (both modes) before transform_bank.py, which now resolves the account by label and fails fast if it is missing. API: - ?bankAccountId= required on list/stats/facets/summary — not optional with an "all accounts" default, since summing an MXN and a USD register repeats the currency-collapsing mistake the billing module exists to prevent. Missing is 400, unknown is 404. - facets() had no account clause at all and summary() has two raw-SQL rollups; all three are now parameterised. Scoping only one of summary's queries would leave the year list and its drill-down describing different books. - New bank/accounts + bank/banks sub-resource under a MANAGER bank:manage-accounts ability. currency is absent from the update DTO: booked movements are denominated in it, so editing would re-denominate history. Capture into a closed account is rejected. Web: - /banco gains an account picker (remembered per browser) and reads every figure in the selected account's currency; the "single currency (MXN)" doc-comment and the hardcoded MXN formatting are gone. - New /banco/cuentas for banks and accounts. Accounts are closed, never deleted — the FK is required, so deleting one would destroy its register. - /inicio's chequera card names the account it is reading instead of implying a single register. Verified against dev + browser: a second USD account showed full read/write isolation from the MXN register, whose totals were unchanged (22,669 movements, net 1,014,266.97). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
1b79b43a54 |
fix(migration): make Phase B additive sync actually work + verify end-to-end
The --sync path had never been run and was broken in several ways. Fixed and verified against the dev DB (two consecutive syncs, both exit 0, 32/32 assertions: stable PKs, manual-row preservation, changed-row updates, legacy-delete, no child duplication, zero FK orphans; idempotent). - policies/properties: reuse each legacy row's existing id (by provenance) BEFORE building child rows, so children no longer point at a discarded fresh uuid; rebuild legacy-owned children via scoped delete + reinsert. - customers: replace zip(customers, refs) (mispaired almost every row) with a ref-grouped id remap; names now restore and no spurious customers appear. - drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert. - upsert lookup tables (policy_types, insurance_providers, type_transactions, adjusters) by natural name and remap child FKs instead of inserting fresh uuids that nothing points at. - transactions: drop updatedAt=NOW() (no such column); guard report formatting on NULL legacySourceTable (manual rows). Same report guard in bank. - add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune only legacy-owned empties, never manually-added customers. web: customer-detail mini tx list now strikes voided rows with an "(anulado)" tag (was the last void-UI rendering gap; /estado-cuenta already handled it). docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI browser pass recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
f1ef1c70b3 |
wip: ops admin panel + migration sync + crud/rbac phase-5 snapshot
Working-tree checkpoint of in-progress work carried across prior sessions on the feat/crud-rbac branch, committed so it lands on the remote alongside the CI changes. - Operaciones admin panel: apps/api/src/ops (ingest upload, backup / restore / re-import jobs) wired into app.module + RBAC abilities, and the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR / BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups. - migration/sync.py plus transform_*.py / run_all / config / dbenv / blob_extract adjustments for the additive sync path. - crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals. - schema.prisma + PLAN/RESUME doc updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
fa9b696752 |
Migration: prune customers with no business records
144 customers owned zero properties, zero policies and zero transactions — the legacy DATGRAL row exists but nothing in either business line ever attached to it. They padded the staff customer list with rows that can't be acted on. 27 were also nameless (dead ID slots); the other 117 have real names and sometimes contact details, and read as never-activated prospects or lapsed clients rather than junk. Removing both sets is a deliberate call. Implemented as a separate step rather than a filter inside transform_customers.py: emptiness is only knowable after properties, policies and transactions have loaded, and deciding it there would mean re-deriving each downstream transform's source-matching logic against the staged Parquet. Runs after transform_transactions.py in run_all.py. Safe by construction — a customer with no rows in any of the three tables has nothing pointing at it, so the delete cannot orphan anything; only its own customer_legacy_refs go with it. The step asserts zero orphans afterwards. Every pruned customer is written to output/pruned_customers.csv with its legacy provenance before the delete, and --dry-run reports without touching anything. Nothing is unrecoverable: the Access sources are untouched and a pipeline run without this step brings them all back. Verified: full run_all.py pass ends at 1538 customers (from 1682), with 1519 properties / 2378 policies / 45861 transactions all intact and zero orphans. 17 nameless customers remain, all of which carry real records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
594ee7cfca |
Migration: recover blank customer names from secondary legacy tables
DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance), which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)". The blank is real — those cells are empty in the Access files, not lost in extraction — but the rows mostly are not junk: 176 of the 257 carry a property, a policy, or transactions. The old PHP importer handled this by skipping blank-name rows outright (jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was worse than it looks: every other adapter resolved its customer FK through the customer_mapping table those skipped rows never entered, so their properties and policies were silently dropped (customerServiceAdapter.php:45) and their transactions were written against customer_id 0 (customerBalanceAdapter.php:52). So: recover the name instead of skipping. Names come from the secondary tables that still carry them, most trustworthy first — UTILSEG (the office's own hand-maintained name <-> id cross-reference spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow the name its insurance record resolved to. Result: 213 of 257 recovered, 44 still genuinely nameless anywhere in the source. customers.nameSource records which table each recovered name came from, so a reconstructed name is never mistaken for one that was really on the record — the list tags it "nombre recuperado", the detail header names the source, and a still-unnamed customer renders muted italic instead of as a normal name. Also fixes run_all.py: transform_properties and transform_policies truncate service_documents/policy_documents, but blob_extract.py was not in the step list, so a full re-run left the uploaded MinIO objects with no rows pointing at them. Hit exactly that while reloading for this change. Verified end-to-end: full pipeline re-run against dev reproduces every prior count (1682 customers, 1519 properties, 2378 policies, 45861 transactions, 22354 bank rows, 70 documents) with zero orphans, and both apps build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
83e3cb8f47 |
Transform+load: shared ledger + SCOTHIA bank register (step 3 complete)
migration/transform_transactions.py unions every cash/billing ledger into `transactions` per the reconciliation rules: both EFECTIVO tables (no folio de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3 fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and exchange_rates lookups. Customer FK resolves through customer_legacy_refs; rows with no resolvable customer/date are skipped and counted. Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans), 79 type_transactions, 2301 exchange_rates. migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as signed amounts (income +, expense -) and TABLA RAMODOS into business_line_categories. Deliberately customer-independent (office's own checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77), 66 categories; categoryId left null (concept->ramo classifier is future work). run_all.py: pipeline now customers -> properties -> policies -> transactions -> bank, all idempotent. Verified full end-to-end run against dev. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
21899e99bb |
Transform+load: consolidate all insurance lines into policies (step 3)
migration/transform_policies.py folds every insurance Access table into one `policies` table (policy_types discriminator) plus child tables, via a per-table declarative mapping that absorbs the column-name variance (num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source column not explicitly modeled — the type-specific coverage amounts — is preserved verbatim in coveragesJson, so consolidation loses nothing. Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago + pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries (by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and builds policy_types + insurance_providers lookups. Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 / M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678 installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15 providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a customer carrying both a utility property and MULT policies (the unified cross-line view). run_all.py: add policies to the ordered pipeline. Customer FK resolves through insurance customer_legacy_refs, so this runs after customers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rmancinasandClaude Opus 4.8
|
a680ad2bb0 |
Transform+load: properties/services/trust + env-parameterize migration
migration/transform_properties.py loads properties, property_services and trust_accounts from staged DATMEX/PROFILE, resolving each property's customer FK through customer_legacy_refs. Services are derived from DATMEX's own account/route/meter fields (the authoritative data); PROFILE flags — merged best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only refine each service's `active`. Trust accounts are 1:1 from DATMEX trust fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4. Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped), 3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 / OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts — counts track the PROFILE enrollment flags. Reproducibility (asked: dev must be redoable in prod): - migration/dbenv.py: single DB-target source = deploy/.env.<env>'s DATABASE_URL; connect(env) + env_arg() (--env, default dev). - transform_customers.py / transform_properties.py now take --env instead of hardcoding .env.dev. - migration/run_all.py: runs every step in dependency order against --env (optional --stage re-extracts from Access first). Reproducing dev->prod is `run_all.py --env prod` after deploying the prod stack + prisma db push. All steps are idempotent (truncate+rebuild); re-run yields identical counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |