Working-tree checkpoint of in-progress work carried across prior
sessions on the feat/crud-rbac branch, committed so it lands on the
remote alongside the CI changes.
- Operaciones admin panel: apps/api/src/ops (ingest upload, backup /
restore / re-import jobs) wired into app.module + RBAC abilities, and
the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR
/ BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups.
- migration/sync.py plus transform_*.py / run_all / config / dbenv /
blob_extract adjustments for the additive sync path.
- crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals.
- schema.prisma + PLAN/RESUME doc updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the office's own bank-register browser over the migrated SCOTHIA
data (22,354 bank_transactions), the last self-contained feature module.
API (apps/api/src/bank):
- GET /bank register browser: search over concepto/reference/notes/
amountInWords; direction (income|expense|void), cleared and
date-range filters; 5 sorts; income/expense/net totals for
the whole filtered set, not just the page
- GET /bank/stats headline income/expense/net + counts, date span, pending
- GET /bank/facets year list for the period filter
- GET /bank/summary year and month rollups with a running net-movement figure
Web (/banco): "Movimientos" register + "Resumen por periodo" with year->month
drill-down; added to the AppShell nav as "Chequera".
Deliberately kept OUT of /estado-cuenta: this is the office's own money, not
customer balances, and the two are never summed or shown together.
No category/ramo dimension, and the deferred concept->ramo classifier is
dropped as won't-build: concepto is a payee name (0 of 22,354 match a
category) and TABLA RAMODOS is an expense chart of accounts + owner names,
not the insurance/servicios/fideicomiso split it was assumed to be, so a
classifier would invent data. Single currency (MXN); the "acumulado" is net
movement since the register opened (no opening balance in the source), not a
bank balance. Verified end-to-end in the browser; totals reconcile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
with the corrected de-dup rule, plus a box explaining why the original
verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
summed.
- Build sequencing step 6 marked done.
RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
"not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
worth knowing — run_all.py vs single transforms, `next build` clobbering a
running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
many sessions ago. Renumbered, with an explicit "next" block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 5. Properties, services and trust accounts become a first-class
browser the way /polizas is for insurance.
API (apps/api/src/properties):
GET /properties search over address, customer, service account
number, meter, trust number and phones; filters for
service kind, municipality, trust bank, trust bucket
(with|without|active|expiring|expired|undated) and
hasServices; 5 sorts
GET /properties/stats properties/owners/services/trusts, renewal counts,
service mix per kind
GET /properties/facets kinds, municipalities, banks — all with counts
GET /properties/:id services, fideicomiso, linked policy, owner and
sibling properties, owner-level utility ledger
Web: /servicios (renewals-first browser, clickable stat cells and service-mix
strip) and /servicios/[id]. Property cards on /clientes/[id] and linked
properties on /polizas/[id] now navigate into it.
Data findings baked into the design:
- The trust deadline staff chase is trust_accounts.dueDate2 (DATMEX vence2),
one year after vence1 on 531 of 541 dated trusts: 18 due within 30 days,
119 already overdue. Every renewal bucket keys off dueDate2 alone.
- properties.zone is dead (1444 of 1519 null, the rest near-unique), so the
geographic filter is the municipality carried in the predial service's
notes (ROSARITO 566 / TIJUANA 221 / ENSENADA 152, 939/939 populated).
- PropertyService.notes means a different thing per kind (municipality, CFE
PAR/IMPAR cycle, gas supply type, cable provider) and is labelled as such.
- 240 of 1519 properties have no service rows at all — its own bucket.
Sorting by trust due date scopes to properties that have a trust, since MySQL
would otherwise float the ~966 trust-less NULLs above every real due date;
the sort label and the result meta both say so.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 4. Adds the policies API and the Spanish-first /polizas pages on
top of the customer records the customer module already exposes.
API (apps/api/src/policies):
- GET /policies — search over policy number, customer name, agent, vehicle
license plate, insured-driver name and legacy id; filters for vigencia
bucket, ramo, aseguradora and liquidation state; five sort orders.
- GET /policies/stats — bucket counts plus premium in force split by
currency (MXN and USD can't be summed).
- GET /policies/facets — ramos/aseguradoras with counts for the dropdowns.
- GET /policies/:id — full policy plus the owning customer.
Vigencia is derived from policyTo as active/expiring/expired/undated.
"undated" is a real bucket rather than an error case: 528 of the 2378
migrated policies carry no end date at all.
Web:
- /polizas — renewals-first browser; the stat cells double as vigencia
filters, with a secondary row for ramo, aseguradora and sort order.
- /polizas/[id] — vigencia hero, condiciones y primas, pagos, vehículos,
asegurados/beneficiarios, siniestros, the verbatim legacy coverage
columns, and documents.
- Nav gains Clientes | Pólizas with a real active state, and the two
modules cross-link in both directions.
Also fixes a display bug on the customer detail page: it headlined
policies.total, which is dead data — only 2 of 2378 rows are non-zero
(1585 are literally 0, 791 null), and one of those two is lower than its
own net premium. That rendered "$0.00 Total" on 1585 policies. Premium
headlines and the premium sort now use netPremium (2377/2378 populated);
total is shown only where it is non-zero, as raw source data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.
Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.
Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.
migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).
run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.
Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.
Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).
run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).
Loaded and validated against the dev DB (192.168.4.212:3307):
1682 customers (1172 utilities master + 510 insurance-only)
2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
560 insurance rows linked via num_util, 0 broken cross-refs
542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.
RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
docs/LEGACY_DATABASES.md documents all three source Access databases
(every table, column, type, and known data-quality quirk) generated
from a live read of the real files, so no Windows/Access driver is
needed to understand their structure going forward.
New migration/ tooling: catalog_schema.py connects to the real files
and walks every table (including excluded scratch tables);
render_catalog_md.py renders that into the doc's appendix. Raw output
checked in at migration/catalog.json so the doc can be regenerated
without touching Access again.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 4.8