- New reports backend (registry, service, controller, outputs, types)
with catalog endpoint + slug/CSV/XLSX/PDF/print outputs.
- /reportes catalog + /reportes/[slug] runner; ReportRunner + ContextReports
components wire pre-filtered links from domain pages.
- Fix: /reportes/[slug] now reads searchParams and forwards initialParams to
ReportRunner so /reportes/edo-cuenta-datos?customerId=... auto-runs
instead of dropping the id and forcing a manual customer search.
- /inicio landing page; root + login redirect to /inicio.
- Company header env vars + logo asset for PDF/print rendering.
- exceljs + pdfkit deps.
Replace the JC text mark in AppShell and login with the real
company_logo.png. Restyle .brand-mark to host the image (white
rounded bg, object-fit contain). Appbar 38px, login panel 46px.
Add the missing api/web deployment path on top of the existing image build CI.
- deploy/jorgecuadros-app.stack.yml: PROD app stack (api + web) pulling the
git.mancinas.io registry images. Does not ship mysql/minio (separate stacks);
API reaches them via DATABASE_URL / S3_ENDPOINT. API pinned to the
jorgecuadros_db node for stable ingest/backup volumes; web is stateless.
- deploy/jorgecuadros-app.env.example: documented stack env template.
- .gitea/workflows/deploy.yml: manual (workflow_dispatch) deploy to Portainer
via cssnr/portainer-stack-deploy-action. Inputs: image tag + scope
(app = web+api, full = db+minio+app, applied db->minio->app).
Make the web API origin runtime-configurable instead of build-baked: the root
layout injects window.__API_ORIGIN__ from the API_ORIGIN env (force-dynamic) and
lib/api.ts resolves it at runtime, so one built image serves any deployment.
Also: dev.sh to run both dev servers (frees stale ports first) and move local
dev to ports web 4500 / api 4501.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The schema has carried `storageKey` pointers and the migration has written
blobs to MinIO since day one, but the API had no S3 client — documents could
only be deleted, never uploaded or retrieved. This adds the missing wiring.
API
- StorageModule/StorageService (@aws-sdk/client-s3, path-style for MinIO):
put/getStream/delete, best-effort bucket ensure on boot, gracefully disabled
when S3 env is absent (ServiceUnavailable on use).
- Reads S3_ENDPOINT/S3_BUCKET + S3_ACCESS_KEY/S3_SECRET_KEY, falling back to
MINIO_ROOT_USER/MINIO_ROOT_PASSWORD so one credential set drives both the
migration and the API.
- Property service documents: POST :id/documents (multipart), GET
:id/documents/:childId/download (streamed), delete now also drops the blob.
- Policy documents: same upload/download/delete (previously had none).
- Keys stay under the service/<id>/… and policy/<id>/… prefixes the migration
established.
Web
- api.ts: shared uploadFile() helper (uploadIngest refactored onto it),
upload/download/remove helpers for property & policy documents.
- Servicios, polizas, clientes detail pages: real Descargar links and an
upload control (gated by policy:update / property:update) replacing the
"storage pending" notes.
Infra
- docker-compose: minio service (9000/9001, healthcheck, named volume) + S3
env wired into the api service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Working-tree checkpoint of in-progress work carried across prior
sessions on the feat/crud-rbac branch, committed so it lands on the
remote alongside the CI changes.
- Operaciones admin panel: apps/api/src/ops (ingest upload, backup /
restore / re-import jobs) wired into app.module + RBAC abilities, and
the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR
/ BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups.
- migration/sync.py plus transform_*.py / run_all / config / dbenv /
blob_extract adjustments for the additive sync path.
- crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals.
- schema.prisma + PLAN/RESUME doc updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add comprehensive Docker image versioning and a Gitea Actions workflow
that builds and pushes both the API and web images to the
git.mancinas.io registry.
Versioning: both Dockerfiles take APP_VERSION / GIT_SHA / BUILD_DATE
build-args, surfaced as runtime ENV + OCI labels, so a running container
self-reports the exact commit it was built from. metadata-action emits a
tag set per build: semver (from vX.Y.Z git tags), branch ref,
sha-<short>, and latest (default branch only).
Also fix the Dockerfiles for the pnpm workspace: the old npm install
could not resolve the "@jorgecuadros/database": "workspace:*" protocol
dep and would abort the API build. Now pin pnpm 9.15.9 via corepack,
install --frozen-lockfile with node-linker=hoisted (flat tree so the
runtime stage copies a single node_modules), and build via --filter. The
API build stage gets python3/make/g++ for argon2's musl source compile.
Add .dockerignore to keep the build context lean and deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Covers stack, repo layout, local dev (pnpm install, env, MySQL via
docker, prisma db push, seed admin, run api+web, login), full-stack
Docker path, common commands, auth/roles, legacy migration, and prod
notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The add/edit form was appended after the whole table, so on long lists
(e.g. /catalogos aseguradoras, 16+ rows) clicking Editar on a top row
opened the form far below the fold — appearing to do nothing. Render the
edit form in place of its row, and the add form as the first table row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes phase 5 — the ledger and chequera pages get the append+void
UI on top of the phase-5 API.
Web:
- Shared MovementForm (customer picker + línea + cargo/abono sign + amount
+ moneda + concepto facet + periodo/referencia/cheque/mensaje). Used by
both /estado-cuenta (cross-customer, picker) and /estado-cuenta/[id]
(customer prefilled).
- /estado-cuenta and /estado-cuenta/[id]: "Capturar movimiento" toggle
gated ledger:create; per-row "Anular" gated ledger:void; voided rows
struck-through. Save/void refresh the list + stats.
- /banco: inline BankCaptureForm (ingreso/egreso sign, cheque, operado,
transferencia, monto en letras) gated bank:create; per-row "Anular"
gated bank:void; voided rows struck-through.
- api.ts: createMovement/voidMovement, createBankMovement/voidBankMovement;
CreateMovementInput/CreateBankMovementInput types; `voided` on the
movement/statement/bank list items.
Also: lookups.controller.ts now audit-logs provider/policy-type/adjuster
create/update/delete (parity with the other write controllers).
API + web compile clean. This is the last piece of the feat/crud-rbac
branch — all five sections plus users are now full CRUD with role gating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Transactions and the bank register become append-only with a void
(reversal) action — never edited or hard-deleted. This is the API half of
phase 5; the capture/void web UI is the remaining piece.
Schema:
- Transaction and BankTransaction gain voidedAt + voidedById. A non-null
voidedAt reverses the row. Pushed to dev.
Correctness (the high-stakes part):
- Every aggregate excludes voided rows: billing movements totals, the raw
balances SQL, stats (groupBy + the sides/crossLine raw subqueries +
first/last), facets (types/sources/years); the statement's running
balance freezes on a voided row and its per-currency/per-domain/per-type
summaries skip them; customers.detail and property owner-ledger groupBy;
and every bank total (totalsFor, stats counts/bounds, facets + summary
raw SQL). List views still return voided rows with a `voided` flag so
the UI can strike them through.
- Bank's legacy zero-amount "void" cheques are unchanged and distinct from
app voids (voidedAt).
API:
- POST /billing + POST /billing/:id/void (ledger:create / ledger:void);
POST /bank + POST /bank/:id/void (bank:create / bank:void). Create needs
STAFF+, void needs MANAGER+. Double-void -> 400, unknown id -> 404,
bad date -> 400. Mutations audited. DTOs added.
Verified against dev end-to-end: a -500 MXN charge moved a customer
balance 31082.08 -> 30582.08, and voiding it returned it to 31082.08 to
the cent; a +1234.56 bank ingreso moved net 899375.77 -> 900610.33 and
voiding returned it to 899375.77. VIEWER create/void both 403,
double-void 400. API compiles clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Utilities section becomes create/edit/archive-able, with its child data.
API:
- Property gains archivedAt (soft-delete); list/browser default to
archivedAt=null with ?includeArchived opt-in.
- PropertiesService: header create/update/archive/restore (customer FK
validated); PropertyService add/update/remove scoped to the property;
TrustAccount upsert (1:1) + remove; ServiceDocument pointer delete.
- Controller write routes: create needs STAFF+ (property:create), archive
MANAGER+ (property:delete), every service/trust/document route
property:update. Mutations audited. DTOs added.
- Document *upload* deliberately deferred: it needs the object-storage
client wired into the API (today only the migration writes to MinIO);
removing an existing pointer row is supported and the UI says so.
Web:
- PropertyForm (header) with CustomerPicker; /servicios/nuevo (accepts
?customerId prefill) and /servicios/[id]/editar.
- Property detail: gated action bar (Editar/Archivar) + "Administrar
propiedad" — services via the shared ChildCollection editor, an inline
1:1 TrustEditor (create/update/clear), and document-row delete.
- "Nueva propiedad" buttons on the list and customer detail (prefilled).
api.ts + types for all of it.
Verified against dev: property create (archivedAt null), service
add/update, VIEWER service-add 403, trust upsert (create then update the
same row), trust/service remove, cross-property child guard 404, archive
drops from the default list and includeArchived surfaces it. Both apps
compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Policy header, all five child collections, and the insurance reference
catalogs become create/edit/delete-able on the RBAC foundation.
API:
- Policy gains archivedAt (soft-delete); list/browser default to
archivedAt=null with ?includeArchived opt-in.
- PoliciesService: header create/update/archive/restore (customer FK
validated for a clean 404); add/update/remove for installments,
vehicles, drivers, beneficiaries, claims — each scoped to its policy so
one policy's id can't touch another's rows; lookups CRUD for providers,
policy types, adjusters.
- PoliciesController write routes: header create/update need STAFF+
(policy:create/update), archive/restore need MANAGER+ (policy:delete),
every child route needs policy:update. New LookupsController at /lookups
(read open; mutate needs lookup:manage / MANAGER+). Mutations audited.
- DTOs (policy header, children, lookups); dates coerced; shared coerce.ts.
Web:
- Generic ChildCollection editor (config-driven add/edit/remove table),
reused by both the policy detail child editors and the catalogs screen.
- PolicyForm (header) with type/provider selects and a debounced
CustomerPicker; /polizas/nuevo (accepts ?customerId prefill) and
/polizas/[id]/editar. Policy detail: gated action bar (Editar/Archivar)
+ "Administrar detalles" child editors for all five collections.
- /catalogos admin screen (aseguradoras/tipos/ajustadores), nav-gated on
lookup:manage. "Nueva póliza" buttons on the list and on the customer
detail (prefilled). api.ts + types for all of the above.
Verified against dev: policy create (dates coerced, archivedAt null),
installment/vehicle add, VIEWER child-add 403, cross-policy child guard
404, lookups CRUD with VIEWER 403 / MANAGER 201, archive drops from the
default list and includeArchived surfaces it. Both apps compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First master-data CRUD module on the phase-1 RBAC foundation.
API:
- Customer gains archivedAt (soft-delete marker, distinct from the legacy
`status` business flag); pushed to dev (nullable, non-destructive).
- CustomersService: create/update/archive/restore. list() and the browser
default to archivedAt=null; ?includeArchived=true opts in. App-created
rows set nameMissing=false and leave legacy provenance null.
- CustomersController write routes guarded per the matrix: create/update
need STAFF+ (customer:create/update), archive/restore need ADMIN
(customer:delete). Every mutation audit-logged.
- create/update DTOs (class-validator); date strings coerced to Date.
Web:
- Shared CustomerForm (create + edit) with identity/address/account
sections; new routes /clientes/nuevo and /clientes/[id]/editar, each
self-gated on the ability.
- List page: ability-gated "Nuevo cliente" button. Detail page: gated
Editar / Archivar (Restaurar) action bar; archived badge.
- api.ts create/update/archive/restore; CustomerInput type; archived flag
on list items.
Verified against dev: create (dates coerced, archivedAt null), edit 200,
VIEWER create 403, STAFF create 201 but archive 403, ADMIN archive drops
the row from the default list and includeArchived surfaces it, restore
returns it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the RBAC foundation the CRUD phases build on, and the first write
module (users). The platform was read-only: every controller was guarded
only by AuthenticatedGuard and UserRole was ADMIN|STAFF. The old PHP app
stored level+role but enforced neither, so this is a fresh design.
Permission model (server-authoritative):
- UserRole expanded to an ordered rank ADMIN > MANAGER > STAFF > VIEWER.
VIEWER is the read-only role; STAFF+ can write.
- auth/abilities.ts: ROLE_RANK + ABILITY_MIN matrix + can()/abilitiesFor().
- @RequireAbility decorator + AbilityGuard enforce it on write routes;
reads stay on AuthenticatedGuard so any logged-in user can read.
- /auth/login and /auth/me now return the resolved abilities map, so the
web gates its UI off one payload instead of duplicating the rules.
User management (ADMIN-only, ability "user:manage"):
- UsersService gains list/create/update/resetPassword (argon2), never
returns passwordHash; blocks self-deactivation and self-demotion;
maps duplicate email to 409.
- UsersController: GET/POST /users, PATCH /users/:id,
POST /users/:id/reset-password.
- Every mutation logged via new AuditService over the existing
ActivityLog model (global CommonModule).
Web:
- AuthContext + useAuth/useCan; AppShell provides the user and gates the
new "Usuarios" nav entry on user:manage; shows the user's role.
- /usuarios admin page: list + create/edit form + password reset +
active toggle, Spanish-first, reusing existing card/table/field styles.
Schema pushed to dev (enum only, non-destructive). Verified end-to-end
against dev: admin CRUD works, VIEWER writes 403 while reads 200,
self-lockout guards and duplicate-email 409 all hold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TRASPASOS PAYPAL's 7-figure balances exposed three layout bugs on
/estado-cuenta/[id], all invisible on normal small-figure customers:
- summary/línea cards: `summary-grid` uses auto-fill, so a single- or
two-currency card never widens past the min track (~190px) however wide
the page is. The 24px nowrap headline (-$7,028,533.44) overflowed the
card border. Widen the min track to 260px so the figure fits in full,
and drop the size to 22px. Explicitly no ellipsis — a truncated money
figure reads as a wrong number.
- `concept-list` had no horizontal padding (`.card` carries none), so the
concept totals sat flush on the card border. Pad it.
- the Cargo/Abono direction label was an inline span glued to the amount;
make it a block so it drops onto its own line. Same fix applied to the
movement-list page for consistency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the office's own bank-register browser over the migrated SCOTHIA
data (22,354 bank_transactions), the last self-contained feature module.
API (apps/api/src/bank):
- GET /bank register browser: search over concepto/reference/notes/
amountInWords; direction (income|expense|void), cleared and
date-range filters; 5 sorts; income/expense/net totals for
the whole filtered set, not just the page
- GET /bank/stats headline income/expense/net + counts, date span, pending
- GET /bank/facets year list for the period filter
- GET /bank/summary year and month rollups with a running net-movement figure
Web (/banco): "Movimientos" register + "Resumen por periodo" with year->month
drill-down; added to the AppShell nav as "Chequera".
Deliberately kept OUT of /estado-cuenta: this is the office's own money, not
customer balances, and the two are never summed or shown together.
No category/ramo dimension, and the deferred concept->ramo classifier is
dropped as won't-build: concepto is a payee name (0 of 22,354 match a
category) and TABLA RAMODOS is an expense chart of accounts + owner names,
not the insurance/servicios/fideicomiso split it was assumed to be, so a
classifier would invent data. Single currency (MXN); the "acumulado" is net
movement since the register opened (no opening balance in the source), not a
bank balance. Verified end-to-end in the browser; totals reconcile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
with the corrected de-dup rule, plus a box explaining why the original
verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
summed.
- Build sequencing step 6 marked done.
RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
"not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
worth knowing — run_all.py vs single transforms, `next build` clobbering a
running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
many sessions ago. Renumbered, with an explicit "next" block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 6 — the payoff of the unified customer record: a utility charge
and an insurance payment finally sit on the same page, under the same
person, with a running balance.
API (apps/api/src/billing/):
- GET /billing — cross-customer movement browser. Search over customer,
referencia, cheque, concepto and periodo; filters for business line,
currency, charge-vs-credit, concept, origin table and a from/to date
range; 5 sorts. Returns totals for the whole filtered set, not just the
page, so a filtered view can't be misread as the full ledger.
- GET /billing/balances — per-customer receivables worklist with
owing/credit/settled buckets and 4 sorts. Raw SQL (parameterized via
Prisma.sql): needs conditional sums per currency and per direction in
one pass plus ordering and pagination on a computed balance, none of
which groupBy expresses.
- GET /billing/stats, /billing/facets, /billing/customers/:id.
Web:
- /estado-cuenta — two views over the same ledger, because staff ask two
different questions: "Saldos por cliente" (who owes what) and
"Movimientos" (every charge and credit).
- /estado-cuenta/[id] — the statement: balance per currency, the same
balance split by business line, charges broken out by concept, and the
full movement list with a running balance.
- Cross-linked from the customer and property detail pages.
Two data findings shape the whole module:
1. transactions.amount is a signed ledger. Every charge type is negative
without exception (WATER 3115/3117, ELECTRIC 2191/2191, PROPERTY TAXES
926/926, TRUST FEE 188/188) and every deposit type positive (CHECK and
CASH DEPOSIT, PAYPAL, all of EFECTIVO). So SUM(amount) is the balance
and negative means the customer owes the office.
2. Currency is not summable. 912 of the 1269 customers with a ledger move
in both MXN and USD, the charge side is MXN-only while receipts arrive
in both, and no per-movement exchange rate was ever stored. A single
"total balance" would be a figure that never existed in the books, so
every total is reported per currency and the balance filter/sort takes
a currency argument rather than collapsing.
Also: type_transactions.nameEs is entirely null (the legacy TYPE OF TRX
ESPAÑOL column is empty in all 79 rows), so Spanish concept names come
from a label map in labels.ts; the entries that are payee names rather
than categories fall through untranslated, which is correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.
reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.
Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.
- reconcile.py: add canon(), which parses a key column to a number when
nearly every populated cell parses and re-emits it at fixed precision.
Applied in keyset() and in the folio-conflict comparison. Rewrite the
group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
two efectivo_like() calls. EFECTIVO loads first and wins collisions.
De-dup on the business key, never on folio — folio is per-table
sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
their verdicts are unchanged.
Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 5. Properties, services and trust accounts become a first-class
browser the way /polizas is for insurance.
API (apps/api/src/properties):
GET /properties search over address, customer, service account
number, meter, trust number and phones; filters for
service kind, municipality, trust bank, trust bucket
(with|without|active|expiring|expired|undated) and
hasServices; 5 sorts
GET /properties/stats properties/owners/services/trusts, renewal counts,
service mix per kind
GET /properties/facets kinds, municipalities, banks — all with counts
GET /properties/:id services, fideicomiso, linked policy, owner and
sibling properties, owner-level utility ledger
Web: /servicios (renewals-first browser, clickable stat cells and service-mix
strip) and /servicios/[id]. Property cards on /clientes/[id] and linked
properties on /polizas/[id] now navigate into it.
Data findings baked into the design:
- The trust deadline staff chase is trust_accounts.dueDate2 (DATMEX vence2),
one year after vence1 on 531 of 541 dated trusts: 18 due within 30 days,
119 already overdue. Every renewal bucket keys off dueDate2 alone.
- properties.zone is dead (1444 of 1519 null, the rest near-unique), so the
geographic filter is the municipality carried in the predial service's
notes (ROSARITO 566 / TIJUANA 221 / ENSENADA 152, 939/939 populated).
- PropertyService.notes means a different thing per kind (municipality, CFE
PAR/IMPAR cycle, gas supply type, cable provider) and is labelled as such.
- 240 of 1519 properties have no service rows at all — its own bucket.
Sorting by trust due date scopes to properties that have a trust, since MySQL
would otherwise float the ~966 trust-less NULLs above every real due date;
the sort label and the result meta both say so.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 4. Adds the policies API and the Spanish-first /polizas pages on
top of the customer records the customer module already exposes.
API (apps/api/src/policies):
- GET /policies — search over policy number, customer name, agent, vehicle
license plate, insured-driver name and legacy id; filters for vigencia
bucket, ramo, aseguradora and liquidation state; five sort orders.
- GET /policies/stats — bucket counts plus premium in force split by
currency (MXN and USD can't be summed).
- GET /policies/facets — ramos/aseguradoras with counts for the dropdowns.
- GET /policies/:id — full policy plus the owning customer.
Vigencia is derived from policyTo as active/expiring/expired/undated.
"undated" is a real bucket rather than an error case: 528 of the 2378
migrated policies carry no end date at all.
Web:
- /polizas — renewals-first browser; the stat cells double as vigencia
filters, with a secondary row for ramo, aseguradora and sort order.
- /polizas/[id] — vigencia hero, condiciones y primas, pagos, vehículos,
asegurados/beneficiarios, siniestros, the verbatim legacy coverage
columns, and documents.
- Nav gains Clientes | Pólizas with a real active state, and the two
modules cross-link in both directions.
Also fixes a display bug on the customer detail page: it headlined
policies.total, which is dead data — only 2 of 2378 rows are non-zero
(1585 are literally 0, 791 null), and one of those two is lower than its
own net premium. That rendered "$0.00 Total" on 1585 policies. Premium
headlines and the premium sort now use netPremium (2377/2378 populated);
total is shown only where it is non-zero, as raw source data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
144 customers owned zero properties, zero policies and zero transactions —
the legacy DATGRAL row exists but nothing in either business line ever
attached to it. They padded the staff customer list with rows that can't be
acted on. 27 were also nameless (dead ID slots); the other 117 have real
names and sometimes contact details, and read as never-activated prospects
or lapsed clients rather than junk. Removing both sets is a deliberate call.
Implemented as a separate step rather than a filter inside
transform_customers.py: emptiness is only knowable after properties, policies
and transactions have loaded, and deciding it there would mean re-deriving
each downstream transform's source-matching logic against the staged Parquet.
Runs after transform_transactions.py in run_all.py.
Safe by construction — a customer with no rows in any of the three tables has
nothing pointing at it, so the delete cannot orphan anything; only its own
customer_legacy_refs go with it. The step asserts zero orphans afterwards.
Every pruned customer is written to output/pruned_customers.csv with its
legacy provenance before the delete, and --dry-run reports without touching
anything. Nothing is unrecoverable: the Access sources are untouched and a
pipeline run without this step brings them all back.
Verified: full run_all.py pass ends at 1538 customers (from 1682), with
1519 properties / 2378 policies / 45861 transactions all intact and zero
orphans. 17 nameless customers remain, all of which carry real records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 44 customers with no recoverable name render as "(SIN NOMBRE)", and
ordering the list by name alone floated all of them to the top — "(" sorts
before every letter — so the first two screens of the customer browser were
nothing but placeholders. Small number, worst possible position.
Adds customers.nameMissing, set by the transform and used as the primary sort
key so those records land at the end of the list. Denormalized rather than
computed in the query because the list is paginated in SQL, so the ordering
has to be expressible as a column.
Applied to the dev DB as an ALTER + UPDATE in place (no truncate), so the
existing loaded data and its FKs were left alone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance),
which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)".
The blank is real — those cells are empty in the Access files, not lost in
extraction — but the rows mostly are not junk: 176 of the 257 carry a
property, a policy, or transactions.
The old PHP importer handled this by skipping blank-name rows outright
(jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was
worse than it looks: every other adapter resolved its customer FK through
the customer_mapping table those skipped rows never entered, so their
properties and policies were silently dropped (customerServiceAdapter.php:45)
and their transactions were written against customer_id 0
(customerBalanceAdapter.php:52). So: recover the name instead of skipping.
Names come from the secondary tables that still carry them, most trustworthy
first — UTILSEG (the office's own hand-maintained name <-> id cross-reference
spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy
rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow
the name its insurance record resolved to. Result: 213 of 257 recovered, 44
still genuinely nameless anywhere in the source.
customers.nameSource records which table each recovered name came from, so a
reconstructed name is never mistaken for one that was really on the record —
the list tags it "nombre recuperado", the detail header names the source, and
a still-unnamed customer renders muted italic instead of as a normal name.
Also fixes run_all.py: transform_properties and transform_policies truncate
service_documents/policy_documents, but blob_extract.py was not in the step
list, so a full re-run left the uploaded MinIO objects with no rows pointing
at them. Hit exactly that while reloading for this change.
Verified end-to-end: full pipeline re-run against dev reproduces every prior
count (1682 customers, 1519 properties, 2378 policies, 45861 transactions,
22354 bank rows, 70 documents) with zero orphans, and both apps build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First real frontend feature against the live Customer module API.
- login/ — session login form posting to /auth/login with credentials
included; the session cookie is what every subsequent request rides on.
- clientes/ — customer list with search and the cross-line stats header
(customers, utilities/insurance split, both-lines count).
- clientes/[id]/ — unified detail view: identity, properties + services,
policies, and transaction history for one customer, which is the whole
point of the migration (one record spanning both business lines).
- components/AppShell.tsx, lib/{api,labels,types}.ts — shared fetch wrapper
(always credentials: "include"), Spanish label maps for the enum values
the API returns, and the API response types.
- globals.css + layout.tsx — Spanish-first document (lang="es"), the type
scale, and the design tokens the pages share. Fonts load via <link> so an
offline build still renders on the system fallback stacks.
- page.tsx now redirects / to /clientes.
Also fixes pnpm-workspace.yaml: the allowBuilds map held pnpm's literal
placeholder text ("set this to true or false"), which made every install
fail with ERR_PNPM_IGNORED_BUILDS. Since pnpm 11 auto-installs before
running a script, that broke `pnpm start:dev` outright. Set the values to
true and dropped the superseded onlyBuiltDependencies list.
Verified: both apps build clean, and login -> /auth/me -> /customers/stats
round-trips against the dev database (1682 customers, 526 on both lines).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the unified Customer API against the migrated data:
- customers.service: list (search across name/email/phone/city/legacy id,
business-line filter, pagination + per-row _count flags), detail (identity +
legacyRefs + properties/services/trust + policies with installments/vehicles/
drivers/beneficiaries/claims/docs + recent transactions + a per-domain/
currency ledger summary), and stats.
- customers.controller: GET /customers, /customers/:id, /customers/stats,
guarded by AuthenticatedGuard. Registered in AppModule.
- Fix LocalAuthGuard to call super.logIn so a session is actually established
(login previously succeeded but persisted no session -> 403 afterwards).
- apps/api/scripts/seed-user.mjs: idempotent Argon2 admin seed.
Tooling: adopt pnpm as the package manager (machine npm is a pnpm shim that
ignores the workspaces field). Add pnpm-workspace.yaml (+ onlyBuiltDependencies
for argon2/prisma/nest native builds), switch the api's @jorgecuadros/database
dep to workspace:*, add @types/passport, track pnpm-lock.yaml, drop the stale
package-lock.json.
Verified end-to-end against the dev DB: login sets a session cookie; stats
returns 1682 customers / 526 both-lines / 45861 transactions; search + detail
return the full cross-line customer view (properties+services AND policies AND
a unified transaction statement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.
Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.
Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-minio.stack.yml: S3-compatible object storage (MinIO) for
the platform's document blobs, deployed to the cubex Swarm with the same
statefulness rules as the DB stack (named volume, pinned to the labeled node).
Parametrized for dev/prod as two stacks (dev API 9100/console 9101, prod
9000/9001). Dev deployed + bucket jorgecuadros-documents created.
migration/blob_extract.py: re-reads the LONGBINARY columns via mdb-export
-b hex (staging used -b strip), carves the embedded file out of the Access
OLE wrapper by locating its magic bytes (JPEG/PNG/PDF/GIF/TIFF) and trimming
trailing OLE junk, uploads to MinIO, and writes service_documents /
policy_documents pointer rows. Row->parent alignment uses mdb-export's
deterministic order (== staged _row_num) for policies and numer_id for
properties. Idempotent (truncate doc tables + overwrite by deterministic key);
--limit/--tables for test passes.
Validated on a limited pass: carved blobs are valid JPEGs (ffd8ff..ffd9)
correctly linked to their policies.
requirements.txt: add boto3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.
migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).
run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.
Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.
Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).
run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_properties.py loads properties, property_services and
trust_accounts from staged DATMEX/PROFILE, resolving each property's customer
FK through customer_legacy_refs. Services are derived from DATMEX's own
account/route/meter fields (the authoritative data); PROFILE flags — merged
best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only
refine each service's `active`. Trust accounts are 1:1 from DATMEX trust
fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4.
Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped),
3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 /
OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts —
counts track the PROFILE enrollment flags.
Reproducibility (asked: dev must be redoable in prod):
- migration/dbenv.py: single DB-target source = deploy/.env.<env>'s
DATABASE_URL; connect(env) + env_arg() (--env, default dev).
- transform_customers.py / transform_properties.py now take --env instead of
hardcoding .env.dev.
- migration/run_all.py: runs every step in dependency order against --env
(optional --stage re-extracts from Access first). Reproducing dev->prod is
`run_all.py --env prod` after deploying the prod stack + prisma db push.
All steps are idempotent (truncate+rebuild); re-run yields identical counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).
Loaded and validated against the dev DB (192.168.4.212:3307):
1682 customers (1172 utilities master + 510 insurance-only)
2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
560 insurance rows linked via num_util, 0 broken cross-refs
542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.
RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-db.stack.yml: canonical internal MySQL for the platform,
targeting the Portainer local endpoint on cubex (3-node Swarm). Parametrized
(MYSQL_PORT / MYSQL_SERVER_ID) so one file deploys both environments as two
stacks with Swarm-namespaced volumes:
dev -> jorgecuadros-dev-db :3307 server-id 11
prod -> jorgecuadros-prod-db :3306 server-id 1 (replication source)
Swarm-correct: named volume (no bind mount), pinned to one node via
node.labels.jorgecuadros_db==true (cubex labeled), binlog+GTID enabled from
the start so prod can be the VPS replication source without reconfigure.
Dev deployed and verified: MySQL 8.4.10 reachable at 192.168.4.212:3307,
gtid_mode ON, database jorgecuadros present. Secrets live in gitignored
deploy/.env.dev, injected via Portainer stack env at deploy time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macOS has no Access ODBC driver, so the pyodbc-based extract.py could not
run. Rewrite it to shell out to mdbtools (mdb-tables/mdb-export) while
keeping the public interface (connect/list_tables/read_table/
sanitize_column_name) unchanged, so load_staging.py and config.py are
untouched. connect() now returns the file path as the handle (mdbtools has
no persistent connection).
Behavior details:
- -b strip drops LONGBINARY/OLE bytes so blobs never corrupt the CSV
(documents are extracted separately in migration step 4).
- ISO date/datetime output (-D/-T); staging read as text (dtype=str),
only empty fields treated as null (keep_default_na=False) so literal
"NA"/"NULL" data strings survive.
- mdbtools reads deleted/corrupted records mdbtools omits rather than
aborting, so the old per-row skip loop is no longer needed.
Verified end-to-end: full staging load reproduces the original Windows
run (82 tables, 0 errors) with exact row counts (datos2 16000, EFECTIVO
13697, DATGRAL 1172/1070, DATMEX 1520) and recovers all 764 MULT rows
(the pyodbc path lost 1 to HY109 corruption). Accented-column tables
(PROPANO) read cleanly.
requirements.txt: drop pyodbc/pywin32 (Windows-only), keep pandas/pyarrow/
sqlalchemy/pymysql; document the Windows-only DAO catalog as historical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
docs/LEGACY_DATABASES.md documents all three source Access databases
(every table, column, type, and known data-quality quirk) generated
from a live read of the real files, so no Windows/Access driver is
needed to understand their structure going forward.
New migration/ tooling: catalog_schema.py connects to the real files
and walks every table (including excluded scratch tables);
render_catalog_md.py renders that into the doc's appendix. Raw output
checked in at migration/catalog.json so the doc can be regenerated
without touching Access again.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinas
