The first deploy attempt (run 705) died on "Input required and not supplied:
token", which names the action's input rather than the secret that was unset —
the repo had only REGISTRY_USERNAME and REGISTRY_PASSWORD, so every deploy
secret was missing on both workflows. That is also why the endpoint_id /
pull_image input-name bug had gone unnoticed: neither workflow had ever got
far enough to use them.
Both workflows now check their required secrets up front and fail listing the
ones that are empty. The scope=full-only secrets are only required when the
dispatch is actually scope=full.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the gap between "what tag did I deploy" and "what is actually running",
and gives the schema a history that can be reasoned about across releases.
Migrations
- Baseline the existing schema as 0000_init (migrate diff --from-empty). The
schema had only ever been applied with `prisma db push`, so no history
existed and schema state was disconnected from app version. Existing
databases must be baselined once with `migrate resolve --applied 0000_init`;
the workflows print this remedy on P3005.
- Run `prisma migrate deploy` as a deploy STEP, not the container CMD — as a
CMD, N replicas would race each other applying the same migration.
Version reporting
- GET /version on the API reports the APP_VERSION / GIT_SHA / BUILD_DATE that
build.yml already baked into both images but nothing ever read.
- The web footer shows the web build and flags an api/web mismatch. The two
cannot drift at build time (one matrix run) but can at deploy time.
- Both deploy workflows now fail if the running API does not report the tag
that was dispatched — a stack naming a tag is not proof of what is running.
- scripts/set-version.mjs stamps every package.json, which had all sat at
0.1.0 while real releases shipped as v1.x.
Pre-migrate backup
- deploy/scripts/pre-migrate-backup.mjs dumps the database from INSIDE the
still-running old API container over Portainer's Docker API, so the file
lands in the volume the Operaciones restore screen reads. A dump taken on
the CI runner would be unreachable by the only restore path we have.
Verifies the artefact with `gzip -t` before letting the migration proceed.
galactus
- deploy/galactus/*.compose.yml: standalone-Docker ports of the Swarm stacks.
Plain compose silently ignores `deploy:`, so restart_policy becomes
`restart: unless-stopped` — without it nothing returns after a host reboot.
- .gitea/workflows/deploy-galactus.yml drives endpoint 3 with its own secrets.
Fixes
- deploy.yml passed `endpoint_id` and `pull_image` to
cssnr/portainer-stack-deploy-action, which has no such inputs (they are
`endpoint` and `pull`). The endpoint was silently never set.
docs/DEPLOY_AND_MIGRATIONS.md documents expand/contract as the rule for schema
changes: Prisma has no down-migrations, so a code rollback never rolls the
schema back, and restoring the replication master from a dump diverges every
replica.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The office keeps more than one operating account (Utilities banks in MXN,
Seguros in USD), but bank_transactions was a single implicit MXN register by
design. Adds Bank/BankAccount and makes every read and write in the module
scoped to exactly one account.
Schema:
- Bank / BankAccount. Currency is fixed per account and BankTransaction has
no currency column of its own — a movement inherits its account's, the way
a real bank account doesn't mix currencies.
- BankTransaction.bankAccountId, required. A movement with no known account
isn't reconcilable against a statement.
- @@index([bankAccountId, transactionDate]): every read now filters by
account and orders/groups by date.
Migration:
- backfill_bank_accounts.py seeds Scotiabank + "Utilities — Scotiabank (MXN)"
and backfills all 22,669 existing rows onto it, then promotes the column to
NOT NULL and attaches the FK. Standalone because prisma db push cannot add
a required column to a populated table. Idempotent; re-running once a second
account exists does not re-point rows.
- run_all.py runs it (both modes) before transform_bank.py, which now resolves
the account by label and fails fast if it is missing.
API:
- ?bankAccountId= required on list/stats/facets/summary — not optional with an
"all accounts" default, since summing an MXN and a USD register repeats the
currency-collapsing mistake the billing module exists to prevent. Missing is
400, unknown is 404.
- facets() had no account clause at all and summary() has two raw-SQL rollups;
all three are now parameterised. Scoping only one of summary's queries would
leave the year list and its drill-down describing different books.
- New bank/accounts + bank/banks sub-resource under a MANAGER
bank:manage-accounts ability. currency is absent from the update DTO: booked
movements are denominated in it, so editing would re-denominate history.
Capture into a closed account is rejected.
Web:
- /banco gains an account picker (remembered per browser) and reads every
figure in the selected account's currency; the "single currency (MXN)"
doc-comment and the hardcoded MXN formatting are gone.
- New /banco/cuentas for banks and accounts. Accounts are closed, never
deleted — the FK is required, so deleting one would destroy its register.
- /inicio's chequera card names the account it is reading instead of implying
a single register.
Verified against dev + browser: a second USD account showed full read/write
isolation from the MXN register, whose totals were unchanged (22,669
movements, net 1,014,266.97).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two follow-ups to the text-size control.
Spacing now scales with the text. All padding, margin, gap and min-height
declarations in globals.css move from px to rem (263 declarations, converted
mechanically), so --ui-scale drives the whole layout rather than just the
glyphs. Deliberately left in px: border widths, which must stay hairlines;
box-shadow offsets; border-radius, which reads as bloated when scaled on large
cards; --shell-max, a container cap that must not outgrow the viewport; and
media-query breakpoints, which are conditions rather than declarations. With
spacing following along, the presets gain a 1.5 "Máximo" step and MAX_UI_SCALE
rises from 1.4.
The preference now lives on the account instead of only in one browser.
User.uiScale (Float, default 1) is added to the schema and to the safe select,
so it rides along on /auth/login and /auth/me. PATCH /auth/preferences writes
it, guarded by AuthenticatedGuard only — every role including VIEWER may set
their own, and the target is always the session's user id, never a body
parameter, so this cannot be used to touch another account. The global
ValidationPipe's whitelist rejects any extra field, so role cannot ride in
alongside uiScale.
localStorage stays, demoted to a pre-paint cache for the layout.tsx script;
AppShell reconciles it against the account once /auth/me answers, with the
account winning. FontScaleControl becomes a controlled component since the
same value is now edited from the appbar and the drawer.
Verified against the dev API: PATCH persists and is reflected by a subsequent
/auth/me, out-of-range values are rejected 400, and an extra "role" field in
the body is rejected 400.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The appbar had grown to 11 flat links with no responsive behaviour, and
overflowed below ~1100px.
Nav is now 7 top-level entries: Inicio, Clientes, Pólizas, Propiedades,
Reportes stay one click away, while the movement screens (Captura, Estado de
cuenta, Chequera) and the admin screens (Catálogos, Usuarios, Operaciones)
collapse into "Cobranza" and "Admin" dropdowns. Groups are ability-filtered
and disappear entirely when the user can see none of their items, so VIEWER
never renders an empty Admin menu. activeHref now scans the flattened link
list, and a group trigger highlights while one of its children is current.
Below 980px the nav collapses to a burger drawer that lists every group
expanded, closing on navigation and on Escape.
Text size is user-adjustable app-wide. Every font-size in globals.css is
converted from px to rem (mechanically, 133 declarations) and the root size
becomes calc(100% * var(--ui-scale)), so one variable on <html> rescales the
whole UI. The preference persists in localStorage and is applied by a
pre-hydration script in layout.tsx to avoid a flash at the default size; the
Aa control lives in the appbar and, as a segmented row, in the drawer.
Spacing stays in px by design, which is why 1.3 is the largest preset.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implements docs/RECEIPT_CAPTURE_SPEC.md §1, the legacy "Editor"
replacement, on top of the single-movement capture from plan step 6.
No new abilities: batching and resolving are both capturing.
- outstanding (legacy NOPAGO): capture flag, ?outstanding= filter, and
POST /billing/:id/resolve-outstanding (gated ledger:create, not
ledger:void — resolving completes a capture rather than reversing
one). Outstanding rows are excluded from every balance aggregate,
matching the legacy SALDOS ULTIMO 0 query's HAVING NOPAGO = 0, but
still count in the movement browser's filtered totals.
- POST /billing/batch: many customers' receipts against one check, in
one $transaction. Deliberately not a persisted batch entity —
checkNumber is already a column and grouping by it answers every
legacy by-check query.
- GET /billing/by-check + a cheque-count report, replacing REPORTE
CHEQUE COUNT / REPORTE POR CHEQUE / EDITA CHEQUE ALF|COUNT|NUM. Print,
PDF, CSV and XLSX come free from the existing /reportes/:slug machinery.
- Web: /estado-cuenta/lote (the Editor screen, with live reconciliation
against the physical check amount), an "Estado de pago" filter, a
"sin fondos" row tag and a Resolver dialog, plus a top-level "Captura"
nav entry.
Integration seam for the OCR auto-capture module (spec §2), which is
required to post through createBatch rather than writing Transaction
rows itself: items[i] maps to lines[i] so postedTransactionId can be
zipped back on; opts.refs[i] stamps captureRef with a duplicate-post
guard that a voided row deliberately does not block; opts.source is
service-level only, so an HTTP client cannot label hand-keyed rows as
machine-captured. captureSource/captureRef are nullable so the 40,136
migrated rows stay NULL rather than being mislabelled.
Fixes two pre-existing bugs found while building this:
- statement() filtered legacySourceTable with `notIn`, which compiles to
SQL NOT IN — and `NULL NOT IN (...)` is NULL, so every app-captured
movement was invisible on the customer statement (438 rows in the
movement browser vs 392 on the statement) while showing everywhere
else. This would have made the whole capture feature look broken.
- The balances count query omitted the void filter its own page query
applied, so the total disagreed with the rows.
Nav highlighting now resolves by longest match; the previous
first-startsWith logic lit up both the parent and any nested entry.
Verified end-to-end against the dev DB, API and browser; all test rows
removed afterwards. Also corrects RESUME.md, which documented the dev
ports as :3001/:3000 — they are :4501/:4500, from the env files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Companion to docs/RECEIPT_CAPTURE_SPEC.md — the insurance half of the
2026-07-25/26 meeting. Documentation only; no application code.
Verified against the code and a live query of the dev DB rather than
designed from the meeting notes alone, which changed several conclusions:
- Renewal emails and the liquidación batch are much smaller than they
look. RenewalNotice + its @@unique([policyId, generation]) idempotency
key and the aviso-renovacion letter body already exist; the per-policy
liquidation fields are wired end to end. What's missing is a scheduler,
a mail client, and the batch layer.
- Carrier research: ANA and GMX are one company (Grupo Valore). ANA
exposes a live SOAP service with a published operation list; GMX
publishes no machine interface at all. Every ANA operation serves
new-business quoting/issuance, not "list my book" — so the direction
question decides whether the feature is buildable.
- UTILSEG is unusable for Utilities↔Seguros reconciliation and the spec
closes that long-standing open question: DATGRAL.[NUM UTIL] is
authoritative (name match 298/563 vs 58/1024), and where the two
sources overlap they contradict on 170 of 218 shared ids.
Also records two live defects found while verifying: policy_types is
missing its INCENDIO and M_EMPR rows (the FK is ON DELETE SET NULL, so 5
m_empr policies silently lost their ramo), and the legacy settlement
slots don't match the target model (MULT/INCENDIO carry two, M EMPR
carries four, Policy collapses to one).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Points PLAN.md at docs/RECEIPT_CAPTURE_SPEC.md and surfaces its open
design questions (OCR provider, Seguros bank details, clave catastral
vs. predial, recycling triggers) separately from the existing ops-only
open items list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Forward implementation spec covering the legacy "Editor" receipt-capture
workflow plus three net-new requests from the 2026-07-25/26 meeting with
Jorge: PDF/OCR auto-capture, multi-bank chequera support, and
customer-number recycling. Matching logic and data-model gaps for each
were verified against the actual migration scripts and API code, not
just the schema comments.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`r.netPremium` / `r.total` are strings, so an empty-string value leaked
""` into the JSX instead of rendering nothing. Wrap in Boolean() so the
guard is a real conditional.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces ~40 legacy Access renewal-notice report clones (one per carrier
per coverage tier, e.g. AMPL/RC/LIC RENEW X MES/VENCE ATLAS 13/2013) with
one parameterized aviso-renovacion report driven by real Policy/Vehicle/
coveragesJson data instead of hand-typed label text per clone.
- schema.prisma: add RenewalNotice, replacing the legacy CONTROL <ramo>
RENEW[2/3] X MES paper log of which notice generation was sent
- reports: new "letter" ReportFormat + aviso-renovacion registry entry +
LetterLayout renderer in ReportRunner.tsx
- docs/RENEWAL_NOTICES.md + migration/legacy_report_defs/: extracted (via
Application.SaveAsText, since the VBA project wouldn't load) and
documented the legacy report/query chain this replaces
Coveragesjson key names and a mark-as-sent mutation are still unverified/
unbuilt — see caveats in docs/RENEWAL_NOTICES.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The --sync path had never been run and was broken in several ways. Fixed and
verified against the dev DB (two consecutive syncs, both exit 0, 32/32
assertions: stable PKs, manual-row preservation, changed-row updates,
legacy-delete, no child duplication, zero FK orphans; idempotent).
- policies/properties: reuse each legacy row's existing id (by provenance)
BEFORE building child rows, so children no longer point at a discarded fresh
uuid; rebuild legacy-owned children via scoped delete + reinsert.
- customers: replace zip(customers, refs) (mispaired almost every row) with a
ref-grouped id remap; names now restore and no spurious customers appear.
- drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy
policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert.
- upsert lookup tables (policy_types, insurance_providers, type_transactions,
adjusters) by natural name and remap child FKs instead of inserting fresh
uuids that nothing points at.
- transactions: drop updatedAt=NOW() (no such column); guard report formatting
on NULL legacySourceTable (manual rows). Same report guard in bank.
- add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune
only legacy-owned empties, never manually-added customers.
web: customer-detail mini tx list now strikes voided rows with an "(anulado)"
tag (was the last void-UI rendering gap; /estado-cuenta already handled it).
docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI
browser pass recorded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New reports backend (registry, service, controller, outputs, types)
with catalog endpoint + slug/CSV/XLSX/PDF/print outputs.
- /reportes catalog + /reportes/[slug] runner; ReportRunner + ContextReports
components wire pre-filtered links from domain pages.
- Fix: /reportes/[slug] now reads searchParams and forwards initialParams to
ReportRunner so /reportes/edo-cuenta-datos?customerId=... auto-runs
instead of dropping the id and forcing a manual customer search.
- /inicio landing page; root + login redirect to /inicio.
- Company header env vars + logo asset for PDF/print rendering.
- exceljs + pdfkit deps.
Replace the JC text mark in AppShell and login with the real
company_logo.png. Restyle .brand-mark to host the image (white
rounded bg, object-fit contain). Appbar 38px, login panel 46px.
Add the missing api/web deployment path on top of the existing image build CI.
- deploy/jorgecuadros-app.stack.yml: PROD app stack (api + web) pulling the
git.mancinas.io registry images. Does not ship mysql/minio (separate stacks);
API reaches them via DATABASE_URL / S3_ENDPOINT. API pinned to the
jorgecuadros_db node for stable ingest/backup volumes; web is stateless.
- deploy/jorgecuadros-app.env.example: documented stack env template.
- .gitea/workflows/deploy.yml: manual (workflow_dispatch) deploy to Portainer
via cssnr/portainer-stack-deploy-action. Inputs: image tag + scope
(app = web+api, full = db+minio+app, applied db->minio->app).
Make the web API origin runtime-configurable instead of build-baked: the root
layout injects window.__API_ORIGIN__ from the API_ORIGIN env (force-dynamic) and
lib/api.ts resolves it at runtime, so one built image serves any deployment.
Also: dev.sh to run both dev servers (frees stale ports first) and move local
dev to ports web 4500 / api 4501.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The schema has carried `storageKey` pointers and the migration has written
blobs to MinIO since day one, but the API had no S3 client — documents could
only be deleted, never uploaded or retrieved. This adds the missing wiring.
API
- StorageModule/StorageService (@aws-sdk/client-s3, path-style for MinIO):
put/getStream/delete, best-effort bucket ensure on boot, gracefully disabled
when S3 env is absent (ServiceUnavailable on use).
- Reads S3_ENDPOINT/S3_BUCKET + S3_ACCESS_KEY/S3_SECRET_KEY, falling back to
MINIO_ROOT_USER/MINIO_ROOT_PASSWORD so one credential set drives both the
migration and the API.
- Property service documents: POST :id/documents (multipart), GET
:id/documents/:childId/download (streamed), delete now also drops the blob.
- Policy documents: same upload/download/delete (previously had none).
- Keys stay under the service/<id>/… and policy/<id>/… prefixes the migration
established.
Web
- api.ts: shared uploadFile() helper (uploadIngest refactored onto it),
upload/download/remove helpers for property & policy documents.
- Servicios, polizas, clientes detail pages: real Descargar links and an
upload control (gated by policy:update / property:update) replacing the
"storage pending" notes.
Infra
- docker-compose: minio service (9000/9001, healthcheck, named volume) + S3
env wired into the api service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Working-tree checkpoint of in-progress work carried across prior
sessions on the feat/crud-rbac branch, committed so it lands on the
remote alongside the CI changes.
- Operaciones admin panel: apps/api/src/ops (ingest upload, backup /
restore / re-import jobs) wired into app.module + RBAC abilities, and
the apps/web/src/app/operaciones page. docker-compose gets INGEST_DIR
/ BACKUP_DIR volumes; .gitignore excludes migration/ingest + backups.
- migration/sync.py plus transform_*.py / run_all / config / dbenv /
blob_extract adjustments for the additive sync path.
- crud/rbac phase-5 web bits: AppShell, api/labels/types libs, globals.
- schema.prisma + PLAN/RESUME doc updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add comprehensive Docker image versioning and a Gitea Actions workflow
that builds and pushes both the API and web images to the
git.mancinas.io registry.
Versioning: both Dockerfiles take APP_VERSION / GIT_SHA / BUILD_DATE
build-args, surfaced as runtime ENV + OCI labels, so a running container
self-reports the exact commit it was built from. metadata-action emits a
tag set per build: semver (from vX.Y.Z git tags), branch ref,
sha-<short>, and latest (default branch only).
Also fix the Dockerfiles for the pnpm workspace: the old npm install
could not resolve the "@jorgecuadros/database": "workspace:*" protocol
dep and would abort the API build. Now pin pnpm 9.15.9 via corepack,
install --frozen-lockfile with node-linker=hoisted (flat tree so the
runtime stage copies a single node_modules), and build via --filter. The
API build stage gets python3/make/g++ for argon2's musl source compile.
Add .dockerignore to keep the build context lean and deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Covers stack, repo layout, local dev (pnpm install, env, MySQL via
docker, prisma db push, seed admin, run api+web, login), full-stack
Docker path, common commands, auth/roles, legacy migration, and prod
notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The add/edit form was appended after the whole table, so on long lists
(e.g. /catalogos aseguradoras, 16+ rows) clicking Editar on a top row
opened the form far below the fold — appearing to do nothing. Render the
edit form in place of its row, and the add form as the first table row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes phase 5 — the ledger and chequera pages get the append+void
UI on top of the phase-5 API.
Web:
- Shared MovementForm (customer picker + línea + cargo/abono sign + amount
+ moneda + concepto facet + periodo/referencia/cheque/mensaje). Used by
both /estado-cuenta (cross-customer, picker) and /estado-cuenta/[id]
(customer prefilled).
- /estado-cuenta and /estado-cuenta/[id]: "Capturar movimiento" toggle
gated ledger:create; per-row "Anular" gated ledger:void; voided rows
struck-through. Save/void refresh the list + stats.
- /banco: inline BankCaptureForm (ingreso/egreso sign, cheque, operado,
transferencia, monto en letras) gated bank:create; per-row "Anular"
gated bank:void; voided rows struck-through.
- api.ts: createMovement/voidMovement, createBankMovement/voidBankMovement;
CreateMovementInput/CreateBankMovementInput types; `voided` on the
movement/statement/bank list items.
Also: lookups.controller.ts now audit-logs provider/policy-type/adjuster
create/update/delete (parity with the other write controllers).
API + web compile clean. This is the last piece of the feat/crud-rbac
branch — all five sections plus users are now full CRUD with role gating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Transactions and the bank register become append-only with a void
(reversal) action — never edited or hard-deleted. This is the API half of
phase 5; the capture/void web UI is the remaining piece.
Schema:
- Transaction and BankTransaction gain voidedAt + voidedById. A non-null
voidedAt reverses the row. Pushed to dev.
Correctness (the high-stakes part):
- Every aggregate excludes voided rows: billing movements totals, the raw
balances SQL, stats (groupBy + the sides/crossLine raw subqueries +
first/last), facets (types/sources/years); the statement's running
balance freezes on a voided row and its per-currency/per-domain/per-type
summaries skip them; customers.detail and property owner-ledger groupBy;
and every bank total (totalsFor, stats counts/bounds, facets + summary
raw SQL). List views still return voided rows with a `voided` flag so
the UI can strike them through.
- Bank's legacy zero-amount "void" cheques are unchanged and distinct from
app voids (voidedAt).
API:
- POST /billing + POST /billing/:id/void (ledger:create / ledger:void);
POST /bank + POST /bank/:id/void (bank:create / bank:void). Create needs
STAFF+, void needs MANAGER+. Double-void -> 400, unknown id -> 404,
bad date -> 400. Mutations audited. DTOs added.
Verified against dev end-to-end: a -500 MXN charge moved a customer
balance 31082.08 -> 30582.08, and voiding it returned it to 31082.08 to
the cent; a +1234.56 bank ingreso moved net 899375.77 -> 900610.33 and
voiding returned it to 899375.77. VIEWER create/void both 403,
double-void 400. API compiles clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Utilities section becomes create/edit/archive-able, with its child data.
API:
- Property gains archivedAt (soft-delete); list/browser default to
archivedAt=null with ?includeArchived opt-in.
- PropertiesService: header create/update/archive/restore (customer FK
validated); PropertyService add/update/remove scoped to the property;
TrustAccount upsert (1:1) + remove; ServiceDocument pointer delete.
- Controller write routes: create needs STAFF+ (property:create), archive
MANAGER+ (property:delete), every service/trust/document route
property:update. Mutations audited. DTOs added.
- Document *upload* deliberately deferred: it needs the object-storage
client wired into the API (today only the migration writes to MinIO);
removing an existing pointer row is supported and the UI says so.
Web:
- PropertyForm (header) with CustomerPicker; /servicios/nuevo (accepts
?customerId prefill) and /servicios/[id]/editar.
- Property detail: gated action bar (Editar/Archivar) + "Administrar
propiedad" — services via the shared ChildCollection editor, an inline
1:1 TrustEditor (create/update/clear), and document-row delete.
- "Nueva propiedad" buttons on the list and customer detail (prefilled).
api.ts + types for all of it.
Verified against dev: property create (archivedAt null), service
add/update, VIEWER service-add 403, trust upsert (create then update the
same row), trust/service remove, cross-property child guard 404, archive
drops from the default list and includeArchived surfaces it. Both apps
compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Policy header, all five child collections, and the insurance reference
catalogs become create/edit/delete-able on the RBAC foundation.
API:
- Policy gains archivedAt (soft-delete); list/browser default to
archivedAt=null with ?includeArchived opt-in.
- PoliciesService: header create/update/archive/restore (customer FK
validated for a clean 404); add/update/remove for installments,
vehicles, drivers, beneficiaries, claims — each scoped to its policy so
one policy's id can't touch another's rows; lookups CRUD for providers,
policy types, adjusters.
- PoliciesController write routes: header create/update need STAFF+
(policy:create/update), archive/restore need MANAGER+ (policy:delete),
every child route needs policy:update. New LookupsController at /lookups
(read open; mutate needs lookup:manage / MANAGER+). Mutations audited.
- DTOs (policy header, children, lookups); dates coerced; shared coerce.ts.
Web:
- Generic ChildCollection editor (config-driven add/edit/remove table),
reused by both the policy detail child editors and the catalogs screen.
- PolicyForm (header) with type/provider selects and a debounced
CustomerPicker; /polizas/nuevo (accepts ?customerId prefill) and
/polizas/[id]/editar. Policy detail: gated action bar (Editar/Archivar)
+ "Administrar detalles" child editors for all five collections.
- /catalogos admin screen (aseguradoras/tipos/ajustadores), nav-gated on
lookup:manage. "Nueva póliza" buttons on the list and on the customer
detail (prefilled). api.ts + types for all of the above.
Verified against dev: policy create (dates coerced, archivedAt null),
installment/vehicle add, VIEWER child-add 403, cross-policy child guard
404, lookups CRUD with VIEWER 403 / MANAGER 201, archive drops from the
default list and includeArchived surfaces it. Both apps compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First master-data CRUD module on the phase-1 RBAC foundation.
API:
- Customer gains archivedAt (soft-delete marker, distinct from the legacy
`status` business flag); pushed to dev (nullable, non-destructive).
- CustomersService: create/update/archive/restore. list() and the browser
default to archivedAt=null; ?includeArchived=true opts in. App-created
rows set nameMissing=false and leave legacy provenance null.
- CustomersController write routes guarded per the matrix: create/update
need STAFF+ (customer:create/update), archive/restore need ADMIN
(customer:delete). Every mutation audit-logged.
- create/update DTOs (class-validator); date strings coerced to Date.
Web:
- Shared CustomerForm (create + edit) with identity/address/account
sections; new routes /clientes/nuevo and /clientes/[id]/editar, each
self-gated on the ability.
- List page: ability-gated "Nuevo cliente" button. Detail page: gated
Editar / Archivar (Restaurar) action bar; archived badge.
- api.ts create/update/archive/restore; CustomerInput type; archived flag
on list items.
Verified against dev: create (dates coerced, archivedAt null), edit 200,
VIEWER create 403, STAFF create 201 but archive 403, ADMIN archive drops
the row from the default list and includeArchived surfaces it, restore
returns it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the RBAC foundation the CRUD phases build on, and the first write
module (users). The platform was read-only: every controller was guarded
only by AuthenticatedGuard and UserRole was ADMIN|STAFF. The old PHP app
stored level+role but enforced neither, so this is a fresh design.
Permission model (server-authoritative):
- UserRole expanded to an ordered rank ADMIN > MANAGER > STAFF > VIEWER.
VIEWER is the read-only role; STAFF+ can write.
- auth/abilities.ts: ROLE_RANK + ABILITY_MIN matrix + can()/abilitiesFor().
- @RequireAbility decorator + AbilityGuard enforce it on write routes;
reads stay on AuthenticatedGuard so any logged-in user can read.
- /auth/login and /auth/me now return the resolved abilities map, so the
web gates its UI off one payload instead of duplicating the rules.
User management (ADMIN-only, ability "user:manage"):
- UsersService gains list/create/update/resetPassword (argon2), never
returns passwordHash; blocks self-deactivation and self-demotion;
maps duplicate email to 409.
- UsersController: GET/POST /users, PATCH /users/:id,
POST /users/:id/reset-password.
- Every mutation logged via new AuditService over the existing
ActivityLog model (global CommonModule).
Web:
- AuthContext + useAuth/useCan; AppShell provides the user and gates the
new "Usuarios" nav entry on user:manage; shows the user's role.
- /usuarios admin page: list + create/edit form + password reset +
active toggle, Spanish-first, reusing existing card/table/field styles.
Schema pushed to dev (enum only, non-destructive). Verified end-to-end
against dev: admin CRUD works, VIEWER writes 403 while reads 200,
self-lockout guards and duplicate-email 409 all hold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TRASPASOS PAYPAL's 7-figure balances exposed three layout bugs on
/estado-cuenta/[id], all invisible on normal small-figure customers:
- summary/línea cards: `summary-grid` uses auto-fill, so a single- or
two-currency card never widens past the min track (~190px) however wide
the page is. The 24px nowrap headline (-$7,028,533.44) overflowed the
card border. Widen the min track to 260px so the figure fits in full,
and drop the size to 22px. Explicitly no ellipsis — a truncated money
figure reads as a wrong number.
- `concept-list` had no horizontal padding (`.card` carries none), so the
concept totals sat flush on the card border. Pad it.
- the Cargo/Abono direction label was an inline span glued to the amount;
make it a block so it drops onto its own line. Same fix applied to the
movement-list page for consistency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the office's own bank-register browser over the migrated SCOTHIA
data (22,354 bank_transactions), the last self-contained feature module.
API (apps/api/src/bank):
- GET /bank register browser: search over concepto/reference/notes/
amountInWords; direction (income|expense|void), cleared and
date-range filters; 5 sorts; income/expense/net totals for
the whole filtered set, not just the page
- GET /bank/stats headline income/expense/net + counts, date span, pending
- GET /bank/facets year list for the period filter
- GET /bank/summary year and month rollups with a running net-movement figure
Web (/banco): "Movimientos" register + "Resumen por periodo" with year->month
drill-down; added to the AppShell nav as "Chequera".
Deliberately kept OUT of /estado-cuenta: this is the office's own money, not
customer balances, and the two are never summed or shown together.
No category/ramo dimension, and the deferred concept->ramo classifier is
dropped as won't-build: concepto is a payee name (0 of 22,354 match a
category) and TABLA RAMODOS is an expense chart of accounts + owner names,
not the insurance/servicios/fideicomiso split it was assumed to be, so a
classifier would invent data. Single currency (MXN); the "acumulado" is net
movement since the register opened (no opening balance in the source), not a
bank balance. Verified end-to-end in the browser; totals reconcile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PLAN.md:
- Migration step 2: replace the "near-disjoint ledgers, migrate both" rule
with the corrected de-dup rule, plus a box explaining why the original
verdict was wrong so the reversal is auditable rather than silent.
- Note that transactions.amount is signed and that currencies are never
summed.
- Build sequencing step 6 marked done.
RESUME.md — the execution queue still stated the reverted EFECTIVO verdict
verbatim, so a fresh session reading top-to-bottom would have hit the old
rule in step 3 and the correction in step 4 with no way to tell which won.
Beyond that fix, several sections still described the pre-macOS-move world:
- §2: every source path was C:\Users\ricar\...; the repo was described as
"not yet a git repository".
- §4.4: described the pyodbc + Access ODBC extraction rather than mdbtools.
- §6: four of five "open items" were already resolved.
- §7: documented the old Windows box. Now the macOS machine, plus the traps
worth knowing — run_all.py vs single transforms, `next build` clobbering a
running dev server's .next, and the mdb-export numeric formatting trap.
- §8: items were mis-numbered (5b before 5) and item 5 was work finished
many sessions ago. Renumbered, with an explicit "next" block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 6 — the payoff of the unified customer record: a utility charge
and an insurance payment finally sit on the same page, under the same
person, with a running balance.
API (apps/api/src/billing/):
- GET /billing — cross-customer movement browser. Search over customer,
referencia, cheque, concepto and periodo; filters for business line,
currency, charge-vs-credit, concept, origin table and a from/to date
range; 5 sorts. Returns totals for the whole filtered set, not just the
page, so a filtered view can't be misread as the full ledger.
- GET /billing/balances — per-customer receivables worklist with
owing/credit/settled buckets and 4 sorts. Raw SQL (parameterized via
Prisma.sql): needs conditional sums per currency and per direction in
one pass plus ordering and pagination on a computed balance, none of
which groupBy expresses.
- GET /billing/stats, /billing/facets, /billing/customers/:id.
Web:
- /estado-cuenta — two views over the same ledger, because staff ask two
different questions: "Saldos por cliente" (who owes what) and
"Movimientos" (every charge and credit).
- /estado-cuenta/[id] — the statement: balance per currency, the same
balance split by business line, charges broken out by concept, and the
full movement list with a running balance.
- Cross-linked from the customer and property detail pages.
Two data findings shape the whole module:
1. transactions.amount is a signed ledger. Every charge type is negative
without exception (WATER 3115/3117, ELECTRIC 2191/2191, PROPERTY TAXES
926/926, TRUST FEE 188/188) and every deposit type positive (CHECK and
CASH DEPOSIT, PAYPAL, all of EFECTIVO). So SUM(amount) is the balance
and negative means the customer owes the office.
2. Currency is not summable. 912 of the 1269 customers with a ledger move
in both MXN and USD, the charge side is MXN-only while receipts arrive
in both, and no per-movement exchange rate was ever stored. A single
"total balance" would be a figure that never existed in the books, so
every total is reported per currency and the balance filter/sort takes
a currency argument rather than collapsing.
Also: type_transactions.nameEs is entirely null (the legacy TYPE OF TRX
ESPAÑOL column is empty in all 79 rows), so Spanish concept names come
from a label map in labels.ts; the entries that are payee names rather
than categories fall through untranslated, which is correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconciliation pass ruled EFECTIVO and EFECTIVO_BACKUP "near-disjoint
ledgers" and the transform loaded both in full. That verdict was a bug, not
a finding.
reconcile.py compared the business key (cl, fecha, monto, conepto) as raw
strings, on the stated premise that "every table went through the same
mdb-export path, so identical source values serialize identically". They
don't: mdb-export formats a numeric column from its Access column type, so
the same amount is emitted as `5000` from one table and `27000.0000` from
the other. No two rows could ever match on `monto`, which is why the pass
reported 2 overlapping rows.
Canonicalizing numeric key columns first shows 12386 of EFECTIVO_BACKUP's
12387 rows already exist verbatim in EFECTIVO — same customer, same
timestamp to the second, same amount, same concept text — leaving exactly
one genuinely new row. The ledger was carrying 12386 duplicated payments,
roughly doubling every customer's historical receipt total.
- reconcile.py: add canon(), which parses a key column to a number when
nearly every populated cell parses and re-emits it at fixed precision.
Applied in keyset() and in the folio-conflict comparison. Rewrite the
group-1 verdict and the module docstring's method note.
- transform_transactions.py: share a business-key `seen` set between the
two efectivo_like() calls. EFECTIVO loads first and wins collisions.
De-dup on the business key, never on folio — folio is per-table
sequential and collides on 12204 different payments.
- Regenerate RECONCILIATION.md. Groups 2 and 3 re-checked under the fix;
their verdicts are unchanged.
Ledger after re-running run_all.py --env dev: 45861 -> 33475 rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 5. Properties, services and trust accounts become a first-class
browser the way /polizas is for insurance.
API (apps/api/src/properties):
GET /properties search over address, customer, service account
number, meter, trust number and phones; filters for
service kind, municipality, trust bank, trust bucket
(with|without|active|expiring|expired|undated) and
hasServices; 5 sorts
GET /properties/stats properties/owners/services/trusts, renewal counts,
service mix per kind
GET /properties/facets kinds, municipalities, banks — all with counts
GET /properties/:id services, fideicomiso, linked policy, owner and
sibling properties, owner-level utility ledger
Web: /servicios (renewals-first browser, clickable stat cells and service-mix
strip) and /servicios/[id]. Property cards on /clientes/[id] and linked
properties on /polizas/[id] now navigate into it.
Data findings baked into the design:
- The trust deadline staff chase is trust_accounts.dueDate2 (DATMEX vence2),
one year after vence1 on 531 of 541 dated trusts: 18 due within 30 days,
119 already overdue. Every renewal bucket keys off dueDate2 alone.
- properties.zone is dead (1444 of 1519 null, the rest near-unique), so the
geographic filter is the municipality carried in the predial service's
notes (ROSARITO 566 / TIJUANA 221 / ENSENADA 152, 939/939 populated).
- PropertyService.notes means a different thing per kind (municipality, CFE
PAR/IMPAR cycle, gas supply type, cable provider) and is labelled as such.
- 240 of 1519 properties have no service rows at all — its own bucket.
Sorting by trust due date scopes to properties that have a trust, since MySQL
would otherwise float the ~966 trust-less NULLs above every real due date;
the sort label and the result meta both say so.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plan step 4. Adds the policies API and the Spanish-first /polizas pages on
top of the customer records the customer module already exposes.
API (apps/api/src/policies):
- GET /policies — search over policy number, customer name, agent, vehicle
license plate, insured-driver name and legacy id; filters for vigencia
bucket, ramo, aseguradora and liquidation state; five sort orders.
- GET /policies/stats — bucket counts plus premium in force split by
currency (MXN and USD can't be summed).
- GET /policies/facets — ramos/aseguradoras with counts for the dropdowns.
- GET /policies/:id — full policy plus the owning customer.
Vigencia is derived from policyTo as active/expiring/expired/undated.
"undated" is a real bucket rather than an error case: 528 of the 2378
migrated policies carry no end date at all.
Web:
- /polizas — renewals-first browser; the stat cells double as vigencia
filters, with a secondary row for ramo, aseguradora and sort order.
- /polizas/[id] — vigencia hero, condiciones y primas, pagos, vehículos,
asegurados/beneficiarios, siniestros, the verbatim legacy coverage
columns, and documents.
- Nav gains Clientes | Pólizas with a real active state, and the two
modules cross-link in both directions.
Also fixes a display bug on the customer detail page: it headlined
policies.total, which is dead data — only 2 of 2378 rows are non-zero
(1585 are literally 0, 791 null), and one of those two is lower than its
own net premium. That rendered "$0.00 Total" on 1585 policies. Premium
headlines and the premium sort now use netPremium (2377/2378 populated);
total is shown only where it is non-zero, as raw source data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
144 customers owned zero properties, zero policies and zero transactions —
the legacy DATGRAL row exists but nothing in either business line ever
attached to it. They padded the staff customer list with rows that can't be
acted on. 27 were also nameless (dead ID slots); the other 117 have real
names and sometimes contact details, and read as never-activated prospects
or lapsed clients rather than junk. Removing both sets is a deliberate call.
Implemented as a separate step rather than a filter inside
transform_customers.py: emptiness is only knowable after properties, policies
and transactions have loaded, and deciding it there would mean re-deriving
each downstream transform's source-matching logic against the staged Parquet.
Runs after transform_transactions.py in run_all.py.
Safe by construction — a customer with no rows in any of the three tables has
nothing pointing at it, so the delete cannot orphan anything; only its own
customer_legacy_refs go with it. The step asserts zero orphans afterwards.
Every pruned customer is written to output/pruned_customers.csv with its
legacy provenance before the delete, and --dry-run reports without touching
anything. Nothing is unrecoverable: the Access sources are untouched and a
pipeline run without this step brings them all back.
Verified: full run_all.py pass ends at 1538 customers (from 1682), with
1519 properties / 2378 policies / 45861 transactions all intact and zero
orphans. 17 nameless customers remain, all of which carry real records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 44 customers with no recoverable name render as "(SIN NOMBRE)", and
ordering the list by name alone floated all of them to the top — "(" sorts
before every letter — so the first two screens of the customer browser were
nothing but placeholders. Small number, worst possible position.
Adds customers.nameMissing, set by the transform and used as the primary sort
key so those records land at the end of the list. Denormalized rather than
computed in the query because the list is paginated in SQL, so the ordering
has to be expressible as a column.
Applied to the dev DB as an ALTER + UPDATE in place (no truncate), so the
existing loaded data and its FKs were left alone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATGRAL.NOMBRE is blank on 266 legacy rows (140 utilities, 126 insurance),
which surfaced in the UI as 257 customers literally named "(SIN NOMBRE)".
The blank is real — those cells are empty in the Access files, not lost in
extraction — but the rows mostly are not junk: 176 of the 257 carry a
property, a policy, or transactions.
The old PHP importer handled this by skipping blank-name rows outright
(jorgecuadros-intra-webapp/src/tools/customerAdapter.php:47,81). That was
worse than it looks: every other adapter resolved its customer FK through
the customer_mapping table those skipped rows never entered, so their
properties and policies were silently dropped (customerServiceAdapter.php:45)
and their transactions were written against customer_id 0
(customerBalanceAdapter.php:52). So: recover the name instead of skipping.
Names come from the secondary tables that still carry them, most trustworthy
first — UTILSEG (the office's own hand-maintained name <-> id cross-reference
spanning both lines), then the billing runs (IVA 2015, COBRO3) and the policy
rows' NOMBRE ASEG (MULT, M EMPR, INCENDIO). A linked customer can also borrow
the name its insurance record resolved to. Result: 213 of 257 recovered, 44
still genuinely nameless anywhere in the source.
customers.nameSource records which table each recovered name came from, so a
reconstructed name is never mistaken for one that was really on the record —
the list tags it "nombre recuperado", the detail header names the source, and
a still-unnamed customer renders muted italic instead of as a normal name.
Also fixes run_all.py: transform_properties and transform_policies truncate
service_documents/policy_documents, but blob_extract.py was not in the step
list, so a full re-run left the uploaded MinIO objects with no rows pointing
at them. Hit exactly that while reloading for this change.
Verified end-to-end: full pipeline re-run against dev reproduces every prior
count (1682 customers, 1519 properties, 2378 policies, 45861 transactions,
22354 bank rows, 70 documents) with zero orphans, and both apps build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First real frontend feature against the live Customer module API.
- login/ — session login form posting to /auth/login with credentials
included; the session cookie is what every subsequent request rides on.
- clientes/ — customer list with search and the cross-line stats header
(customers, utilities/insurance split, both-lines count).
- clientes/[id]/ — unified detail view: identity, properties + services,
policies, and transaction history for one customer, which is the whole
point of the migration (one record spanning both business lines).
- components/AppShell.tsx, lib/{api,labels,types}.ts — shared fetch wrapper
(always credentials: "include"), Spanish label maps for the enum values
the API returns, and the API response types.
- globals.css + layout.tsx — Spanish-first document (lang="es"), the type
scale, and the design tokens the pages share. Fonts load via <link> so an
offline build still renders on the system fallback stacks.
- page.tsx now redirects / to /clientes.
Also fixes pnpm-workspace.yaml: the allowBuilds map held pnpm's literal
placeholder text ("set this to true or false"), which made every install
fail with ERR_PNPM_IGNORED_BUILDS. Since pnpm 11 auto-installs before
running a script, that broke `pnpm start:dev` outright. Set the values to
true and dropped the superseded onlyBuiltDependencies list.
Verified: both apps build clean, and login -> /auth/me -> /customers/stats
round-trips against the dev database (1682 customers, 526 on both lines).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the unified Customer API against the migrated data:
- customers.service: list (search across name/email/phone/city/legacy id,
business-line filter, pagination + per-row _count flags), detail (identity +
legacyRefs + properties/services/trust + policies with installments/vehicles/
drivers/beneficiaries/claims/docs + recent transactions + a per-domain/
currency ledger summary), and stats.
- customers.controller: GET /customers, /customers/:id, /customers/stats,
guarded by AuthenticatedGuard. Registered in AppModule.
- Fix LocalAuthGuard to call super.logIn so a session is actually established
(login previously succeeded but persisted no session -> 403 afterwards).
- apps/api/scripts/seed-user.mjs: idempotent Argon2 admin seed.
Tooling: adopt pnpm as the package manager (machine npm is a pnpm shim that
ignores the workspaces field). Add pnpm-workspace.yaml (+ onlyBuiltDependencies
for argon2/prisma/nest native builds), switch the api's @jorgecuadros/database
dep to workspace:*, add @types/passport, track pnpm-lock.yaml, drop the stale
package-lock.json.
Verified end-to-end against the dev DB: login sets a session cookie; stats
returns 1682 customers / 526 both-lines / 45861 transactions; search + detail
return the full cross-line customer view (properties+services AND policies AND
a unified transaction statement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.
Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.
Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-minio.stack.yml: S3-compatible object storage (MinIO) for
the platform's document blobs, deployed to the cubex Swarm with the same
statefulness rules as the DB stack (named volume, pinned to the labeled node).
Parametrized for dev/prod as two stacks (dev API 9100/console 9101, prod
9000/9001). Dev deployed + bucket jorgecuadros-documents created.
migration/blob_extract.py: re-reads the LONGBINARY columns via mdb-export
-b hex (staging used -b strip), carves the embedded file out of the Access
OLE wrapper by locating its magic bytes (JPEG/PNG/PDF/GIF/TIFF) and trimming
trailing OLE junk, uploads to MinIO, and writes service_documents /
policy_documents pointer rows. Row->parent alignment uses mdb-export's
deterministic order (== staged _row_num) for policies and numer_id for
properties. Idempotent (truncate doc tables + overwrite by deterministic key);
--limit/--tables for test passes.
Validated on a limited pass: carved blobs are valid JPEGs (ffd8ff..ffd9)
correctly linked to their policies.
requirements.txt: add boto3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.
migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).
run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_policies.py folds every insurance Access table into one
`policies` table (policy_types discriminator) plus child tables, via a
per-table declarative mapping that absorbs the column-name variance
(num_id/numer_id, no_poliza/poliza, p_neta/prima_neta/prima1). Any source
column not explicitly modeled — the type-specific coverage amounts — is
preserved verbatim in coveragesJson, so consolidation loses nothing.
Unpivots the hardcoded repeated slots: 4 payment installments (c_1er_pago +
pago_subsec x3), up to 3 vehicles (auto tables + MCA2), up to 3 named
insured drivers (MCA2 + LICENCIAS). Also loads BENEF -> policy_beneficiaries
(by policy number), DATOS -> claims, AJUSTADORES(+ATLAS) -> adjusters, and
builds policy_types + insurance_providers lookups.
Loaded/validated (dev): 2378 policies (AUTO 1307 / MULT 760 / LICENCIAS 306 /
M_EMPR 5; 10 skipped for unresolved customer FK, 0 orphans), 4678
installments, 1110 vehicles, 513 drivers, 126 beneficiaries, 1 claim, 15
providers, 17 adjusters — all child FKs verified 0 orphans. Spot-checked a
customer carrying both a utility property and MULT policies (the unified
cross-line view).
run_all.py: add policies to the ordered pipeline. Customer FK resolves
through insurance customer_legacy_refs, so this runs after customers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_properties.py loads properties, property_services and
trust_accounts from staged DATMEX/PROFILE, resolving each property's customer
FK through customer_legacy_refs. Services are derived from DATMEX's own
account/route/meter fields (the authoritative data); PROFILE flags — merged
best-effort on (numer_id,casa,direccion), which matched 1519/1519 — only
refine each service's `active`. Trust accounts are 1:1 from DATMEX trust
fields; TRUSTVENCE (overlapping) deferred to reconciliation; blobs are step 4.
Loaded/validated (dev): 1519 properties (0 orphans, 1 blank id skipped),
3486 services (ELECTRIC 1118 / PROPERTY_TAX 939 / WATER 859 / GAS 335 /
OTHER 115 / FEDERAL_ZONE 76 / CABLE 41 / ALARM 3), 553 trust accounts —
counts track the PROFILE enrollment flags.
Reproducibility (asked: dev must be redoable in prod):
- migration/dbenv.py: single DB-target source = deploy/.env.<env>'s
DATABASE_URL; connect(env) + env_arg() (--env, default dev).
- transform_customers.py / transform_properties.py now take --env instead of
hardcoding .env.dev.
- migration/run_all.py: runs every step in dependency order against --env
(optional --stage re-extracts from Access first). Reproducing dev->prod is
`run_all.py --env prod` after deploying the prod stack + prisma db push.
All steps are idempotent (truncate+rebuild); re-run yields identical counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
migration/transform_customers.py builds `customers` + `customer_legacy_refs`
from staged DATGRAL, implementing the reconciliation rules: utilities DATGRAL
is the customer master; insurance DATGRAL folds in via its num_util
cross-reference (matches enrich the master with the ID-document fields
utilities lacks); COBRO3 excluded as a charge batch. Every legacy row gets a
provenance ref, so the load is auditable and idempotent (truncate+rebuild).
Loaded and validated against the dev DB (192.168.4.212:3307):
1682 customers (1172 utilities master + 510 insurance-only)
2242 legacy refs (1172 utilities + 1070 insurance) — 0 orphans
560 insurance rows linked via num_util, 0 broken cross-refs
542 merged identities spanning both business lines
Spot-checked a merged customer: single record carrying utilities fee +
insurance passport/ID enriched in, both provenance refs present.
RESUME.md: mark customers done, record dev-DB infra + the pnpm/npm caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy/jorgecuadros-db.stack.yml: canonical internal MySQL for the platform,
targeting the Portainer local endpoint on cubex (3-node Swarm). Parametrized
(MYSQL_PORT / MYSQL_SERVER_ID) so one file deploys both environments as two
stacks with Swarm-namespaced volumes:
dev -> jorgecuadros-dev-db :3307 server-id 11
prod -> jorgecuadros-prod-db :3306 server-id 1 (replication source)
Swarm-correct: named volume (no bind mount), pinned to one node via
node.labels.jorgecuadros_db==true (cubex labeled), binlog+GTID enabled from
the start so prod can be the VPS replication source without reconfigure.
Dev deployed and verified: MySQL 8.4.10 reachable at 192.168.4.212:3307,
gtid_mode ON, database jorgecuadros present. Secrets live in gitignored
deploy/.env.dev, injected via Portainer stack env at deploy time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add migration/reconcile.py (reads staged Parquet) and the generated
migration/RECONCILIATION.md. Naive full-row matching across the suspected
"duplicate" groups gives a misleading ~0 overlap, so each group is probed on
a deliberate business key instead. The evidence overturns all three of the
plan's original assumptions:
- EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table
sequential number that collides (12,363 shared folio numbers, every one a
different transaction); real business-key (cl,fecha,monto,conepto) overlap
is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule:
migrate both, keyed by (source_table, folio) provenance, no folio de-dup,
don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream.
- datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods
(2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three,
no de-dup; keep datos2.due_date.
- DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a
filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL
is the sole customer master, COBRO3 contributes zero customers.
Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization
at transform time.
Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and
RESUME.md (queue: reconciliation done, transform+load next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macOS has no Access ODBC driver, so the pyodbc-based extract.py could not
run. Rewrite it to shell out to mdbtools (mdb-tables/mdb-export) while
keeping the public interface (connect/list_tables/read_table/
sanitize_column_name) unchanged, so load_staging.py and config.py are
untouched. connect() now returns the file path as the handle (mdbtools has
no persistent connection).
Behavior details:
- -b strip drops LONGBINARY/OLE bytes so blobs never corrupt the CSV
(documents are extracted separately in migration step 4).
- ISO date/datetime output (-D/-T); staging read as text (dtype=str),
only empty fields treated as null (keep_default_na=False) so literal
"NA"/"NULL" data strings survive.
- mdbtools reads deleted/corrupted records mdbtools omits rather than
aborting, so the old per-row skip loop is no longer needed.
Verified end-to-end: full staging load reproduces the original Windows
run (82 tables, 0 errors) with exact row counts (datos2 16000, EFECTIVO
13697, DATGRAL 1172/1070, DATMEX 1520) and recovers all 764 MULT rows
(the pyodbc path lost 1 to HY109 corruption). Accented-column tables
(PROPANO) read cleanly.
requirements.txt: drop pyodbc/pywin32 (Windows-only), keep pandas/pyarrow/
sqlalchemy/pymysql; document the Windows-only DAO catalog as historical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project moved Windows -> macOS, retiring the pyodbc + Access ODBC driver
extraction path in favor of mdbtools (verified against the real files).
Record the four now-locked decisions and the resolved utility_dbo blocker.
- PLAN.md: rework migration step 1 for mdbtools; replace "Open decisions"
with locked Decisions (extraction=mdbtools, i18n=Spanish-first,
CI=Gitea Actions, utility_dbo=available); update Status for the macOS
move + portal DB in hand; finalize sync steps against real utility_dbo.
- RESUME.md: rewrite next-steps as a locked execution queue.
- migration/config.py: point SOURCE_ROOT at ~/Downloads/JorgeCuadros-Legacy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
docs/LEGACY_DATABASES.md documents all three source Access databases
(every table, column, type, and known data-quality quirk) generated
from a live read of the real files, so no Windows/Access driver is
needed to understand their structure going forward.
New migration/ tooling: catalog_schema.py connects to the real files
and walks every table (including excluded scratch tables);
render_catalog_md.py renders that into the doc's appendix. Raw output
checked in at migration/catalog.json so the doc can be regenerated
without touching Access again.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 5