fix(migration): make Phase B additive sync actually work + verify end-to-end
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m37s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m9s

The --sync path had never been run and was broken in several ways. Fixed and
verified against the dev DB (two consecutive syncs, both exit 0, 32/32
assertions: stable PKs, manual-row preservation, changed-row updates,
legacy-delete, no child duplication, zero FK orphans; idempotent).

- policies/properties: reuse each legacy row's existing id (by provenance)
  BEFORE building child rows, so children no longer point at a discarded fresh
  uuid; rebuild legacy-owned children via scoped delete + reinsert.
- customers: replace zip(customers, refs) (mispaired almost every row) with a
  ref-grouped id remap; names now restore and no spurious customers appear.
- drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy
  policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert.
- upsert lookup tables (policy_types, insurance_providers, type_transactions,
  adjusters) by natural name and remap child FKs instead of inserting fresh
  uuids that nothing points at.
- transactions: drop updatedAt=NOW() (no such column); guard report formatting
  on NULL legacySourceTable (manual rows). Same report guard in bank.
- add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune
  only legacy-owned empties, never manually-added customers.

web: customer-detail mini tx list now strikes voided rows with an "(anulado)"
tag (was the last void-UI rendering gap; /estado-cuenta already handled it).

docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI
browser pass recorded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-24 13:37:22 -07:00
co-authored by Claude Opus 4.8
parent 8802f08d4f
commit 1b79b43a54
11 changed files with 211 additions and 76 deletions
+42 -18
View File
@@ -127,8 +127,14 @@ To rerun (from `migration/`, venv at `migration/.venv`):
```bash
./.venv/bin/python load_staging.py --output-dir ./output # re-extract from Access (needs mdbtools + the source files)
./.venv/bin/python run_all.py --env dev # full transform+load; add --stage to re-extract first
./.venv/bin/python run_all.py --env dev --sync # additive sync: upsert legacy by provenance, keep manual rows, prune legacy empties
```
`--sync` mode (Phase B) upserts legacy-owned rows by their provenance keys and preserves
manual rows (`legacyId IS NULL`); every transform reuses each row's existing PK, rebuilds
legacy-owned children by scoped delete + reinsert, and drops legacy rows gone from source.
Verified end-to-end against dev 2026-07-24 — see §6 item 6.
## 5. Infrastructure & sync architecture (designed, not yet built)
- **Internal server** — on-prem, private IP `192.168.1.xx`, no inbound internet exposure. Runs the platform + canonical MySQL (source of truth).
@@ -162,17 +168,30 @@ the reconciliation pass (done, then corrected) are all closed. See §3 and §8.
5. **`TRASPASOS PAYPAL` is a clearing account, not a customer** — carries -7.03M MXN over
309 movements and therefore tops the adeudo worklist. Deliberately not special-cased in
code; needs a business decision on how to model it.
6. **DB Operations — Phase B (additive sync) — IMPLEMENTED, verification pending.** Phase A provides
the admin-only `/operaciones` page + `ops` API module (ability `db:manage`, ADMIN), ingest
folder, backup, restore, and destructive re-import. Phase B now enables `SYNC`: `OpsService`
creates a safety backup and runs `run_all.py --sync`; transforms upsert legacy-owned rows by
provenance keys while preserving existing PKs and rows whose `legacyId IS NULL` (manual).
Prisma now enforces provenance uniqueness for properties, policies, transactions, vehicles,
and bank transactions. Sync intentionally skips prune/blob steps so manual customers and
document pointers are not removed. Python compilation plus API/web production builds pass;
still required before production use: push updated Prisma schema and run an end-to-end sync
against a disposable/dev DB proving stable PKs, manual-row preservation, changed-row updates,
and legacy-delete handling.
6. **DB Operations — Phase B (additive sync) — VERIFIED END-TO-END against dev DB 2026-07-24.**
Phase A provides the admin-only `/operaciones` page + `ops` API module (ability `db:manage`,
ADMIN), ingest folder, backup, restore, and destructive re-import. Phase B enables `SYNC`:
`OpsService` creates a safety backup and runs `run_all.py --sync`; transforms upsert
legacy-owned rows by provenance keys while preserving existing PKs and rows whose
`legacyId IS NULL` (manual). Prisma enforces provenance uniqueness for properties, policies,
transactions, and bank transactions (the vehicle unique was **removed** — one legacy policy
row carries up to 3 vehicles that share a `legacyId`, so provenance is not unique per
vehicle; vehicles are rebuilt by scoped delete + reinsert). Sync skips blob extraction, and
runs a **manual-safe prune** (`prune_empty_customers.py --sync` — only prunes empties that
carry a legacy ref, never manually-added customers) because the customer upsert otherwise
re-creates every previously-pruned empty from Parquet.
**The as-written sync was broken and had never been run; a batch of bugs were fixed on
2026-07-24 before it passed** (fresh-uuid child FKs in policies/properties, unconditional
child inserts, a `zip(customers, refs)` mispairing in transform_customers, invalid vehicle
unique, lookup tables built with fresh uuids but never upserted, a `updatedAt=NOW()` on a
table with no such column, and report crashes on NULL `legacySourceTable` for manual rows).
Verified with `migration/` `verify_sync.py`-style harness: two consecutive `run_all.py --sync`
runs both exit 0 and pass 32/32 assertions (stable PKs, manual-row preservation, changed-row
updates, legacy-delete, no child duplication, zero FK orphans), idempotent (customers stable
at 1537). Schema pushed to dev, Prisma client regenerated, API build clean. Migration/web
changes uncommitted as of this update. Still open before production: run the same sync from
the `/operaciones` UI (OpsService path) and against a prod-shaped DB.
## 7. Environment notes (current macOS machine)
@@ -385,15 +404,20 @@ for what's actually next.
---
- **Sync implementation — DONE, validation pending.** `run_all.py --sync` performs the
non-destructive legacy upsert path for customers, properties, policies, transactions, and
bank rows. It preserves manual rows and stable legacy-owned primary keys; the admin SYNC job
automatically creates a pre-sync backup. Next validation: apply schema changes, then exercise
sync against a disposable DB with added, changed, removed, and manually-created rows.
- **Sync implementation — DONE + VALIDATED end-to-end against dev 2026-07-24.** `run_all.py
--sync` performs the non-destructive legacy upsert path for customers, properties, policies,
transactions, and bank rows, plus a manual-safe empty-customer prune. It preserves manual rows
and stable legacy-owned primary keys; the admin SYNC job auto-creates a pre-sync backup. The
as-written code was broken and had never been run — a batch of bugs was fixed before it passed
(see §6 item 6). Two consecutive syncs both exit 0 and pass 32/32 assertions (added, changed,
removed, and manually-created rows), idempotent. Remaining: exercise the same path from the
`/operaciones` admin UI and against a prod-shaped DB.
- **Plan step 9: portal sync worker** remains separate and blocked on VPS provisioning. This
Phase B feature synchronizes Access source files into the internal platform; it does not yet
poll `utility_dbo` inbox tables or replicate portal-facing data to a VPS.
- **Small / open:** (a) `TRASPASOS PAYPAL` clearing account still tops the adeudo worklist
(§6.4d) — a business modelling call, not code. (b) Credential rotation on the old repo's
exposed MySQL password. (c) The `/estado-cuenta` browser visual pass`/banco` was verified
in-browser this session; `/estado-cuenta` still worth a look.
exposed MySQL password. (c) ~~The `/estado-cuenta` browser visual pass.~~ **DONE 2026-07-24** —
verified vs dev: Anular buttons admin-gated, voided rows struck + excluded from totals,
clicking Anular voids end-to-end (note: it uses a blocking `window.confirm`). Customer-detail
mini tx list now also strikes voided rows ("(anulado)" tag) — was the last void-UI gap.