fix(migration): make Phase B additive sync actually work + verify end-to-end
The --sync path had never been run and was broken in several ways. Fixed and verified against the dev DB (two consecutive syncs, both exit 0, 32/32 assertions: stable PKs, manual-row preservation, changed-row updates, legacy-delete, no child duplication, zero FK orphans; idempotent). - policies/properties: reuse each legacy row's existing id (by provenance) BEFORE building child rows, so children no longer point at a discarded fresh uuid; rebuild legacy-owned children via scoped delete + reinsert. - customers: replace zip(customers, refs) (mispaired almost every row) with a ref-grouped id remap; names now restore and no spurious customers appear. - drop the invalid Vehicle @@unique(legacySourceTable, legacyId) — one legacy policy row carries up to 3 vehicles sharing a legacyId; handle via delete+reinsert. - upsert lookup tables (policy_types, insurance_providers, type_transactions, adjusters) by natural name and remap child FKs instead of inserting fresh uuids that nothing points at. - transactions: drop updatedAt=NOW() (no such column); guard report formatting on NULL legacySourceTable (manual rows). Same report guard in bank. - add manual-safe prune (prune_empty_customers.py --sync, in SYNC_STEPS): prune only legacy-owned empties, never manually-added customers. web: customer-detail mini tx list now strikes voided rows with an "(anulado)" tag (was the last void-UI rendering gap; /estado-cuenta already handled it). docs: RESUME.md updated — Phase B sync marked verified end-to-end, void-UI browser pass recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -127,8 +127,14 @@ To rerun (from `migration/`, venv at `migration/.venv`):
|
||||
```bash
|
||||
./.venv/bin/python load_staging.py --output-dir ./output # re-extract from Access (needs mdbtools + the source files)
|
||||
./.venv/bin/python run_all.py --env dev # full transform+load; add --stage to re-extract first
|
||||
./.venv/bin/python run_all.py --env dev --sync # additive sync: upsert legacy by provenance, keep manual rows, prune legacy empties
|
||||
```
|
||||
|
||||
`--sync` mode (Phase B) upserts legacy-owned rows by their provenance keys and preserves
|
||||
manual rows (`legacyId IS NULL`); every transform reuses each row's existing PK, rebuilds
|
||||
legacy-owned children by scoped delete + reinsert, and drops legacy rows gone from source.
|
||||
Verified end-to-end against dev 2026-07-24 — see §6 item 6.
|
||||
|
||||
## 5. Infrastructure & sync architecture (designed, not yet built)
|
||||
|
||||
- **Internal server** — on-prem, private IP `192.168.1.xx`, no inbound internet exposure. Runs the platform + canonical MySQL (source of truth).
|
||||
@@ -162,17 +168,30 @@ the reconciliation pass (done, then corrected) are all closed. See §3 and §8.
|
||||
5. **`TRASPASOS PAYPAL` is a clearing account, not a customer** — carries -7.03M MXN over
|
||||
309 movements and therefore tops the adeudo worklist. Deliberately not special-cased in
|
||||
code; needs a business decision on how to model it.
|
||||
6. **DB Operations — Phase B (additive sync) — IMPLEMENTED, verification pending.** Phase A provides
|
||||
the admin-only `/operaciones` page + `ops` API module (ability `db:manage`, ADMIN), ingest
|
||||
folder, backup, restore, and destructive re-import. Phase B now enables `SYNC`: `OpsService`
|
||||
creates a safety backup and runs `run_all.py --sync`; transforms upsert legacy-owned rows by
|
||||
provenance keys while preserving existing PKs and rows whose `legacyId IS NULL` (manual).
|
||||
Prisma now enforces provenance uniqueness for properties, policies, transactions, vehicles,
|
||||
and bank transactions. Sync intentionally skips prune/blob steps so manual customers and
|
||||
document pointers are not removed. Python compilation plus API/web production builds pass;
|
||||
still required before production use: push updated Prisma schema and run an end-to-end sync
|
||||
against a disposable/dev DB proving stable PKs, manual-row preservation, changed-row updates,
|
||||
and legacy-delete handling.
|
||||
6. **DB Operations — Phase B (additive sync) — VERIFIED END-TO-END against dev DB 2026-07-24.**
|
||||
Phase A provides the admin-only `/operaciones` page + `ops` API module (ability `db:manage`,
|
||||
ADMIN), ingest folder, backup, restore, and destructive re-import. Phase B enables `SYNC`:
|
||||
`OpsService` creates a safety backup and runs `run_all.py --sync`; transforms upsert
|
||||
legacy-owned rows by provenance keys while preserving existing PKs and rows whose
|
||||
`legacyId IS NULL` (manual). Prisma enforces provenance uniqueness for properties, policies,
|
||||
transactions, and bank transactions (the vehicle unique was **removed** — one legacy policy
|
||||
row carries up to 3 vehicles that share a `legacyId`, so provenance is not unique per
|
||||
vehicle; vehicles are rebuilt by scoped delete + reinsert). Sync skips blob extraction, and
|
||||
runs a **manual-safe prune** (`prune_empty_customers.py --sync` — only prunes empties that
|
||||
carry a legacy ref, never manually-added customers) because the customer upsert otherwise
|
||||
re-creates every previously-pruned empty from Parquet.
|
||||
|
||||
**The as-written sync was broken and had never been run; a batch of bugs were fixed on
|
||||
2026-07-24 before it passed** (fresh-uuid child FKs in policies/properties, unconditional
|
||||
child inserts, a `zip(customers, refs)` mispairing in transform_customers, invalid vehicle
|
||||
unique, lookup tables built with fresh uuids but never upserted, a `updatedAt=NOW()` on a
|
||||
table with no such column, and report crashes on NULL `legacySourceTable` for manual rows).
|
||||
Verified with `migration/` `verify_sync.py`-style harness: two consecutive `run_all.py --sync`
|
||||
runs both exit 0 and pass 32/32 assertions (stable PKs, manual-row preservation, changed-row
|
||||
updates, legacy-delete, no child duplication, zero FK orphans), idempotent (customers stable
|
||||
at 1537). Schema pushed to dev, Prisma client regenerated, API build clean. Migration/web
|
||||
changes uncommitted as of this update. Still open before production: run the same sync from
|
||||
the `/operaciones` UI (OpsService path) and against a prod-shaped DB.
|
||||
|
||||
## 7. Environment notes (current macOS machine)
|
||||
|
||||
@@ -385,15 +404,20 @@ for what's actually next.
|
||||
|
||||
---
|
||||
|
||||
- **Sync implementation — DONE, validation pending.** `run_all.py --sync` performs the
|
||||
non-destructive legacy upsert path for customers, properties, policies, transactions, and
|
||||
bank rows. It preserves manual rows and stable legacy-owned primary keys; the admin SYNC job
|
||||
automatically creates a pre-sync backup. Next validation: apply schema changes, then exercise
|
||||
sync against a disposable DB with added, changed, removed, and manually-created rows.
|
||||
- **Sync implementation — DONE + VALIDATED end-to-end against dev 2026-07-24.** `run_all.py
|
||||
--sync` performs the non-destructive legacy upsert path for customers, properties, policies,
|
||||
transactions, and bank rows, plus a manual-safe empty-customer prune. It preserves manual rows
|
||||
and stable legacy-owned primary keys; the admin SYNC job auto-creates a pre-sync backup. The
|
||||
as-written code was broken and had never been run — a batch of bugs was fixed before it passed
|
||||
(see §6 item 6). Two consecutive syncs both exit 0 and pass 32/32 assertions (added, changed,
|
||||
removed, and manually-created rows), idempotent. Remaining: exercise the same path from the
|
||||
`/operaciones` admin UI and against a prod-shaped DB.
|
||||
- **Plan step 9: portal sync worker** remains separate and blocked on VPS provisioning. This
|
||||
Phase B feature synchronizes Access source files into the internal platform; it does not yet
|
||||
poll `utility_dbo` inbox tables or replicate portal-facing data to a VPS.
|
||||
- **Small / open:** (a) `TRASPASOS PAYPAL` clearing account still tops the adeudo worklist
|
||||
(§6.4d) — a business modelling call, not code. (b) Credential rotation on the old repo's
|
||||
exposed MySQL password. (c) The `/estado-cuenta` browser visual pass — `/banco` was verified
|
||||
in-browser this session; `/estado-cuenta` still worth a look.
|
||||
exposed MySQL password. (c) ~~The `/estado-cuenta` browser visual pass.~~ **DONE 2026-07-24** —
|
||||
verified vs dev: Anular buttons admin-gated, voided rows struck + excluded from totals,
|
||||
clicking Anular voids end-to-end (note: it uses a blocking `window.confirm`). Customer-detail
|
||||
mini tx list now also strikes voided rows ("(anulado)" tag) — was the last void-UI gap.
|
||||
|
||||
Reference in New Issue
Block a user