feat(migration): refuse a full re-import that would delete native rows
Build and Push Images / Build jorgecuadros-web (push) Successful in 1m51s
Build and Push Images / Build jorgecuadros-api (push) Successful in 2m42s

A full run_all.py pass truncates and rebuilds every table it owns from the
Access extract. That was harmless while the platform was a read-only mirror --
every row came from the extract, so wiping and rebuilding lost nothing. It
stopped being harmless once the platform started minting rows Access has never
heard of: allocated portal NUMids, customers created in the staff UI,
OCR-captured policies, app-booked ledger rows, uploaded documents.

REIMPORT is a button in /operaciones, so that was one click away.

native_guard.py counts what only exists here and exits 3; run_all.py runs it
before the first truncate and stops. Detecting an allocated NUMid needs the
staged Parquet -- the customer holds an ordinary-looking (utilities, DATGRAL,
'1172') ref, so "customer has no refs" cannot see it and only comparing against
the extract can. Missing staging is therefore treated as blocking rather than
as "nothing to protect".

The guard does not teach full mode to preserve anything: --sync already upserts
legacy rows against the existing refs and leaves the rest alone, and rebuilding
that inside full mode would re-implement it. --force-full (checkbox in the
REIMPORT confirm, recorded in the audit log) deletes them deliberately.

Verified against dev: clean before, exit 3 listing utilities/1172 with a
synthetic ref present, clean again after removing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 21:01:01 -07:00
co-authored by Claude Opus 5
parent 6a97242fc3
commit 17d83291c3
10 changed files with 319 additions and 20 deletions
+40
View File
@@ -15,6 +15,12 @@ Then:
./.venv/bin/python run_all.py --env dev # data only (staging already present)
./.venv/bin/python run_all.py --env prod --stage # re-extract from Access first, then load
A full pass truncates and rebuilds every table it owns from the Access extract,
so anything the platform minted itself — allocated portal NUMids, customers
created in the staff UI, OCR-captured policies, app-booked ledger rows, uploaded
documents — is destroyed. native_guard.py runs first and refuses when the target
database holds any of it; --force-full overrides and deletes them.
--sync swaps the truncate+rebuild steps for the additive upsert ones. It reads
the same staged Parquet, so it needs --stage too unless a previous run left
migration/output populated on this machine — which is never true in a
@@ -83,6 +89,33 @@ SYNC_STEPS = [
]
# native_guard.py exits with this when the target database holds rows that only
# exist in the platform. Kept in step with the constant there.
GUARD_BLOCKED = 3
def guard(env: str, force: bool) -> None:
"""Stop a full pass that would delete platform-native rows.
Only the full path needs this: --sync upserts legacy rows against the
existing refs and leaves everything else alone, so it cannot lose anything.
Run after staging, because the guard verifies legacy refs against the staged
Parquet and cannot tell an allocated NUMid from an Access one without it.
"""
cmd = [PY, str(HERE / "native_guard.py"), "--env", env]
print("+ " + " ".join(cmd), flush=True)
r = subprocess.run(cmd)
if r.returncode == GUARD_BLOCKED and not force:
sys.exit(r.returncode)
if r.returncode == GUARD_BLOCKED and force:
print(
"\n! --force-full: continuando y BORRANDO las filas nativas listadas.\n",
flush=True,
)
elif r.returncode:
sys.exit(r.returncode)
def run(cmd: list[str], step: int | None = None, total: int | None = None) -> None:
# The "[paso i/N] name" marker is a contract with the Operaciones screen,
# which parses the last one to show progress. Emitting it here rather than
@@ -103,6 +136,8 @@ def main() -> None:
help="re-run the raw staging load first (needs the Access files + mdbtools)")
ap.add_argument("--sync", action="store_true",
help="upsert legacy rows and archive removed legacy rows; preserve manual rows")
ap.add_argument("--force-full", action="store_true",
help="run the full truncate+rebuild even when it deletes platform-native rows")
args = ap.parse_args()
steps = SYNC_STEPS if args.sync else STEPS
@@ -116,6 +151,11 @@ def main() -> None:
run([PY, str(HERE / "load_staging.py"), "--output-dir", str(HERE / "output")],
step=1, total=total)
# Deliberately not counted as a step: it is a precondition, it takes a
# second, and the Operaciones progress bar parses those numbers.
if not args.sync:
guard(args.env, args.force_full)
for i, step in enumerate(steps, start=1 + offset):
cmd = [PY, str(HERE / step), "--env", args.env]
if args.sync: