Blob extraction: fix DATMEX document columns; migration step 4 complete

DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.

Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.

Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 18:56:49 -07:00
co-authored by Claude Opus 4.8
parent feb6bc91a7
commit 46d75473ba
2 changed files with 18 additions and 5 deletions
+7 -2
View File
@@ -42,8 +42,13 @@ SOURCE_ROOT = Path.home() / "Downloads" / "JorgeCuadros-Legacy"
# (key, access_file, access_table, staged_table_name, blob_cols, model)
SOURCES = [
# DATMEX's real scanned bills live in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-
# image columns (per-service bill scans), not doc_1/doc_2 (which are empty).
# In practice only a handful are populated — the .accdb is mostly bloat.
dict(key="datmex", file="UTILITIES.accdb", table="DATMEX", staged="DATMEX",
blobs=["doc_1", "doc_2"], model="service"),
blobs=["iluz", "iagua", "ipredial", "itel", "doc_1", "doc_2"], model="service",
doctypes={"iluz": "ELECTRIC_BILL", "iagua": "WATER_BILL",
"ipredial": "PROPERTY_TAX_BILL", "itel": "PHONE_BILL"}),
dict(key="mult", file="SEGUROS 16_be.mdb", table="MULT", staged="mult",
blobs=["foto1", "docs_1", "docs_2"], model="policy"),
dict(key="autos_ampl", file="SEGUROS 16_be.mdb", table="TABLA AUTOS AMPL",
@@ -171,7 +176,7 @@ def main():
prefix = "service" if src["model"] == "service" else "policy"
key = f"{prefix}/{parent}/{src['staged']}_{ri}_{col}.{ext}"
s3.put_object(Bucket=bucket, Key=key, Body=data, ContentType=ct)
dtype = col.upper()
dtype = src.get("doctypes", {}).get(col, col.upper())
if src["model"] == "service":
svc_rows.append((str(uuid.uuid4()), parent, dtype, key))
else: