Blob extraction: fix DATMEX document columns; migration step 4 complete
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not doc_1/doc_2 (which are empty). Add them to the extractor with meaningful document types. Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3 DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents + 67 policy_documents, 0 orphans, storageKeys resolve. Migration steps 1-4 (staging, reconciliation, transform+load, documents) are complete; RESUME.md updated. Next: the Customer module (API/web). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -42,8 +42,13 @@ SOURCE_ROOT = Path.home() / "Downloads" / "JorgeCuadros-Legacy"
|
||||
|
||||
# (key, access_file, access_table, staged_table_name, blob_cols, model)
|
||||
SOURCES = [
|
||||
# DATMEX's real scanned bills live in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-
|
||||
# image columns (per-service bill scans), not doc_1/doc_2 (which are empty).
|
||||
# In practice only a handful are populated — the .accdb is mostly bloat.
|
||||
dict(key="datmex", file="UTILITIES.accdb", table="DATMEX", staged="DATMEX",
|
||||
blobs=["doc_1", "doc_2"], model="service"),
|
||||
blobs=["iluz", "iagua", "ipredial", "itel", "doc_1", "doc_2"], model="service",
|
||||
doctypes={"iluz": "ELECTRIC_BILL", "iagua": "WATER_BILL",
|
||||
"ipredial": "PROPERTY_TAX_BILL", "itel": "PHONE_BILL"}),
|
||||
dict(key="mult", file="SEGUROS 16_be.mdb", table="MULT", staged="mult",
|
||||
blobs=["foto1", "docs_1", "docs_2"], model="policy"),
|
||||
dict(key="autos_ampl", file="SEGUROS 16_be.mdb", table="TABLA AUTOS AMPL",
|
||||
@@ -171,7 +176,7 @@ def main():
|
||||
prefix = "service" if src["model"] == "service" else "policy"
|
||||
key = f"{prefix}/{parent}/{src['staged']}_{ri}_{col}.{ext}"
|
||||
s3.put_object(Bucket=bucket, Key=key, Body=data, ContentType=ct)
|
||||
dtype = col.upper()
|
||||
dtype = src.get("doctypes", {}).get(col, col.upper())
|
||||
if src["model"] == "service":
|
||||
svc_rows.append((str(uuid.uuid4()), parent, dtype, key))
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user