rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Blob extraction: fix DATMEX document columns; migration step 4 complete

Browse Source 
DATMEX's scanned bills are in the ILUZ/IAGUA/IPREDIAL/ITEL invoice-image OLE
columns (typed ELECTRIC_BILL/WATER_BILL/PROPERTY_TAX_BILL/PHONE_BILL), not
doc_1/doc_2 (which are empty). Add them to the extractor with meaningful
document types.

Data finding: the LONGBINARY columns are almost entirely unpopulated — only 3
DATMEX blob cells across 1520 rows, and 67 policy blobs (MULT/TABLA AUTOS
AMPL foto/docs). The large .accdb/.mdb file sizes are Access bloat, not
documents. Final: 70 documents in MinIO (~290 MB), 3 service_documents +
67 policy_documents, 0 orphans, storageKeys resolve.

Migration steps 1-4 (staging, reconciliation, transform+load, documents) are
complete; RESUME.md updated. Next: the Customer module (API/web).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 18:56:49 -07:00
co-authored by Claude Opus 4.8
parent feb6bc91a7
commit 46d75473ba
2 changed files with 18 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESUME.md
+11 -3
View File
@@ -195,9 +195,17 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
SCOTHIA DATOS I/E as signed amounts (income +, expense -; net +899,375.77), 66
`business_line_categories`. No customer FK; categoryId left null (concept->ramo classifier
is a later enhancement).
- **NEXT:** document extraction (step 4) — pull the LONGBINARY blobs (DATMEX doc_1/2, policy
docs_1/2/foto1, DATOS docs) to object storage + write *_documents rows. Then the Customer
module API/web (Spanish-first).
- **Documents — DONE** (`migration/blob_extract.py`, migration step 4): carves embedded files
out of the Access OLE wrapper (magic-byte detection) and uploads to MinIO on cubex (stack
`jorgecuadros-dev-minio`, S3 at 192.168.4.212:9100, bucket jorgecuadros-documents), writing
service_documents/policy_documents pointer rows. Loaded 70 documents (3 service bills +
67 policy foto/docs, 0 orphans, ~290 MB). **Data finding:** the LONGBINARY columns are
almost entirely empty — DATMEX's real bill-scan columns are ILUZ/IAGUA/IPREDIAL/ITEL (not
doc_1/doc_2), but only 3 cells populated across 1520 rows; the *_MENS tables are mail-merge
templates (correctly excluded). The 538MB/882MB source files are mostly Access bloat, not
documents. **Migration steps 1-4 COMPLETE.**
- **NEXT:** the Customer module API/web (Spanish-first) — all relational data + documents are
loaded, so build the unified customer list/search/detail view.
- Full pipeline reproducible in one command: `run_all.py --env <env>` runs customers ->
properties -> policies -> transactions -> bank in order (all idempotent); add `--stage`
to re-extract from the Access files first. Verified end-to-end against dev.