deploy/jorgecuadros-minio.stack.yml: S3-compatible object storage (MinIO) for
the platform's document blobs, deployed to the cubex Swarm with the same
statefulness rules as the DB stack (named volume, pinned to the labeled node).
Parametrized for dev/prod as two stacks (dev API 9100/console 9101, prod
9000/9001). Dev deployed + bucket jorgecuadros-documents created.
migration/blob_extract.py: re-reads the LONGBINARY columns via mdb-export
-b hex (staging used -b strip), carves the embedded file out of the Access
OLE wrapper by locating its magic bytes (JPEG/PNG/PDF/GIF/TIFF) and trimming
trailing OLE junk, uploads to MinIO, and writes service_documents /
policy_documents pointer rows. Row->parent alignment uses mdb-export's
deterministic order (== staged _row_num) for policies and numer_id for
properties. Idempotent (truncate doc tables + overwrite by deterministic key);
--limit/--tables for test passes.
Validated on a limited pass: carved blobs are valid JPEGs (ffd8ff..ffd9)
correctly linked to their policies.
requirements.txt: add boto3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macOS has no Access ODBC driver, so the pyodbc-based extract.py could not
run. Rewrite it to shell out to mdbtools (mdb-tables/mdb-export) while
keeping the public interface (connect/list_tables/read_table/
sanitize_column_name) unchanged, so load_staging.py and config.py are
untouched. connect() now returns the file path as the handle (mdbtools has
no persistent connection).
Behavior details:
- -b strip drops LONGBINARY/OLE bytes so blobs never corrupt the CSV
(documents are extracted separately in migration step 4).
- ISO date/datetime output (-D/-T); staging read as text (dtype=str),
only empty fields treated as null (keep_default_na=False) so literal
"NA"/"NULL" data strings survive.
- mdbtools reads deleted/corrupted records mdbtools omits rather than
aborting, so the old per-row skip loop is no longer needed.
Verified end-to-end: full staging load reproduces the original Windows
run (82 tables, 0 errors) with exact row counts (datos2 16000, EFECTIVO
13697, DATGRAL 1172/1070, DATMEX 1520) and recovers all 764 MULT rows
(the pyodbc path lost 1 to HY109 corruption). Accented-column tables
(PROPANO) read cleanly.
requirements.txt: drop pyodbc/pywin32 (Windows-only), keep pandas/pyarrow/
sqlalchemy/pymysql; document the Windows-only DAO catalog as historical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ODBC only exposes Tables and non-hidden SELECT queries, so this used
DAO COM automation (migration/catalog_objects.py, requires pywin32)
to catalog Reports, Forms, and full Query SQL text across all four
files instead.
Key finding: SEGUROS 16.mdb, previously noted as having zero data
tables, turns out to hold all 212 Reports/149 Forms/857 Queries for
the insurance line - SEGUROS 16_be.mdb is confirmed pure data storage
with zero saved objects. The renewal-notice reports also reveal a
RENEW/RENEW2/RENEW3 multi-notice reminder cycle not visible in the
table schema.
docs/LEGACY_DATABASES_OBJECTS.md documents all of this with the full
751 real queries' SQL text (business logic: billing math, year-
rollover batches, duplicate/delinquency detection). Raw output at
migration/objects.json. Cross-linked from RESUME.md and the existing
table-only LEGACY_DATABASES.md.
Next.js + NestJS + Prisma (MySQL) monorepo replacing the legacy PHP
internal app. Includes a session-based auth module with Argon2 password
hashing and global input validation (replacing the old app's SQL
injection and plaintext password comparison), the full target Prisma
schema for customers/insurance/utilities/shared ledger/bank register,
Docker Compose + Dockerfiles, and an Access-to-staging migration
pipeline (migration/) already run against the real source databases.
See PLAN.md and RESUME.md for the full architecture and session history.
rmancinasandClaude Opus 4.8