rmancinas/jorgecuadros-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Transform+load: shared ledger + SCOTHIA bank register (step 3 complete)

Browse Source 
migration/transform_transactions.py unions every cash/billing ledger into
`transactions` per the reconciliation rules: both EFECTIVO tables (no folio
de-dup, near-disjoint), all three billing tables (disjoint periods), the FM3
fee stream (amount = fee+tax+multa), IVA 2015 (nominal date), and insurance
EFECTIVO (domain INSURANCE). Also loads the type_transactions (EN/ES) and
exchange_rates lookups. Customer FK resolves through customer_legacy_refs;
rows with no resolvable customer/date are skipped and counted.
Loaded (dev): 45861 transactions (UTILITY 45566 / INSURANCE 295, 0 orphans),
79 type_transactions, 2301 exchange_rates.

migration/transform_bank.py loads SCOTHIA DATOS I/E into bank_transactions as
signed amounts (income +, expense -) and TABLA RAMODOS into
business_line_categories. Deliberately customer-independent (office's own
checking account). Loaded (dev): 22354 bank_transactions (net +899,375.77),
66 categories; categoryId left null (concept->ramo classifier is future work).

run_all.py: pipeline now customers -> properties -> policies -> transactions
-> bank, all idempotent. Verified full end-to-end run against dev.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rmancinas
2026-07-22 18:43:34 -07:00
co-authored by Claude Opus 4.8
parent 21899e99bb
commit 83e3cb8f47
4 changed files with 397 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESUME.md
+16 -5
View File
@@ -185,11 +185,22 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
adjusters. Unmodeled coverage columns preserved verbatim in `coveragesJson`. Verified a
unified customer (EARWOOD, DAVID) carrying both a utility property+services and 2 MULT
policies — the cross-line customer view works at the data layer.
- **NEXT:** the shared ledger union per the reconciliation rules (both EFECTIVO tables, all
three billing tables, provenance-keyed; normalize `monedas` variants), then SCOTHIA bank
register, then document extraction (step 4, LONGBINARY blobs -> object storage).
- Migration is env-parameterized + reproducible: `run_all.py --env <env>` runs customers ->
properties -> policies in order; add `--stage` to re-extract from Access first.
- **Shared ledger — DONE** (`migration/transform_transactions.py`): 45861 transactions
(UTILITY 45566 / INSURANCE 295, 0 orphans) unioning both EFECTIVO tables (13696+12386,
no folio de-dup), all three billing tables (datos2/FEE ANUAL/fee15), the FM3 fee stream
(amount=fee+tax+multa), IVA 2015 (nominal date), and insurance EFECTIVO — per the
reconciliation rules; plus 79 `type_transactions` (EN/ES) and 2301 `exchange_rates`.
Skipped 22 no-customer + 303 no-date (mostly datos2 blanks).
- **Bank register — DONE** (`migration/transform_bank.py`): 22354 `bank_transactions` from
SCOTHIA DATOS I/E as signed amounts (income +, expense -; net +899,375.77), 66
`business_line_categories`. No customer FK; categoryId left null (concept->ramo classifier
is a later enhancement).
- **NEXT:** document extraction (step 4) — pull the LONGBINARY blobs (DATMEX doc_1/2, policy
docs_1/2/foto1, DATOS docs) to object storage + write *_documents rows. Then the Customer
module API/web (Spanish-first).
- Full pipeline reproducible in one command: `run_all.py --env <env>` runs customers ->
properties -> policies -> transactions -> bank in order (all idempotent); add `--stage`
to re-extract from the Access files first. Verified end-to-end against dev.
5b. **Infra done:** dev MySQL deployed to the cubex Swarm via Portainer API as stack
`jorgecuadros-dev-db` (MySQL 8.4, `192.168.4.212:3307`, node `cubex` labeled
`jorgecuadros_db=true`); Prisma schema pushed (26 tables). Stack file: