Reconciliation pass (migration step 2): decide union/de-dup rules
Add migration/reconcile.py (reads staged Parquet) and the generated migration/RECONCILIATION.md. Naive full-row matching across the suspected "duplicate" groups gives a misleading ~0 overlap, so each group is probed on a deliberate business key instead. The evidence overturns all three of the plan's original assumptions: - EFECTIVO vs EFECTIVO_BACKUP: NOT a live/backup pair. `folio` is a per-table sequential number that collides (12,363 shared folio numbers, every one a different transaction); real business-key (cl,fecha,monto,conepto) overlap is 2. Near-disjoint ledgers (BACKUP ~2017-2022, EFECTIVO recent). Rule: migrate both, keyed by (source_table, folio) provenance, no folio de-dup, don't drop BACKUP. FM3 tables are a separate fee/tax/multa stream. - datos2 vs FEE ANUAL vs fee15: disjoint billing runs from different periods (2025-26 / 2018 / 2017), zero real-identity overlap. Rule: union all three, no de-dup; keep datos2.due_date. - DATGRAL vs COBRO3: COBRO3.fee is a constant 75 (a charge batch), not a filtered customer snapshot; every num_id already in DATGRAL. Rule: DATGRAL is the sole customer master, COBRO3 contributes zero customers. Also flags monedas currency variants (PESOS/Pesos/DOLLARS) for normalization at transform time. Update PLAN.md (migration step 2 outcome + corrected inventory bullets) and RESUME.md (queue: reconciliation done, transform+load next). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -34,10 +34,10 @@ Extracted live via `pyodbc` + the Windows Access ODBC driver (`dump_schema.py`,
|
||||
|
||||
**UTILITIES.accdb** (52 tables, ~538MB, dominated by embedded document blobs):
|
||||
- `DATMEX` (1,520 rows) — one row per property: address, phones, and inline utility fields for water/electric/gas/cable/property-tax/federal-zone/trust, each with its own due-date/route/account-number columns, plus 6 `LONGBINARY` columns holding scanned utility bills/IDs.
|
||||
- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). `COBRO3` looks like a filtered snapshot of `DATGRAL`, not a distinct entity.
|
||||
- `DATGRAL` / `COBRO3` (1,172 / 181 rows) — customer master data (name, MX + US address, phone, email, ID document, client-since date, fee, status). ~~`COBRO3` looks like a filtered snapshot of `DATGRAL`~~ **Reconciliation (step 2) corrected this: `COBRO3` is a charge batch (constant fee 75), not a customer snapshot — `DATGRAL` is the sole customer master.**
|
||||
- `PROFILE` (1,520) — per-property service enrollment flags, joins 1:1 with `DATMEX` by `NUMERID`.
|
||||
- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers, near-identical shape, apparent year/program snapshots rather than distinct data.
|
||||
- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs, again near-duplicate shapes across different periods/exports — needs de-duplication logic, not a straight union.
|
||||
- `EFECTIVO` (13,697), `EFECTIVO FM3` (627), `EFECTIVO_BACKUP` (12,387), `CHEQUE FM3` (157) — cash/check transaction ledgers. ~~apparent year/program snapshots rather than distinct data~~ **Reconciliation (step 2) corrected this: EFECTIVO and EFECTIVO_BACKUP are near-disjoint ledgers (folio collides; only 2 business-key matches) — migrate both, no folio de-dup. FM3 tables are a separate fee stream.**
|
||||
- `FEE ANUAL` (1,082), `datos2` (16,000), `fee15` (1,030), `billing` (0) — recurring billing/fee transaction logs. ~~near-duplicate shapes ... needs de-duplication~~ **Reconciliation (step 2) corrected this: disjoint billing runs from different periods (2017 / 2018 / 2025–26), zero real-identity overlap — straight union, no de-dup.**
|
||||
- `TRUSTVENCE` (549) — bank trust account fee due-dates.
|
||||
- `TIPO HIST` (2,301) — exchange-rate history (date/hour/rate), referenced by the `MONEDAS` (currency) field used throughout.
|
||||
- `TYPE OF TRX` (79) — ES/EN transaction-type lookup (already mirrored as `type_transactions` in the old MySQL schema — reuse that mapping).
|
||||
@@ -108,7 +108,10 @@ All tables get a surrogate `id` (uuid or serial) plus, where the row came from a
|
||||
Given the amount of near-duplicate/overlapping data across snapshot tables (multiple `EFECTIVO*` variants, multiple year-stamped billing tables, `COBRO3` vs `DATGRAL`), doing a direct Access → normalized-MySQL transform in one pass is risky — a bug loses the ability to check itself against the source.
|
||||
|
||||
1. **Raw staging load**: dump every non-scratch Access table 1:1 into a MySQL `staging` (per-source schema/database, e.g. `stg_utilities`/`stg_seguros`/`stg_scothia`) — same columns, minimal type coercion — via a Python script across all four source files. Already built and run against real data as `migration/load_staging.py` in the new repo — see Status below. This is the audit trail — nothing is transformed yet. **Extraction toolchain note:** the original build used `pyodbc` + the Windows Access ODBC driver; the project has since moved to a macOS machine, so the extraction layer (`migration/extract.py`) is being reworked to use **mdbtools** (`mdb-tables`/`mdb-export`, installed via Homebrew) instead. mdbtools has been verified against the real files to read table data, accented-column tables (which broke pyodbc's UTF-16 path — e.g. `PROPANO`), and per-table exports cleanly. mdbtools does **not** extract Forms/Reports/Queries, but those were already captured on Windows via DAO/COM and are frozen in `migration/objects.json` + `docs/LEGACY_DATABASES_OBJECTS.md`, so nothing is lost. The only piece needing extra handling under mdbtools is `LONGBINARY` blob/document extraction (step 4), where mdbtools emits the OLE wrapper — addressed when step 4 runs, not a blocker for steps 1–3.
|
||||
2. **Reconciliation pass**: for each set of overlapping tables (the `EFECTIVO` variants, the billing-period tables, `DATGRAL` vs `COBRO3`), write SQL that diffs them and produces a report of exact duplicates vs. genuinely distinct records, before deciding the union/de-dupe rule. Don't guess the rule up front — the data decides it.
|
||||
2. **Reconciliation pass** — **DONE** (`migration/reconcile.py` → `migration/RECONCILIATION.md`, run against the staged data). For each set of overlapping tables, it probes a deliberate *business key* (not naive full-row match, which gives a misleading ~0 overlap everywhere) and reports what's actually duplicate vs. distinct. **Outcome overturned all three of the plan's original "duplicate" assumptions — the union/de-dup rules below are now decided by the data:**
|
||||
- **`EFECTIVO` vs `EFECTIVO_BACKUP`:** *not* a live/backup duplicate pair. `folio` is a per-table sequential number that **collides** (12,363 shared folio numbers, all carrying different transactions); on the real business key `(cl,fecha,monto,conepto)` only **2 rows** overlap. They are near-disjoint ledgers (BACKUP ≈ 2017–2022, EFECTIVO recent). **Rule: migrate both**, keyed internally by `(legacy_source_table, folio)` provenance; no folio de-dup, don't drop BACKUP. `EFECTIVO FM3`/`CHEQUE FM3` are a separate `fee/tax/multa` stream, migrated distinctly. (`monedas` needs currency normalization — `PESOS`/`Pesos`/`DOLLARS` variants.)
|
||||
- **`datos2` vs `FEE ANUAL` vs `fee15`:** *not* near-duplicate exports. They are **disjoint billing runs from different periods** (`datos2` ≈2025–26, `FEE ANUAL` 2018-01-03, `fee15` 2017-01-10 — each period-table `refer` is a single constant); zero real-identity overlap. **Rule: migrate all three, no de-dup**; keep `datos2.due_date` (null for the others).
|
||||
- **`DATGRAL` vs `COBRO3`:** `COBRO3` is *not* a filtered snapshot of the customer master — its `fee` is a **constant 75** for all 181 rows (a saved charge worklist / "cobro" = collection), and every `num_id` already exists in `DATGRAL`. **Rule: `DATGRAL` is the sole utilities customer master; COBRO3 contributes zero customers** — model its 181 rows as charge transactions if worth keeping, else exclude.
|
||||
3. **Transform + load**: SQL/TypeScript scripts (versioned in the new repo under `migration/`) that read `staging`, apply the customer-matching and unpivot logic described above, and upsert into the real Prisma-managed tables, writing `legacy_*` provenance on every row.
|
||||
4. **Document extraction**: separate one-off script pulls every `LONGBINARY` column out to files (named by provenance key), uploads to object storage, and inserts the corresponding `*_documents` metadata row.
|
||||
5. **Validation**: row-count and spot-check reconciliation between `staging` and final tables (e.g., every legacy customer has exactly one `customers` row via `customer_legacy_refs`; sum of migrated transaction amounts per customer matches sum in `staging`).
|
||||
|
||||
@@ -164,10 +164,16 @@ Homebrew. No Access ODBC driver, `node_modules` not installed, staging Parquet n
|
||||
row (mdb-export's `-b` / error handling; confirm the bad row is skipped, not fatal).
|
||||
2. **Re-run staging** (`python load_staging.py --output-dir ./output`) to regenerate the staged
|
||||
data on this machine, then load into a local MySQL (`docker compose up mysql`) for SQL reconciliation.
|
||||
3. **Reconciliation pass** (plan step 2) against staged data — the `EFECTIVO*` variants, the
|
||||
billing-period tables, `DATGRAL` vs `COBRO3`. Produce diff/dedupe rules from the data.
|
||||
4. **Transform + load** (plan step 3), starting with `Customer`/`CustomerLegacyRef` — every
|
||||
other module depends on it. Use the `NUM UTIL` cross-ref + name/address matching.
|
||||
3. **Reconciliation pass** (plan step 2) — **DONE** (`migration/reconcile.py` → `RECONCILIATION.md`).
|
||||
Overturned all three "duplicate" assumptions: EFECTIVO/BACKUP are near-disjoint ledgers
|
||||
(folio collides; migrate both), the billing tables are disjoint period runs (union all, no
|
||||
de-dup), and COBRO3 is a charge batch not a customer snapshot (DATGRAL is sole master). The
|
||||
decided union/de-dup rules are in `PLAN.md` migration step 2.
|
||||
4. **Transform + load** (plan step 3) — NEXT. Start with `Customer`/`CustomerLegacyRef` (every
|
||||
other module depends on it): DATGRAL (utilities) is the master; join insurance `DATGRAL`
|
||||
via its `NUM UTIL` cross-ref + name/address matching; COBRO3 excluded from customers. Then
|
||||
the ledger union per the reconciliation rules (both EFECTIVO tables, all three billing tables,
|
||||
provenance-keyed; normalize `monedas` currency variants).
|
||||
5. **Customer module** in `apps/api`/`apps/web` (list/search/detail) — first real feature,
|
||||
Spanish-first UI. Run `npm install` at repo root first (node_modules absent here).
|
||||
6. **Sync design finalization** — now unblocked: map the internal→VPS replicated subset and the
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# Reconciliation Report — Overlapping Legacy Tables
|
||||
|
||||
_Migration plan step 2. Generated by `reconcile.py` from staged Parquet (`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py > RECONCILIATION.md`._
|
||||
|
||||
**Headline:** none of the three suspected "duplicate" groups are what the plan assumed. They are disjoint historical/period data or a mislabeled batch — see each group's decided rule.
|
||||
|
||||
## 1. Cash ledger — `EFECTIVO` vs `EFECTIVO_BACKUP`
|
||||
|
||||
- `EFECTIVO`: 13697 rows. `EFECTIVO_BACKUP`: 12387 rows.
|
||||
- `folio` is per-table sequential: 13697 distinct in EFECTIVO (= row count), 12369 in BACKUP. 12363 folio *numbers* appear in both.
|
||||
- **But of those 12363 shared folio numbers, 12363 carry a *different* transaction** (differ on ['cl', 'fecha', 'monto', 'conepto']). → `folio` collides; it is NOT a stable cross-table id.
|
||||
- On the real business key `['cl', 'fecha', 'monto', 'conepto']`: **2 rows in both**, 13663 only in EFECTIVO, 12385 only in BACKUP.
|
||||
- Date ranges are different eras: BACKUP is dominated by 2017–2022 records, EFECTIVO by recent ones.
|
||||
|
||||
**Decided rule:** the two tables are **near-disjoint ledgers**, not a live/backup duplicate pair (only 2 shared payments out of ~13k+12k). Migrate **both** into `transactions`, each row keyed internally by `(legacy_source_table, folio)` provenance — do **not** de-dup on `folio` (it collides) and do **not** drop BACKUP (it holds ~12k older payments absent from EFECTIVO). The 2 business-key matches are the only possible double-counts and should be spot-checked, but at that volume they don't threaten balance integrity.
|
||||
|
||||
> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — `fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct transactions, not reconciled against EFECTIVO.
|
||||
|
||||
> Data-quality note for the transform: `monedas` has case/spelling variants (`PESOS`/`Pesos`, `DOLARES`/`Dolares`/`DOLLARS`) — normalize currency on load.
|
||||
|
||||
## 2. Billing/fee logs — `datos2` vs `FEE ANUAL` vs `fee15`
|
||||
|
||||
- Rows: `datos2` 16000, `FEE ANUAL` 1082, `fee15` 1030.
|
||||
- `date` spans: `datos2` ('2025-01-08 00:00:00', '2026-05-15 00:00:00'), `FEE ANUAL` ('2018-01-03 00:00:00', '2018-01-03 00:00:00'), `fee15` ('2017-01-10 00:00:00', '2017-01-10 00:00:00') — **three different periods**.
|
||||
- `FEE ANUAL.refer` and `fee15.refer` are each a single constant value (1/1 distinct) — these are one-shot per-period fee runs, not general logs.
|
||||
- Real-identity overlap `['numid', 'date', 'chargecredit']`: FEE ANUAL∩datos2 = 0, fee15∩datos2 = 0, FEE ANUAL∩fee15 = 0.
|
||||
|
||||
**Decided rule:** `datos2`, `FEE ANUAL`, and `fee15` are **disjoint historical billing runs from different periods** (≈2025–26, 2018, 2017 respectively), not copies of one another. Migrate **all three** into the billing side of `transactions`, provenance-tagged; no de-dup is needed (zero real-identity overlap). Preserve `datos2.due_date` (the other two lack it — leave null for their rows).
|
||||
|
||||
## 3. Customer master — `DATGRAL` vs `COBRO3`
|
||||
|
||||
- `DATGRAL` 1172 rows / `COBRO3` 181 rows; both unique on `num_id`.
|
||||
- Every `COBRO3.num_id` is in `DATGRAL` (181/181); it adds **no new customer id**.
|
||||
- **`COBRO3.fee` is constant** (1 distinct value: ['75']) while `DATGRAL.fee` varies per customer — so COBRO3 is not a snapshot of the master's fee field.
|
||||
- Sampling shows COBRO3 rows carry the flat charge with names/addresses denormalized (some blank), i.e. a saved *charge worklist*, not the authoritative customer record.
|
||||
|
||||
**Decided rule:** `COBRO3` is a **billing/charge batch** ('cobro' = collection), not a customer table. `DATGRAL` is the sole utilities customer master. **Do not** merge COBRO3 into `customers` or let it overwrite any master field. If the flat charge has value as history, model those 181 rows as charge `transactions` (amount = the constant fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 from the migration entirely. Either way it contributes zero new customers.
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
"""
|
||||
Migration plan step 2: reconciliation pass.
|
||||
|
||||
The legacy Access files carry several sets of near-duplicate / overlapping
|
||||
tables (snapshot copies, per-period exports, filtered views). Before the
|
||||
transform+load (step 3) can union them, we need to know — from the data, not
|
||||
from a guess — which rows are exact duplicates across those tables and which
|
||||
are genuinely distinct, so the de-dup / union rule is decided by evidence.
|
||||
|
||||
Reads the staged Parquet produced by load_staging.py (`--output-dir`) and
|
||||
prints a Markdown reconciliation report. Regenerate with:
|
||||
|
||||
./.venv/bin/python reconcile.py > RECONCILIATION.md
|
||||
|
||||
Method note: naive full-row matching across these tables gives ~0 overlap,
|
||||
which is a trap — it hides *why*. The interesting question is whether two
|
||||
tables hold the SAME economic records under cosmetic differences, or
|
||||
genuinely DIFFERENT records. So each group is probed on a deliberate
|
||||
*business key* (the columns that identify the real-world thing) and the
|
||||
volatile/identity columns are examined separately. Every table went through
|
||||
the same mdb-export path, so identical source values serialize identically —
|
||||
string comparison on a chosen key is a valid identity test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
STG = Path(__file__).parent / "output" / "stg_utilities"
|
||||
_META = ("_legacy_source_table", "_row_num")
|
||||
_NULL = "∅"
|
||||
|
||||
|
||||
def load(name: str) -> pd.DataFrame:
|
||||
df = pd.read_parquet(STG / f"{name}.parquet")
|
||||
df = df[[c for c in df.columns if c not in _META]].copy()
|
||||
for c in df.columns:
|
||||
df[c] = df[c].astype("string").str.strip().fillna(_NULL)
|
||||
return df
|
||||
|
||||
|
||||
def keyset(df: pd.DataFrame, cols: list[str]) -> set[str]:
|
||||
return set(df[cols].agg("\x1f".join, axis=1))
|
||||
|
||||
|
||||
def overlap(a: pd.DataFrame, b: pd.DataFrame, cols: list[str]):
|
||||
ka, kb = keyset(a, cols), keyset(b, cols)
|
||||
return len(ka & kb), len(ka - kb), len(kb - ka)
|
||||
|
||||
|
||||
def constant_cols(df: pd.DataFrame) -> list[str]:
|
||||
return [c for c in df.columns if df[c].nunique(dropna=False) <= 1]
|
||||
|
||||
|
||||
def p(*a):
|
||||
print(*a)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
p("# Reconciliation Report — Overlapping Legacy Tables")
|
||||
p("")
|
||||
p("_Migration plan step 2. Generated by `reconcile.py` from staged Parquet "
|
||||
"(`output/stg_utilities`). Regenerate: `./.venv/bin/python reconcile.py "
|
||||
"> RECONCILIATION.md`._")
|
||||
p("")
|
||||
p("**Headline:** none of the three suspected \"duplicate\" groups are what "
|
||||
"the plan assumed. They are disjoint historical/period data or a "
|
||||
"mislabeled batch — see each group's decided rule.")
|
||||
p("")
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Group 1: EFECTIVO vs EFECTIVO_BACKUP
|
||||
# ------------------------------------------------------------------ #
|
||||
p("## 1. Cash ledger — `EFECTIVO` vs `EFECTIVO_BACKUP`")
|
||||
p("")
|
||||
ef, efb = load("efectivo"), load("efectivo_backup")
|
||||
biz = ['cl', 'fecha', 'monto', 'conepto'] # what identifies a real payment
|
||||
p(f"- `EFECTIVO`: {len(ef)} rows. `EFECTIVO_BACKUP`: {len(efb)} rows.")
|
||||
# folio behaviour
|
||||
fe = set(ef['folio']) - {_NULL}
|
||||
fb = set(efb['folio']) - {_NULL}
|
||||
p(f"- `folio` is per-table sequential: {ef['folio'].nunique()} distinct in "
|
||||
f"EFECTIVO (= row count), {efb['folio'].nunique()} in BACKUP. "
|
||||
f"{len(fe & fb)} folio *numbers* appear in both.")
|
||||
both_folio = fe & fb
|
||||
m = ef[ef['folio'].isin(both_folio)].drop_duplicates('folio').set_index('folio')
|
||||
n = efb[efb['folio'].isin(both_folio)].drop_duplicates('folio').set_index('folio')
|
||||
ci = m.index.intersection(n.index)
|
||||
folio_conflict = int((m.loc[ci, biz] != n.loc[ci, biz]).any(axis=1).sum())
|
||||
p(f"- **But of those {len(ci)} shared folio numbers, {folio_conflict} carry "
|
||||
f"a *different* transaction** (differ on {biz}). → `folio` collides; it is "
|
||||
"NOT a stable cross-table id.")
|
||||
b, oa, ob = overlap(ef, efb, biz)
|
||||
p(f"- On the real business key `{biz}`: **{b} rows in both**, {oa} only in "
|
||||
f"EFECTIVO, {ob} only in BACKUP.")
|
||||
p("- Date ranges are different eras: BACKUP is dominated by 2017–2022 "
|
||||
"records, EFECTIVO by recent ones.")
|
||||
p("")
|
||||
p("**Decided rule:** the two tables are **near-disjoint ledgers**, not a "
|
||||
"live/backup duplicate pair (only " + str(b) + " shared payments out of "
|
||||
"~13k+12k). Migrate **both** into `transactions`, each row keyed "
|
||||
"internally by `(legacy_source_table, folio)` provenance — do **not** "
|
||||
"de-dup on `folio` (it collides) and do **not** drop BACKUP (it holds "
|
||||
"~12k older payments absent from EFECTIVO). The " + str(b) + " business-"
|
||||
"key matches are the only possible double-counts and should be spot-"
|
||||
"checked, but at that volume they don't threaten balance integrity.")
|
||||
p("")
|
||||
p("> `EFECTIVO FM3` (627) and `CHEQUE FM3` (157) are a separate stream — "
|
||||
"`fee`/`tax`/`multa` columns instead of `monto` — and migrate as distinct "
|
||||
"transactions, not reconciled against EFECTIVO.")
|
||||
p("")
|
||||
p("> Data-quality note for the transform: `monedas` has case/spelling "
|
||||
"variants (`PESOS`/`Pesos`, `DOLARES`/`Dolares`/`DOLLARS`) — normalize "
|
||||
"currency on load.")
|
||||
p("")
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Group 2: datos2 vs FEE ANUAL vs fee15
|
||||
# ------------------------------------------------------------------ #
|
||||
p("## 2. Billing/fee logs — `datos2` vs `FEE ANUAL` vs `fee15`")
|
||||
p("")
|
||||
d2, fa, f15 = load("datos2"), load("fee_anual"), load("fee15")
|
||||
p(f"- Rows: `datos2` {len(d2)}, `FEE ANUAL` {len(fa)}, `fee15` {len(f15)}.")
|
||||
|
||||
def date_span(df):
|
||||
d = sorted(x for x in df['date'].unique() if x != _NULL)
|
||||
return (d[0], d[-1]) if d else ("∅", "∅")
|
||||
p(f"- `date` spans: `datos2` {date_span(d2)}, `FEE ANUAL` {date_span(fa)}, "
|
||||
f"`fee15` {date_span(f15)} — **three different periods**.")
|
||||
p(f"- `FEE ANUAL.refer` and `fee15.refer` are each a single constant value "
|
||||
f"({fa['refer'].nunique()}/{f15['refer'].nunique()} distinct) — these are "
|
||||
"one-shot per-period fee runs, not general logs.")
|
||||
idcols = ['numid', 'date', 'chargecredit']
|
||||
_, _, _ = overlap(fa, d2, idcols)
|
||||
b1 = overlap(fa, d2, idcols)[0]
|
||||
b2 = overlap(f15, d2, idcols)[0]
|
||||
b3 = overlap(fa, f15, idcols)[0]
|
||||
p(f"- Real-identity overlap `{idcols}`: FEE ANUAL∩datos2 = {b1}, "
|
||||
f"fee15∩datos2 = {b2}, FEE ANUAL∩fee15 = {b3}.")
|
||||
p("")
|
||||
p("**Decided rule:** `datos2`, `FEE ANUAL`, and `fee15` are **disjoint "
|
||||
"historical billing runs from different periods** (≈2025–26, 2018, 2017 "
|
||||
"respectively), not copies of one another. Migrate **all three** into the "
|
||||
"billing side of `transactions`, provenance-tagged; no de-dup is needed "
|
||||
"(zero real-identity overlap). Preserve `datos2.due_date` (the other two "
|
||||
"lack it — leave null for their rows).")
|
||||
p("")
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Group 3: DATGRAL vs COBRO3
|
||||
# ------------------------------------------------------------------ #
|
||||
p("## 3. Customer master — `DATGRAL` vs `COBRO3`")
|
||||
p("")
|
||||
dg, cb = load("datgral"), load("cobro3")
|
||||
dg_ids, cb_ids = set(dg['num_id']), set(cb['num_id'])
|
||||
p(f"- `DATGRAL` {len(dg)} rows / `COBRO3` {len(cb)} rows; both unique on "
|
||||
f"`num_id`.")
|
||||
p(f"- Every `COBRO3.num_id` is in `DATGRAL` ({len(cb_ids & dg_ids)}/"
|
||||
f"{len(cb_ids)}); it adds **no new customer id**.")
|
||||
fee_const = cb['fee'].nunique()
|
||||
fee_vals = list(cb['fee'].unique())[:3]
|
||||
p(f"- **`COBRO3.fee` is constant** ({fee_const} distinct value: {fee_vals}) "
|
||||
"while `DATGRAL.fee` varies per customer — so COBRO3 is not a snapshot of "
|
||||
"the master's fee field.")
|
||||
p("- Sampling shows COBRO3 rows carry the flat charge with names/addresses "
|
||||
"denormalized (some blank), i.e. a saved *charge worklist*, not the "
|
||||
"authoritative customer record.")
|
||||
p("")
|
||||
p("**Decided rule:** `COBRO3` is a **billing/charge batch** ('cobro' = "
|
||||
"collection), not a customer table. `DATGRAL` is the sole utilities "
|
||||
"customer master. **Do not** merge COBRO3 into `customers` or let it "
|
||||
"overwrite any master field. If the flat charge has value as history, "
|
||||
"model those 181 rows as charge `transactions` (amount = the constant "
|
||||
"fee) keyed to the existing DATGRAL customers — otherwise exclude COBRO3 "
|
||||
"from the migration entirely. Either way it contributes zero new "
|
||||
"customers.")
|
||||
p("")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user