feat(ops): verify the replica against the master, not just its own status

Every field the replication card showed was self-reported by the replica, and
the two most reassuring ones lie in the same failure. Seconds_Behind_Source
reads 0 when the I/O thread is disconnected — with no incoming event there is
nothing to measure staleness against — and Replica_IO_Running only says the
network thread is alive, not that it is receiving.

Two checks that ask the master instead:

- GTID drift, folded into the polled status. GTID_SUBTRACT(master, replica)
  counts transactions the master executed that the replica has not, so a silent
  disconnect shows up as a number that climbs instead of a lag that stays 0.
  It also isolates transactions carried under the replica's OWN server UUID —
  writes that exist nowhere on the master. There are currently 518 of them,
  residue of the seed dump load; inert while log_replica_updates is off, and a
  real divergence the day anyone promotes that box.

- A full row-by-row comparison behind a button, over the eight tables
  my.jorgecuadros.com reads. GTIDs prove the replica applied everything the
  master sent; they say nothing about rows changed here by another route, which
  is the one failure the rest of the card cannot see.

The comparison hashes CONVERT(col USING binary), not CAST(col AS CHAR). CAST
transcodes into the connection character set, and the two servers do not agree
on it: the client inside the master's container negotiates latin1, the replica's
utf8mb4. Every accented character in a Mexican name, street or note then hashes
differently and the tool reports a permanent mismatch on exactly the tables that
hold free text. Caught by building it and running it — customers.name gave
3344437324815 against 3339150372121 under CAST, and 3339150372121 on both under
CONVERT. All eight tables now match byte for byte.

Verify is POST and audited despite reading nothing: it full-scans both servers,
so a prefetch or a refresh must not be able to start one.

Tests cover the GTID interval arithmetic, which is inclusive at both ends and
easy to get wrong by one in the direction that hides a gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

apps/api/src/ops/replication.spec.ts

@@ -1,4 +1,9 @@
-import { applyProgress, replicaField } from "./replication.service";
+import {
+  applyProgress,
+  countGtids,
+  gtidsForUuid,
+  replicaField,
+} from "./replication.service";
 
 /**
  * Verbatim shape of `SHOW REPLICA STATUS\G` from the live replica, trimmed to
@@ -178,3 +183,72 @@ describe("applyProgress", () => {
     expect(applyProgress(positions("binlog.000042", "NULL", "binlog.000042", 400))).toBeNull();
   });
 });
+
+/**
+ * Real GTID sets from the live pair, captured 2026-08-06. The replica's own
+ * server UUID (3b103283…) carries the transactions the seed dump load executed
+ * locally; the master's UUID (defc34e2…) carries the replicated history.
+ */
+const REPLICA_EXECUTED =
+  "3b103283-8f15-11f1-a52b-020017027b33:1-513," +
+  "defc34e2-8c5d-11f1-8e58-52c4c853bce8:1-525";
+const REPLICA_UUID = "3b103283-8f15-11f1-a52b-020017027b33";
+
+describe("countGtids", () => {
+  it("counts an inclusive range at both ends", () => {
+    // 1-5 is five transactions. Off-by-one here understates the gap, which is
+    // the direction that hides a problem.
+    expect(countGtids("defc34e2-8c5d-11f1-8e58-52c4c853bce8:1-5")).toBe(5);
+  });
+
+  it("counts a bare single transaction", () => {
+    expect(countGtids("defc34e2-8c5d-11f1-8e58-52c4c853bce8:7")).toBe(1);
+  });
+
+  it("sums several intervals under one UUID", () => {
+    expect(countGtids("defc34e2-8c5d-11f1-8e58-52c4c853bce8:1-5:8:10-12")).toBe(9);
+  });
+
+  it("sums across UUIDs, including the wrapped form MySQL prints", () => {
+    expect(countGtids(REPLICA_EXECUTED)).toBe(513 + 525);
+    // `gtid_executed` comes back wrapped once it holds more than one UUID.
+    expect(countGtids(REPLICA_EXECUTED.replace(",", ",\n"))).toBe(513 + 525);
+  });
+
+  /** An empty subtraction result is the caught-up case and must be zero. */
+  it("returns 0 for an empty or blank set", () => {
+    expect(countGtids("")).toBe(0);
+    expect(countGtids("   \n ")).toBe(0);
+  });
+
+  /**
+   * MySQL 8.4 allows an alphanumeric tag between the UUID and the intervals.
+   * It is not an interval and must not be counted as one.
+   */
+  it("skips a tag without counting it", () => {
+    expect(countGtids("defc34e2-8c5d-11f1-8e58-52c4c853bce8:mytag:1-3")).toBe(3);
+  });
+});
+
+describe("gtidsForUuid", () => {
+  it("isolates the replica's own transactions from the replicated history", () => {
+    expect(countGtids(gtidsForUuid(REPLICA_EXECUTED, REPLICA_UUID))).toBe(513);
+  });
+
+  it("returns nothing for a UUID that is not in the set", () => {
+    expect(gtidsForUuid(REPLICA_EXECUTED, "00000000-0000-0000-0000-000000000000")).toBe("");
+  });
+
+  /**
+   * The colon matters. Without it a UUID prefix would match a longer UUID that
+   * merely starts the same way, and the replica's local writes would be
+   * over-reported.
+   */
+  it("does not match on a bare prefix", () => {
+    expect(gtidsForUuid(REPLICA_EXECUTED, "3b103283")).toBe("");
+  });
+
+  it("returns nothing when the UUID is blank", () => {
+    expect(gtidsForUuid(REPLICA_EXECUTED, "")).toBe("");
+  });
+});