feat(ops): verify the replica against the master, not just its own status

Every field the replication card showed was self-reported by the replica, and
the two most reassuring ones lie in the same failure. Seconds_Behind_Source
reads 0 when the I/O thread is disconnected — with no incoming event there is
nothing to measure staleness against — and Replica_IO_Running only says the
network thread is alive, not that it is receiving.

Two checks that ask the master instead:

- GTID drift, folded into the polled status. GTID_SUBTRACT(master, replica)
  counts transactions the master executed that the replica has not, so a silent
  disconnect shows up as a number that climbs instead of a lag that stays 0.
  It also isolates transactions carried under the replica's OWN server UUID —
  writes that exist nowhere on the master. There are currently 518 of them,
  residue of the seed dump load; inert while log_replica_updates is off, and a
  real divergence the day anyone promotes that box.

- A full row-by-row comparison behind a button, over the eight tables
  my.jorgecuadros.com reads. GTIDs prove the replica applied everything the
  master sent; they say nothing about rows changed here by another route, which
  is the one failure the rest of the card cannot see.

The comparison hashes CONVERT(col USING binary), not CAST(col AS CHAR). CAST
transcodes into the connection character set, and the two servers do not agree
on it: the client inside the master's container negotiates latin1, the replica's
utf8mb4. Every accented character in a Mexican name, street or note then hashes
differently and the tool reports a permanent mismatch on exactly the tables that
hold free text. Caught by building it and running it — customers.name gave
3344437324815 against 3339150372121 under CAST, and 3339150372121 on both under
CONVERT. All eight tables now match byte for byte.

Verify is POST and audited despite reading nothing: it full-scans both servers,
so a prefetch or a refresh must not be able to start one.

Tests cover the GTID interval arithmetic, which is inclusive at both ends and
easy to get wrong by one in the direction that hides a gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

apps/web/src/lib/api.ts

@@ -60,6 +60,7 @@ import type {
   OpsJob,
   OpsJobKind,
   ReplicationStatus,
+  VerifyResult,
   IngestFile,
   BackupFile,
   PropertyDetail,
@@ -964,6 +965,18 @@ export function getReplicationStatus(): Promise<ReplicationStatus> {
   return apiFetch<ReplicationStatus>("/ops/replication");
 }
 
+/**
+ * Compare every customer-visible table against the master, row by row.
+ *
+ * Slow by nature — it is a full scan of both servers — so it is a button, not
+ * part of the poll. Answers the question replication status cannot: GTIDs prove
+ * the replica applied everything the master sent, not that nothing else changed
+ * the rows here.
+ */
+export function verifyReplication(): Promise<VerifyResult> {
+  return apiFetch<VerifyResult>("/ops/replication/verify", { method: "POST" });
+}
+
 export function listOpsJobs(): Promise<OpsJob[]> {
   return apiFetch<OpsJob[]>("/ops/jobs");
 }

apps/web/src/lib/types.ts

@@ -101,10 +101,52 @@ export interface ReplicationStatus {
   lastSqlError: string | null;
   sourceHost: string | null;
   apply: ApplyProgress | null;
+  drift: GtidDrift | null;
   problem: string | null;
   checkedAt: string;
 }
 
+/**
+ * Executed-history gap between master and replica, in transactions.
+ *
+ * The only field on the card that is not self-reported by the replica, and the
+ * only one that catches a silently disconnected I/O thread: with no incoming
+ * events, `secondsBehind` reads 0 because there is nothing to measure staleness
+ * against, so a dead link looks perfectly current. This number grows instead.
+ *
+ * Null when the master could not be reached — "unknown" must not render as
+ * "identical".
+ */
+export interface GtidDrift {
+  missingTransactions: number;
+  missingGtidSet: string | null;
+  /**
+   * Transactions written on the replica under its own server UUID, which exist
+   * nowhere on the master. Non-zero is expected — restoring the seed dump
+   * executed its statements locally — and harmless while nothing replicates
+   * from this node.
+   */
+  localTransactions: number;
+}
+
+/** One table compared on both sides of the link. */
+export interface TableFingerprint {
+  table: string;
+  masterRows: number;
+  replicaRows: number;
+  masterChecksum: string;
+  replicaChecksum: string;
+  matches: boolean;
+}
+
+export interface VerifyResult {
+  identical: boolean;
+  tables: TableFingerprint[];
+  problem: string | null;
+  checkedAt: string;
+  elapsedMs: number;
+}
+
 /**
  * Relay-log apply progress, in source binlog bytes.
  *