feat(ops): verify the replica against the master, not just its own status

Every field the replication card showed was self-reported by the replica, and
the two most reassuring ones lie in the same failure. Seconds_Behind_Source
reads 0 when the I/O thread is disconnected — with no incoming event there is
nothing to measure staleness against — and Replica_IO_Running only says the
network thread is alive, not that it is receiving.

Two checks that ask the master instead:

- GTID drift, folded into the polled status. GTID_SUBTRACT(master, replica)
  counts transactions the master executed that the replica has not, so a silent
  disconnect shows up as a number that climbs instead of a lag that stays 0.
  It also isolates transactions carried under the replica's OWN server UUID —
  writes that exist nowhere on the master. There are currently 518 of them,
  residue of the seed dump load; inert while log_replica_updates is off, and a
  real divergence the day anyone promotes that box.

- A full row-by-row comparison behind a button, over the eight tables
  my.jorgecuadros.com reads. GTIDs prove the replica applied everything the
  master sent; they say nothing about rows changed here by another route, which
  is the one failure the rest of the card cannot see.

The comparison hashes CONVERT(col USING binary), not CAST(col AS CHAR). CAST
transcodes into the connection character set, and the two servers do not agree
on it: the client inside the master's container negotiates latin1, the replica's
utf8mb4. Every accented character in a Mexican name, street or note then hashes
differently and the tool reports a permanent mismatch on exactly the tables that
hold free text. Caught by building it and running it — customers.name gave
3344437324815 against 3339150372121 under CAST, and 3339150372121 on both under
CONVERT. All eight tables now match byte for byte.

Verify is POST and audited despite reading nothing: it full-scans both servers,
so a prefetch or a refresh must not be able to start one.

Tests cover the GTID interval arithmetic, which is inclusive at both ends and
easy to get wrong by one in the direction that hides a gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

apps/api/src/ops/ops.controller.ts

@@ -104,6 +104,24 @@ export class OpsController {
     return this.replication.status();
   }
 
+  /**
+   * Full row-by-row comparison of the customer-visible tables against the master.
+   *
+   * POST rather than GET despite reading nothing: it is a full scan of both
+   * servers and must not be something a browser prefetch, a retry, or a refresh
+   * can set off. Audited for the same reason — it is a deliberate, costly act,
+   * and "who ran this while the site was slow" is a question worth answering.
+   */
+  @Post("replication/verify")
+  async verifyReplication(@Req() req: Request) {
+    const result = await this.replication.verify();
+    void this.audit.log(this.actingId(req), "ops.replication.verify", {
+      identical: result.identical,
+      elapsedMs: result.elapsedMs,
+    });
+    return result;
+  }
+
   @Get("jobs")
   listJobs() {
     return this.ops.listJobs();