An autonomous recovery loop that closes on its own

Observe. Veritas samples every poll and keeps the *slope* of the lag, not just the instantaneous value. A replica two seconds behind on three consecutive cycles is a different signal from one two seconds behind on a single isolated cycle, even when both readings trip the same absolute threshold.

Diagnose. The diagnose step correlates the lag slope with WAL pressure on the primary, replica liveness, and the recent commit rate. If the trend is compounding inside the threshold band, Veritas flags the cluster as drifting well before any individual reading crosses an absolute trip-wire.

Observe. The observe step reads quorum state, replica liveness, and lease ownership *across* the affected nodes in parallel — both sides of a partition report a healthy leader at the same instant, and that disagreement is itself the signal.

Diagnose. Diagnosis refuses any conclusion that cannot prove exactly one eligible leader. If two nodes claim the lease simultaneously, the loop labels the cluster ambiguous and refuses to act; the "true" answer is that there is no canonical leader to act on yet.

Observe. Consensus metrics look clean — the failover succeeded, replication has re-established, the alert has cleared. Veritas does not treat a successful promote as a closed incident; the observe step keeps watching read-view freshness on the new leader even after the consensus layer reports green.

Diagnose. The diagnose step checks the read view against the last-known-committed state and flags a recovery as incomplete unless an end-to-end read against the new leader returns the data the cluster most recently wrote. A heartbeat is not a probe.

Act. The act step generates a remediation plan: pause the lowest-priority write classes long enough for the replica to drain, throttle the WAL receiver if the pressure is bidirectional, and keep the primary accepting critical-path traffic. The plan is risk-scored against the current commit rate and refused only if the throttle itself would breach a write-side SLO.

Verify. Verification replays a write–read probe from the application layer against the replica, not against the leader, and confirms read-your-writes holds across both nodes before closing the incident. A second probe runs five minutes later to catch drift that re-compounded after the throttle released.

Act. Recovery refuses to run. The risk-scored plan is generated but only accepted when the diagnose step can prove a single canonical leader; ambiguous quorum state is a hard refusal, not a degraded execution. The agent holds the same plan in memory and waits for the partition to resolve before acting.

Verify. Once the partition heals, exactly one promote is executed against the node the verify step proves is canonical, and the action is written to the tamper-evident ledger with the leader-ownership proof as evidence. The plan is single-shot; a second promote is never issued against any node that has not been re-proven canonical at the moment of action.

Act. The act step rolls the recovery back if the verify probe fails — promotion is reversed, the old leader is re-eligible, and the plan retries against a different replica with the freshest combined WAL + read-view consistency. Each retry is logged to the audit trail with the exact signal that failed the probe.

Verify. Verification is a real, end-to-end read against the new leader returning the data most recently committed cluster-wide, not a monitoring heartbeat. The incident only closes when that probe succeeds; the recovery is explicitly "incomplete" until the read view on the promoted node is proven current.