Update ClickHouse Keeper Prometheus Alert Rules

fix #1328

Summary

This PR updates the ClickHouse Keeper Prometheus alert rules to use modern metrics exposed by recent ClickHouse Keeper versions (23.x+) and adds several new critical alerts for better observability.

Problem

The existing alert rules were using outdated ZooKeeper-compatible metrics (zk_*) that are no longer exposed by modern ClickHouse Keeper versions. This caused the alerts to fail silently, leaving the cluster without proper monitoring.

Fixes #1328

Changes

Updated Existing Alerts (5)

1. ClickHouseKeeperDown

   • Removed obsolete zk_ruok metric
   • Now uses only up{} metric for availability monitoring

2. ClickHouseKeeperHighLatency

   • Replaced zk_max_latency → ClickHouseAsyncMetrics_KeeperMaxLatency
   • Updated unit from "ticks" to "ms" for clarity

3. ClickHouseKeeperOutstandingRequests

   • Replaced zk_outstanding_requests → ClickHouseMetrics_KeeperOutstandingRequests

4. ClickHouseKeeperHighEphemeralNodes

   • Replaced zk_ephemerals_count → ClickHouseAsyncMetrics_KeeperEphemeralsCount
   • Updated documentation link to ClickHouse Keeper docs

5. ClickHouseKeeperHighFileDescriptors

   • Removed (old metric zk_open_file_descriptor_count not available)
   • Replaced with new percentage-based alert (see below)

New Alerts Added (5)

6. ClickHouseKeeperCommitsFailed (Critical)

   • Detects failed Raft commits indicating consensus issues
   • Uses: ClickHouseProfileEvents_KeeperCommitsFailed

7. ClickHouseKeeperSnapshotCreationsFailed (High)

   • Monitors failed snapshot creations that could lead to log accumulation
   • Uses: ClickHouseProfileEvents_KeeperSnapshotCreationsFailed

8. ClickHouseKeeperLostQuorum (Critical)

   • Alerts when leader loses quorum (cluster cannot commit operations)
   • Uses: ClickHouseAsyncMetrics_KeeperSyncedFollowers + KeeperIsLeader

9. ClickHouseKeeperMemorySoftLimitExceeded (Warning)

   • Monitors memory soft limit violations
   • Uses: ClickHouseAsyncMetrics_KeeperIsExceedingMemorySoftLimitHit

10. ClickHouseKeeperHighFileDescriptorUsage (Warning)

    • Alerts when file descriptor usage exceeds 80% of available FDs
    • Uses: ClickHouseAsyncMetrics_KeeperOpenFileDescriptorCount / KeeperMaxFileDescriptorCount

Alert Severity Distribution

• Critical (3): Down, CommitsFailed, LostQuorum
• High (2): OutstandingRequests, SnapshotCreationsFailed
• Warning (5): HighLatency, HighEphemeralNodes, MemorySoftLimit, HighFileDescriptorUsage

Regards,
Kilian