Prometheus AlertManager

Static vs Dynamic Thresholds

Most of the default alert rules compare the value of a metric to a static hardcoded value.

Example of the hw_battery_charge_ratio metric which represents the charge percentage of a battery (between 0 and 1):

• a warn alert is active when the battery charge is below 0.5 (50%)
• a critical alert is active when the battery charge is below 0.3 (30%)

Note: When the value of hw_battery_charge_ratio falls below 0.3, both warn and critical alerts are active, since the condition matches both above alert rules.

However, some alerts cannot be configured with hardcoded values, like for temperature sensors. For such metrics, 2 additional metrics have been added to represent the warning and alarm thresholds. The default alert rules compare the base metric to the corresponding threshold metrics.

Example of the hw_temperature_celsius metric:

• a warn alert is active when the temperature is greater than the value of hw_temperature_limit_celsius{limit_type="high.degraded"}
• a critical alert is active when the temperature is greater than the value of hw_temperature_limit_celsius{limit_type="high.critical"}

- name: Temperature
  rules:
    - alert: Temperature-High-warn
      expr: hw_temperature_celsius >= ignoring(limit_type) hw_temperature_limit_celsius{limit_type="high.degraded"}
      labels:
        severity: "warn"
    - alert: Temperature-High-critical
      expr: hw_temperature_celsius >= ignoring(limit_type) hw_temperature_limit_celsius{limit_type="high.critical"}
      labels:
        severity: "critical"

The table below summarizes the metrics that should be compared to their corresponding dynamic threshold metrics:

Install

Copy the config/metricshub-alertmanager-rules-example.yaml file in your Prometheus installation folder and rename it metricshub-alertmanager-rules.yaml.

In the prometheus.yaml file, add the alerting configuration, as shown below:

rule_files:
  - metricshub-alertmanager-rules.yaml

Restart your Prometheus server to taken the new rules into account.