Alerting #

Monitoring without alerting is like a dashboard nobody watches — problems can happen without anyone knowing. Alerting is the layer above monitoring that proactively notifies the team when abnormal conditions are detected. But bad alerting can be as dangerous as no alerting at all: too many non-actionable notifications make the team numb and ignore the alerts that actually matter. This article discusses how to automate Alertmanager setup with Ansible, design structured notification routes, and build actionable alert patterns without flooding the on-call team.

Alerting Architecture #

The alerting flow in the Prometheus stack is divided into two clear stages. Prometheus is responsible for detection — evaluating alert rules against scraped metrics and turning them into firing or resolved status. Alertmanager is responsible for distribution — receiving alerts from Prometheus, then deciding who should know, on which channel, how often, and what should be suppressed. This separation is important: if we combine them, we lose all the grouping, deduplication, inhibition, and silencing features that make alerting practical at production scale.

flowchart TD
    P[Prometheus] -->|evaluate alert rules| P
    P -->|alert firing| AM[Alertmanager]

    AM --> G1{"Grouping<br/>by alertname+env+job"}
    G1 --> G2{"Dedup<br/>cluster-wide"}
    G2 --> R{"Routing tree<br/>matchers"}

    R -->|"severity=critical"| PD["PagerDuty<br/>on-call paging"]
    R -->|"severity=warning"| SL["Slack<br/>#alerts-warning"]
    R -->|"environment=production"| SP["Slack<br/>#alerts-production"]
    R -->|"job=db"| SD["Slack<br/>#team-database"]
    R -->|daily digest| EM["Email<br/>nightly digest"]

    AM -. "inhibit" .-> AM
    AM -. "silence" .-> AM

The three core Alertmanager features that distinguish it from “raw notification sending” are grouping (combining related alerts into one notification), inhibition (suppressing derived alerts when the parent alert is already firing), and silencing (hiding alerts for a certain period, usually during maintenance). All three will be discussed in depth in the following sections.

Alertmanager Installation and Configuration #

The Ansible role for Alertmanager must handle three things separately: binary download, configuration deployment, and systemd service setup. This separation lets each component be redeployed without repeating the others:

# roles/alertmanager/tasks/main.yml
---
- name: Create the alertmanager user
  user:
    name: alertmanager
    system: true
    shell: /usr/sbin/nologin
    home: /var/lib/alertmanager
    create_home: true

- name: Download Alertmanager
  get_url:
    url: >
      https://github.com/prometheus/alertmanager/releases/download/
      v{{ alertmanager_version }}/alertmanager-{{ alertmanager_version }}.linux-amd64.tar.gz      
    dest: /tmp/alertmanager.tar.gz
    checksum: "sha256:{{ alertmanager_checksum }}"

- name: Extract and install the binary
  unarchive:
    src: /tmp/alertmanager.tar.gz
    dest: /tmp/
    remote_src: true

- name: Copy the binary to /usr/local/bin
  copy:
    src: "/tmp/alertmanager-{{ alertmanager_version }}.linux-amd64/alertmanager"
    dest: /usr/local/bin/alertmanager
    owner: root
    group: root
    mode: '0755'
    remote_src: true

- name: Create the configuration and data directories
  file:
    path: "{{ item }}"
    state: directory
    owner: alertmanager
    group: alertmanager
    mode: '0750'
  loop:
    - /etc/alertmanager
    - /etc/alertmanager/templates
    - /var/lib/alertmanager

- name: Deploy the Alertmanager configuration
  template:
    src: alertmanager.yml.j2
    dest: /etc/alertmanager/alertmanager.yml
    owner: alertmanager
    group: alertmanager
    mode: '0640'
    validate: "alertmanager --config.file=%s --storage.path=/tmp/amtest check-config"
  notify: Restart Alertmanager

- name: Deploy the systemd unit file
  template:
    src: alertmanager.service.j2
    dest: /etc/systemd/system/alertmanager.service
  notify:
    - Reload systemd
    - Restart Alertmanager

- name: Run Alertmanager
  systemd:
    name: alertmanager
    state: started
    enabled: true

Notice the use of validate on the configuration task — Ansible runs alertmanager --config.file=%s --storage.path=/tmp/amtest check-config against the newly rendered file before moving it to its final location. If the YAML syntax is wrong or a field is invalid, Ansible fails before the broken configuration overwrites the old one. This pattern is mandatory for all critical service configurations.

Routing and Notification Configuration #

The routing tree is Alertmanager’s brain. It determines to whom and where a specific alert should arrive. The configuration below shows a multi-tier pattern common in companies with multiple environments and multiple teams:

{# roles/alertmanager/templates/alertmanager.yml.j2 #}
global:
  resolve_timeout: 5m
  slack_api_url: "{{ vault_slack_webhook_url }}"
  pagerduty_url: "https://events.pagerduty.com/v2/enqueue"
  smtp_smarthost: "{{ smtp_host }}:{{ smtp_port }}"
  smtp_from: "alertmanager@{{ mail_domain }}"
  smtp_require_tls: true

templates:
  - /etc/alertmanager/templates/*.tmpl

route:
  # Default receiver if no route matches
  receiver: slack-warnings
  group_by: ['alertname', 'environment', 'job']
  group_wait: 30s           # Wait 30 seconds before sending the first notification
  group_interval: 5m        # Send updates every 5 minutes if new alerts appear in the group
  repeat_interval: 4h       # Repeat notifications every 4 hours if not resolved
  routes:
    # Critical alerts → PagerDuty (for on-call duty)
    - matchers:
        - severity = critical
      receiver: pagerduty-critical
      repeat_interval: 1h    # Repeat more often for critical
      continue: true         # Also keep evaluating the routes below

    # Alerts for the production environment → separate channel
    - matchers:
        - environment = production
        - severity =~ "warning|critical"
      receiver: slack-production
      continue: true         # Also continue to the next routes

    # Database alerts → database team
    - matchers:
        - job =~ "postgresql|mysql|redis"
      receiver: slack-database-team
      continue: true

    # Nightly digest for informational alerts
    - matchers:
        - severity = info
      receiver: email-digest
      repeat_interval: 24h
      active_time_intervals:
        - business-hours

receivers:
  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-warning"
        send_resolved: true
        title: >
          {{ '{{' }} template "slack.title" . {{ '}}' }}
        text: >
          {{ '{{' }} template "slack.text" . {{ '}}' }}
        actions:
          - type: button
            text: "Acknowledge"
            url: "{{ '{{' }} .CommonAnnotations.ack_url {{ '}}' }}"
          - type: button
            text: "Runbook"
            url: "{{ '{{' }} .CommonAnnotations.runbook_url {{ '}}' }}"

  - name: slack-production
    slack_configs:
      - channel: "#alerts-production"
        send_resolved: true
        icon_emoji: ":fire:"
        mention_users:
          - "{{ vault_oncall_user_id }}"

  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: "{{ vault_pagerduty_routing_key }}"
        description: >
          {{ '{{' }} template "pagerduty.description" . {{ '}}' }}
        severity: critical
        details:
          environment: "{{ '{{' }} .CommonLabels.environment {{ '}}' }}"
          service: "{{ '{{' }} .CommonLabels.job {{ '}}' }}"
          runbook: "{{ '{{' }} .CommonAnnotations.runbook_url {{ '}}' }}"

  - name: slack-database-team
    slack_configs:
      - channel: "#team-database"
        send_resolved: true
        mention_users:
          - "{{ vault_db_oncall_user_id }}"

  - name: email-digest
    email_configs:
      - to: "{{ alerts_digest_recipients | join(',') }}"
        send_resolved: true
        headers:
          Subject: "[Daily Alert Digest] {{ '{{' }} .CommonLabels.environment {{ '}}' }}"

inhibit_rules:
  # If there's a critical alert from the same host, suppress its warning alerts
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: ['instance', 'job']

  # If there's a "HostDown" alert, suppress all metric alerts from that host
  - source_matchers:
      - alertname = HostDown
    target_matchers:
      - severity =~ "warning|critical"
    equal: ['instance']

  # If there's a "ClusterUnreachable" alert, suppress all per-node alerts
  - source_matchers:
      - alertname = ClusterUnreachable
    target_matchers:
      - alertname =~ "Node.*|Host.*"
    equal: ['cluster']

time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '08:00'
            end_time: '18:00'

Important parameters we need to understand:

• group_by — the labels used to combine alerts. Alerts with the same alertname, environment, and job are sent as one notification, not three separate ones.
• group_wait — the pause after the first alert fires before the first notification is sent. Useful for waiting whether more alerts will appear that can be grouped.
• group_interval — how often additional notifications are sent if new alerts appear in the same group.
• repeat_interval — how often notifications are repeated for still-firing alerts. Set shorter for critical (1 hour) and longer for warning (4-24 hours).
• continue: true — also evaluate the routes below it. Without continue, the first matching route stops the evaluation.
• inhibit_rules — when a source alert is firing, target alerts are suppressed if the equal labels match.

Sequence Diagram: Alert Flow from Detection to Notification #

To understand how all components work together, look at the sequence below. Starting from Prometheus evaluating the rule, until the alert appears on the right channel:

sequenceDiagram
    participant App as Application
    participant Prom as Prometheus
    participant AM as Alertmanager
    participant Slack as Slack
    participant PD as PagerDuty

    App->>Prom: "Expose /metrics (scrape every 15s)"
    Prom->>Prom: "Evaluate alert rules<br/>(every 15s)"
    Note over Prom: "Alert rule:<br/>expr: error_rate > 5%<br/>for: 5m"
    Prom->>Prom: "Status: PENDING (less than 5m)"
    Prom->>Prom: "Status: FIRING (after 5m)"
    Prom->>AM: "POST /api/v1/alerts<br/>(firing alert)"
    AM->>AM: "Group by alertname+env+job"
    AM->>AM: Apply inhibition rules
    AM->>AM: Check active silences
    AM->>AM: Apply routing tree
    AM->>AM: "group_wait 30s (wait for other alerts)"
    AM->>Slack: "POST webhook to #alerts-warning"
    AM->>PD: POST event to PagerDuty
    Slack-->>AM: 200 OK
    PD-->>AM: 202 Accepted
    Note over Prom,AM: "Repeat interval 4h<br/>while the alert is still firing"
    Prom->>AM: "Status update (still firing)"
    AM->>Slack: "Send reminder (resolved=false)"

Notice two critical things: (1) for: 5m in the alert rule makes the alert only fire after the condition holds for 5 minutes — this prevents false alarms from momentary spikes. (2) group_wait in Alertmanager waits 30 seconds before sending, giving related alerts a chance to be grouped.

Informative Notification Templates #

An alert that only says “something went wrong” is useless. A good template includes all the information needed to start an investigation — host, description, start time, and a link to the runbook. Here’s a Slack template that separates information into structured fields:

{# /etc/alertmanager/templates/slack.tmpl #}
{{ '{{' }} define "slack.title" {{ '}}' }}
[{{ '{{' }} .Status | toUpper {{ '}}' }}{{ '{{' }} if eq .Status "firing" {{ '}}' }}:{{ '{{' }} .Alerts.Firing | len {{ '}}' }}{{ '{{' }} end {{ '}}' }}]
{{ '{{' }} .CommonLabels.alertname {{ '}}' }} — {{ '{{' }} .CommonLabels.environment {{ '}}' }}
{{ '{{' }} end {{ '}}' }}

{{ '{{' }} define "slack.text" {{ '}}' }}
{{ '{{' }} range .Alerts {{ '}}' }}
*Host:* `{{ '{{' }} .Labels.instance {{ '}}' }}`
*Service:* {{ '{{' }} .Labels.job {{ '}}' }}
*Severity:* {{ '{{' }} .Labels.severity | toUpper {{ '}}' }}
*Description:* {{ '{{' }} .Annotations.description {{ '}}' }}
*Started:* {{ '{{' }} .StartsAt.Format "2006-01-02 15:04:05 WIB" {{ '}}' }}
*Runbook:* <{{ '{{' }} .Annotations.runbook_url {{ '}}' }}|Open Runbook>
{{ '{{' }} if .Annotations.dashboard_url {{ '}}' }}
*Dashboard:* <{{ '{{' }} .Annotations.dashboard_url {{ '}}' }}|View Dashboard>
{{ '{{' }} end {{ '}}' }}
---
{{ '{{' }} end {{ '}}' }}
{{ '{{' }} end {{ '}}' }}

The PagerDuty template must be more concise because its display is different — focus on the short description (one sentence), severity (paging policy depends on it), and payload details that appear on the incident page:

{# /etc/alertmanager/templates/pagerduty.tmpl #}
{{ '{{' }} define "pagerduty.description" {{ '}}' }}
[{{ '{{' }} .Status | toUpper {{ '}}' }}] {{ '{{' }} .CommonLabels.alertname {{ '}}' }} on {{ '{{' }} .CommonLabels.environment {{ '}}' }}: {{ '{{' }} .CommonAnnotations.summary {{ '}}' }}
{{ '{{' }} end {{ '}}' }}

{{ '{{' }} define "pagerduty.details" {{ '}}' }}
{
  "firing": {{ '{{' }} .Alerts.Firing | len {{ '}}' }},
  "resolved": {{ '{{' }} .Alerts.Resolved | len {{ '}}' }},
  "environment": "{{ '{{' }} .CommonLabels.environment {{ '}}' }}",
  "service": "{{ '{{' }} .CommonLabels.job {{ '}}' }}",
  "instance": "{{ '{{' }} (index .Alerts 0).Labels.instance {{ '}}' }}",
  "runbook_url": "{{ '{{' }} .CommonAnnotations.runbook_url {{ '}}' }}"
}
{{ '{{' }} end {{ '}}' }}

Separate templates per channel: Slack needs Markdown and interactivity (mentions, buttons), PagerDuty needs a JSON payload that automation can parse, email needs plain text. Avoid one big template trying to serve everything — the formatting will clash.

Managing Silences with Ansible #

During scheduled maintenance, we don’t want alerts from nodes being restarted to flood the channel. A silence is the official way to tell Alertmanager: “don’t send alerts matching this matcher during this period”. Ansible can automate the silence lifecycle — created before maintenance, expired after it:

# playbooks/create-silence.yml
---
- name: Create a silence in Alertmanager during maintenance
  hosts: localhost
  vars:
    alertmanager_url: "http://alertmanager.internal:9093"
    silence_duration_hours: 4
    silence_comment: "Scheduled maintenance — {{ ansible_date_time.date }}"

  tasks:
    - name: Calculate the silence end time
      set_fact:
        silence_end: >-
          {{ (ansible_date_time.epoch | int + silence_duration_hours * 3600) | strftime('%Y-%m-%dT%H:%M:%S.000Z') }}          

    - name: Create the silence in Alertmanager
      uri:
        url: "{{ alertmanager_url }}/api/v2/silences"
        method: POST
        body_format: json
        body:
          matchers:
            - name: environment
              value: "{{ env }}"
              isRegex: false
            - name: job
              value: "{{ silence_job | default('.*') }}"
              isRegex: "{{ silence_job is regex('\\\\*') or silence_job is regex('\\\\.') }}"
          startsAt: "{{ ansible_date_time.iso8601 }}"
          endsAt: "{{ silence_end }}"
          comment: "{{ silence_comment }}"
          createdBy: "ansible-automation"
        status_code: 200
      register: silence_result
      no_log: true

    - name: Display the created silence ID
      debug:
        msg: "Silence created with ID: {{ silence_result.json.silenceID }}"

    - name: Save the silence ID to expire later
      copy:
        content: "{{ silence_result.json.silenceID }}"
        dest: "/tmp/silence-{{ env }}-{{ silence_job | default('all') }}.id"
        mode: '0600'
      delegate_to: localhost

Playbook to remove the silence after maintenance completes:

# playbooks/expire-silence.yml
---
- name: Expire the silence after maintenance completes
  hosts: localhost
  vars:
    alertmanager_url: "http://alertmanager.internal:9093"
    silence_id_file: "/tmp/silence-{{ env }}-{{ silence_job | default('all') }}.id"

  tasks:
    - name: Read the silence ID
      slurp:
        src: "{{ silence_id_file }}"
      register: silence_id_b64
      ignore_errors: true

    - name: Decode the silence ID
      set_fact:
        silence_id: "{{ silence_id_b64.content | b64decode | trim }}"
      when: silence_id_b64 is succeeded

    - name: Expire the silence via the API
      uri:
        url: "{{ alertmanager_url }}/api/v2/silences/{{ silence_id }}"
        method: DELETE
        status_code: 200
      when: silence_id_b64 is succeeded
      ignore_errors: true

    - name: Remove the silence ID file
      file:
        path: "{{ silence_id_file }}"
        state: absent

For maintenance workflow integration, call the create-silence.yml playbook before maintenance starts, and expire-silence.yml after it finishes — either manually or as part of a Jenkins/GitHub Actions job.

Decision Tree: Choosing the Right Severity #

Wrong severity is the main source of alert fatigue. The rule of thumb: critical = needs response within 15 minutes (paging), warning = needs response within working hours (ticket), info = no immediate response needed (context only). The following decision tree helps teams decide severity when creating a new alert:

flowchart TD
    A["Create a new alert"] --> B{"Is the service<br/>unusable<br/>by users?"}
    B -->|"Yes"| C{"Does this need<br/>a 24/7<br/>response?"}
    C -->|"Yes"| D["severity: critical<br/>route to PagerDuty"]
    C -->|"No"| E["severity: warning<br/>route to Slack #alerts"]

    B -->|"No"| F{"Could this become<br/>a big problem<br/>if left alone?"}
    F -->|"Yes"| G{"Is there a performance<br/>impact or<br/>data loss risk?"}
    G -->|"Yes"| E
    G -->|"No"| H["severity: warning<br/>route to team Slack channel"]

    F -->|"No"| I{"Is this useful<br/>information for<br/>investigation?"}
    I -->|"Yes"| J["severity: info<br/>route to email digest"]
    I -->|"No"| K["Remove the alert<br/>not needed"]

A frequently violated rule: “info” alerts will still be sent to a channel if there’s a matching route. If we create a severity=info alert but don’t have a route accepting severity=info, that alert falls into the default receiver (slack-warnings) and becomes noise. Always create a dedicated receiver for info if we actually want to send info alerts.

Notification Channel Comparison #

Not all alerts are suitable for the same channel. This table compares the three main channels and when each is most appropriate:

Practical recommendation: PagerDuty only for severity=critical that truly pages, Slack for real-time warnings and info, email for nightly digests of info alerts that need to be seen during working hours. Mixing all three without rules only produces ignored alerts on every channel.

Alert Fatigue Reduction Patterns #

Alert fatigue happens when teams receive too many non-actionable notifications. Several proven patterns significantly reduce noise:

# Good alert rules have:
# 1. Meaningful thresholds (not arbitrary)
# 2. Sufficient 'for' duration (avoiding flapping)
# 3. Correct severity
# 4. Actionable annotations
# 5. A runbook link

# ANTI-PATTERN: an overly sensitive alert
- alert: HighCPU
  expr: cpu_usage > 80
  # No 'for' — alerts on every momentary spike
  # 80% threshold is too low — CPU often passes 80% during normal bursts
  # No runbook — on-call doesn't know what to do

# CORRECT: an alert with duration, meaningful threshold, and runbook
- alert: SustainedHighCPU
  expr: >
    avg by(instance) (
      rate(node_cpu_seconds_total{mode!="idle"}[5m])
    ) * 100 > 90    
  for: 15m              # Must persist 15 minutes before alerting
  labels:
    severity: warning
  annotations:
    summary: "Sustained high CPU usage on {{ '{{' }} $labels.instance {{ '}}' }}"
    description: "CPU {{ '{{' }} $value | printf "%.1f" {{ '}}' }}% for the last 15 minutes"
    runbook_url: "https://wiki.company.com/runbooks/high-cpu"

# ANTI-PATTERN: an alert that only mentions the problem without context
- alert: DiskFull
  expr: node_filesystem_avail_bytes{mountpoint="/"} < 1000000000
  annotations:
    summary: "Disk almost full"

# CORRECT: an alert that mentions impact and next steps
- alert: DiskSpaceCritical
  expr: >
    (node_filesystem_avail_bytes{mountpoint="/"}
     / node_filesystem_size_bytes{mountpoint="/"}) < 0.05    
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Critical disk on {{ '{{' }} $labels.instance {{ '}}' }} (remaining {{ '{{' }} $value | humanizePercentage {{ '}}' }})"
    description: >
      Only {{ '{{' }} $value | humanizePercentage {{ '}}' }} disk remaining.
      Services will start failing within 30 minutes if no action is taken.      
    runbook_url: "https://wiki.company.com/runbooks/disk-full"
    dashboard_url: "https://grafana.company.com/d/disk-usage?var-host={{ '{{' }} $labels.instance {{ '}}' }}"

# ANTI-PATTERN: duplicate alerts (one problem triggers many alerts)
# Disk usage + inode usage + write latency + service crash — all firing together
# when the disk is truly full

# CORRECT: use inhibition to suppress derived alerts
# In alertmanager.yml
inhibit_rules:
  - source_matchers:
      - alertname = DiskSpaceCritical
    target_matchers:
      - alertname =~ "ServiceCrashed|HighLatency|WriteTimeout"
    equal: ['instance']
# When DiskSpaceCritical fires, derived alerts from the same instance are suppressed
# On-call only sees ONE main alert, not 5

Three main principles from the examples above: (1) thresholds must be impact-based, not round numbers that look nice. (2) every alert has a realistic for: duration — minimum 5 minutes for warnings, 2-3 minutes for critical. (3) inhibit_rules clean up alert bursts when there’s a parent problem.

High Availability Alertmanager #

Alertmanager supports clustering for high availability. Without clustering, if one Alertmanager instance dies, firing alerts won’t be sent until the instance returns — this can be fatal. Cluster mode configuration uses the gossip protocol:

{# Add to alertmanager.yml.j2 for cluster mode #}
{% if alertmanager_cluster_enabled | default(false) %}
cluster:
  listen-address: ""
  # Cluster peers generated from the alertmanager inventory group
  {% for host in groups['alertmanager'] %}
  - {{ hostvars[host]['ansible_default_ipv4']['address'] }}:9094
  {% endfor %}
{% endif %}

To automate Alertmanager cluster setup with Ansible:

# roles/alertmanager/tasks/cluster.yml
---
- name: Open the Alertmanager cluster port
  ufw:
    rule: allow
    port: '9094'
    proto: tcp
  when: ansible_facts['os_family'] == 'Debian'

- name: Deploy the configuration with cluster enabled
  template:
    src: alertmanager.yml.j2
    dest: /etc/alertmanager/alertmanager.yml
    owner: alertmanager
    group: alertmanager
    mode: '0640'
  vars:
    alertmanager_cluster_enabled: true
  notify: Restart Alertmanager

The Alertmanager cluster uses the gossip protocol for state synchronization — when one node receives a silence, all nodes know about it. When a new alert arrives, one of the nodes sends the notification (based on consensus), so there’s no duplication.

Testing Alert Rules with Prometheus #

Before deploying to production, alert rules must be tested to ensure the PromQL expressions are correct and thresholds are reasonable. promtool provides two important commands:

# Validate the syntax and structure of alert rules
promtool check rules /etc/prometheus/rules/*.yml

# Unit test alert rules with PromQL scenarios
promtool test rules /etc/prometheus/tests/alerts.test.yml

The alert rules test file in YAML format defines expected input and output scenarios:

# tests/alerts.test.yml
rule_files:
  - /etc/prometheus/rules/slo-recording.yml

evaluation_interval: 1m

tests:
  # Test 1: Alert does not fire when the error rate is below the threshold
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api",status="200"}'
        values: '1000x10'
      - series: 'http_requests_total{job="api",status="500"}'
        values: '5x10'    # 0.5% error rate
    alert_rule_test:
      - eval_time: 10m
        alertname: HighErrorBudgetBurnRate
        exp_alerts: []   # Must not fire at a 0.5% error rate

  # Test 2: Alert fires when the burn rate is high
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api",status="200"}'
        values: '100x10'
      - series: 'http_requests_total{job="api",status="500"}'
        values: '900x10'  # 90% error rate — disaster
    alert_rule_test:
      - eval_time: 10m
        alertname: HighErrorBudgetBurnRate
        exp_alerts:
          - exp_labels:
              severity: critical
              job: api
            exp_annotations:
              summary: "Error budget burning fast"
              runbook_url: "https://wiki.company.com/runbooks/error-budget-burn"

Run these tests in the CI pipeline every time alert rules change:

# .github/workflows/alert-tests.yml (concept)
- name: Test alert rules
  run: |
    docker run --rm -v $(pwd):/etc/prometheus \
      prom/prometheus:latest \
      promtool test rules /etc/prometheus/tests/alerts.test.yml    

The SLO & SLA article discusses in more detail the error budget-based alerts we saw in the test cases above. For a broader context on why specific alerts are created, see also the Monitoring article discussing alert rules as code, and the Incident Response article discussing the runbooks that must be linked on every alert.

Summary #

• Alertmanager handles routing, grouping, deduplication, inhibition, and silencing — separate this responsibility from Prometheus which is only responsible for detection.
• group_wait, group_interval, and repeat_interval are the three main knobs determining when and how often notifications are sent — set according to severity and urgency.
• inhibit_rules suppresses derived alerts when the parent alert is already firing — for example, suppress warnings when a critical from the same host is already active, or suppress all service alerts when HostDown is active.
• Correct severity is critical for 24/7 paging, warning for working-hour tickets, info for daily digests — mixing without rules only produces alerts ignored on every channel.
• Good notification templates include: host, problem description, start time, runbook link, and dashboard link — everything needed to start an investigation without opening other tools.
• Use for: in alert rules — alerts without a minimum duration fire on every momentary spike and become noise. Realistic minimums: 2-3 minutes for critical, 5-15 minutes for warning.
• Silences via Ansible enable scheduled maintenance without flooding false-alarm notifications — create a silence before starting maintenance, expire it after it completes.
• High availability Alertmanager is mandatory in production — cluster mode with the gossip protocol ensures alerts are sent even if one node dies.

← Previous: Monitoring Next: Dashboard →