SLO & SLA #

Monitoring shows that a service is slow. Alerts tell that latency is rising. But the most important question for business is: how many incidents can we tolerate before violating our commitment to customers? SLO (Service Level Objective) and SLA (Service Level Agreement) provide a measurable, structured answer. SLO is an internal target maintained by the engineering team; SLA is an external promise to customers with legal/commercial consequences if violated. Ansible can automate the entire measurement infrastructure — from recording rules in Prometheus to SLO dashboards in Grafana — so compliance with targets is always measured in real-time and monthly customer reports can be generated without manual work.

Basic Concepts: SLI, SLO, Error Budget, and SLA #

These four concepts are often mixed up, even though each has a different role in the hierarchy:

flowchart TD
    SLI["SLI<br/>Service Level Indicator<br/>the actual metric measured"] --> SLO["SLO<br/>Service Level Objective<br/>the target to achieve"]
    SLO --> EB["Error Budget<br/>100% - SLO target<br/>the tolerated failure margin"]
    EB --> BR["Burn Rate<br/>how fast the budget burns"]
    EB --> SLA["SLA<br/>Service Level Agreement<br/>a formal contract to customers"]

    SLI -. "example" .-> SLIex["99.5% request success rate"]
    SLO -. "example" .-> SLOex["99.9% target<br/>over 30 days"]
    EB -. "example" .-> EBex["0.1% x 30 days<br/>= ~43 minutes downtime"]
    BR -. "example" .-> BRex["1x = normal<br/>14x = budget exhausted in ~2 days"]
    SLA -. "example" .-> SLAex["99.5% uptime<br/>10% monthly fee refund<br/>if violated"]

Formal definitions we need to hold onto:

• SLI (Service Level Indicator) — the actual measured metric. Example: the percentage of requests with a non-5xx response code in a 5-minute window, or P99 latency in the last hour. SLIs are always measurable and queryable from Prometheus.
• SLO (Service Level Objective) — the target to achieve for that SLI. Example: the success rate SLI must be ≥ 99.9% in a 30-day window. The SLO is an internal contract between the engineering team and product/business.
• Error Budget — the allowed failure margin. If the SLO is 99.9%, the error budget = 0.1%. In 30 days, that means 0.1% × 30 × 24 × 60 = 43.2 minutes of downtime. The team may fail up to that much; after it’s exceeded, action must be taken (postmortem, feature moratorium, refactor).
• SLA (Service Level Agreement) — a formal contract to customers with commercial consequences. Usually the SLO is stricter than the SLA: if the SLA is 99.5% and the SLO is 99.9%, the team has a 0.4% margin for incidents that don’t violate the SLA.

Never set the SLO equal to or stricter than the SLA. The SLA is a promise that if violated has compensation consequences; the SLO is an internal target. If SLO = SLA, we have no buffer during incidents — customers will file complaints every time the internal target isn’t met, which should still be within the tolerated margin.

Choosing the Right SLI #

The chosen SLI must represent the user experience, not internal metrics that are easy to achieve. The three most common SLI categories:

Choosing an SLI is a product decision, not a technical one. Ask the product manager: “what metric best represents users being happy with this service?” The right answer is usually availability or latency — not CPU usage.

Recording Rules for Efficient SLIs #

Complex SLI calculations can be very expensive if recalculated every time they’re queried from dashboards or alerts. Recording rules calculate and store the results periodically — the result is a new time series that can be queried cheaply:

# roles/prometheus/tasks/slo-rules.yml
---
- name: Deploy recording rules for SLO
  template:
    src: rules/slo-recording.yml.j2
    dest: /etc/prometheus/rules/slo-recording.yml
    owner: prometheus
    mode: '0644'
    validate: promtool check rules %s
  notify: Reload Prometheus

{# templates/rules/slo-recording.yml.j2 #}
groups:
  # Group 1: SLI metrics (measurement basis)
  - name: slo_recording_rules
    interval: 30s
    rules:
      # SLI: Availability — percentage of successful requests (non-5xx)
      - record: job:http_requests_total:success_rate5m
        expr: >
          sum(rate(http_requests_total{status!~"5.."}[5m])) by (job, service)
          /
          sum(rate(http_requests_total[5m])) by (job, service)

      # SLI: P99 latency from the histogram
      - record: job:http_request_duration_seconds:p99_5m
        expr: >
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, service, le)
          )

      # SLI: P95 latency
      - record: job:http_request_duration_seconds:p95_5m
        expr: >
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, service, le)
          )

  # Group 2: Error budget metrics (derived from SLIs)
  - name: error_budget_rules
    interval: 30s
    rules:
      # Error budget burn rate (multi-window, multi-burn-rate)
      # Page an alert if the burn rate is 14x faster than normal — will be exhausted in ~2 days
      - record: job:error_budget_burn_rate:1h
        expr: >
          (1 - job:http_requests_total:success_rate5m)
          /
          (1 - {{ slo_availability_target | default(0.999) }})

      # Remaining error budget in the last 30 days
      - record: job:error_budget_remaining:30d
        expr: >
          1 - (
            (1 - avg_over_time(job:http_requests_total:success_rate5m[30d]))
            /
            (1 - {{ slo_availability_target | default(0.999) }})
          )

      # Prediction of when the error budget will be exhausted if the current burn rate persists
      - record: job:error_budget_exhaustion_days:current
        expr: >
          (1 - {{ slo_availability_target | default(0.999) }})
          /
          (1 - job:http_requests_total:success_rate5m) * 30

Three groups we need to understand: (1) SLI metrics are raw measurements — always recalculate from the original data. (2) Error budget metrics are derived — calculated from SLI metrics. (3) Predictions help teams plan — “if the current condition persists, the budget runs out in X days”.

Always keep SLIs at the job/service level, not the instance level. An SLO is about the service as a whole — a single failing instance isn’t an SLO violation if other services are still healthy and the load balancer fails over correctly.

Error Budget-Based Alerts (Multi-Window, Multi-Burn-Rate) #

Traditional alerts (CPU > 80%, latency > 200ms) often produce false alarms. Error budget-based alerts are far more precise — only alert when the error budget is truly burning fast. The multi-window, multi-burn-rate pattern from the Google SRE workbook is the industry standard:

{# templates/rules/slo-alerts.yml.j2 #}
groups:
  - name: slo_alerts
    rules:
      # === PAGE ALERT (critical, needs response within 15 minutes) ===
      # Two short windows — high burn rate in a fast window
      - alert: SLO_HighBurnRate_Fast
        expr: >
          job:error_budget_burn_rate:1h > 14.4
          and
          job:error_budget_burn_rate:5m > 14.4
        for: 2m
        labels:
          severity: critical
          slo: availability
          team: platform
        annotations:
          summary: "Very high SLO burn rate: {{ '{{' }} $labels.service {{ '}}' }}"
          description: >
            Burn rate {{ '{{' }} $value | humanize {{ '}}' }}x of normal.
            The error budget will be exhausted in ~2 days if this burn rate persists.
            RESPOND WITHIN 15 MINUTES.
          runbook_url: "https://wiki.company.com/runbooks/slo-burn-fast"
          dashboard_url: "https://grafana.company.com/d/slo-burn?var-service={{ '{{' }} $labels.service {{ '}}' }}"

      # === TICKET ALERT (warning, response within working hours) ===
      # Medium window — high burn rate but not urgent
      - alert: SLO_MediumBurnRate
        expr: >
          job:error_budget_burn_rate:6h > 6
          and
          job:error_budget_burn_rate:30m > 6
        for: 15m
        labels:
          severity: warning
          slo: availability
        annotations:
          summary: "High SLO burn rate: {{ '{{' }} $labels.service {{ '}}' }}"
          description: >
            Burn rate {{ '{{' }} $value | humanize {{ '}}' }}x of normal in the last 6 hours.
            The error budget will be exhausted in ~5 days. MUST BE INVESTIGATED TODAY.
          runbook_url: "https://wiki.company.com/runbooks/slo-burn-medium"

      # === SLOW BURN (warning, response within weeks) ===
      # Long window — low but sustained burn rate
      - alert: SLO_SlowBurnRate
        expr: >
          job:error_budget_burn_rate:3d > 1
          and
          job:error_budget_burn_rate:6h > 1
        for: 1h
        labels:
          severity: warning
          slo: availability
        annotations:
          summary: "SLO slow burn detected: {{ '{{' }} $labels.service {{ '}}' }}"
          description: >
            Burn rate {{ '{{' }} $value | humanize {{ '}}' }}x of normal over 3 days.
            The error budget will be exhausted in ~30 days. PLAN A FIX.
          runbook_url: "https://wiki.company.com/runbooks/slo-burn-slow"

The logic behind multi-window: if the burn rate is consistently high in both a short window AND a long window, the alarm is valid. This avoids false positives from a single short window (like a 5-minute traffic spike) or a single long window (like a momentary metric scrape error).

The table below explains the commonly used burn rate thresholds:

Decision Tree: Choose a Realistic SLO Target #

Choosing an SLO target isn’t an arbitrary number. The main trade-off: a higher target = more reliable for users, but more expensive (needs redundancy, faster rollback, more SRE). The following decision tree helps determine the right target:

flowchart TD
    A["Just starting to<br/>define SLO"] --> B{"Is there historical<br/>reliability<br/>data?"}
    B -- "Yes" --> C{"Look at the worst<br/>percentile of the<br/>last 30 days"}
    B -- "No" --> D["Start measuring the SLI<br/>for 30 days<br/>before setting the SLO"]

    C --> E{"Where is the current<br/>P99 latency?"}
    E -- "50ms" --> F["SLO target:<br/>99.9%<br/>latency 100ms"]
    E -- "200ms" --> G["SLO target:<br/>99.5%<br/>latency 500ms"]
    E -- "2 seconds" --> H["SLO target:<br/>99%<br/>latency 5 seconds"]

    D --> I["After 30 days<br/>of collected data"]
    I --> C

    F --> J{"Is there an SLA<br/>to customers?"}
    G --> J
    H --> J

    J -- "Yes" --> K["SLA = SLO - 0.3%<br/>there's an incident buffer"]
    J -- "No" --> L["SLO is an internal<br/>target only"]

Four questions that must be answered before setting an SLO:

1. What is the current reliability? — If the service has already been running at 99.5% over the last 30 days, set the SLO at 99.5% (realistic), not 99.99% (impossible without improvement).
2. What is the impact of failure? — A payment service going down = direct revenue loss. A blog going down = inconvenience. The SLO target must be proportional.
3. How much does raising 0.1% cost? — From 99.9% to 99.95% might require multi-region active-active, database redundancy, and a 24/7 on-call team. Is the business willing to pay?
4. What SLA is already promised to customers? — The internal SLO must be stricter than the external SLA to have a buffer.

SLO Dashboards in Grafana #

A good SLO dashboard displays everything needed for trust that the service meets its targets. Recommended panel structure:

To deploy the SLO dashboard via Ansible:

- name: Deploy the SLO dashboard to Grafana
  copy:
    src: files/dashboards/slo-overview.json
    dest: /var/lib/grafana/dashboards/slo-overview.json
    owner: grafana
    mode: '0640'
  notify: Reload Grafana dashboards

Important pattern: error budget remaining must be the most prominent panel — not success rate, not latency. The error budget is the number that directly answers “are we safe this month?”.

Sequence Diagram: SLO Flow from Measurement to Decision #

To understand how SLO data flows from raw metrics to business decisions, look at the sequence below:

sequenceDiagram
    participant App as Application
    participant Prom as Prometheus
    participant RR as Recording Rules
    participant AR as Alert Rules
    participant AM as Alertmanager
    participant Dash as Grafana SLO Dashboard
    participant PM as Product Manager

    App->>Prom: "Expose /metrics (request count, duration, status)"
    Prom->>Prom: "Scrape every 15s"
    Prom->>RR: "Evaluate recording rules (every 30s)"
    RR->>Prom: "Save SLI & error budget series"
    Prom->>AR: "Evaluate alert rules (every 15s)"
    AR->>AR: "Calculate the multi-window burn rate"
    AR->>AM: "Send an alert if the burn rate > threshold"
    AM->>AM: "Route the alert by severity"
    AM-->>PM: "PagerDuty / Slack notification"

    PM->>Dash: "Open the SLO dashboard (weekly review)"
    Dash->>Prom: "Query the SLI series"
    Prom-->>Dash: "Last 30 days of data"
    Dash-->>PM: "Visualize error budget, burn rate, trend"

    PM->>PM: "Decision:"
    Note over PM: "Budget > 50%? Safe, continue the roadmap\nBudget 20-50%? Be cautious, defer non-critical features\nBudget < 20%? Stop features, focus on reliability"
    PM->>AR: "Update the SLO threshold if needed"
    PM->>App: "Trigger a refactor if slow burn"

The important pattern from this sequence: a well-measured SLO allows product managers to make decisions based on data, not intuition. Without error budget numbers, weekly meetings usually turn into endless “is the service healthy or not” debates.

Automatic SLA Reports #

For SLAs promised to customers, monthly reports must be sent on time. Ansible + the Prometheus API can automate data collection and PDF/HTML report generation:

# playbooks/generate-sla-report.yml
---
- name: Generate the monthly SLA report
  hosts: localhost
  vars:
    report_month: "{{ lookup('pipe', 'date +%Y-%m') }}"
    prometheus_url: "https://prometheus.company.com"
    slo_availability_target: 0.999
    slo_latency_target_ms: 200

  tasks:
    - name: Fetch the availability data for this month
      uri:
        url: "{{ prometheus_url }}/api/v1/query_range"
        method: GET
        body_format: form-urlencoded
        body:
          query: 'avg_over_time(job:http_requests_total:success_rate5m{service="myapp"}[30d])'
          start: "{{ lookup('pipe', 'date -d\"first day of this month\" +%s') }}"
          end: "{{ lookup('pipe', 'date +%s') }}"
          step: "3600"
        headers:
          Authorization: "Bearer {{ vault_prometheus_token }}"
        return_content: true
      register: availability_data
      no_log: true

    - name: Fetch the P99 latency data for this month
      uri:
        url: "{{ prometheus_url }}/api/v1/query_range"
        method: GET
        body_format: form-urlencoded
        body:
          query: 'avg_over_time(job:http_request_duration_seconds:p99_5m{service="myapp"}[30d])'
          start: "{{ lookup('pipe', 'date -d\"first day of this month\" +%s') }}"
          end: "{{ lookup('pipe', 'date +%s') }}"
          step: "3600"
        headers:
          Authorization: "Bearer {{ vault_prometheus_token }}"
        return_content: true
      register: latency_data
      no_log: true

    - name: Calculate the average availability rate
      set_fact:
        avg_availability: >-
          {{ (availability_data.json.data.result[0].values
              | map(attribute=1) | map('float') | sum
              / availability_data.json.data.result[0].values | length * 100) | round(4) }}          

    - name: Calculate the average P99 latency
      set_fact:
        avg_latency_ms: >-
          {{ (latency_data.json.data.result[0].values
              | map(attribute=1) | map('float') | sum
              / latency_data.json.data.result[0].values | length * 1000) | round(2) }}          

    - name: Determine the SLA status
      set_fact:
        sla_status: >-
          {{ 'COMPLIANT' if (avg_availability | float >= 99.9 and avg_latency_ms | float < slo_latency_target_ms | float)
             else 'BREACH' }}          

    - name: Generate the Markdown report file
      template:
        src: sla-report.md.j2
        dest: "/var/reports/sla-{{ report_month }}-{{ inventory_hostname }}.md"

    - name: Generate the PDF from Markdown
      command: >
        pandoc /var/reports/sla-{{ report_month }}-{{ inventory_hostname }}.md
        -o /var/reports/sla-{{ report_month }}-{{ inventory_hostname }}.pdf
        --pdf-engine=xelatex
        -V geometry:margin=1in
        --toc        
      when: sla_install_pandoc | default(false)

    - name: Send the report to the team email
      community.general.mail:
        host: "{{ smtp_host }}"
        port: "{{ smtp_port }}"
        to: "{{ sla_report_recipients }}"
        subject: "SLA Report {{ report_month }} — {{ sla_status }}"
        body: "{{ lookup('file', '/var/reports/sla-' + report_month + '-' + inventory_hostname + '.md') }}"
        attach:
          - "/var/reports/sla-{{ report_month }}-{{ inventory_hostname }}.md"
      when: sla_send_email | default(false)

An informative SLA report template:

{# sla-report.md.j2 #}
# SLA Report — {{ report_month }}

**Period:** 1 {{ report_month }} — {{ lookup('pipe', 'date +%d %B %Y') }}
**Status:** `{{ sla_status }}`

## Summary

| Metric | SLA Target | Actual | Status |
|---|---|---|---|
| Availability | ≥ 99.9% | {{ avg_availability }}% | {{ '✓' if avg_availability | float >= 99.9 else '✗' }} |
| P99 Latency | ≤ {{ slo_latency_target_ms }}ms | {{ avg_latency_ms }}ms | {{ '✓' if avg_latency_ms | float < slo_latency_target_ms else '✗' }} |

## Significant Incidents

{% for incident in monthly_incidents | default([]) %}
- **{{ incident.date }}** — {{ incident.summary }} (duration: {{ incident.duration }})
{% endfor %}
_(No incidents exceeding 5 minutes occurred during this period)_

## Trends

Attach the 30-day availability graph and the 30-day P99 latency graph
from the SLO dashboard for visual context.

## Commitment

We {{ 'MEET' if sla_status == 'COMPLIANT' else 'DO NOT MEET' }}
the SLA commitment for the {{ report_month }} period.
{{ 'No compensation is scheduled.' if sla_status == 'COMPLIANT' else 'Our team will send the applicable compensation details per the contract.' }}

Generate SLA reports via a cron job at the end of the month, not manually. Add 0 9 1 * * to the crontab to run generate-sla-report.yml on the 1st of every month at 9 AM. The report is available before the monthly meeting, and nobody forgets to generate it.

ANTI-PATTERN vs CORRECT in SLO Implementation #

Several common traps when adopting SLOs. Understanding them early will save weeks of time:

# ANTI-PATTERN: SLO without an error budget calculation
# "Our SLO is 99.9%" - ok, but 99.9% of what? Per hour? Per day? Per month?
# Without an explicit error budget, an SLO becomes jargon without consequences

# CORRECT: define SLO + window + error budget explicitly
# Format: SLI + target + window + error budget
slo_definitions:
  - name: api-availability
    sli: "percentage of HTTP requests with status < 500"
    target: 99.9
    window: 30d
    error_budget: "43.2 minutes per 30 days"   # Explicit, not just a number
    owner: platform-team

# ANTI-PATTERN: an alert that pages immediately when the SLO is slightly below target
- alert: SLOBreach
  expr: job:http_requests_total:success_rate5m < 0.999
  for: 1m
  labels:
    severity: critical
  # Problem: a 99.8% success rate in 5 minutes does NOT mean the 99.9%/30d SLO was violated
  # This is a false positive that will bombard on-call

# CORRECT: alert based on burn rate, not absolute value
- alert: SLO_HighBurnRate_Fast
  expr: >
    job:error_budget_burn_rate:1h > 14.4
    and
    job:error_budget_burn_rate:5m > 14.4    
  for: 2m
  labels:
    severity: critical
  # Alert only if the budget burns FAST (will be exhausted in 2 days)
  # A brief 99.8% SLO isn't a problem; a 99.5% SLO for 6 hours = a problem

# ANTI-PATTERN: unrealistic SLO targets
- alert: CriticalServiceDown
  expr: up{job="critical"} == 0
  # Implication: even 1 minute of downtime pages on-call
  # A 100% availability internal target = no error budget = every incident is an "SLO breach"
  # On-call burnout within 3 months

# CORRECT: set an SLO that leaves margin
# Reality: 99.99% availability requires multi-region active-active, 3x redundancy, 24/7 on-call
# For most services, 99.9% is the sweet spot between reliability and cost
# The "down" definition must also be clear: 5xx > 1% of traffic? Or a single endpoint failing?
slo_target: 99.9           # Realistic
sla_target: 99.5           # Looser than the SLO
error_budget_30d: 43m      # The team has margin

# ANTI-PATTERN: SLOs that only exist in a spreadsheet
# "Our target is 99.9%" - written in Google Sheets, not visible on dashboards
# The team doesn't know whether they're on track until the end of the month

# CORRECT: SLO visible on real-time dashboards
# - A prominent "Error Budget Remaining" panel on the service dashboard
# - Alerts page on-call if the burn rate is high
# - SLO status reviewed weekly in meetings
# - SLA reports to customers automatically generated from real-time data

Three patterns to take away: (1) An SLO without an error budget is just a number — an explicit error budget gives the SLO consequences. (2) Alerts based on absolute values = false positives — burn rate alerts reflect the impact on the SLO. (3) An SLO that isn’t visible is an SLO that’s forgotten — dashboards and alerts make SLOs part of daily operations.

Integrating SLO with Alerts and Incident Response #

A well-defined SLO has direct implications for alert policy and incident handling. Three key connections:

1. Burn rate alert → runbook linkage — every burn rate alert has a runbook_url explaining the diagnosis and mitigation steps. See Alerting for details on alert routing.

2. SLO violation → incident severity classification — when an SLO is violated, determine the incident severity based on how much error budget was burned:

   • Page alert (14x burn rate) = severity 1 incident, needs immediate on-call
   • Ticket alert (6x burn rate) = severity 2, fix within working hours
   • Slow burn (1x for 3 days) = severity 3, backlog item

3. Error budget policy → feature freeze — when the remaining error budget is < 20%, teams usually agree to a “feature freeze” — all engineering effort is focused on reliability, not new features. This is a product decision, not a technical one — the playbook can be automated for notifications.

More details on the incident response workflow are in Incident Response. For the Prometheus setup that’s the basis of SLI measurement, see Monitoring.

Testing SLO Alert Rules #

Just like regular alert rules, SLO alert rules must be tested to ensure thresholds and windows produce the right alerts. Use promtool test rules:

# tests/slo-alerts.test.yml
rule_files:
  - /etc/prometheus/rules/slo-recording.yml
  - /etc/prometheus/rules/slo-alerts.yml

evaluation_interval: 1m

tests:
  # Test 1: SLO alert does NOT fire at a low error rate
  - interval: 1m
    name: "low error rate stays calm"
    input_series:
      - series: 'http_requests_total{job="api",status="200"}'
        values: '9990x60'
      - series: 'http_requests_total{job="api",status="500"}'
        values: '10x60'    # 0.1% error rate — below the 99.9% SLO
    alert_rule_test:
      - eval_time: 1h
        alertname: SLO_HighBurnRate_Fast
        exp_alerts: []     # Should NOT fire

  # Test 2: SLO alert fires at a high error rate
  - interval: 1m
    name: "high error rate triggers page"
    input_series:
      - series: 'http_requests_total{job="api",status="200"}'
        values: '500x60'
      - series: 'http_requests_total{job="api",status="500"}'
        values: '500x60'    # 50% error rate — disaster
    alert_rule_test:
      - eval_time: 10m
        alertname: SLO_HighBurnRate_Fast
        exp_alerts:
          - exp_labels:
              severity: critical
              slo: availability
              job: api
            exp_annotations:
              summary: "Very high SLO burn rate"
              runbook_url: "https://wiki.company.com/runbooks/slo-burn-fast"

  # Test 3: Medium burn rate fires the ticket alert, not the page
  - interval: 1m
    name: "medium burn rate fires ticket"
    input_series:
      - series: 'http_requests_total{job="api",status="200"}'
        values: '900x360'
      - series: 'http_requests_total{job="api",status="500"}'
        values: '100x360'    # 10% error rate over 6 hours — 6x burn rate
    alert_rule_test:
      - eval_time: 6h
        alertname: SLO_MediumBurnRate
        exp_alerts:
          - exp_labels:
              severity: warning
              slo: availability

Run these tests in the CI pipeline every time a threshold or window is changed:

promtool test rules /etc/prometheus/tests/slo-alerts.test.yml

Summary #

• SLI → SLO → Error Budget → Burn Rate is a logical chain: measure the SLI, set the SLO target, calculate the remaining error budget, monitor the burn rate to predict budget exhaustion.
• SLIs must represent the user experience — availability and latency are the most common. CPU usage and memory usage aren’t good SLIs because they don’t correlate directly with user satisfaction.
• Recording rules for complex SLI calculations — far more efficient than real-time queries on every alert evaluation or dashboard render.
• Multi-window burn rates (1h+5m for page, 6h+30m for ticket, 3d+6h for slow burn) catch both fast and slow problems without too many false alarms — single window = noisy, multi-window = robust.
• Alerts based on burn rate, not absolute values — a 99.8% success rate in 5 minutes does NOT mean the 30-day SLO was violated. A 14x burn rate means the budget is exhausted in 2 days, that’s what should page.
• SLO dashboards must show error budget remaining as the most prominent panel — this is the most meaningful number for engineering prioritization decisions.
• SLO stricter than SLA — SLA 99.5% = SLO 99.9%. The 0.4% buffer gives room for incidents that don’t violate the contract but still need investigation.
• Automatic SLA reports via Ansible + the Prometheus API eliminate the manual work of compiling monthly reports to customers or management — generate via a cron job at the start of the month.
• Test SLO alert rules with promtool test rules in CI — ensure thresholds and windows produce the right alerts for various error rate scenarios.
• An SLO without an error budget is just a number — an explicit error budget with a time unit (43 minutes/month) gives the SLO consequences and becomes discussion material in planning meetings.

← Previous: Incident Response Next: Best Practice →