Incident Response #

An alert has fired. Someone on on-call duty must act. Without preparation, every incident starts from zero: open the terminal, SSH to the server, try to remember the relevant diagnostic commands, read scattered logs, try a few solutions. Often what happens is missteps, unnecessary escalations, and hours wasted on things that were actually debugged before. With executable runbooks — Ansible playbooks that document the response steps while also executing them — teams can respond faster, more consistently, and with fewer errors. This article discusses how to build an effective runbook infrastructure.

The Complete Incident Response Flow #

Incident response isn’t just about “fix the thing that’s broken”. There’s a structured flow from detection to post-mortem, and each phase has different goals and outputs:

sequenceDiagram
    participant Mon as Monitoring
    participant OnCall as On-Call Engineer
    participant Run as Runbook
    participant Sys as Affected System
    participant EsC as Escalation
    participant PM as Post-Mortem

    Mon->>OnCall: "Alert fired"
    Note over OnCall: "Phase 1: DETECTION"
    OnCall->>Run: "Acknowledge alert, run diagnose.yml"
    Run->>Sys: "Collect data"
    Sys-->>Run: "Logs, metrics, status"
    Run-->>OnCall: "Diagnostic report"
    Note over OnCall: "Phase 2: TRIAGE"
    OnCall->>OnCall: "Determine severity"
    OnCall->>Run: "Choose the runbook matching the symptoms"
    Note over OnCall: "Phase 3: MITIGATION"
    Run->>Sys: "Execute remediation steps"
    alt Successful
        Sys-->>Run: "Service recovered"
        OnCall->>Mon: "Confirm resolved"
    else Failed
        Run-->>OnCall: "Runbook didn't resolve"
        OnCall->>EsC: "Escalate (severity rises)"
    end
    Note over OnCall,PM: "Phase 4: POST-MORTEM"
    OnCall->>PM: "Write a blameless post-mortem"
    PM->>PM: "Identify root cause + action items"

These four phases — Detection, Triage, Mitigation, Post-Mortem — are the foundation of effective incident response. Ansible automates phases 1 and 3 (diagnostics and mitigation), and documents phases 2 and 4 through comments in the runbook playbooks.

Severity Levels: Who Gets Called When #

Not all incidents are the same. Severity determines how fast a response is needed, who must be called, and whether escalation is required. The following table is a common reference:

Having a clear severity matrix prevents two common problems: (1) engineers burning out because all alerts are treated as SEV-1, and (2) incidents being under-escalated because the engineer isn’t sure whether to call the lead.

Severity Escalation State Diagram #

Severity can go up and down depending on the situation. An incident starting as SEV-3 can become SEV-1 if the business impact turns out bigger than initially assumed:

stateDiagram-v2
    [*] --> SEV3: "Alert received"
    SEV3 --> SEV2: "Impact spreading"
    SEV2 --> SEV1: "Total outage"
    SEV2 --> SEV3: "Impact limited"
    SEV1 --> SEV2: "Partial mitigation"
    SEV1 --> Resolved: "Service recovered"
    SEV2 --> Resolved: "Workaround active"
    SEV3 --> Resolved: "Fixed"
    Resolved --> PostMortem: "Any severity"
    PostMortem --> [*]
    SEV1 --> SEV1: "Self-escalate if impact worsens"

Decision Tree: The Right Initial Severity #

flowchart TD
    A["Alert comes in"] --> B{"User-facing impact?"}
    B -- "Yes" --> C{"All users or some?"}
    C -- "All users" --> D["SEV-1: Total outage"]
    C -- "Some" --> E{"Is there a workaround?"}
    E -- "No" --> F["SEV-1: Customer-facing down"]
    E -- "Yes" --> G["SEV-2: Degraded"]
    B -- "No" --> H{"Data loss or security?"}
    H -- "Yes" --> I["SEV-1: Data/Security"]
    H -- "No" --> J{"Can it be postponed this hour?"}
    J -- "No" --> K["SEV-2: Internal blocker"]
    J -- "Yes" --> L["SEV-3: Minor"]

Diagnostic Playbooks: Collect First, Decide Later #

The first step during an incident is not making changes — it’s collecting information to understand the situation. Many incidents actually only need a service restart, but without proper diagnostics, engineers instead change configuration or do unnecessary redeploys that worsen the problem:

# playbooks/diagnose.yml
---
- name: Collect diagnostic information during an incident
  hosts: "{{ target_hosts | default('all') }}"
  gather_facts: true

  tasks:
    - name: Collect the status of all critical services
      systemd:
        name: "{{ item }}"
      register: service_status
      loop: "{{ critical_services }}"
      ignore_errors: true
      changed_when: false

    - name: Collect resource usage
      command: "{{ item.cmd }}"
      register: "{{ item.name }}"
      loop:
        - { name: top_processes, cmd: "ps aux --sort=-%cpu | head -15" }
        - { name: disk_usage,    cmd: "df -h" }
        - { name: memory_usage,  cmd: "free -h" }
        - { name: network_conn,  cmd: "ss -tnp | head -30" }
        - { name: open_files,    cmd: "lsof | wc -l" }
      ignore_errors: true
      changed_when: false

    - name: Collect the latest error logs
      command: "journalctl -u {{ item }} --since '30 minutes ago' --no-pager -p err"
      register: recent_errors
      loop: "{{ critical_services }}"
      ignore_errors: true
      changed_when: false

    - name: Save all diagnostics to a local file
      local_action:
        module: copy
        content: |
          ===== INCIDENT DIAGNOSTICS =====
          Host: {{ inventory_hostname }}
          Time: {{ ansible_date_time.iso8601 }}

          === SERVICE STATUS ===
          {% for result in service_status.results %}
          {{ result.item }}: {{ result.status.ActiveState | default('unknown') }}
          {% endfor %}

          === RESOURCE USAGE ===
          {{ disk_usage.stdout }}

          {{ memory_usage.stdout }}

          === TOP PROCESSES ===
          {{ top_processes.stdout }}

          === LOG ERRORS (LAST 30 MINUTES) ===
          {% for result in recent_errors.results %}
          --- {{ result.item }} ---
          {{ result.stdout | default('(no errors)') }}
          {% endfor %}          
        dest: "/tmp/incident-diag-{{ inventory_hostname }}-{{ ansible_date_time.date }}.txt"
      delegate_to: localhost

- name: Display the diagnostic summary
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Diagnostic file location
      debug:
        msg: "Diagnostic files saved to /tmp/incident-diag-*.txt"

The important point of this playbook: no task changes system state. All tasks only register data and run command with changed_when: false. The purpose is purely information gathering. Engineers running this playbook can rest easy knowing there’s no risk of breaking something that’s actually still working.

When running diagnostic playbooks, always save the output to a file with a timestamp, then commit it to Git or upload it to the internal wiki. This file becomes evidence for the post-mortem and a reference for similar incidents in the future. Diagnostics that aren’t archived = lost learning.

Self-Healing: Automation for Common Conditions #

Some recurring, predictable incident conditions can be handled automatically without human intervention. Self-healing automation reduces recovery time, reduces alert fatigue, and lets on-call engineers focus on incidents that truly need human reasoning:

# playbooks/self-heal.yml
---
- name: Self-healing for common conditions
  hosts: appservers
  become: true

  tasks:
    - name: Collect the current condition information
      gather_facts: true

    # Condition 1: Disk almost full — clean up old logs and Docker images
    - name: Check the disk usage
      command: df / --output=pcent
      register: disk_pcent
      changed_when: false

    - name: Clean up old logs if disk > 85%
      block:
        - name: Find old logs (older than 7 days)
          find:
            paths: /var/log
            age: 7d
            patterns: "*.log.*"
            recurse: true
          register: old_logs

        - name: Remove old log files
          file:
            path: "{{ item.path }}"
            state: absent
          loop: "{{ old_logs.files }}"

        - name: Clean up unused Docker images
          community.docker.docker_prune:
            images: true
            volumes: true
            builder_cache: true
          ignore_errors: true

        - name: Record the self-healing action
          debug:
            msg: "Self-heal: disk cleanup performed (disk usage > 85%)"
      when: disk_pcent.stdout | trim | replace('%', '') | int > 85

    # Condition 2: Service crash — automatic restart
    - name: Check the critical service status
      systemd:
        name: "{{ item }}"
      register: svc_check
      loop: "{{ critical_services }}"
      changed_when: false
      ignore_errors: true

    - name: Restart inactive services
      systemd:
        name: "{{ item.item }}"
        state: restarted
      loop: "{{ svc_check.results }}"
      when:
        - item.status is defined
        - item.status.ActiveState != "active"
      loop_control:
        label: "{{ item.item }}"

    # Condition 3: Memory pressure — restart the service with a memory leak
    - name: Check the memory usage
      command: free -m --output=used
      register: mem_used
      changed_when: false

    - name: Restart the application if memory is very high
      systemd:
        name: myapp
        state: restarted
      when:
        - ansible_memtotal_mb > 0
        - (mem_used.stdout_lines[-1] | trim | int) / ansible_memtotal_mb > 0.90

When Self-Healing Is Appropriate vs Dangerous #

Self-healing is a double-edged sword. On one hand it reduces noise and MTTR; on the other, it can hide systemic problems that should be investigated:

SELF-HEAL IS SUITABLE for:
  ✓ Disk cleanup (log rotation, image prune) — reversible, low risk
  ✓ Restarting crashed services — usually recover on their own
  ✓ Killing zombie processes — clear and reversible
  ✓ Rotating large log files — idempotent

SELF-HEAL IS DANGEROUS for:
  ✗ Restarting a production database without strict thresholds — can trigger failover
  ✗ Auto-scaling resources up without analysis — hides memory leaks
  ✗ Clearing alert state — hides problems
  ✗ Modifying service configuration — could make misconfiguration worse
  ✗ Restarting Kubernetes nodes — can trigger pod eviction cascades

Every self-heal action must send a notification to the on-call channel, even if it succeeds. Engineers need to know self-heal happened — not to act at that moment, but to analyze patterns: if self-heal for service X happens 5 times a day, there’s a fundamental problem in service X (memory leak, config bug) that must be investigated and permanently fixed. Frequent self-heal is a symptom, not a solution.

Runbooks as Documented Playbooks #

The most effective runbooks are the ones that can be executed while also explaining what they do. Documentation separated from execution quickly goes stale — runbooks that live in Ansible playbooks are always up-to-date because they’re tested every time they run:

# playbooks/runbooks/database-connection-exhausted.yml
---
# RUNBOOK: Database Connection Pool Exhausted
#
# Symptoms: Error "remaining connection slots are reserved for non-replication superuser connections"
#           or the application times out when connecting to the database
# Common causes: Connection leak in the application, traffic spike, or pool size too small
# Escalation: If this runbook doesn't resolve the problem within 10 minutes,
#             contact the database team and raise the severity to SEV-2

- name: Runbook — Database Connection Pool Exhausted
  hosts: dbservers
  become: true

  tasks:
    - name: "[Diagnostics] View all active connections to the database"
      command: >
        psql -U postgres -c
        "SELECT client_addr, state, count(*) as count
         FROM pg_stat_activity
         GROUP BY client_addr, state
         ORDER BY count DESC"        
      register: db_connections
      changed_when: false
      become_user: postgres

    - name: "[Info] Display connections per client"
      debug:
        var: db_connections.stdout_lines

    - name: "[Check] How many long-idle connections are there?"
      command: >
        psql -U postgres -c
        "SELECT count(*) FROM pg_stat_activity
         WHERE state = 'idle'
         AND state_change < now() - interval '10 minutes'"        
      register: idle_connections
      changed_when: false
      become_user: postgres

    - name: "[Action] Terminate long-idle connections if > 20"
      command: >
        psql -U postgres -c
        "SELECT pg_terminate_backend(pid)
         FROM pg_stat_activity
         WHERE state = 'idle'
         AND state_change < now() - interval '10 minutes'
         AND pid <> pg_backend_pid()"        
      become_user: postgres
      when: idle_connections.stdout | regex_search('\d+') | int > 20
      register: terminate_result

    - name: "[Result] Display the number of terminated connections"
      debug:
        msg: "Terminated connections: {{ terminate_result.stdout | default('0 (nothing to terminate)') }}"

    - name: "[Verify] Check the connection count now"
      command: >
        psql -U postgres -c
        "SELECT count(*) FROM pg_stat_activity"        
      register: current_connections
      changed_when: false
      become_user: postgres

    - name: "[Status] Current connection condition"
      debug:
        var: current_connections.stdout_lines

Notice the structure in the playbook: every task has a [Diagnostics], [Check], [Action], [Verify], [Status] label at the start of its name. This pattern makes it easy for on-call engineers reading log output to understand which phase is running, and also speeds up post-mortems — the logs directly show the sequence of steps already taken.

Anti-Pattern: Vague Runbook vs Concrete Runbook #

# ANTI-PATTERN: Vague runbook without step details
# playbooks/fix-database.yml
---
- name: Fix database issue
  hosts: dbservers
  tasks:
    - name: Restart postgres if needed
      systemd:
        name: postgresql
        state: restarted
      when: "something is wrong"
# Problems:
# 1. "if needed" and "something is wrong" are undefined.
# 2. An engineer running this at 3 AM has to guess when to restart.
# 3. No verification after the restart.
# 4. No record of what was checked.
# 5. Not reversible if the restart actually worsens the problem.

# CORRECT: A runbook with concrete steps, clear thresholds, and verification
# playbooks/runbooks/database-connection-exhausted.yml
- name: Runbook — Database Connection Pool Exhausted
  hosts: dbservers
  tasks:
    - name: "[Diagnostics] Count idle connections > 10 minutes"
      command: "psql -c \"SELECT count(*) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '10 minutes'\""
      register: idle_count
      changed_when: false

    - name: "[Action] Terminate if > 20 idle connections"
      command: "psql -c \"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '10 minutes' AND pid <> pg_backend_pid()\""
      when: idle_count.stdout | regex_search('\d+') | int > 20

    - name: "[Verify] Ensure the total connection count drops below the threshold"
      command: "psql -c \"SELECT count(*) FROM pg_stat_activity\""
      register: total_count
      failed_when: total_count.stdout | regex_search('\d+') | int > 100
# Advantages:
# - Explicit thresholds (> 20 idle connections, total < 100)
# - Has a verification phase (failed_when for the rollback indicator)
# - No service restart — reversible, low-risk
# - Clear output for the post-mortem

Escalation Workflow #

When a runbook doesn’t resolve the problem, escalation must happen automatically — the on-call engineer doesn’t need to waste time deciding who to call. The system must make escalation decisions based on severity and incident duration:

# playbooks/escalate.yml
---
- name: Send the incident escalation
  hosts: localhost
  vars:
    incident_id: "{{ lookup('pipe', 'date +%Y%m%d%H%M%S') }}"

  tasks:
    - name: Create an incident ticket in the ticketing system
      uri:
        url: "{{ pagerduty_events_url }}"
        method: POST
        body_format: json
        body:
          routing_key: "{{ vault_pagerduty_routing_key }}"
          event_action: trigger
          dedup_key: "ansible-incident-{{ incident_id }}"
          payload:
            summary: "{{ incident_summary }}"
            severity: "{{ incident_severity | default('critical') }}"
            source: "ansible-runbook"
            custom_details:
              environment: "{{ env }}"
              affected_hosts: "{{ ansible_play_hosts | join(', ') }}"
              runbook: "{{ playbook_dir | basename }}"
        status_code: 202
      no_log: true

    - name: Send the emergency Slack notification
      uri:
        url: "{{ vault_slack_webhook_url }}"
        method: POST
        body_format: json
        body:
          text: ":rotating_light: *CRITICAL INCIDENT*"
          attachments:
            - color: danger
              title: "{{ incident_summary }}"
              fields:
                - title: Environment
                  value: "{{ env }}"
                  short: true
                - title: Incident ID
                  value: "{{ incident_id }}"
                  short: true
                - title: Runbook
                  value: "{{ playbook_dir | basename }}"
                  short: true
                - title: Affected Hosts
                  value: "{{ ansible_play_hosts | join(', ') }}"
                  short: false
      no_log: true

Decision Tree: Who Gets Called #

flowchart TD
    A[Incident triggered] --> B{Severity?}
    B -- SEV-1 --> C[On-Call: call<br/>+ lead engineer]
    C --> D{Ack < 5 min?}
    D -- No --> E[Auto-escalate:<br/>engineering manager]
    E --> F{Ack < 10 min?}
    F -- No --> G[VP Engineering +<br/>comms team]
    B -- SEV-2 --> H[On-Call handles<br/>+ notify lead in Slack]
    H --> I{Resolved < 1 hour?}
    I -- No --> J[Escalate to SEV-1]
    I -- Yes --> K[Close + post-mortem]
    B -- SEV-3 --> L[On-Call handles alone]
    L --> M[Resolved? ticket close]
    B -- SEV-4 --> N[Backlog, no escalation]

This workflow can be automated through timers in PagerDuty or rules in alerting (see Alerting). The important thing: escalation is never a manual decision during an incident — that decision was made beforehand and written in the policy.

Post-Mortem: Blameless and Actionable #

A post-mortem isn’t a document to find who was wrong — it’s a document to find the systemic conditions that allowed the incident to happen. A “blameless” culture is crucial: if people fear being blamed during incidents, they’ll hide information, and learning never happens:

Anti-Pattern: Skipping Post-Mortems vs Blameless Post-Mortems #

ANTI-PATTERN: Skip post-mortem for "small" incidents
  After a SEV-3 finishes, the engineer returns to regular work.
  No post-mortem. No root cause analysis.
  3 months later, the same SEV-3 happens again, to a different person.
  Again no post-mortem. The pattern is never seen.

  Risk: The same incident repeats with the same cost every time.
  The team never learns. The "fix and forget" culture wins.

CORRECT: Blameless post-mortem for all SEV-1 and SEV-2 incidents
  Within 48 hours after resolved, schedule a post-mortem meeting.
  Attendees: on-call engineer, lead, SRE, product manager.

  Format:
  1. Timeline (from alert to resolved, with precise timestamps)
  2. Root cause analysis (5-Whys or fishbone diagram)
  3. Contributing factors (what made this incident possible)
  4. What went well (fast response, effective runbook)
  5. What went wrong (slow alert, missing documentation, obsolete runbook)
  6. Action items with owners and deadlines

  Output: Public document in the internal wiki + action items in the backlog.

  Risk: a post-mortem eats 1-2 hours per incident.
  Benefit: the same incident doesn't repeat → saves 5-10 hours of incident response
           in the future → positive ROI after the 2nd incident.

Post-Mortem Template via Ansible #

For consistency, provide a post-mortem template that can be generated from a playbook:

# playbooks/templates/post-mortem-template.md.j2
# Post-Mortem: {{ incident_summary }}

**Incident ID:** {{ incident_id }}
**Date:** {{ incident_date }}
**Severity:** {{ incident_severity }}
**Status:** Resolved
**Duration:** {{ incident_duration }}

## Timeline

| Time (UTC) | Event |
|---|---|
{% for event in timeline %}
| {{ event.time }} | {{ event.description }} |
{% endfor %}

## Root Cause

{{ root_cause }}

## Contributing Factors

{% for factor in contributing_factors %}
- {{ factor }}
{% endfor %}

## What Went Well

{% for item in went_well %}
- {{ item }}
{% endfor %}

## What Went Wrong

{% for item in went_wrong %}
- {{ item }}
{% endfor %}

## Action Items

| Action | Owner | Deadline | Status |
|---|---|---|---|
{% for item in action_items %}
| {{ item.action }} | {{ item.owner }} | {{ item.deadline }} | Open |
{% endfor %}

Anti-Pattern: Post-Mortems That Blame Individuals #

ANTI-PATTERN: A post-mortem focused on "who"
  "Andi accidentally dropped the production table during maintenance."
  "Toni was late acknowledging the alert because he was in a meeting."
  "Siti deployed during peak hours without approval."

  Effects:
  - The team becomes defensive, afraid of open discussion in post-mortems
  - Information is hidden in the next post-mortem
  - Focus is on "person X must be more careful" instead of the system
  - No systemic change → incidents repeat

CORRECT: A post-mortem focused on systemic conditions
  "The deployment process has no pre-check that validates the environment."
  "The alert has no clear escalation policy for this severity."
  "The on-call schedule doesn't overlap with team meetings, so there's a 30-minute gap."
  "The runbook for scenario X hasn't been written, and the engineer who first responded
   has never faced this scenario before."

  Advantages:
  - Nobody feels personally blamed
  - Focus is on systems, processes, and tooling that can be improved
  - Action items are systemic changes (not "Andi must be more careful")
  - Psychological safety culture → incidents are reported earlier

Integration with Alerting and SLO #

Effective incident response starts with good alerts and clearly defined SLOs. Alerts without SLOs = noise. SLOs without runbooks = goals without execution. See Alerting for actionable alert setup, and SLO & SLA for defining measurable service targets:

flowchart LR
    A["Monitoring & SLO"] --> B{"SLO breach?"}
    B -- "Yes" --> C["Alert fired"]
    C --> D["Severity assigned"]
    D --> E{"Escalation policy"}
    E --> SEV1["SEV-1: PagerDuty"]
    E --> SEV2["SEV-2: Slack + on-call"]
    E --> SEV3["SEV-3: Ticket"]
    SEV1 --> F["On-Call responds"]
    SEV2 --> F
    SEV3 --> F
    F --> G{"Runbook resolves?"}
    G -- "Yes" --> H["Service restored"]
    H --> I["Post-Mortem"]
    G -- "No" --> J["Escalation rises"]
    J --> K["Lead/Manager called"]
    K --> F

A good alert is an alert that has a runbook link in its annotation. When clicking the alert in PagerDuty, the engineer is directly taken to the relevant runbook. This saves lookup time and reduces “what should I do” decisions during incidents.

When Incident Response Automation Isn’t Suitable #

Not all incidents are suitable for automation. There’s a class of incidents needing human reasoning and situational decisions:

DON'T automate:
  ✗ Security incidents (data breaches, unauthorized access) — need forensic investigation
  ✗ Data corruption or potential data loss — need manual assessment
  ✗ Deployment rollback decisions — need business judgment
  ✗ Customer communication about outages — need crafted messages
  ✗ Root cause analysis — needs humans with domain context
  ✗ Post-mortem action items — need team discussion, not scripts

AUTOMATION IS SUITABLE for:
  ✓ Collecting diagnostics (logs, metrics, status) during alerts
  ✓ Restarting crashed services with clear thresholds
  ✓ Cleaning up full resources (disk, memory)
  ✓ Terminating idle/overloaded database connections
  ✓ Sending escalation notifications to PagerDuty/Slack
  ✓ Generating post-mortem templates with timelines
  ✓ Verifying conditions after remediation

The clear boundary between automation and judgment is the key: overly aggressive automation can hide problems; overly minimal automation doesn’t provide value.

Summary #

• Collect diagnostics before acting — diagnostic playbooks that gather service status, resource usage, and error logs provide a picture of the situation without making changes.
• A clear severity matrix (SEV-1 to SEV-4) prevents under-escalation and over-escalation — determine response time thresholds and escalation paths before incidents happen.
• Self-healing for recurring, predictable conditions (full disk, crashed services, idle connections) — reduce manual intervention for things that can be automated with low risk.
• Runbooks as playbooks are the best way to document incident procedures — documentation and execution live in the same place and are always in sync.
• Add narrative comments in runbook playbooks (Symptoms, Causes, Escalation) — people on-call at 3 AM need context, not just commands.
• Use task labels like [Diagnostics], [Action], [Verify] to make log output reading and post-mortem analysis easier.
• Escalation workflows must be automatic — if a runbook doesn’t resolve the problem, the system must send an escalation alert without needing a manual decision.
• Blameless post-mortems focus on systemic conditions, not individuals — this creates psychological safety and prevents recurring incidents.
• All SEV-1 and SEV-2 post-mortems must produce action items with owners and deadlines — documentation without action items = learning without implementation.
• Every self-heal action must send a notification to on-call, even if successful — frequent self-heal patterns are a symptom of systemic problems.
• Keep all runbooks in Git and review them periodically — runbooks that aren’t updated become misleading documents when most needed.
• A good alert has a runbook link in its annotation — one click from PagerDuty straight to the relevant playbook.
• See also Alerting for actionable alert setup, and SLO & SLA for defining measurable service targets.

← Previous: Health Check Next: SLO & SLA →