Error Handling #

Playbooks that only run in perfect conditions aren’t resilient enough to manage production infrastructure. In the real world, we face unstable networks, package repositories suddenly going offline, full disks, or unresponsive third-party services. If we don’t design playbooks with mature error handling strategies, a small failure in one task can stop the entire deployment process, leave target servers in an inconsistent state (half-configured), and make recovery difficult. Ansible provides various built-in features to detect, ignore, modify, and automatically recover from failures to maintain our system integrity.

Why Error Handling Strategies Matter #

When Ansible executes a playbook, its default behavior is very strict: if a task fails (returns a non-zero exit code or experiences a connection error), Ansible immediately stops all subsequent tasks for that failed host, but continues execution for other successful hosts. This behavior is designed for safety so we don’t pile new configuration on top of an already broken system. However, in complex automation scenarios like zero-downtime application deployments or database migrations, this abrupt stop can be a disaster if not anticipated with rollback or cleanup.

Here’s a flow diagram visualizing how Ansible handles execution flow when facing a task failure by default:

flowchart TD
    Start["Start Playbook"] --> T1["Task 1: Successful Execution"]
    T1 --> T2{"Task 2: Was It Successful?"}
    T2 -- "Yes" --> T3["Task 3: Continue Execution"]
    T2 -- "No" --> StopHost["Stop Execution on the Failed Host"]
    T3 --> End["Done (All Hosts Up-to-date)"]
    StopHost --> FailState["Host Enters Failed State (Next Tasks Ignored)"]

By understanding this basic flow, we can start applying more flexible mechanisms to control when a task is considered failed and how the system should respond to that failure.

The block, rescue, and always Constructs #

Ansible provides a task grouping structure called block. This structure is very similar to the try-catch-finally exception handling blocks we commonly find in modern programming languages like Java, Python, or C#. Using block, we can group several logically related tasks, define recovery steps (rescue) if one of the tasks inside the block fails, and specify cleanup tasks (always) that must execute regardless of whether the main block succeeds or fails.

Using block to Group Tasks #

Blocks allow us to apply task-level directives (like become, when, or vars) to several tasks at once efficiently. However, its main use is limiting the error scope. If a task inside the block returns an error, Ansible stops the remaining tasks inside that block and immediately jumps to the rescue section.

The Role of rescue in Automatic Rollback #

The rescue section only executes if one of the tasks inside the block fails. Inside rescue, we can place tasks responsible for rollback, like restoring backup configuration files, restarting services that went down, or sending emergency notifications to the operations team. After the tasks inside rescue finish executing successfully, Ansible considers the play back on track (not considered failed) and continues execution to the task after that block.

The Role of always for Cleanup #

The always section executes in any condition: both when all tasks inside the block succeed, and after the rescue section finishes handling the error. This is a very ideal place for cleanup tasks like removing temporary files, closing dedicated database connections, or recording deployment audit logs.

Here’s an example implementation of block, rescue, and always for safe application deployment with automatic rollback:

# ANTI-PATTERN: Running tasks linearly without error handling.
# If the deployment fails midway, the system is left in a broken/stuck state.
- name: Deploy the application naively
  hosts: app_servers
  tasks:
    - name: Stop the application service
      ansible.builtin.systemd:
        name: webapp
        state: stopped

    - name: Deploy new code from Git
      ansible.builtin.git:
        repo: "[email protected]:us/webapp.git"
        dest: /var/www/webapp
        version: release-1.2.0

    - name: Run the database migration
      ansible.builtin.command: /var/www/webapp/bin/migrate
      # DON'T: If the migration fails here, the webapp service stays dead forever
      # and the half-deployed new code is left as is.

    - name: Restart the application service
      ansible.builtin.systemd:
        name: webapp
        state: started

Now let’s compare it with the correct implementation using structured error handling:

# CORRECT: Wrapping the critical deployment flow with block-rescue-always.
- name: Deploy the application with rollback protection
  hosts: app_servers
  vars:
    backup_dir: /var/backups/webapp
    app_dir: /var/www/webapp
  tasks:
    - name: Main Deployment Execution Block
      block:
        - name: Create the backup directory
          ansible.builtin.file:
            path: "{{ backup_dir }}"
            state: directory
            mode: '0755'

        - name: Back up the old application code before the update
          ansible.builtin.archive:
            path: "{{ app_dir }}"
            dest: "{{ backup_dir }}/app_prev.tar.gz"
            format: gz

        - name: Stop the application service
          ansible.builtin.systemd:
            name: webapp
            state: stopped

        - name: Deploy new code from Git
          ansible.builtin.git:
            repo: "[email protected]:us/webapp.git"
            dest: "{{ app_dir }}"
            version: release-1.2.0

        - name: Run the database migration
          ansible.builtin.command: "{{ app_dir }}/bin/migrate"
          register: migration_result

        - name: Restart the application service
          ansible.builtin.systemd:
            name: webapp
            state: started

      rescue:
        - name: Failure warning detected
          ansible.builtin.debug:
            msg: "An error occurred in the main block! Starting the automatic rollback process..."

        - name: Restore the old application code from the backup
          ansible.builtin.unarchive:
            src: "{{ backup_dir }}/app_prev.tar.gz"
            dest: "{{ app_dir }}"
            remote_src: true

        - name: Restart the service with the old code (rollback state)
          ansible.builtin.systemd:
            name: webapp
            state: started

        - name: Report the deployment failure status
          ansible.builtin.fail:
            msg: "The deployment failed and the system was successfully rolled back to the previous version."

      always:
        - name: Clean up the temporary backup file if present
          ansible.builtin.file:
            path: "{{ backup_dir }}/app_prev.tar.gz"
            state: absent

        - name: Log the final deployment status to the control node
          delegate_to: localhost
          ansible.builtin.lineinfile:
            path: /var/log/ansible-deploy.log
            line: "{{ ansible_date_time.iso8601 }} - Host {{ inventory_hostname }} finished processing."
            create: true
            mode: '0644'

The execution flow of the code block above is illustrated through the following flow diagram:

flowchart TD
    subgraph "Main Block (&quot;block&quot;)"
        A1["Backup Old Code"] --> A2["Stop Service"]
        A2 --> A3["Deploy New Code"]
        A3 --> A4["DB Migration"]
        A4 --> A5["Start Service"]
    end
    subgraph "Recovery Block (&quot;rescue&quot;)"
        R1["Restore Code from Backup"] --> R2["Start Old Service"]
        R2 --> R3["Trigger Playbook Failure"]
    end
    subgraph "Cleanup Block (&quot;always&quot;)"
        L1["Remove Temp Backup File"] --> L2["Write Deployment Log"]
    end

    Start["Start Task"] --> A1
    A4 -- "Error Occurs" --> R1
    A5 -- "Success" --> L1
    R3 --> L1
    L2 --> End["Done"]

failed_when: Customizing Failure Criteria #

By default, Ansible considers a task failed if the program or module run returns a non-zero exit code. However, this criterion doesn’t always match our real operational needs. Sometimes a command runs successfully with exit code 0 but outputs error text in stdout indicating the operation failed. Conversely, sometimes a program produces a non-zero exit code (for example grep returns exit code 1 if it doesn’t find a text match), even though logically in our workflow that’s a normal condition we want to accept.

This is where we use the failed_when directive. With failed_when, we can evaluate a task’s return status (registered variable) using Jinja2 logic expressions to precisely determine when that task should be considered failed.

Identifying Errors Through Text Output #

For example, when we run a cli command to check an external API’s health, the cli might always return exit code 0 as long as the network is connected, even though the server responds with "Status: 500 Internal Server Error". We can monitor that task’s stdout variable:

# CORRECT: Determining failure based on specific strings in the stdout
- name: Check the external API health
  ansible.builtin.command: curl -s http://api.external.com/health
  register: api_check
  failed_when: 
    - "'ERROR' in api_check.stdout"
    - "'500' in api_check.stdout"
  changed_when: false

Handling Specific Exit Code Tolerance #

If we use utilities like pg_isready to check Postgres status, we know this tool can return exit code 1 if the server is still processing initial startup. If we want to consider exit code 1 as safe and only consider it failed if the exit code is 2 (fatal error), we can use failed_when flexibly:

# CORRECT: Modifying the exit code tolerance
- name: Check the PostgreSQL database readiness
  ansible.builtin.command: pg_isready -h localhost -p 5432
  register: pg_status
  failed_when: "pg_status.rc == 2"
  changed_when: false

Let’s compare the anti-pattern often done by developers with a more elegant solution using failed_when.

# ANTI-PATTERN: Using an additional task with the 'fail' module to check status.
# This makes the code very verbose and produces extra tasks in log outputs.
- name: Run the database query naively
  ansible.builtin.command: mysql -e "SELECT * FROM users;"
  register: db_query
  ignore_errors: true

- name: Validate the query results manually
  ansible.builtin.fail:
    msg: "The database query failed to execute!"
  when: db_query.rc != 0 or 'Connection refused' in db_query.stderr


# CORRECT: Combining the failure validation directly inside the task using failed_when.
- name: Run the database query efficiently
  ansible.builtin.command: mysql -e "SELECT * FROM users;"
  register: db_query_optimal
  failed_when:
    - db_query_optimal.rc != 0 or 'Connection refused' in db_query_optimal.stderr
  changed_when: false

ignore_errors: Continuing Despite Failure #

The ignore_errors: true directive tells Ansible that if this task fails, ignore it and continue to the next task as if nothing happened. This is useful for optional non-blocking tasks, like cleaning an optional cache directory that may have already been removed, or trying to send optional statistics to an internal log server.

Misusing ignore_errors: true is a big danger! Don’t use it as a shortcut to hide misconfigurations or system errors you don’t understand. If we ignore critical errors (like disk mount failures or security library installation failures), the playbook keeps running and eventually causes the system to collapse at the next task without a clear error indication at the start.

Better Alternatives to ignore_errors #

Most ignore_errors uses can actually be replaced with state checks using the stat module or checking conditions with when.

For example, if we want to remove an old configuration file but don’t want the task to fail if that file truly doesn’t exist, we can use a modular approach instead of blindly ignoring errors:

# ANTI-PATTERN: Using ignore_errors for uncertain file operations.
- name: Remove the old configuration file if present
  ansible.builtin.file:
    path: /etc/nginx/conf.d/old_site.conf
    state: absent
  ignore_errors: true  # DON'T: If this errors because of a disk permission issue, we won't know.

# CORRECT: Checking the file existence first before taking action.
- name: Check whether the old configuration file exists
  ansible.builtin.stat:
    path: /etc/nginx/conf.d/old_site.conf
  register: old_site_file

- name: Remove the configuration file if proven to exist
  ansible.builtin.file:
    path: /etc/nginx/conf.d/old_site.conf
    state: absent
  when: old_site_file.stat.exists

This way, if a real file system failure occurs (like permission denied on the /etc/nginx/ directory), Ansible raises the error precisely so we can detect access right misconfigurations on the server.

force_handlers: Guaranteeing State Maintenance #

When a task changes system state (like updating a configuration template), that task usually triggers a notify to a handler (for example "Restart Nginx"). However, per Ansible’s built-in rules, all triggered handlers are only executed at the end of the play section. The problem arises if a system failure occurs mid-execution of tasks after the template trigger.

By default, if a play fails midway, Ansible ignores all handlers scheduled to run at the end of the play. As a result, our server can be in an inconsistent state: the new configuration file is already written to disk, but the service hasn’t been restarted to load that new configuration. This is very dangerous because if the server is manually restarted later, the service might fail to start due to the unvalidated new configuration.

Using force_handlers #

To prevent this scenario, we can set the force_handlers: true directive at the play level. This directive forces Ansible to still execute all already-notified handlers, even if playbook execution stops due to a failure in a subsequent task.

We can configure this inside our playbook:

# CORRECT: Setting force_handlers at the play level to guarantee service consistency
- name: Manage the webapp web server configuration
  hosts: web_servers
  force_handlers: true  # ✓ Guarantees all notified handlers still run even if a later task fails
  tasks:
    - name: Update the Nginx configuration file
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        mode: '0644'
      notify: Restart Nginx Service

    - name: Run the external asset verification task
      ansible.builtin.command: /usr/local/bin/verify-assets.sh
      # If this verification task fails, the 'Restart Nginx Service' handler is still executed by Ansible
      # so the Nginx configuration change above isn't left hanging without a restart.

  handlers:
    - name: Restart Nginx Service
      ansible.builtin.systemd:
        name: nginx
        state: restarted

Besides defining it at the playbook level, we can also force this setting globally inside our ansible.cfg configuration file:

# ansible.cfg
[defaults]
# Enabling force_handlers by default for all playbooks
force_handlers = True

any_errors_fatal and max_fail_percentage for Large Scale #

When managing dozens or hundreds of servers at once, allowing playbook execution to continue on healthy hosts while some other hosts fail can be a poor decision. Especially in interdependent microservices architecture deployments, or when running distributed database schema updates. A failure on one or a small subset of nodes can break the entire cluster’s consistency.

Ansible provides two advanced ways to control error tolerance limits in large-scale infrastructure: any_errors_fatal and max_fail_percentage.

any_errors_fatal: Atomic Death #

If we set any_errors_fatal: true on a play, Ansible instantly stops the entire playbook execution on all active hosts as soon as just one host experiences a task failure. This is crucial in early preparation stages (pre-tasks) or when running main database migrations where one node’s failure must cancel the entire deployment process to maintain global data integrity.

max_fail_percentage: Threshold Tolerance #

If we’re doing gradual rolling updates on a large web server cluster (for example using the serial directive), stopping the entire deployment just because one server experiences a small failure might be too sensitive. We can determine the maximum failure percentage limit using max_fail_percentage. As long as the number of failed servers is still below that limit percentage, Ansible continues execution to the next server group.

Here’s a comparative illustration of their use in one playbook:

# CORRECT: Using any_errors_fatal on database setup and max_fail_percentage on webapp rolling updates.
- name: Critical Database Initialization
  hosts: database_cluster
  any_errors_fatal: true  # ✓ Stop all database nodes if just one node fails during bootstrap
  tasks:
    - name: Verify the active database replication
      ansible.builtin.command: check_replication_status.sh
      changed_when: false

- name: Web Server Cluster Rolling Update
  hosts: web_servers
  serial: 10%                  # Process 10% of hosts at a time gradually
  max_fail_percentage: 20      # ✓ Allow continuing as long as failed servers don't exceed 20% of total hosts
  tasks:
    - name: Install the webapp security package updates
      ansible.builtin.apt:
        name: webapp-package
        state: latest

The following flow diagram describes how execution decisions are made based on the max_fail_percentage criteria:

flowchart TD
    Start["Start New Batch (Serial 10%)"] --> Exec["Execute Tasks in the Current Batch"]
    Exec --> Eval{"Are There Failed Nodes?"}
    Eval -- "No" --> BatchSuccess["Batch Successful"]
    Eval -- "Yes" --> Calc{"Is the Failure Percentage > max_fail_percentage (20%)?"}
    Calc -- "Yes" --> Terminate["Stop the Playbook Globally (Fatal Error)"]
    Calc -- "No" --> BatchSuccess
    BatchSuccess --> CheckNext{"Are There Remaining Batches?"}
    CheckNext -- "Yes" --> Start
    CheckNext -- "No" --> End["All Batches Finished Processing"]

Automatic Rollback Design Patterns in Production #

To build self-healing enterprise-level infrastructure automation, we must not only rely on basic error detection. We must structure our playbooks with a complete failure handling architecture, covering status metric collection before execution (pre-flight checks), dynamic storage of previous version states, safe deployment using transaction blocks, post-deployment health verification (health checks), and automatic recovery (rollback) if that verification fails.

Below is an example production playbook design applying all those concepts comprehensively:

# CORRECT: Web application deployment design pattern integrated with automatic rollback and health status verification
---
- name: Self-Healing Deployment with an Automatic Recovery System
  hosts: web_servers
  become: true
  force_handlers: true  # Ensuring handlers still run to tidy up services
  vars:
    app_root: /opt/production_app
    backup_root: /opt/backups_app
    app_version_target: "v2.1.0"
    health_check_url: "http://127.0.0.1:8080/health"

  pre_tasks:
    - name: Collect the current application version information
      ansible.builtin.slurp:
        src: "{{ app_root }}/version.txt"
      register: current_version_raw
      ignore_errors: true

    - name: Define the backup version for rollback
      ansible.builtin.set_fact:
        app_version_previous: "{{ (current_version_raw.content | b64decode | trim) if current_version_raw.content is defined else 'none' }}"

    - name: Display the detected version information
      ansible.builtin.debug:
        msg: "Detected old application version: {{ app_version_previous }}. Update target: {{ app_version_target }}"

  tasks:
    - name: Application Deployment Transaction Block
      block:
        - name: Create the backup directory if it doesn't exist
          ansible.builtin.file:
            path: "{{ backup_root }}"
            state: directory
            mode: '0700'

        - name: Back up the current application directory if the previous version exists
          ansible.builtin.archive:
            path: "{{ app_root }}"
            dest: "{{ backup_root }}/app_backup_{{ app_version_previous }}.tar.gz"
            format: gz
          when: app_version_previous != 'none'

        - name: Stop the old application safely
          ansible.builtin.systemd:
            name: production_app
            state: stopped

        - name: Clean the old application directory to prepare for new code
          ansible.builtin.file:
            path: "{{ app_root }}"
            state: absent

        - name: Recreate the empty application directory
          ansible.builtin.file:
            path: "{{ app_root }}"
            state: directory
            mode: '0755'

        - name: Download and extract the new version application package
          ansible.builtin.unarchive:
            src: "https://artifactory.our.internal/apps/release-{{ app_version_target }}.tar.gz"
            dest: "{{ app_root }}"
            remote_src: true

        - name: Write the new version to the application identity file
          ansible.builtin.copy:
            content: "{{ app_version_target }}"
            dest: "{{ app_root }}/version.txt"
            mode: '0644'

        - name: Run the application module updates
          ansible.builtin.command: npm install --production
          args:
            chdir: "{{ app_root }}"

        - name: Restart the application service with the new code
          ansible.builtin.systemd:
            name: production_app
            state: started
            enabled: true

        - name: Wait for the application to do its initial booting
          ansible.builtin.pause:
            seconds: 5

        - name: Test the application health (Post-deployment Health Check)
          ansible.builtin.uri:
            url: "{{ health_check_url }}"
            status_code: 200
          register: health_status
          until: health_status.status == 200
          retries: 5
          delay: 3
          # If this health check API request fails within 5 attempts,
          # this task is considered failed and automatically triggers the rescue section execution.

      rescue:
        - name: Critical warning - Health check failed! Starting system recovery...
          ansible.builtin.debug:
            msg: "Deployment failure on host {{ inventory_hostname }}. Starting automatic recovery to version {{ app_version_previous }}."

        - name: Run the Rollback if the old version is available
          block:
            - name: Stop the service that failed to run
              ansible.builtin.systemd:
                name: production_app
                state: stopped

            - name: Remove the broken application code directory
              ansible.builtin.file:
                path: "{{ app_root }}"
                state: absent

            - name: Recreate the application directory
              ansible.builtin.file:
                path: "{{ app_root }}"
                state: directory
                mode: '0755'

            - name: Re-extract the old version backup file
              ansible.builtin.unarchive:
                src: "{{ backup_root }}/app_backup_{{ app_version_previous }}.tar.gz"
                dest: "{{ app_root }}"
                remote_src: true

            - name: Restart the application service with the stable old version
              ansible.builtin.systemd:
                name: production_app
                state: started
              # We only run this rescue block if there's indeed a successfully saved old version backup
          when: app_version_previous != 'none'

        - name: Report the final deployment error to the monitoring dashboard
          ansible.builtin.fail:
            msg: "Deployment to version {{ app_version_target }} failed on host {{ inventory_hostname }}. The system has been restored to version {{ app_version_previous }}."

      always:
        - name: Clean up the old backup archive to save server storage capacity
          ansible.builtin.file:
            path: "{{ backup_root }}/app_backup_{{ app_version_previous }}.tar.gz"
            state: absent
          when: app_version_previous != 'none'

Through the framework above, our infrastructure is never left dead or non-functional. The block-rescue existence ensures that dependency module installation failures (npm install) or application port booting failures (health checks) are immediately responded to by restoring the old code folder from the tarball archive and restarting the stable application daemon.

Summary #

block / rescue / always is the try-catch-finally foundation structure in Ansible for grouping critical tasks and executing automatic recovery (rollback) actions when errors occur.
failed_when allows us to define our own precise task failure rules using Jinja2 expressions, instead of only relying on non-zero exit codes.
ignore_errors must be used very carefully and selectively; avoid using it blindly to cover up real configuration errors.
force_handlers: true guarantees all notified handlers (like service restarts) are still executed by Ansible even if a later task in the play errors out.
any_errors_fatal: true is useful for atomic deployment scenarios where a failure on one node must cancel playbook execution globally for data safety.
max_fail_percentage sets a failure tolerance threshold on large-scale infrastructure, allowing deployments to continue if the number of failed servers is still below the tolerance limit.
A good production deployment architecture always includes pre-flight check steps to store the old state before modifying files, so the rollback process can run accurately.

← Previous: Scheduled Task Next: Delegation & Local Action →