Rollback Strategy #

A deployment that can’t be rolled back is a deployment that isn’t ready for production. No matter how much testing is done, there’s always the possibility of problems that only appear when code runs in production with real traffic — database query plans that differ on real data, network latency between regions, or race conditions that only appear under load. The ability to return to a working version quickly and reliably is a system property that must not be an afterthought — it must be designed from the start. This article discusses effective rollback strategies using Ansible, from automatic rollback when health checks fail, database rollback with expand-contract, to blue-green and canary for instant rollback.

Deployment Lifecycle States #

Before discussing rollback strategies, it’s important to understand the states a deployment goes through during its lifetime. A state diagram visualization helps us see at which point rollback enters as a transition:

stateDiagram-v2
    [*] --> Idle
    Idle --> Deploying : "deploy start"
    Deploying --> Healthy : "health check pass"
    Deploying --> Degraded : "health check fail"
    Healthy --> Degraded : "alert triggered"
    Degraded --> RollingBack : "rollback trigger"
    RollingBack --> Healthy : "previous version verified"
    Healthy --> [*]
    RollingBack --> Failed : "rollback error"
    Failed --> Manual : "operator takeover"
    Manual --> [*]
    Idle --> Idle : "rollback within window"

This diagram shows four critical states: Deploying (transition), Healthy (successful steady state), Degraded (problem detection), and RollingBack (recovery transition). Rollback is a controlled transition from Degraded back to Healthy by reverting to the previous version. Without a clear state machine, we don’t know when to roll back and when to forward-fix.

The Degraded state is usually triggered by one of: a health check failing within the observation window (usually 5–15 minutes after deploy), the error rate rising above a threshold (e.g. >1% for 5 consecutive minutes), or p99 latency rising significantly. These thresholds must be defined explicitly, not from subjective feelings.

Storing State for Rollback #

The first step of effective rollback is ensuring the required information is available. Before a new version deployment starts, we must know with certainty: what version is currently running, when that version was deployed, and how to revert.

# playbooks/deploy.yml
---
- name: Deploy with rollback capability
  hosts: appservers
  vars:
    deploy_version: "{{ version | mandatory }}"

  pre_tasks:
    - name: Record the currently running version
      command: cat /opt/app/VERSION
      register: current_version_file
      changed_when: false
      ignore_errors: true

    - name: Save the current version to a rollback variable
      set_fact:
        rollback_version: "{{ current_version_file.stdout | default('unknown') | trim }}"

    - name: Save the rollback info to a file
      copy:
        content: |
          version={{ rollback_version }}
          deployed_at={{ ansible_date_time.iso8601 }}
          deployed_by={{ lookup('env', 'USER') | default('ci-pipeline') }}          
        dest: /opt/app/PREVIOUS_VERSION
        mode: '0644'

    - name: Log the deployment start
      lineinfile:
        path: /var/log/deployments.log
        line: "{{ ansible_date_time.iso8601 }} START v{{ deploy_version }} (rollback: v{{ rollback_version }}) @ {{ inventory_hostname }}"
        create: true
      delegate_to: localhost

The PREVIOUS_VERSION file is a simple but critical record. When a deployment fails and the team panics trying to figure out “where do we need to revert to?”, the answer available on the server itself is much faster than digging through commit history or asking on Slack. More importantly, this file is written at the start of the deployment — when the server is still healthy, not when things are already chaotic.

Logging deployments to a centralized file (via delegate_to: localhost) provides a global audit trail. During incidents, we can quickly see: what deployments happened in the last 10 minutes, by whom, on which servers. This cuts investigation time from hours to minutes.

Automatic Rollback When Health Checks Fail #

A health check failure isn’t a situation needing manual intervention. If we can detect it, we should also be able to respond automatically. Ansible’s block/rescue enables this elegantly: if the block part fails, the rescue part executes for recovery, and always executes for logging — whatever happens.

  tasks:
    - block:
        - name: Deploy the new version
          git:
            repo: https://github.com/company/app.git
            dest: /opt/app
            version: "v{{ deploy_version }}"

        - name: Install dependencies
          pip:
            requirements: /opt/app/requirements.txt
            virtualenv: /opt/app/venv

        - name: Restart the application
          systemd:
            name: myapp
            state: restarted

        - name: Wait for the application to be ready (health check)
          uri:
            url: "http://localhost:{{ app_port }}/health"
            status_code: 200
          register: health_check
          until: health_check.status == 200
          retries: 12
          delay: 10

        - name: Verify the running version
          uri:
            url: "http://localhost:{{ app_port }}/api/version"
            return_content: true
          register: version_check
          failed_when: deploy_version not in version_check.content

        - name: Write the new VERSION file
          copy:
            content: "{{ deploy_version }}\n"
            dest: /opt/app/VERSION

      rescue:
        - name: Health check failed — starting automatic rollback
          debug:
            msg: >
              Deployment v{{ deploy_version }} failed on {{ inventory_hostname }}.
              Rolling back to v{{ rollback_version }}.              

        - name: Roll back to the previous version
          git:
            repo: https://github.com/company/app.git
            dest: /opt/app
            version: "v{{ rollback_version }}"
          when: rollback_version != 'unknown'

        - name: Restart with the old version
          systemd:
            name: myapp
            state: restarted
          when: rollback_version != 'unknown'

        - name: Verify the rollback succeeded
          uri:
            url: "http://localhost:{{ app_port }}/health"
            status_code: 200
          retries: 6
          delay: 10

        - name: Fail the play to notify the pipeline
          fail:
            msg: >
              Deployment v{{ deploy_version }} failed and rollback to
              v{{ rollback_version }} has been performed. Check the logs for details.              

      always:
        - name: Log the deployment result
          lineinfile:
            path: /var/log/deployments.log
            line: >
              {{ ansible_date_time.iso8601 }}
              {{ 'ROLLBACK' if ansible_failed_result is defined else 'SUCCESS' }}
              v{{ deploy_version }} @ {{ inventory_hostname }}              
            create: true
          delegate_to: localhost

A sequence diagram illustrates this flow clearly:

sequenceDiagram
    participant Pipe as "CI Pipeline"
    participant Ans as "Ansible"
    participant Srv as "Server"
    participant Reg as "Registry"
    Pipe->>Ans: "deploy v2.1.0"
    Ans->>Srv: "pull source v2.1.0"
    Ans->>Srv: "restart myapp"
    Ans->>Srv: "GET /health"
    Srv-->>Ans: "503 Service Unavailable"
    Ans->>Srv: "GET /health (retry 1)"
    Srv-->>Ans: "503"
    Ans->>Srv: "GET /health (retry 2)"
    Srv-->>Ans: "503"
    Note over Ans: "retries exhausted"
    Ans->>Srv: "checkout v2.0.5 (rollback)"
    Ans->>Srv: "restart myapp"
    Ans->>Srv: "GET /health"
    Srv-->>Ans: "200 OK"
    Ans-->>Pipe: "fail (rollback successful)"
    Pipe->>Pipe: "notify team, page oncall"

failed_when: deploy_version not in version_check.content is an additional sanity check. Sometimes the health check can return 200 (e.g. the old server is still running or the application didn’t restart properly). By verifying the /api/version endpoint returns the newly deployed version, we ensure the deployment actually happened, not a silent failure.

A too-permissive health check (returning 200 as long as the server responds) can fool the system. Health checks must test critical functionality: database connections, downstream API calls, or main business logic. A health check that only returns {"status": "ok"} without any tests provides no information about whether the application actually works.

ANTI-PATTERN: No Rollback Plan vs Automated Rollback #

Imagine this scenario: a production deploy happens on Friday afternoon, the health check fails 10 minutes later, and there’s no rollback playbook. The on-call engineer must dig through Git history to find the last stable commit, then debug how to revert because of merge conflicts. This process takes 45 minutes. During those 45 minutes, production is down and revenue is lost. This situation is completely avoidable.

# ANTI-PATTERN: deploy without a rollback plan
# playbooks/deploy-naive.yml
---
- name: Deploy without rollback capability
  hosts: appservers
  tasks:
    - name: Pull the latest code
      git:
        repo: https://github.com/company/app.git
        dest: /opt/app
        version: "{{ branch }}"  # Could be anything, even a half-finished commit

    - name: Restart
      systemd:
        name: myapp
        state: restarted

    # Done. No health check, no rollback plan,
    # no audit trail. Good luck.

The problem: the deployment above “succeeded” from Ansible’s perspective (tasks finished), but there’s no verification that the application actually runs. If it fails, the engineer must manually revert from memory or Git log. No automation, no fast way back to the previous state.

# CORRECT: deploy with full rollback capability
# playbooks/deploy-safe.yml (like above, summarized)
---
- name: Deploy with full rollback capability
  hosts: appservers
  vars:
    deploy_version: "{{ version | mandatory }}"

  pre_tasks:
    - name: Record the previous version
      command: cat /opt/app/VERSION
      register: current_version
      changed_when: false
      ignore_errors: true
    - set_fact:
        rollback_version: "{{ current_version.stdout | default('unknown') | trim }}"

  tasks:
    - block:
        - name: Pull and restart
          # ... deployment tasks ...

        - name: Health check
          uri:
            url: "http://localhost:{{ app_port }}/health"
            status_code: 200
          retries: 12
          delay: 5

        - name: Update VERSION
          copy:
            content: "{{ deploy_version }}\n"
            dest: /opt/app/VERSION

      rescue:
        - name: Automatic rollback to v{{ rollback_version }}
          git:
            repo: https://github.com/company/app.git
            dest: /opt/app
            version: "v{{ rollback_version }}"
        - name: Restart
          systemd:
            name: myapp
            state: restarted

      always:
        - name: Log to the centralized log
          lineinfile:
            path: /var/log/deployments.log
            line: "{{ ansible_date_time.iso8601 }} {{ 'ROLLBACK' if ansible_failed_result is defined else 'SUCCESS' }} v{{ deploy_version }}"
            create: true

The crucial difference: the previous version is always recorded before a new deployment, the health check is mandatory, and rollback happens automatically in the rescue block if the health check fails. When a deploy fails, the playbook knows exactly where to revert, without asking or guessing.

Manual Rollback via Pipeline #

Automation doesn’t eliminate the need for manual rollback. There are situations where automatic rollback is too risky (e.g. automatic rollback could lose data that just came in, or deep investigation is needed before reverting). For that, we must provide a separate rollback playbook triggerable from the pipeline.

# playbooks/rollback.yml
---
- name: Manual deployment rollback
  hosts: appservers
  vars:
    # target_version must be passed: -e target_version=2.0.5
    target_version: "{{ target_version | mandatory }}"

  pre_tasks:
    - name: Confirm the version to roll back to exists in the registry
      command: "docker manifest inspect registry.company.com/myapp:{{ target_version }}"
      delegate_to: localhost
      changed_when: false

    - name: Record the currently running version (before rollback)
      command: cat /opt/app/VERSION
      register: pre_rollback_version
      changed_when: false
      ignore_errors: true

  tasks:
    - name: Log the rollback start
      debug:
        msg: >
          Rolling back from v{{ pre_rollback_version.stdout | default('unknown') }}
          to v{{ target_version }}          

    - name: Pull the rollback target image
      community.docker.docker_image:
        name: "registry.company.com/myapp:{{ target_version }}"
        source: pull

    - name: Run the rollback version container
      community.docker.docker_container:
        name: myapp
        image: "registry.company.com/myapp:{{ target_version }}"
        state: started
        restart_policy: unless-stopped
        recreate: true

    - name: Wait for the application to be ready
      uri:
        url: "http://localhost:{{ app_port }}/health"
        status_code: 200
      retries: 12
      delay: 5

    - name: Update the VERSION file
      copy:
        content: "{{ target_version }}\n"
        dest: /opt/app/VERSION

    - name: Log the rollback completion
      debug:
        msg: "Rollback successful to v{{ target_version }}"

A separate rollback playbook lets on-call engineers revert even in the middle of the night without remembering deployment details. Just run the command ansible-playbook -i inventory/production/ rollback.yml -e target_version=2.0.5 and go back to sleep. The important thing: the rollback playbook must be independent — it must not depend on state or variables from the deploy playbook.

Database Rollback: A Different Problem #

Code rollback tends to be easy: swap the image or binary, restart the service, done in minutes. Database schema rollback is far more complex because data is persistent — dropped columns delete data, changed types can permanently break constraints.

The main principle: code can be rolled back easily, databases can’t. Design migrations that don’t require database rollback.

# Principles for rollback-able databases:
#
# 1. Backward-compatible migrations — old code must be able to run
#    with the new schema BEFORE a rollback happens
#
# 2. Separate destructive migrations — drop columns/tables in a separate
#    migration, not together with adding new columns
#
# 3. Expand-Contract pattern:
#    Phase 1 (Expand): Add the new column, deploy new code writing to BOTH columns
#    Phase 2 (Contract): Drop the old column after verifying nobody needs it
#
# If down migrations exist (Django, Alembic, etc.):
- name: Roll back the database migration
  command: "python manage.py migrate {{ app_name }} {{ target_migration }}"
  args:
    chdir: /opt/app
  when:
    - rollback_db | default(false) | bool
    - target_migration is defined

The Expand-Contract pattern is the only safe way for significant database schema changes. Its visualization:

flowchart TD
    A["Initial State: old_column"] --> B["Add new_column"]
    B --> C["Deploy code: write to BOTH columns"]
    C --> D["Backfill data from old_column to new_column"]
    D --> E{"Is the data consistent?"}
    E -- "Yes" --> F["Deploy code: read from new_column"]
    F --> G["Deploy code: stop writing to old_column"]
    G --> H["Drop old_column in a separate migration"]
    E -- "No" --> I["Investigate, fix, redo the backfill"]

In the Expand phase, new code is written to write to both columns (old and new) so the old application version can still run. After all code is updated and data is backfilled, only then in the Contract phase is the old column dropped. If the deployment is rolled back midway, old code can still read from the old column, and the new column still exists (even if not fully populated) — no data loss.

Never combine “add column” and “drop column” in one deployment. If the deployed code fails midway, we could lose data because the column was already dropped while the rolled-back old code doesn’t know where to store data. Always separate into at least two deployments: one expand, one contract.

Blue-Green Deployment for Instant Rollback #

Blue-green is a deployment strategy enabling rollback by switching a pointer, not redoing the deployment. Two parallel environments (blue and green), only one receives traffic. On deploy, switch traffic to the new slot. On rollback, switch back to the old slot — done in seconds.

# playbooks/blue-green-deploy.yml
---
- name: Blue-Green Deployment
  hosts: loadbalancer
  vars:
    current_slot: "{{ lookup('file', '/etc/app/active-slot') | default('blue') }}"
    new_slot: "{{ 'green' if current_slot == 'blue' else 'blue' }}"

  tasks:
    - name: Deploy to the inactive slot ({{ new_slot }})
      include_tasks: deploy-to-slot.yml
      vars:
        slot: "{{ new_slot }}"

    - name: Verify the new slot is healthy
      uri:
        url: "http://{{ new_slot }}.internal:{{ app_port }}/health"
        status_code: 200
      retries: 12
      delay: 5

    - name: Switch traffic to the new slot
      template:
        src: nginx-upstream.conf.j2
        dest: /etc/nginx/conf.d/upstream.conf
      vars:
        active_slot: "{{ new_slot }}"
      notify: Reload nginx

    - name: Wait for the nginx reload
      meta: flush_handlers

    - name: Save the active slot
      copy:
        content: "{{ new_slot }}\n"
        dest: /etc/app/active-slot

    - name: Instant rollback instructions if needed
      debug:
        msg: >
          Deployment successful to slot {{ new_slot }}.
          For instant rollback: ansible-playbook rollback-blue-green.yml
          (returns traffic to slot {{ current_slot }} without re-deploy)          

The blue-green state diagram looks like this:

stateDiagram-v2
    [*] --> BlueActive
    BlueActive --> BlueDeployingGreen : "deploy"
    BlueDeployingGreen --> GreenVerifying : "deploy done"
    GreenVerifying --> GreenActive : "health check pass"
    GreenVerifying --> BlueActive : "health check fail"
    GreenActive --> BlueActive : "rollback (instant)"
    GreenActive --> GreenActive : "monitor"

The biggest advantage of blue-green: rollback is a pointer switch, not a re-deploy. In seconds, traffic moves from slot A to slot B. On rollback, traffic moves back from B to A — done in the time it takes the load balancer to reload its configuration (usually < 5 seconds with nginx). No image downloads, no migrations, no additional risk.

The trade-off: requires 2x resources because of the two parallel environments. For small systems or non-mission-critical staging, this is wasteful. For production with strict SLAs, blue-green is the industry standard.

ANTI-PATTERN: Only Forward-Fix vs Bidirectional Recovery #

There are two extreme camps in rollback: the “always forward-fix” camp that refuses reverts believing bugs can be patched in minutes, and the “excessive rollback” camp that reverts the smallest problems without investigation. Both are suboptimal.

# ANTI-PATTERN: forward-fix only, no rollback
# playbooks/deploy-no-rollback.yml
---
- name: Deploy without rollback
  hosts: appservers
  tasks:
    - name: Deploy the new version
      git:
        repo: https://github.com/company/app.git
        dest: /opt/app
        version: "{{ version }}"

    - name: Restart
      systemd:
        name: myapp
        state: restarted

    # Done. No rollback plan.
    # If it fails, the engineer must hotfix to main, push, and
    # hope the hotfix is correct. No shortcuts.

This approach is problematic because: bugs in production take time to investigate, fix, and deploy. Meanwhile, the service stays down. Forward-fix is a good strategy for minor bugs (e.g. a UI typo), but bad for major regressions (e.g. a query locking a table for 30 seconds).

# CORRECT: bidirectional recovery — rollback first, fix later
# playbooks/deploy-with-bidirectional-recovery.yml
---
- name: Deploy with rollback option
  hosts: appservers
  vars:
    deploy_version: "{{ version | mandatory }}"

  pre_tasks:
    - name: Record the rollback version
      command: cat /opt/app/VERSION
      register: current
      changed_when: false
      ignore_errors: true
    - set_fact:
        rollback_version: "{{ current.stdout | default('unknown') | trim }}"

  tasks:
    - block:
        - name: Deploy
          git:
            repo: https://github.com/company/app.git
            dest: /opt/app
            version: "v{{ deploy_version }}"

        - name: Restart and health check
          systemd:
            name: myapp
            state: restarted
          notify: wait for health

        - name: Wait for the health check
          uri:
            url: "http://localhost:{{ app_port }}/health"
            status_code: 200
          retries: 12
          delay: 5

      rescue:
        - name: Automatic rollback
          git:
            repo: https://github.com/company/app.git
            dest: /opt/app
            version: "v{{ rollback_version }}"
        - name: Restart with the old version
          systemd:
            name: myapp
            state: restarted

      always:
        - name: Send the deployment status alert
          uri:
            url: "{{ vault_slack_webhook }}"
            method: POST
            body_format: json
            body:
              text: "Deploy v{{ deploy_version }}: {{ 'ROLLBACK' if ansible_failed_result is defined else 'SUCCESS' }}"
            status_code: 200

The correct strategy: rollback first, fix later. Service availability matters more than fix speed. When a deployment fails and we have automatic rollback, the service returns to normal in minutes. Root cause investigation and forward-fix can happen without the pressure of production being down. Once the fix is ready, deploy the forward-fix as a new deployment. This is bidirectional: can rollback, can forward-fix, decision based on context.

Comparing Rollback Strategies #

There’s no one universal rollback strategy. Choose based on downtime tolerance, deployment complexity, and release frequency.

The best rollback strategy for most teams is a combination: blue-green for application code (rollback in seconds) + expand-contract for databases (rollback without data loss) + feature flags for quick kill switches (rollback without redeploy). Start with the simplest (image redeploy with a rollback playbook) and add complexity only when there’s an explicit need.

Canary Rollback for Risk-Sensitive Deployments #

For deployments to large user bases, we might want to limit the exposure of a new version to a small portion of traffic first. Canary deployment enables this: deploy the new version to a small subset of servers, monitor, and if safe, promote to all servers. If unsafe, roll back only the canary portion.

# playbooks/canary-deploy.yml
---
- name: Canary deployment
  hosts: localhost
  vars:
    canary_percentage: 10
    canary_hosts_group: canary_appservers
    production_hosts_group: production_appservers

  tasks:
    - name: Deploy to the canary subset
      include_tasks: deploy-app.yml
      vars:
        target_hosts: "{{ canary_hosts_group }}"
        version: "{{ deploy_version }}"

    - name: Wait for the canary to stabilize
      pause:
        minutes: 5
        prompt: "Wait 5 minutes, then check the error rate"

    - name: Check the canary error rate
      uri:
        url: "https://monitoring.internal/api/canary-error-rate?app=myapp&version={{ deploy_version }}"
        return_content: true
      register: canary_metrics

    - name: Decision based on the error rate
      block:
        - name: Promote the canary to production
          include_tasks: deploy-app.yml
          vars:
            target_hosts: "{{ production_hosts_group }}"
            version: "{{ deploy_version }}"

        - name: Success notification
          debug:
            msg: "Canary succeeded, promoted to {{ canary_percentage }}% traffic"

      rescue:
        - name: Roll back the canary
          include_tasks: rollback-app.yml
          vars:
            target_hosts: "{{ canary_hosts_group }}"

        - name: Stop the promotion
          fail:
            msg: "Canary failed, rolled back, promotion to production cancelled"

Canary rollback is the most conservative strategy: only a small portion of users is exposed to the bug, and rollback happens on a small subset without re-deploying to all servers. This strategy suits businesses whose reputation is sensitive to downtime (payment, healthcare, etc.) where new releases need extra validation.

Summary #

• Always record the running version before the deployment starts — the VERSION and PREVIOUS_VERSION files are prerequisites for reliable rollback.
• block/rescue for automatic rollback when health checks fail — don’t let a failed deployment leave the system in a half-way state.
• Create a separate rollback playbook triggerable from the pipeline or manually — rollback must not require special technical skills during an incident.
• Database rollback is a separate problem from code rollback — use the expand-contract pattern for migrations that are safe to roll back.
• Blue-green deployment enables instant rollback via pointer switching, not re-deploy — ideal for systems that can’t tolerate rollback downtime.
• Canary deployment for risk-sensitive releases — limit exposure to a small traffic portion, roll back the subset without disturbing the majority.
• Log every deployment and rollback with timestamps and versions — a complete audit trail is invaluable during incident investigation.
• Bidirectional recovery: roll back first to restore the service, fix later in a separate deployment. Forward-fix isn’t the only option.
• Health checks must test critical functionality (DB connections, downstream APIs), not just return 200.
• Don’t combine “add column” and “drop column” in one deployment — always separate them so rollback doesn’t lose data.

← Previous: Environment Management Next: Artifact Management →