Rolling Update #

Applying application updates in modern production environments demands zero downtime. In the past, releasing a new version often required scheduled maintenance in the middle of the night that forced us to temporarily shut down applications. Kubernetes offers an elegant solution through its built-in rolling update mechanism. However, a reliable update isn’t just about replacing the container image — it involves strict health checks, network traffic management, and the ability to detect failures in real-time. Through integration with Ansible, we can automate this entire deployment workflow, test application health post-update, and perform instant recovery if anomalies occur.

Zero-Downtime Rolling Update Strategy #

The rolling update mechanism works by gradually replacing old Pod instances with new Pod instances. Kubernetes ensures that during this transition process, the application always maintains enough capacity to serve incoming user traffic.

Two main parameters control the rolling update behavior on Deployment objects: maxSurge and maxUnavailable.

maxSurge determines the maximum number of new Pods that can be created above the desired replica count during the update process. This value can be a percentage (for example 25%) or a whole number (for example 1).
maxUnavailable determines the maximum number of Pods allowed to be unavailable or dead during the update process.

Here’s the rolling update process flow tightly coordinated between health checks, smoke testing, and automatic rollback:

flowchart TD
    A["Start New Deployment"] --> B["Record Old Image Version (Rollback Fact)"]
    B --> C["Apply New Deployment Manifest (Update Image)"]
    C --> D["Monitor Rollout (k8s_rollout_status)"]
    D --> E{"Did the Rollout Succeed?"}
    E -- "No (Timeout)" --> H["Run Rollback (rollout undo)"]
    E -- "Yes" --> F["Run Smoke Test (Access /health Endpoint)"]
    F --> G{"Is the HTTP Status 200?"}
    G -- "Yes" --> I["Deployment Successful & Done"]
    G -- "No" --> H
    H --> J["Return to Old Version & Report Error"]

Setting Rollout Tolerance Limits (maxSurge & maxUnavailable) #

The configuration choice for these two parameters depends heavily on the cluster’s resource availability and our application’s characteristics.

If we set maxUnavailable: 0 and maxSurge: 25%, we guarantee the cluster capacity never drops below 100% of the desired replica count. Kubernetes creates new Pods first before shutting down old Pods. This is the ideal configuration for public web applications sensitive to performance degradation from reduced active instances. Conversely, if we have limited computing capacity (CPU/RAM) on worker nodes, we can set maxUnavailable: 25% and maxSurge: 0 so Kubernetes shuts down some old instances first before allocating new Pods.

Here’s an example of a safe Kubernetes deployment manifest configuration for rolling updates:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1       # Create a maximum of 1 extra Pod above normal capacity
      maxUnavailable: 0 # Don't let old Pods die before new Pods are Ready
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-server
          image: myregistry.local/web-app:v2.1.0

With the maxUnavailable: 0 configuration, we ensure service availability stays fully maintained throughout the deployment transition.

Applying Liveness and Readiness Probes #

The most common mistake in applying rolling updates is ignoring container health probe configuration (health probes). Without probes, Kubernetes only relies on the main process status inside the container. If that process runs, Kubernetes considers the container healthy and immediately directs user data traffic to it, even though the application inside is still initializing database connections or reading large configuration files. As a result, users receive 502 (Bad Gateway) or 503 (Service Unavailable) error messages for several seconds.

Types of Probes in Kubernetes #

To prevent the scenario above, we must configure three probe types diligently:

Startup Probe: Used for applications requiring a long initial startup time. Other checks (liveness & readiness) are suspended until the startup probe successfully passes the threshold.
Readiness Probe: Used to determine whether a container is ready to accept network traffic. If the readiness probe fails, the Pod’s IP address is removed from the Kubernetes Service endpoint list so no traffic is directed there.
Liveness Probe: Used to monitor the application’s internal health during its running period. If the liveness probe experiences consecutive failures, Kubernetes kills the container and restarts it automatically.

Anti-Pattern vs Probe Configuration Solution #

Let’s compare a container configuration without probe protection with a configuration applying best practices.

# ANTI-PATTERN: Container defined without any health check mechanism
spec:
  containers:
    - name: backend-api
      image: myregistry.local/api:v1.0.0
      # DON'T do this! Kubernetes immediately considers the container
      # ready once the container engine detects the entrypoint is running.

# CORRECT: Applying startup, readiness, and liveness probes completely
spec:
  containers:
    - name: backend-api
      image: myregistry.local/api:v1.0.0
      ports:
        - containerPort: 8080
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        failureThreshold: 30 # Give initialization time up to 5 minutes (30 * 10s)
        periodSeconds: 10
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 15
        periodSeconds: 10
        failureThreshold: 3

Through the configuration above, the rolling update process becomes very safe. Kubernetes never shuts down old Pods before the new Pods successfully pass the readinessProbe check.

Rolling Update Orchestration and Rollout Monitoring with Ansible #

Now we’ll design a complete Ansible playbook tasked with updating the application image, monitoring the rollout process to completion in real-time, running post-deployment tests (smoke tests), and performing automatic rollback if a failure is detected in the new application.

# playbooks/deploy-app.yml
---
- name: Zero-Downtime Deployment Orchestration
  hosts: localhost
  connection: local
  vars:
    app_name: "web-app"
    app_namespace: "production"
    target_version: "{{ version | mandatory }}" # Must be passed via CLI: -e version=v2.2.0
    app_image: "myregistry.local/web-app"
    kubeconfig_path: "~/.kube/config"
    app_url: "https://webapp.production.local/healthz"

  tasks:
    - name: Get the current Deployment metadata
      kubernetes.core.k8s_info:
        kubeconfig: "{{ kubeconfig_path }}"
        kind: Deployment
        name: "{{ app_name }}"
        namespace: "{{ app_namespace }}"
      register: current_deploy_info

    - name: Verify the Deployment exists and save the current version
      set_fact:
        rollback_image: "{{ current_deploy_info.resources[0].spec.template.spec.containers[0].image }}"
      when: current_deploy_info.resources | length > 0

    - name: Display the transition version information
      debug:
        msg:
          - "Old image detected: {{ rollback_image | default('None (Fresh install)') }}"
          - "Heading to target image: {{ app_image }}:{{ target_version }}"

    - name: Update the application image on the Deployment
      kubernetes.core.k8s:
        kubeconfig: "{{ kubeconfig_path }}"
        state: present
        definition:
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: "{{ app_name }}"
            namespace: "{{ app_namespace }}"
          spec:
            template:
              metadata:
                annotations:
                  # Forcing new Pod creation even if the image tag is the same (e.g. latest)
                  deployment-triggered-by: "Ansible {{ ansible_date_time.iso8601 }}"
              spec:
                containers:
                  - name: web-server
                    image: "{{ app_image }}:{{ target_version }}"

    - name: Wait for the rollout status to complete (Timeout: 5 minutes)
      kubernetes.core.k8s_rollout_status:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ app_name }}"
        namespace: "{{ app_namespace }}"
        kind: Deployment
        timeout: 300
      register: rollout_result
      ignore_errors: true

    - name: Evaluate the rollout results
      block:
        - name: Fail the execution if the rollout status failed or timed out
          fail:
            msg: "The Kubernetes rollout failed or timed out!"
          when: rollout_result.failed or (rollout_result.status is not defined)

        - name: Run the post-rollout Smoke Test (Send HTTP Request)
          uri:
            url: "{{ app_url }}"
            method: GET
            status_code: 200
            validate_certs: false
            timeout: 10
          register: smoke_test_result
          retries: 6
          delay: 10
          # We try accessing the endpoint 6 times with a 10-second pause
          # to give tolerance for the backend application startup loading.

      rescue:
        - name: Critical warning - Starting automatic rollback
          debug:
            msg: "An error occurred during the rollout or the smoke test failed! Restoring the version..."

        - name: Cancel the update and return to the previous version (Rollback)
          command: >
            kubectl rollout undo deployment/{{ app_name }}
            -n {{ app_namespace }}
            --kubeconfig {{ kubeconfig_path }}            
          register: rollback_execution
          changed_when: true

        - name: Wait for the rollback status to complete
          kubernetes.core.k8s_rollout_status:
            kubeconfig: "{{ kubeconfig_path }}"
            name: "{{ app_name }}"
            namespace: "{{ app_namespace }}"
            kind: Deployment
            timeout: 180

        - name: Stop the playbook with an error message
          fail:
            msg: >
              The deployment to version {{ target_version }} failed!
              The system has been automatically rolled back to version {{ rollback_image }}.

In the playbook above, we design an error handling structure using block and rescue blocks. If the steps inside the main block (like waiting for the rollout status or smoke test verification through the uri module) fail, Ansible immediately jumps to the rescue section. Inside the rescue, we execute the kubectl rollout undo cancellation command to restore the cluster to the state before the playbook ran. This strategy is crucial for preventing our cluster from staying in a half-broken state.

Canary Deployments and Traffic Management #

Although Kubernetes’ built-in rolling update is very safe, sometimes we want to test new features limited to a small portion of real users before rolling them out to the entire cluster. This technique is called Canary Deployment (named after the canary birds miners used to detect toxic gas).

Canary Deployment Concept in Kubernetes #

Canary deployment can be implemented in Kubernetes simply without using a service mesh (like Istio or Linkerd) by creating two different Deployment objects that share the same label selector on the Service connecting them.

For example, we have a main Service named app-service directing data traffic to Pods with the label app: web-app. We create:

A Stable (Main) Deployment: Running version v2.1.0 with 9 Pod replicas.
A Canary Deployment: Running version v2.2.0 with 1 Pod replica.

Because the Service distributes data traffic evenly to all Pods matching the label selector, automatically about 10% (1 of 10 Pods) of total user traffic requests go to the new Canary version, while the remaining 90% stays directed to the safe stable version.

Canary Orchestration with Ansible #

Let’s create an Ansible playbook that automates Canary Deployment creation, verifies log/error anomalies over a certain duration, and upgrades the Stable version if the Canary testing succeeds.

# playbooks/canary-deploy.yml
---
- name: Canary Deployment Workflow Management
  hosts: localhost
  connection: local
  vars:
    app_name_stable: "web-app-stable"
    app_name_canary: "web-app-canary"
    app_namespace: "production"
    stable_version: "v2.1.0"
    canary_version: "v2.2.0"
    kubeconfig_path: "~/.kube/config"

  tasks:
    - name: Create the Canary Deployment (Small capacity: 1 Replica)
      kubernetes.core.k8s:
        kubeconfig: "{{ kubeconfig_path }}"
        state: present
        definition:
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: "{{ app_name_canary }}"
            namespace: "{{ app_namespace }}"
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: web-app # Sharing the same label selector as stable
                track: canary
            template:
              metadata:
                labels:
                  app: web-app
                  track: canary
              spec:
                containers:
                  - name: web-server
                    image: "myregistry.local/web-app:{{ canary_version }}"
                    ports:
                      - containerPort: 8080
                    readinessProbe:
                      httpGet:
                        path: /ready
                        port: 8080

    - name: Wait for the Canary Deployment to be ready to serve traffic
      kubernetes.core.k8s_rollout_status:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ app_name_canary }}"
        namespace: "{{ app_namespace }}"
        kind: Deployment
        timeout: 180

    - name: Observe data traffic (Evaluation Phase)
      pause:
        minutes: 2
      # During these 2 minutes, we let users use the application.
      # In the real world, this step can be integrated with a Prometheus query
      # to check the HTTP 5xx error rate on the canary pod.

    - name: Fetch failure metrics from the canary pod logging (Simulation)
      command: >
        kubectl logs -n {{ app_namespace }} -l track=canary --tail=100
        --kubeconfig {{ kubeconfig_path }}        
      register: canary_logs
      changed_when: false

    - name: Evaluate the Canary version stability
      block:
        - name: Check whether there are critical error patterns in the logs
          fail:
            msg: "Critical errors detected in the canary logs!"
          when: "'ERROR' in canary_logs.stdout or 'FATAL' in canary_logs.stdout"

        - name: Promote the Canary version to Stable
          kubernetes.core.k8s:
            kubeconfig: "{{ kubeconfig_path }}"
            state: present
            definition:
              apiVersion: apps/v1
              kind: Deployment
              metadata:
                name: "{{ app_name_stable }}"
                namespace: "{{ app_namespace }}"
              spec:
                template:
                  spec:
                    containers:
                      - name: web-server
                        image: "myregistry.local/web-app:{{ canary_version }}"

        - name: Wait for the stable version update to complete
          kubernetes.core.k8s_rollout_status:
            kubeconfig: "{{ kubeconfig_path }}"
            name: "{{ app_name_stable }}"
            namespace: "{{ app_namespace }}"
            kind: Deployment
            timeout: 300

        - name: Remove the Canary Deployment after the promotion completes
          kubernetes.core.k8s:
            kubeconfig: "{{ kubeconfig_path }}"
            state: absent
            kind: Deployment
            name: "{{ app_name_canary }}"
            namespace: "{{ app_namespace }}"

      rescue:
        - name: Remove the Canary Deployment because it was detected as unstable
          kubernetes.core.k8s:
            kubeconfig: "{{ kubeconfig_path }}"
            state: absent
            kind: Deployment
            name: "{{ app_name_canary }}"
            namespace: "{{ app_namespace }}"

        - name: Stop the playbook and report the Canary testing failure
          fail:
            msg: "The Canary version testing failed because errors were detected in the logs. The Canary Deployment has been withdrawn."

The Canary orchestration above provides an additional very strong protection layer for our production systems. We don’t need to guess whether our new version has hidden bugs that only appear under real-world data traffic.

Deployment Strategy Comparison #

As a reference for determining our application pipeline design, let’s summarize the comparison of several common deployment strategies along with their advantages and disadvantages:

Deployment Strategy	Additional Resource Needs	Downtime Impact	Management Complexity	Main Advantage	Main Disadvantage
Recreate	0% (No additional resources needed)	Downtime exists (during the Pod replacement period)	Very Low	Very simple, no dual-version data conflict potential.	Users experience several minutes of downtime.
Rolling Update	Low-Medium (Depends on the maxSurge value)	Zero-Downtime	Low-Medium	Kubernetes built-in, automatic handling, safe.	The application must handle database schema backward compatibility.
Canary	Low (Only needs a minimum of 1 extra new Pod)	Zero-Downtime	Medium-High	Limits the damage radius if there are bugs, can validate real user experience.	Requires a precise monitoring system for error rate detection.
Blue-Green	100% (Needs a full-size parallel cluster/infra)	Zero-Downtime	High	Instant traffic switching via DNS/Load Balancer, instant rollback.	Very resource-hungry, infrastructure costs double.

Summary #

The maxUnavailable: 0 Configuration — Is the best option for guaranteeing zero-downtime service availability, because Kubernetes won’t shut down old Pods before new Pods are Ready.
Mandate Health Probe Checks — Apply startup, readiness, and liveness probes on every container manifest so Kubernetes can accurately monitor application internal readiness.
Leverage k8s_rollout_status — Always monitor the deployment rollout progress status using this Ansible module so the playbook doesn’t continue to the next step while the deployment process is still running.
Implement Automatic Rollback — Use error handling blocks (block/rescue) to trigger instant rollback via kubectl rollout undo if post-deployment smoke tests fail.
Evaluate Canary for Critical Releases — Use the Canary strategy by separating the track label, run log observation over a certain duration before promoting it to the stable version.
Database Schema Backward Compatibility — When running gradual deployments, make sure our application always supports backward compatibility because old and new Pod versions run side by side during the rollout transition.

← Previous: Cluster Maintenance Next: CI/CD Integration →