Cluster Maintenance #

Managing Kubernetes clusters in production environments requires extra attention to reliability and service availability aspects. Running infrastructure isn’t static; we must routinely perform maintenance like node OS updates, Kubernetes version upgrades, and periodic backups of the etcd database. Doing these maintenance tasks manually on dozens or hundreds of nodes isn’t just exhausting, but also increases the risk of human configuration errors that can cause downtime. By leveraging Ansible, we can build orchestrated, consistent, documented, and safely repeatable maintenance workflows without sacrificing the stability of running applications.

Node Lifecycle Management: Cordon and Drain #

Before we touch the physical servers or virtual machines acting as worker nodes for maintenance like kernel updates or hardware replacement, we must ensure no workloads are disrupted. This workload migration process in Kubernetes is known as cordon (marking a node so it doesn’t accept new Pods) and drain (gracefully evicting running Pods to other nodes).

If we directly shut down or reboot a server without this process, the applications running on it experience sudden connection termination. Dead Pods take time to be detected by the control plane before finally being rescheduled on other nodes. During that gap, users experience service disruption.

Why Must Cordon and Drain Be Sequential? #

The node maintenance process must follow a strict logical order. First, we mark the node as unschedulable (cordon). This prevents the Kubernetes scheduler from placing new Pods on that node while the old Pod migration process is underway. Second, we drain the node, which evacuates all active Pods to other healthy worker nodes.

Here’s a visualization of the safe node maintenance management workflow:

flowchart TD
    A["Start Node Maintenance"] --> B["Cordon Node (spec.unschedulable)"]
    B --> C["Drain Node (Eviction)"]
    C --> D{"Are there DaemonSets / emptyDir?"}
    D -- "Yes" --> E["Use --ignore-daemonsets & --delete-emptydir-data"]
    D -- "No" --> F["Pod Evacuation Complete"]
    E --> F
    F --> G["Do OS / Package Update (Kubeadm, Kubelet)"]
    G --> H["Reboot Node (If Needed)"]
    H --> I["Uncordon Node"]
    I --> J["Verify Node Ready"]
    J --> K["Done"]

Anti-Pattern vs Cordon-Drain Automation Solution #

Let’s study an example of a common node maintenance mistake and the right solution using Ansible modules.

# ANTI-PATTERN: Doing node maintenance directly with a reboot or without integrated drain
- name: Reboot the worker node directly
  hosts: worker_nodes
  become: true
  tasks:
    - name: Restart the server directly
      reboot:
        reboot_timeout: 300
      # DON'T do this! Pods will die suddenly, traffic will drop,
      # and the scheduler needs up to 5 minutes (pod-eviction-timeout)
      # to reschedule Pods to other nodes.

# CORRECT: Using integrated coordination to cordon and drain gracefully
- name: Cordon and drain the node gracefully before maintenance
  hosts: localhost
  vars:
    target_node: "k8s-worker-01"
  tasks:
    - name: Mark the node as unschedulable (Cordon)
      kubernetes.core.k8s:
        kubeconfig: "~/.kube/config"
        state: present
        definition:
          apiVersion: v1
          kind: Node
          metadata:
            name: "{{ target_node }}"
          spec:
            unschedulable: true

    - name: Evacuate the workload from the node (Drain)
      command: >
        kubectl drain {{ target_node }}
        --ignore-daemonsets
        --delete-emptydir-data
        --grace-period=60
        --timeout=300s
        --kubeconfig ~/.kube/config        
      register: drain_result
      changed_when: true

In the solution example above, we use the kubernetes.core.k8s module to manipulate the node specification directly to enable the unschedulable status. After that, we use the kubectl drain command wrapped in the command module because the native k8s module doesn’t yet provide a drain wrapper as reliable as the built-in CLI. The --ignore-daemonsets parameter must be included because DaemonSet-managed Pods run on every node and can’t be moved. The --delete-emptydir-data parameter ensures Pods using temporary local storage (emptyDir) can still be evacuated even though their data will be deleted.

Overcoming Drain Obstacles (DaemonSets, Local Storage, and Grace Period) #

When running the drain process, sometimes we encounter failures due to Pods not managed by a Controller (like Deployment, StatefulSet, or ReplicaSet), or Pods using local storage. We must handle these cases with additional parameters.

If there are standalone Pods created directly without a controller, Kubernetes by default refuses the drain process to avoid permanent data loss. If we’re confident those Pods are safe to delete, we can add the --force parameter to our drain command. Additionally, the --grace-period=60 parameter gives applications inside Pods time to finish remaining processes or active connections (graceful shutdown) before the system sends the SIGKILL signal.

After the maintenance task is done, we must return the node to accepting Pods again. This process is called uncordon. Here’s an Ansible playbook to restore node scheduling functionality:

- name: Return the node to active status (Uncordon)
  hosts: localhost
  vars:
    target_node: "k8s-worker-01"
  tasks:
    - name: Reschedule Pods to the node (Uncordon)
      kubernetes.core.k8s:
        kubeconfig: "~/.kube/config"
        state: present
        definition:
          apiVersion: v1
          kind: Node
          metadata:
            name: "{{ target_node }}"
          spec:
            unschedulable: false

    - name: Wait for the node status to become Ready
      kubernetes.core.k8s_info:
        kubeconfig: "~/.kube/config"
        kind: Node
        name: "{{ target_node }}"
      register: node_info
      until: >
        node_info.resources[0].status.conditions 
        | selectattr('type', 'equalto', 'Ready') 
        | map(attribute='status') 
        | first == 'True'        
      retries: 20
      delay: 10

With this flow, we guarantee that the node transition before and after maintenance is fully monitored by our Ansible monitoring system.

Kubernetes Component Upgrade Strategy (Kubeadm, Kubelet, and Kubectl) #

The Kubernetes cluster version upgrade process is a routine task that must be run to ensure we get security updates, bug fixes, and the latest features. Kubernetes recommends upgrading a maximum of one minor version at a time (for example from version 1.28 to 1.29, not jumping directly from 1.28 to 1.30).

The upgrade process must start from the control plane first before continuing to worker nodes. The first component upgraded is kubeadm, then the control plane configuration, followed by kubelet and kubectl on control plane nodes, and only after that do we update worker nodes one by one.

Orchestrated Control Plane Upgrade Procedure #

Let’s create a safe Ansible playbook to process component upgrades on the control plane. This playbook updates the package repository, installs the new kubeadm version, applies the Kubernetes upgrade configuration, and updates kubelet.

# playbooks/upgrade-control-plane.yml
---
- name: Kubernetes Control Plane Version Upgrade
  hosts: control_plane
  become: true
  serial: 1
  vars:
    target_version: "1.29.2"
    debian_version: "1.29.2-1.1" # Adjust to the distro package naming format
  tasks:
    - name: Update the apt package repository cache
      apt:
        update_cache: true

    - name: Remove the hold protection on kubeadm
      dpkg_selections:
        name: kubeadm
        selection: install

    - name: Install the target kubeadm version
      apt:
        name: "kubeadm={{ debian_version }}"
        state: present
        update_cache: true

    - name: Re-apply the hold protection on kubeadm
      dpkg_selections:
        name: kubeadm
        selection: hold

    - name: Run the upgrade verification simulation (Upgrade Plan)
      command: "kubeadm upgrade plan v{{ target_version }}"
      register: upgrade_plan
      changed_when: false

    - name: Display the upgrade simulation results
      debug:
        var: upgrade_plan.stdout_lines

    - name: Apply the control plane upgrade
      command: "kubeadm upgrade apply v{{ target_version }} --yes"
      register: upgrade_apply_result
      changed_when: "'Upgrade complete!' in upgrade_apply_result.stdout"

    - name: Remove the hold protection on kubelet and kubectl
      dpkg_selections:
        name: "{{ item }}"
        selection: install
      loop:
        - kubelet
        - kubectl

    - name: Upgrade kubelet and kubectl
      apt:
        name:
          - "kubelet={{ debian_version }}"
          - "kubectl={{ debian_version }}"
        state: present

    - name: Re-apply the hold protection on kubelet and kubectl
      dpkg_selections:
        name: "{{ item }}"
        selection: hold
      loop:
        - kubelet
        - kubectl

    - name: Reload the systemd daemon
      systemd:
        daemon_reload: true

    - name: Restart the kubelet service
      systemd:
        name: kubelet
        state: restarted
        enabled: true

The playbook above uses the dpkg_selections command to temporarily remove the hold status on APT packages. This hold status is very important in production environments so Kubernetes system packages don’t accidentally upgrade when administrators run daily apt-get upgrade commands. After installing the right version, we must restore the hold status.

Worker Node Rolling Upgrade Orchestration #

After the control plane is successfully upgraded, we can upgrade the package versions on all worker nodes. We must not update all worker nodes at once because it causes cluster capacity exhaustion and application downtime. We must orchestrate them in rotation (rolling upgrade) using the serial: 1 parameter at the play level.

Here’s the playbook for doing a rolling upgrade on all worker nodes:

# playbooks/upgrade-worker-nodes.yml
---
- name: Worker Nodes Package Rolling Upgrade
  hosts: worker_nodes
  become: true
  serial: 1  # Process nodes one by one to prevent downtime
  vars:
    target_version: "1.29.2"
    debian_version: "1.29.2-1.1"
    kubeconfig_path: "/home/ansible/.kube/config"

  tasks:
    - name: Drain the worker node from local control (Control Node)
      delegate_to: localhost
      become: false
      command: >
        kubectl drain {{ inventory_hostname }}
        --ignore-daemonsets
        --delete-emptydir-data
        --grace-period=60
        --timeout=300s
        --kubeconfig {{ kubeconfig_path }}        
      changed_when: true

    - name: Remove the hold protection on node components
      dpkg_selections:
        name: "{{ item }}"
        selection: install
      loop:
        - kubeadm
        - kubelet
        - kubectl

    - name: Upgrade kubeadm on the worker node
      apt:
        name: "kubeadm={{ debian_version }}"
        state: present
        update_cache: true

    - name: Run the worker node upgrade configuration
      command: kubeadm upgrade node
      register: node_upgrade_result
      changed_when: "'Successfully upgraded' in node_upgrade_result.stdout"

    - name: Upgrade kubelet and kubectl on the worker node
      apt:
        name:
          - "kubelet={{ debian_version }}"
          - "kubectl={{ debian_version }}"
        state: present

    - name: Restore the hold status on node components
      dpkg_selections:
        name: "{{ item }}"
        selection: hold
      loop:
        - kubeadm
        - kubelet
        - kubectl

    - name: Reload the systemd configuration
      systemd:
        daemon_reload: true

    - name: Restart the kubelet service
      systemd:
        name: kubelet
        state: restarted

    - name: Return the node to be schedulable again (Uncordon)
      delegate_to: localhost
      become: false
      command: >
        kubectl uncordon {{ inventory_hostname }}
        --kubeconfig {{ kubeconfig_path }}        
      changed_when: true

    - name: Verify node stability
      delegate_to: localhost
      become: false
      kubernetes.core.k8s_info:
        kubeconfig: "{{ kubeconfig_path }}"
        kind: Node
        name: "{{ inventory_hostname }}"
      register: node_status
      until: >
        node_status.resources[0].status.conditions 
        | selectattr('type', 'equalto', 'Ready') 
        | map(attribute='status') 
        | first == 'True'        
      retries: 15
      delay: 10

In this playbook, the delegate_to: localhost technique is used to run administrative kubectl commands directly from the control node (the machine where Ansible runs) that has kubeconfig file authorization. This avoids the need to copy the sensitive kubeconfig file to all worker nodes. This step significantly improves our infrastructure’s security posture.

Distributed etcd Backup and Recovery #

The etcd component is the main pillar of a Kubernetes cluster acting as a consistent, distributed key-value data store. All Kubernetes object definitions, cluster states, network configurations, and application secrets are stored in etcd. If fatal etcd data failure occurs without a backup, our cluster can’t be saved. Therefore, building an automatic etcd backup system is an absolute obligation for operations teams.

How etcd Snapshots Work #

etcd backup is done by creating a point-in-time snapshot of the data store using the etcdctl command. Because etcd in production clusters is usually configured with TLS for data traffic encryption, the backup process requires access to cluster encryption certificates like the root Certificate Authority (CA), server certificate, and its private key.

Here’s a visualization of the distributed etcd backup flow using an automatic controller:

flowchart TD
    A["Trigger Backup Scheduler"] --> B["Get etcd Cert/Key Credentials"]
    B --> C["Run etcdctl snapshot save"]
    C --> D{"Verify Integrity?"}
    D -- "Yes (Valid)" --> E["Copy Snapshot to Control Node (Fetch)"]
    D -- "No (Failed)" --> F["Send Alert Notification"]
    E --> G["Apply Retention Policy (Delete Backups >7 Days)"]
    G --> H["Done"]

Ansible Playbook for Automatic etcd Backup #

Let’s create an Ansible playbook that periodically executes etcd backups on the control plane node, verifies the backup file integrity, downloads it to a centralized storage location, and applies the storage retention policy.

# playbooks/backup-etcd.yml
---
- name: Kubernetes etcd Database Backup
  hosts: control_plane[0] # Just run on one of the control plane nodes
  become: true
  vars:
    etcd_backup_dir: "/var/backups/etcd"
    etcd_certs_dir: "/etc/kubernetes/pki/etcd"
    local_backup_dest: "/opt/backup-center/k8s-etcd"
  tasks:
    - name: Create the local backup storage directory on the target host
      file:
        path: "{{ etcd_backup_dir }}"
        state: directory
        mode: '0700'
        owner: root
        group: root

    - name: Get the timestamp for a unique file name
      set_fact:
        backup_timestamp: "{{ lookup('pipe', 'date +%Y%m%d-%H%M%S') }}"

    - name: Run the etcdctl snapshot
      command: >
        etcdctl snapshot save {{ etcd_backup_dir }}/etcd-snap-{{ backup_timestamp }}.db
        --endpoints=https://127.0.0.1:2379
        --cacert={{ etcd_certs_dir }}/ca.crt
        --cert={{ etcd_certs_dir }}/server.crt
        --key={{ etcd_certs_dir }}/server.key        
      environment:
        ETCDCTL_API: "3"
      register: etcd_backup_result

    - name: Verify the integrity of the newly created snapshot file
      command: >
        etcdctl snapshot status {{ etcd_backup_dir }}/etcd-snap-{{ backup_timestamp }}.db
        --write-out=table        
      environment:
        ETCDCTL_API: "3"
      register: etcd_status_result
      changed_when: false

    - name: Ensure the backup file is valid before further processing
      assert:
        that:
          - "'total size' in etcd_status_result.stdout"
        fail_msg: "The etcd snapshot file is corrupted or invalid!"

    - name: Create the backup directory on the Ansible Control machine
      delegate_to: localhost
      become: false
      file:
        path: "{{ local_backup_dest }}"
        state: directory
        mode: '0750'

    - name: Fetch the backup file to centralized storage (Ansible Control Node)
      fetch:
        src: "{{ etcd_backup_dir }}/etcd-snap-{{ backup_timestamp }}.db"
        dest: "{{ local_backup_dest }}/etcd-snap-{{ inventory_hostname }}-{{ backup_timestamp }}.db"
        flat: true

    - name: Find etcd backups older than 7 days on the target host
      find:
        paths: "{{ etcd_backup_dir }}"
        age: "7d"
        patterns: "etcd-snap-*.db"
      register: old_backups

    - name: Remove expired etcd backups on the target host
      file:
        path: "{{ item.path }}"
        state: absent
      loop: "{{ old_backups.files }}"

etcd Restore Procedure in Emergency Conditions #

When our cluster experiences total data corruption, etcd must be restored before the Kubernetes API server can run again. This recovery process must be done very carefully because etcd must be fully stopped on all control plane nodes first to prevent status synchronization conflicts.

The etcd recovery steps are as follows:

Stop all kube-apiserver and etcd static pod manifests by moving their definition files out of the /etc/kubernetes/manifests manifest directory so systemd/kubelet shuts down those containers.
Delete the old corrupted etcd data directory at /var/lib/etcd.
Run the etcdctl snapshot restore command with the node name, peer IP address, and appropriate etcd cluster initialization list parameters.
Move the static pod manifests back to their original directory so kubelet restarts the API server and etcd.

Here’s an example etcd recovery script automated with Ansible:

# playbooks/restore-etcd.yml
---
- name: Kubernetes Cluster etcd Recovery from Backup
  hosts: control_plane
  become: true
  vars:
    etcd_data_dir: "/var/lib/etcd"
    manifest_dir: "/etc/kubernetes/manifests"
    backup_file_path: "/tmp/etcd-snap-restore.db" # Make sure the backup file is already copied to this directory
    etcd_certs_dir: "/etc/kubernetes/pki/etcd"
    cluster_token: "etcd-k8s-cluster"
  tasks:
    - name: Check the backup file availability on the host
      stat:
        path: "{{ backup_file_path }}"
      register: backup_file_status

    - name: Fail if the backup file is not found
      fail:
        msg: "Backup file not found at {{ backup_file_path }}!"
      when: not backup_file_status.stat.exists

    - name: Disable the API Server and etcd by moving the static pod manifests
      file:
        path: "{{ manifest_dir }}/{{ item }}"
        state: absent
      loop:
        - kube-apiserver.yaml
        - etcd.yaml
      # This step is important so kubelet shuts down the current apiserver and etcd pods.

    - name: Pause so the containers fully stop
      pause:
        seconds: 15

    - name: Delete the old etcd data directory (Back up old data first if needed)
      file:
        path: "{{ etcd_data_dir }}"
        state: absent

    - name: Run the local etcd snapshot restore
      command: >
        etcdctl snapshot restore {{ backup_file_path }}
        --name={{ inventory_hostname }}
        --initial-cluster={{ inventory_hostname }}=https://{{ ansible_default_ipv4.address }}:2380
        --initial-cluster-token={{ cluster_token }}
        --initial-advertise-peer-urls=https://{{ ansible_default_ipv4.address }}:2380
        --data-dir={{ etcd_data_dir }}        
      environment:
        ETCDCTL_API: "3"

    - name: Ensure the new etcd data directory ownership is correct
      file:
        path: "{{ etcd_data_dir }}"
        state: directory
        owner: root
        group: root
        recurse: true

    - name: Re-enable the API Server and etcd by restoring the static pod manifests
      # We can copy them back from a safe backup directory
      copy:
        src: "/etc/kubernetes/manifests-backup/{{ item }}"
        dest: "{{ manifest_dir }}/{{ item }}"
        remote_src: true
      loop:
        - etcd.yaml
        - kube-apiserver.yaml

This recovery process must be adjusted to each cluster’s network configuration. It’s very important to periodically test this recovery script on a staging cluster so when a real failure occurs in production, the operations team can act calmly and methodically.

Handling Problematic Nodes and Automatic Recovery #

Besides scheduled maintenance tasks, Kubernetes clusters often face emergency situations where nodes suddenly change status to NotReady or experience resource pressure. The causes can vary, from running out of memory (OOM), disk failures, to container runtime daemon crashes like Docker or Containerd.

As administrators, we can leverage Ansible for early detection (drift detection / health check) and automatic self-healing recovery actions without waiting for complaint tickets to arrive at the IT help desk.

Understanding Node Status Conditions #

Kubernetes exposes several node health conditions. We can monitor these conditions in detail using Ansible modules. Here’s a summary of the main node condition indicators and remediation actions we can take through Ansible scripts:

Node Condition	Problem Description	Recovery Action with Ansible
Ready	The node is healthy and can accept Pods.	No action needed.
DiskPressure	Local storage capacity is running low (<10%).	Run docker cache cleanup, remove unused images, and clean log directories.
MemoryPressure	RAM capacity is running low, OOM risk.	Identify non-essential Pods with high RAM consumption, migrate workloads in rotation.
PIDPressure	Too many processes running on the node.	Check the max PID limit in systemd, restart zombie processes, limit container threads.
NetworkUnavailable	The node’s network configuration is disrupted.	Restart the CNI daemon (like Calico/Cilium), check routing tables, reload network kernel modules.

Automatic Remediation Playbook for Problematic Nodes #

Let’s create a defensive Ansible playbook that monitors the active status of the container runtime service (containerd) and the kubelet agent on every worker node. If those services die, Ansible detects and tries to bring them back up, then verifies the recovery.

# playbooks/node-remediation.yml
---
- name: Automatic Detection and Remediation of Worker Node Services
  hosts: worker_nodes
  become: true
  tasks:
    - name: Check the containerd runtime active status
      ansible.builtin.systemd:
        name: containerd
      register: containerd_status

    - name: Check the kubelet agent active status
      ansible.builtin.systemd:
        name: kubelet
      register: kubelet_status

    - name: Take recovery action if containerd is down
      ansible.builtin.systemd:
        name: containerd
        state: restarted
      when: containerd_status.status.ActiveState != 'active'
      register: containerd_remediation

    - name: Pause 10 seconds so containerd initializes the socket
      pause:
        seconds: 10
      when: containerd_remediation.changed

    - name: Take recovery action if kubelet is down
      ansible.builtin.systemd:
        name: kubelet
        state: restarted
      when: kubelet_status.status.ActiveState != 'active'

    - name: Clean up unused image cache if DiskPressure occurs
      block:
        - name: Get the root disk space information
          ansible.builtin.setup:
            filter: "ansible_mounts"

        - name: Calculate the root disk usage percentage
          set_fact:
            root_disk: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | first }}"

        - name: Run docker/containerd cache cleanup if disk usage > 85%
          command: crictl rmi --prune
          when: "((root_disk.size_total - root_disk.size_available) / root_disk.size_total * 100) > 85"
          changed_when: true
          register: prune_result

        - name: Display the disk cleanup log
          debug:
            var: prune_result.stdout_lines
          when: prune_result.changed
      rescue:
        - name: Warning if the disk cleanup process fails
          debug:
            msg: "Failed to detect disk space or crictl rmi is not available on the node."

The playbook above is an example of a simple self-healing implementation. We can extend this logic by integrating alert systems like Prometheus Alertmanager to automatically trigger Ansible playbooks via webhook when detecting problems on our Kubernetes cluster.

Summary #

Sequential Cordon & Drain Orchestration — Always cordon to mark a node so it doesn’t accept new Pods before running the drain command to migrate workloads gracefully.
Robust Drain Configuration — Use the --ignore-daemonsets option to ignore system daemons and --delete-emptydir-data to approve temporary local data deletion when evacuating Pods.
Gradual Minor Version Upgrades — Run the Kubernetes upgrade process sequentially one minor version at a time. Start the upgrade from control plane nodes then continue to worker nodes.
Package Protection with APT Hold — Lock the kubeadm, kubelet, and kubectl package versions using hold status so they don’t accidentally upgrade during daily OS maintenance.
etcd Backup with TLS Encryption — Do regular etcd backups using the etcdctl snapshot save command including complete valid TLS encryption certificates.
Centralized Snapshot Storage — Leverage Ansible’s fetch module to pull etcd snapshot files to the safe Ansible control node machine for easier disaster recovery management.
Automatic Remediation of Problematic Nodes — Create self-monitoring playbooks to detect container runtime deaths or disk space exhaustion, then apply instant recovery actions like service restarts and cache cleanup.

← Previous: Helm Next: Rolling Update →