Provision Cluster #

Building a Kubernetes cluster from scratch is often considered one of the most challenging rites of passage for a DevOps Engineer or System Administrator. This process involves a series of precise, sequential system-level steps across many servers: disabling swap, loading kernel modules, tuning sysctl parameters, installing the container runtime, installing Kubernetes binaries, initializing the control plane, setting up cluster networking (CNI), and finally joining worker nodes.

Doing these steps manually on 3 or 10 servers isn’t just exhausting — it’s also highly vulnerable to human error. One small mistake like forgetting to enable SystemdCgroup on the container runtime can cause the cluster to mysteriously fail days later. This is where Ansible comes in as a lifesaving solution. By automating the entire cluster bootstrap flow with idempotent Ansible playbooks, we can bring up a consistent, standardized, production-ready Kubernetes cluster with just a single command.

Why Choose Ansible for Cluster Provisioning? #

When we decide to automate the Kubernetes infrastructure lifecycle, we have several tool choices. However, Ansible offers unique advantages for low-level operating system bootstrapping processes.

Manual Bootstrap Challenges:
  ✗ We must SSH into each server one by one to run OS setup commands.
  ✗ Manually editing sensitive configuration files like /etc/fstab or config.toml is prone to typos.
  ✗ kubeadm initialization tokens have a limited validity period (24 hours) — copying these tokens manually between servers is very tedious.
  ✗ It's hard to replicate the exact same configuration if we want to create identical Staging and Production clusters.

Ease with Ansible Automation:
  ✓ One Playbook for All Nodes — run all OS preparation tasks in parallel across all servers.
  ✓ Dynamic Templating — use Jinja2 to generate clean, accurate containerd and sysctl configuration files.
  ✓ Workflow Orchestration — Ansible can register the kubeadm token on the control plane, store it as a memory variable, then forward it to worker nodes automatically.
  ✓ Version Control — our entire cluster infrastructure is defined as code (IaC) that can be stored in a Git repository.

Network Architecture Design and Cluster Inventory #

Before starting to write Ansible tasks, we must design our cluster topology first. We’ll build a cluster with one Control Plane (Master Node) and two Worker Nodes. The communication topology between nodes can be visualized through the following diagram:

flowchart TD
    subgraph ControlPlane["Control Plane Node"]
        M1["k8s-master-01 (10.0.1.10)"]
    end

    subgraph WorkerNodes["Worker Nodes"]
        W1["k8s-worker-01 (10.0.2.10)"]
        W2["k8s-worker-02 (10.0.2.11)"]
    end

    M1 <-->|"Join Token & API Control (Port 6443)"| W1
    M1 <-->|"Join Token & API Control (Port 6443)"| W2
    W1 <-->|"Pod-to-Pod CNI Overlay Network"| W2

To represent the architecture above in Ansible, we create a structured inventory file. This file divides servers by role so we can direct specific tasks to the right server groups.

# inventory/k8s-cluster/hosts.ini
[control_plane]
k8s-master-01 ansible_host=10.0.1.10 ansible_user=ubuntu

[worker_nodes]
k8s-worker-01 ansible_host=10.0.2.10 ansible_user=ubuntu
k8s-worker-02 ansible_host=10.0.2.11 ansible_user=ubuntu

[k8s_cluster:children]
control_plane
worker_nodes

We also define global cluster variables to control package versions and internal cluster network configuration in the group_vars file:

# inventory/k8s-cluster/group_vars/k8s_cluster.yml
---
k8s_version: "1.29.2"
k8s_minor_version: "1.29"
pod_network_cidr: "192.168.0.0/16" # Default CIDR for Calico CNI
control_plane_endpoint: "10.0.1.10:6443"

Step 1: Operating System and Linux Kernel Preparation #

Kubernetes requires very specific low-level operating system configuration before its main components can run. The two most critical configurations are permanently disabling swap and configuring kernel modules to bridge container networking.

Why Does Kubernetes Forbid Swap? #

By default, kubelet refuses to run if swap is active on the server. The main reason is resource allocation guarantees and performance. Kubernetes is designed to schedule applications based on strict resource limits. If the operating system moves Pod memory to disk swap, application performance drops drastically and unpredictably, and Kubernetes can’t accurately calculate the cluster’s real memory usage.

Kernel Modules and Sysctl #

For network traffic from containers inside Pods to be bridged correctly at the host level, we must load the overlay and br_netfilter kernel modules, and configure sysctl parameters so iptables can see Linux network bridge traffic.

Here’s the OS preparation task implementation in our Ansible role:

# roles/k8s-common/tasks/main.yml
---
- name: Disable swap for the current session
  command: swapoff -a
  changed_when: false

- name: Ensure swap is permanently disabled in fstab
  replace:
    path: /etc/fstab
    regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
    replace: '# \1'

- name: Create the kernel module load configuration file
  copy:
    dest: /etc/modules-load.d/k8s.conf
    content: |
      overlay
      br_netfilter      
    owner: root
    group: root
    mode: '0644'

- name: Load the overlay kernel module manually
  modprobe:
    name: overlay
    state: present

- name: Load the br_netfilter kernel module manually
  modprobe:
    name: br_netfilter
    state: present

- name: Configure sysctl parameters for Kubernetes networking
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    sysctl_file: /etc/sysctl.d/99-kubernetes-cri.conf
    reload: true
  loop:
    - { key: "net.bridge.bridge-nf-call-iptables",  value: "1" }
    - { key: "net.bridge.bridge-nf-call-ip6tables", value: "1" }
    - { key: "net.ipv4.ip_forward",                 value: "1" }

Step 2: Container Runtime Configuration (containerd) #

Since Kubernetes deprecated the Docker Shim in version 1.20 and removed it entirely in version 1.24, we must use a standalone container runtime compatible with the CRI (Container Runtime Interface). containerd is the very stable and lightweight industry standard choice.

The biggest challenge when deploying containerd for Kubernetes is cgroup driver configuration. By default, Linux uses systemd as the process init system. If containerd is configured with its own built-in cgroup driver (cgroupfs), the system will have two different cgroup managers. This can cause system instability when the server runs out of resources. We must force containerd to use the systemd cgroup driver (SystemdCgroup = true).

# roles/containerd/tasks/main.yml
---
- name: Install initial dependencies for containerd
  apt:
    name:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg
    state: present
    update_cache: true

- name: Create the keyrings directory for the Docker repository
  file:
    path: /etc/apt/keyrings
    state: directory
    mode: '0755'

- name: Add the official Docker GPG Key
  apt_key:
    url: https://download.docker.com/linux/ubuntu/gpg
    state: present
    keyring: /etc/apt/keyrings/docker.gpg

- name: Add the Docker repository for containerd
  apt_repository:
    repo: "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
    state: present
    filename: docker

- name: Install the containerd.io package
  apt:
    name: containerd.io
    state: present
    update_cache: true

- name: Create the containerd configuration directory
  file:
    path: /etc/containerd
    state: directory
    mode: '0755'

- name: Generate the default containerd configuration
  command: containerd config default
  register: containerd_config_raw
  changed_when: false

- name: Write the default containerd configuration file
  copy:
    content: "{{ containerd_config_raw.stdout }}"
    dest: /etc/containerd/config.toml
    owner: root
    group: root
    mode: '0644'

- name: Configure containerd to use the SystemdCgroup driver
  replace:
    path: /etc/containerd/config.toml
    regexp: 'SystemdCgroup = false'
    replace: 'SystemdCgroup = true'
  notify: Restart containerd

- name: Ensure containerd is active and runs at boot
  systemd:
    name: containerd
    state: started
    enabled: true

We also create a handler to restart the containerd service if there are configuration changes:

# roles/containerd/handlers/main.yml
---
- name: Restart containerd
  systemd:
    name: containerd
    state: restarted

Step 3: Installing the Kubeadm, Kubelet, and Kubectl Components #

After the container runtime is ready, the next step is installing the core Kubernetes components:

  1. kubelet: The main agent service responsible for running containers on every node.
  2. kubeadm: The CLI tool for deploying and initializing Kubernetes clusters following security best practices.
  3. kubectl: The CLI tool for interacting with the Kubernetes API Server.

[!IMPORTANT] Starting mid-2023, Google officially shut down the legacy APT repository (apt.kubernetes.io). We must switch to the new community repository at pkgs.k8s.io with a specific minor version directory naming format.

# roles/k8s-packages/tasks/main.yml
---
- name: Create the keyring directory for Kubernetes
  file:
    path: /etc/apt/keyrings
    state: directory
    mode: '0755'

- name: Download the new Kubernetes repository GPG Key
  apt_key:
    url: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/Release.key"
    state: present
    keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg

- name: Add the new Kubernetes repository to apt sources
  apt_repository:
    repo: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/ /"
    state: present
    filename: kubernetes

- name: Install kubelet, kubeadm, and kubectl together
  apt:
    name:
      - "kubelet={{ k8s_version }}-*"
      - "kubeadm={{ k8s_version }}-*"
      - "kubectl={{ k8s_version }}-*"
    state: present
    update_cache: true

- name: Pin package versions to prevent accidental updates during apt upgrade
  dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop:
    - kubelet
    - kubeadm
    - kubectl

Step 4: Control Plane Initialization (Master Node Bootstrap) #

Now, all dependencies are installed on all nodes. It’s time to initialize our first Control Plane node using the kubeadm init command. This task should only run on the first master server (control_plane[0]).

After initialization completes, we must:

  1. Copy the cluster admin file (admin.conf aka kubeconfig) to our user’s home directory so we can run kubectl commands without root access.
  2. Create a new dynamic join token for worker nodes to join the cluster securely.
# playbooks/init-control-plane.yml
---
- name: Initialize the Kubernetes Control Plane
  hosts: control_plane[0]
  become: true
  tasks:
    - name: Check whether the Control Plane has been initialized before
      stat:
        path: /etc/kubernetes/admin.conf
      register: k8s_init_check

    - name: Run the kubeadm initialization on the first master node
      command: >
        kubeadm init
        --pod-network-cidr={{ pod_network_cidr }}
        --control-plane-endpoint={{ control_plane_endpoint }}
        --kubernetes-version={{ k8s_version }}        
      register: kubeadm_init_output
      when: not k8s_init_check.stat.exists

- name: Configure kubectl Access for the Regular User
  hosts: control_plane[0]
  become: false
  tasks:
    - name: Create the .kube directory in the user home
      file:
        path: "{{ ansible_env.HOME }}/.kube"
        state: directory
        mode: '0700'

    - name: Copy the admin.conf configuration file to the user home directory
      copy:
        src: /etc/kubernetes/admin.conf
        dest: "{{ ansible_env.HOME }}/.kube/config"
        remote_src: true
        owner: "{{ ansible_user }}"
        group: "{{ ansible_user }}"
        mode: '0600'
      become: true

- name: Fetch the Join Command Dynamically
  hosts: control_plane[0]
  become: true
  tasks:
    - name: Generate a new valid join command
      command: kubeadm token create --print-join-command
      register: temp_join_command
      changed_when: false

    - name: Store the join command in the global localhost variable
      set_fact:
        k8s_join_command: "{{ temp_join_command.stdout }}"
      delegate_to: localhost
      delegate_facts: true

Step 5: Container Network Interface (CNI) Setup #

Kubernetes assumes every Pod has its own unique IP address that can communicate directly with other Pods in the cluster, even if those Pods are on different physical hosts. To realize this network model, we must install a CNI (Container Network Interface) plugin.

The two most popular CNI options in the industry:

  1. Flannel: A very simple and lightweight CNI. It uses VXLAN encapsulation to create a basic overlay network. Flannel is great for small clusters or development environments because of its minimal configuration. However, it doesn’t support advanced security features like Network Policies.
  2. Calico: An advanced CNI very popular at the production level. Calico uses pure BGP protocol without additional encapsulation (when configured that way) to provide high-speed network performance. Additionally, it provides a very powerful security rule enforcement engine (Network Policies).
flowchart TD
    subgraph Node_Master["Master Node"]
        API_CNI["kube-apiserver"]
    end

    subgraph Node_Worker1["Worker Node 1"]
        PodA["Pod A (192.168.1.10)"]
        CNI1["Calico Agent (CNI)"]
        PodA --> CNI1
    end

    subgraph Node_Worker2["Worker Node 2"]
        PodB["Pod B (192.168.2.20)"]
        CNI2["Calico Agent (CNI)"]
        PodB --> CNI2
    end

    CNI1 <-->|"Overlay Tunnel VXLAN / BGP"| CNI2

We’ll install the Calico CNI by deploying its official manifest directly from the master control plane after initialization completes:

# playbooks/install-cni.yml
---
- name: Deploy the Calico CNI Network
  hosts: control_plane[0]
  become: false
  tasks:
    - name: Download the official Calico Operator manifest
      get_url:
        url: https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
        dest: /tmp/tigera-operator.yaml
        mode: '0644'

    - name: Apply the Calico Tigera operator to the cluster
      command: kubectl create -f /tmp/tigera-operator.yaml
      register: apply_operator
      failed_when: 
        - apply_operator.rc != 0 
        - "'already exists' not in apply_operator.stderr"
      changed_when: "'created' in apply_operator.stdout"

    - name: Download the Calico Custom Resources configuration manifest
      get_url:
        url: https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml
        dest: /tmp/custom-resources.yaml
        mode: '0644'

    - name: Adjust the Pod CIDR in the Calico manifest to our variable
      replace:
        path: /tmp/custom-resources.yaml
        regexp: 'cidr: 192.168.0.0/16'
        replace: "cidr: {{ pod_network_cidr }}"

    - name: Apply the Calico network configuration
      command: kubectl create -f /tmp/custom-resources.yaml
      register: apply_resources
      failed_when:
        - apply_resources.rc != 0
        - "'already exists' not in apply_resources.stderr"
      changed_when: "'created' in apply_resources.stdout"

Step 6: Joining Worker Nodes to the Cluster #

After the Control Plane and CNI network are properly installed, it’s time to join the worker servers so they can receive Pod scheduling instructions. We’ll use the join command variable previously stored on the localhost host.

After running the join command, the best operational task is to ensure and wait until the new node’s status is reported as Ready by the API Server before ending the playbook.

# playbooks/join-workers.yml
---
- name: Join Worker Nodes to the Cluster
  hosts: worker_nodes
  become: true
  tasks:
    - name: Check whether this node has been connected before
      stat:
        path: /etc/kubernetes/kubelet.conf
      register: kubelet_conf_check

    - name: Run the join command to the Control Plane
      command: "{{ hostvars['localhost']['k8s_join_command'] }}"
      when: not kubelet_conf_check.stat.exists

    - name: Wait for the node status to become 'Ready' on the Control Plane
      command: >
        kubectl get node {{ ansible_hostname }}
        --kubeconfig /etc/kubernetes/admin.conf
        -o jsonpath='{.status.conditions[-1].type}'        
      register: node_status_output
      until: node_status_output.stdout == "Ready"
      retries: 30
      delay: 10
      delegate_to: "{{ groups['control_plane'][0] }}"
      changed_when: false

Step 7: Fetching the Kubeconfig to the Local Control Node #

The last step of cluster provisioning is copying the secure admin.conf file from the first master server to our local machine (control node). This allows us to manage the Kubernetes cluster remotely using local tools like kubectl, k9s, or our subsequent Ansible playbooks without constantly SSHing into the master server.

# playbooks/fetch-kubeconfig.yml
---
- name: Secure and Fetch the Kubeconfig Locally
  hosts: control_plane[0]
  become: true
  tasks:
    - name: Create the local kubeconfig storage directory on the Control Node
      file:
        path: "{{ playbook_dir }}/kubeconfig"
        state: directory
        mode: '0700'
      delegate_to: localhost
      become: false

    - name: Copy the admin.conf file to the local playbook directory
      fetch:
        src: /etc/kubernetes/admin.conf
        dest: "{{ playbook_dir }}/kubeconfig/admin.conf"
        flat: true

    - name: Adjust the API endpoint address in the local kubeconfig file
      replace:
        path: "{{ playbook_dir }}/kubeconfig/admin.conf"
        regexp: 'server: https://127.0.0.1:6443'
        replace: "server: https://{{ control_plane_endpoint.split(':')[0] }}:6443"
      delegate_to: localhost
      become: false

Now we have a secure local cluster configuration file at the kubeconfig/admin.conf path. We can easily use it by setting the environment variable export KUBECONFIG=./kubeconfig/admin.conf in our local terminal.

Summary #

  • Disabling swap must be done permanently on all cluster nodes so kubelet can run stably and calculate resources accurately.
  • The overlay and br_netfilter kernel modules plus the network bridge sysctl parameters must be active so container communication traffic can bridge the Linux host correctly.
  • Use containerd with the SystemdCgroup = true configuration to align the operating system’s cgroup management with the container runtime to avoid OOM crashes.
  • Move the old Kubernetes repository configuration to the new pkgs.k8s.io community repository to get minor version updates above version 1.28+.
  • Use dpkg_selections hold on the kubelet, kubeadm, and kubectl packages so cluster component versions stay locked and don’t accidentally auto-upgrade.
  • Apply the CNI (like Calico) immediately after the control plane is ready to enable dynamic cross-host Pod-to-Pod network communication.
  • Fetch the join command dynamically using host fact memory (set_fact on localhost) instead of hardcoding static tokens that quickly expire in inventory files.
  • Fetch the admin.conf locally with endpoint IP adjustments so the cluster can be controlled remotely without always needing to SSH into the master server.

← Previous: What is Kubernetes? Next: Deploy Manifest →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact