Provision Host #

Before we can run and orchestrate containers in a production environment, the fundamental thing we must do is prepare the host that will act as the container runtime. This process is often called host provisioning. If we do it manually — from package installation, daemon configuration, user creation, to security management — we’re vulnerable to configuration drift problems. One server might have different logging configuration, while another uses a suboptimal storage driver, ultimately triggering hard-to-trace application failures.

With Ansible, we can automate the entire Docker host provisioning flow. From adding the official repository, installing a specific Docker Engine version, dynamically composing the /etc/docker/daemon.json configuration file, configuring the overlay2 storage driver, limiting log size so it doesn’t fill the disk, to setting up a cron job for container garbage collection. This Infrastructure as Code approach ensures every Docker host we manage runs with identical, secure, production-ready configuration.

flowchart TD
    Start["Start Provisioning"] --> CheckOS{"Detect OS Family"}
    CheckOS -->|"Debian / Ubuntu"| SetupAPT["Setup APT Repository"]
    CheckOS -->|"RedHat / CentOS"| SetupYUM["Setup YUM/DNF Repository"]
    SetupAPT --> InstallDocker["Install Docker Engine & Plugins"]
    SetupYUM --> InstallDocker
    InstallDocker --> ConfigDaemon["Deploy daemon.json & Validate JSON"]
    ConfigDaemon --> SetupUsers["Setup User & docker Group"]
    SetupUsers --> EnableService["Enable & Run Docker Service"]
    EnableService --> SetupGC["Setup Cron Garbage Collection"]
    SetupGC --> Verify["Verify & Test Container"]
    Verify --> End["Done (Host Ready)"]

Manual Docker Host Management Challenges #

Managing Docker hosts manually by typing commands one by one in the terminal is a highly unrecommended pattern for production scale. As the number of servers grows, maintenance challenges increase exponentially.

There are several main problems we often face from manual installation:

1. Package Version Mismatch: Without centralized control, one server might run Docker version 24.x while another runs version 26.x. This version difference can cause API runtime behavior inconsistencies or integration module failures.
2. Scattered Daemon Configuration: Vital settings like registry mirror addresses, dns options, live-restore, and logging drivers are often forgotten to be configured uniformly on every host.
3. Disk Full Risk: By default, Docker doesn’t limit container log size and doesn’t remove unused images (dangling images). Without automated log rotation systems and periodic cleanup, our servers will quickly run out of storage space.

To see the impact comparison between the manual approach and Ansible automation, we can look at the table below:

Installing Docker Engine on Ubuntu and CentOS #

The first step in the provisioning process is installing Docker Engine. To ensure reliability, we must avoid installing the distro’s built-in packages whose versions are often far behind. We must use Docker’s official repository.

In an Ansible playbook, we can use a multi-OS approach that detects the OS family (ansible_os_family) using conditionals. Here’s a Docker role structure supporting Ubuntu (Debian) and CentOS (RedHat):

1. Main Task Structure (tasks/main.yml) #

We split the installation logic by OS family and then run the shared configuration that applies to all systems:

# roles/docker/tasks/main.yml
---
- name: Run the Ubuntu-specific installation (Debian)
  include_tasks: install_debian.yml
  when: ansible_os_family == "Debian"

- name: Run the CentOS-specific installation (RedHat)
  include_tasks: install_redhat.yml
  when: ansible_os_family == "RedHat"

- name: Run the shared Docker Daemon configuration
  include_tasks: configure.yml

- name: Run the user and group setup
  include_tasks: users.yml

- name: Configure system logging and garbage collection
  include_tasks: maintenance.yml

- name: Verify the Docker installation
  include_tasks: verify.yml

2. Ubuntu/Debian Installation Path (tasks/install_debian.yml) #

For the Debian family, we must ensure old packages are cleaned up, APT dependencies installed, the Docker GPG key securely added to the /etc/apt/keyrings directory, the repository added, and finally Docker Engine installed with a pinned (locked) version.

# roles/docker/tasks/install_debian.yml
---
- name: Remove old Docker versions to prevent conflicts
  apt:
    name:
      - docker
      - docker-engine
      - docker.io
      - containerd
      - runc
    state: absent

- name: Install the required APT dependency packages
  apt:
    name:
      - ca-certificates
      - curl
      - gnupg
      - lsb-release
      - python3-pip
    state: present
    update_cache: true

- name: Create the keyrings directory if it doesn't exist
  file:
    path: /etc/apt/keyrings
    state: directory
    mode: '0755'

- name: Download and store the official Docker GPG key
  apt_key:
    url: https://download.docker.com/linux/ubuntu/gpg
    keyring: /etc/apt/keyrings/docker.gpg
    state: present

- name: Add the official Docker repository to the APT system
  apt_repository:
    repo: >
      deb [arch={{ ansible_architecture }}
      signed-by=/etc/apt/keyrings/docker.gpg]
      https://download.docker.com/linux/ubuntu
      {{ ansible_distribution_release }} stable      
    filename: docker
    state: present

- name: Install the specific Docker Engine version (Ubuntu)
  apt:
    name:
      - "docker-ce={{ docker_apt_version }}"
      - "docker-ce-cli={{ docker_apt_version }}"
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
    update_cache: true

3. CentOS/RedHat Installation Path (tasks/install_redhat.yml) #

For CentOS or RedHat, the process involves removing old packages, adding the official YUM repository using the yum_repository or get_url module, installing YUM dependencies, and installing Docker Engine with the appropriate DNF/YUM version.

# roles/docker/tasks/install_redhat.yml
---
- name: Remove old Docker versions (CentOS/RedHat)
  dnf:
    name:
      - docker
      - docker-client
      - docker-client-latest
      - docker-common
      - docker-latest
      - docker-latest-logrotate
      - docker-logrotate
      - docker-engine
    state: absent

- name: Install YUM support packages
  dnf:
    name:
      - yum-utils
      - device-mapper-persistent-data
      - lvm2
      - python3-pip
    state: present

- name: Add the official Docker YUM repository
  get_url:
    url: https://download.docker.com/linux/centos/docker-ce.repo
    dest: /etc/yum.repos.d/docker-ce.repo
    owner: root
    group: root
    mode: '0644'

- name: Install the specific Docker Engine version (CentOS/RedHat)
  dnf:
    name:
      - "docker-ce-{{ docker_yum_version }}"
      - "docker-ce-cli-{{ docker_yum_version }}"
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
    update_cache: true

4. Version Variable Definitions (defaults/main.yml) #

To maintain stability, we must pin the Docker version. Don’t let Ansible randomly download the latest version that could potentially break our application compatibility.

# roles/docker/defaults/main.yml
---
# Setting the specific version for Debian/Ubuntu
docker_apt_version: "5:24.0.7-1~ubuntu.22.04~jammy"

# Setting the specific version for CentOS/RHEL
docker_yum_version: "3:24.0.7-1.el9"

# Default daemon configuration
docker_daemon_log_driver: "json-file"
docker_daemon_log_max_size: "50m"
docker_daemon_log_max_file: "3"
docker_daemon_storage_driver: "overlay2"

# Optional registry mirrors
docker_registry_mirrors: []

# Users to be added to the docker group
docker_users: []

Docker Daemon Optimization via daemon.json #

The /etc/docker/daemon.json file is the control center of Docker Engine behavior. This is where we define how containers are isolated, how logs are stored, which registry mirrors are used, and how Docker processes host resources.

Vital daemon.json Configuration Options #

Here are the crucial options we must apply for production needs:

• live-restore: Set to true. This feature is crucial because it allows our containers to keep running smoothly even while the Docker daemon is being restarted or updated. This guarantees server maintenance without application downtime (zero-downtime maintenance).
• userland-proxy: Set to false. By default, Docker uses a userland proxy to redirect network traffic to container ports. Disabling this option makes network traffic handled directly by host iptables rules, significantly improving network performance and saving host memory overhead.
• log-driver and log-opts: We must control log file sizes. Without limits, container log files grow endlessly until storage runs out. We use the json-file driver with a maximum size limit per file and a number of rotation files.
• storage-driver: The industry-standard storage driver that’s very efficient for Docker on modern Linux is overlay2.

daemon.json Deployment Task with JSON Validation #

When deploying JSON files, character typos like missing commas or misplaced curly braces will make the Docker daemon fail to start when restarted. To prevent this, we use the validate feature in Ansible’s template module. We leverage the built-in Python interpreter to check JSON syntax before the file is written to the destination.

# roles/docker/tasks/configure.yml
---
- name: Ensure the Docker configuration directory exists
  file:
    path: /etc/docker
    state: directory
    owner: root
    group: root
    mode: '0755'

- name: Deploy the daemon.json configuration file from a Jinja2 template
  template:
    src: daemon.json.j2
    dest: /etc/docker/daemon.json
    owner: root
    group: root
    mode: '0644'
    # Validate JSON syntax using the Python interpreter before saving
    validate: "python3 -c 'import json; json.load(open(\"%s\"))'"
  notify: Restart Docker Daemon

Jinja2 daemon.json Template (templates/daemon.json.j2) #

This template dynamically composes the JSON configuration based on the variables we’ve set in the role defaults or host group variables:

{# roles/docker/templates/daemon.json.j2 #}
{
  "log-driver": "{{ docker_daemon_log_driver }}",
  "log-opts": {
    "max-size": "{{ docker_daemon_log_max_size }}",
    "max-file": "{{ docker_daemon_log_max_file }}"
  },
  "storage-driver": "{{ docker_daemon_storage_driver }}",
  "live-restore": true,
  "userland-proxy": false,
  "iptables": true,
  "exec-opts": ["native.cgroupdriver=systemd"]
{% if docker_registry_mirrors | length > 0 %}
  ,
  "registry-mirrors": {{ docker_registry_mirrors | to_json }}
{% endif %}
}

We also add a handler to safely restart Docker if the configuration file changes:

# roles/docker/handlers/main.yml
---
- name: Restart Docker Daemon
  systemd:
    name: docker
    state: restarted
    daemon_reload: true

Storage Driver and File System Management #

Docker needs an efficient way to manage image layers and the read-write container filesystems. The storage driver choice has a big impact on container I/O read-write operation speed and host memory usage.

Why overlay2? #

In the past, Docker used drivers like aufs, devicemapper, or overlay. In today’s modern Linux environments, the overlay2 driver is the industry standard. Its main advantages include:

• Inode Efficiency: overlay2 doesn’t consume large numbers of inodes like its predecessors.
• I/O Performance: Uses native Linux kernel features to merge directories (union mounts), minimizing system overhead.
• Low Memory Usage: Containers sharing the same image layers can share kernel page cache memory pages, reducing host RAM load.

Host File System Requirements (Backing File System) #

For the overlay2 driver to work optimally and stably, the host’s backing file system (the file system where the /var/lib/docker directory lives) is recommended to be ext4 or xfs with the ftype=1 option active.

We can use Ansible to check the host file system type and ensure the /var/lib/docker directory is mounted on disk with the right parameters:

- name: Check the backing filesystem of the Docker directory
  command: stat -f -c %T /var/lib/docker
  register: docker_fs_check
  changed_when: false
  failed_when: false

- name: Warn if the filesystem isn't supported for overlay2
  debug:
    msg: "WARNING: Filesystem {{ docker_fs_check.stdout }} detected. ext4 or xfs is recommended for overlay2 stability."
  when: docker_fs_check.stdout not in ['ext4', 'xfs']

Logging System and Container Log Rotation #

Every line of text our application sends to stdout or stderr is captured by Docker and stored in a JSON-format file on the host. In busy production environments, the log volume generated by containers can reach gigabytes per day. If we leave the default configuration unlimited, the server disk will fill up and trigger a full OS crash.

There are two main ways to manage container log sizes: setting a maximum limit on the Docker daemon, or using the built-in Linux logrotate system.

Logging Strategy Comparison #

Strategy 1: daemon.json Limits (Recommended)
  - Handled directly by the Docker daemon.
  - Logs are immediately truncated when hitting the size limit (e.g. 50MB).
  - Very clean and requires no external OS dependencies.

Strategy 2: Host Logrotate
  - Uses the Linux OS's logrotate cron utility.
  - Works outside the Docker daemon lifecycle.
  - Requires special handling because actively written log files can hit file descriptor constraints if not truncated with the copytruncate option.

Logrotate Implementation Using Ansible #

As an additional mitigation step beyond Docker daemon limits, we can deploy a logrotate configuration for Docker container log files. The copytruncate option is very important so logrotate duplicates the original log file’s contents then empties it in place, instead of moving the file directly, because the Docker daemon continuously holds that log file descriptor open.

# roles/docker/tasks/maintenance.yml (Logrotate Section)
- name: Deploy the logrotate configuration for Docker container logs
  template:
    src: docker-logrotate.j2
    dest: /etc/logrotate.d/docker-containers
    owner: root
    group: root
    mode: '0644'

Here’s the Jinja2 template for the logrotate file:

{# roles/docker/templates/docker-logrotate.j2 #}
/var/lib/docker/containers/*/*.log {
    rotate 7
    daily
    compress
    size 50M
    missingok
    delaycompress
    copytruncate
    notifempty
}

The options above have the following functions:

• rotate 7: Keeps a maximum of 7 old log archive files before starting to delete them.
• daily: Performs rotation daily.
• compress: Compresses old log files using gzip to save disk space.
• size 50M: Immediately rotates if a log file reaches 50 Megabytes, without waiting for the daily schedule.
• copytruncate: Copies the active log file then empties it in place, preventing Docker from losing its log write target.

Automatic Garbage Collection for Containers, Images, and Volumes #

On development or production servers undergoing periodic deployments (for example through CI/CD pipelines), our servers accumulate a lot of digital “garbage”. This includes:

• Containers that are dead or exited (exited containers).
• Orphaned volumes not connected to any container.
• Docker networks no longer in use.
• Old untagged images left behind after we build or pull new images (dangling images).

Ansible has a dedicated module for handling this cleanup efficiently, community.docker.docker_prune. We can compose tasks to run this cleanup periodically, for example weekly or daily, using the cron system.

Pruning Tasks with community.docker.docker_prune #

Here are the Ansible tasks for cleaning up unused Docker resources:

# roles/docker/tasks/maintenance.yml (Garbage Collection Section)
---
- name: Remove stopped containers
  community.docker.docker_prune:
    containers: true
    containers_filters:
      until: "24h" # Remove containers dead for more than 24 hours
  register: prune_containers_result

- name: Remove dangling images (untagged) to save disk
  community.docker.docker_prune:
    images: true
    images_filters:
      dangling: true
  register: prune_images_result

- name: Remove orphaned volumes not connected to any container
  community.docker.docker_prune:
    volumes: true
  register: prune_volumes_result

- name: Display the disk space statistics successfully freed
  debug:
    msg: 
      - "Containers cleaned: {{ prune_containers_result.space_reclaimed | default(0) | filesizeformat }}"
      - "Images cleaned: {{ prune_images_result.space_reclaimed | default(0) | filesizeformat }}"
      - "Volumes cleaned: {{ prune_volumes_result.space_reclaimed | default(0) | filesizeformat }}"

Configuring GC Automation with a Cron Job #

So this cleanup runs automatically without manual intervention later, we create a cleanup shell script and register it as a weekly cron job using the cron module in Ansible.

- name: Create the Docker cleanup script on the host
  copy:
    dest: /usr/local/bin/docker-cleanup.sh
    owner: root
    group: root
    mode: '0755'
    content: |
      #!/bin/bash
      # Automatic cleanup for Docker resources
      docker system prune -af --volumes --filter "until=168h"      

- name: Create a cron job to run the cleanup every Sunday at 03.00
  cron:
    name: "Docker Garbage Collection"
    minute: "0"
    hour: "3"
    weekday: "0"
    job: "/usr/local/bin/docker-cleanup.sh > /dev/null 2>&1"

User Management and Rootless Security Aspects #

Running Docker commands requires high administrative access. By default, to interact with the Docker daemon socket (/var/run/docker.sock), we must be the root user or use the sudo command.

The Security Danger of the docker Group #

A very common pattern developers use is adding regular users to the Unix group named docker. This way, those users can run Docker commands without typing sudo. However, this pattern carries a very high security risk.

# ANTI-PATTERN: Adding any user to the docker group without realizing the consequences
- name: Add the developer user to the docker group
  user:
    name: intern_developer
    groups: docker
    append: true

Why is this dangerous? Users in the docker group effectively have root-equivalent authority on the host. An attacker who takes over that user’s account can easily escalate privileges to host root by running a container that mounts the host’s root filesystem (/):

# Simple exploit by a user in the docker group to read host secret files
docker run -v /:/host-root -it alpine cat /host-root/etc/shadow

Practical Solution: Restrict Access and Audit Playbooks #

To secure the host, we must restrict users entering the docker group to only verified deployment service accounts (like the ansible-deploy user or our CI/CD agents) and keep the Docker socket safe.

Here’s a playbook to manage users with strict authorization:

# roles/docker/tasks/users.yml
---
- name: Ensure the docker group exists on the system
  group:
    name: docker
    state: present

- name: Add specific deployment users to the docker group
  user:
    name: "{{ item }}"
    groups: docker
    append: true
  loop: "{{ docker_users }}"
  when: docker_users is defined and (docker_users | length > 0)

For extra production-level protection, we’re advised to implement Rootless Docker. Conceptually, the Docker daemon and containers run entirely inside a user namespace without needing root access at all. We can use Ansible to prepare the rootless Docker dependencies:

- name: Install the support packages for Rootless Docker
  apt:
    name:
      - uidmap
      - dbus-user-session
    state: present
  when: ansible_os_family == "Debian"

Installation Verification #

After all configuration is deployed and the Docker service enabled, we must make sure everything works correctly before marking the host ready for use. This verification process includes ensuring the service is actively running, checking basic functionality by running a lightweight test container, and validating that the installed Docker version matches our expected configuration.

# roles/docker/tasks/verify.yml
---
- name: Ensure the docker service is enabled and running
  systemd:
    name: docker
    state: started
    enabled: true

- name: Test run the hello-world test container
  community.docker.docker_container:
    name: test-hello-world
    image: hello-world:latest
    state: started
  register: test_run
  failed_when: test_run is failed

- name: Remove the hello-world test container
  community.docker.docker_container:
    name: test-hello-world
    state: absent

- name: Fetch the installed Docker version information
  command: docker version --format "{{ '{{' }}.Server.Version{{ '}}' }}"
  register: docker_installed_version
  changed_when: false

- name: Display the host verification status report
  debug:
    msg: 
      - "Operating System: {{ ansible_distribution }} {{ ansible_distribution_version }}"
      - "Docker Engine Active: {{ test_run.container is defined }}"
      - "Installed Docker Version: {{ docker_installed_version.stdout }}"

By running this automatic verification step, we can immediately detect if there are anomalies or installation failures from the start of the playbook run, avoiding unpleasant surprises when the deployment team starts deploying their real applications to this new host.

Summary #

• Use the Official Repository — Avoid installing Docker from the distro’s built-in OS packages. Use Docker’s official repository and pin the version so all servers are uniform.
• Use live-restore — Enabling live-restore: true in /etc/docker/daemon.json is crucial for keeping containers running while the Docker daemon is restarted.
• Disable userland-proxy — Setting userland-proxy: false moves the network forwarding load directly to the kernel iptables, saving RAM and improving network I/O performance.
• Log Rotation Is Mandatory — Always limit Docker logging capacity using the log-driver and log-opts options in the daemon.json file so server storage isn’t threatened by container log files filling up.
• Routine Garbage Collection — Apply garbage resource cleanup automation with the docker_prune module or scheduled cron job scripts to periodically remove dangling images and unused volumes.
• Privilege Security — Membership in the docker group equals host root access rights. Restrict group membership to only trusted deployment automation accounts.

← Previous: Common Mistake Next: Deploy Container →