Drift Handling #
In data center and cloud computing operations, keeping server conditions consistent over time is one of the biggest challenges. The state where target server configuration gradually deviates from the ideal state you defined is called configuration drift. Drift often happens due to manual emergency fixes (quick fixes) via SSH, uncoordinated automatic package updates, or undocumented ad-hoc script executions. Ansible is fundamentally designed with the idempotency principle to detect, prevent, and fix this drift automatically, guaranteeing stability and security compliance across your entire infrastructure.
Understanding the Configuration Drift and Idempotency Concepts #
Configuration drift is the main enemy of infrastructure scalability. Imagine you have a cluster of 50 identical web servers. One day, a sudden traffic spike happens, and an administrator makes a direct SSH connection to one server (web-12) to raise the worker_connections parameter on Nginx to handle the traffic load. After the problem is solved, the manual change is forgotten and never recorded into the infrastructure code repository.
A few months later, you do a global Nginx configuration update using an Ansible playbook. Because the web-12 server has a different manual configuration from the other 49 servers, the application on that server might behave inconsistently or crash due to an incompatible missed parameter. This is a classic example of the danger of configuration drift.
This is where the Idempotency concept plays an important role. Mathematically, an operation is idempotent if you can run it repeatedly in sequence and the final result stays the same as when you first ran it.
All built-in Ansible modules are designed with the idempotency principle in mind. When you run a playbook, Ansible doesn’t directly overwrite or reinstall everything. Each module first performs an inspection phase (state gathering) on the managed node:
- Reads the target server’s current actual state.
- Compares it with the ideal state you want in the playbook (desired state).
- If the actual state already matches the ideal state, Ansible reports
okstatus and makes no modifications. - If the actual state differs, only then does Ansible make changes and report
changedstatus.
flowchart TD
A["Desired State (Playbook/Code)"] --> B["Ansible Run 1: Apply Config"]
B --> C["Server in Perfect Sync"]
C --> D["Drift Event: Manual Change / Auto Update"]
D --> E["Server in Drifted State"]
E --> F["Ansible Run 2 with --check --diff"]
F --> G{"Drift Detected?"}
G -- "Yes" --> H["Remediation Run: Ansible applies desired state"]
G -- "No" --> I["No Action Needed (Status OK)"]
H --> CDetecting Drift with Check Mode and Diff #
To find out whether your servers have drifted from the base configuration you defined, you don’t need to run changes directly. Ansible provides two very powerful command-line parameters for system auditing:
--check(Check Mode / Dry-Run): Runs the playbook without making any physical changes on target servers. Modules report whether they would make changes (producingchangedstatus) if run normally.--diff(Show Differences): Displays detailed text line differences (like thegit diffcommand or unified diff format) between the actual configuration file on the server and the Ansible-rendered configuration file.
By combining both, you get an outstanding security and compliance audit tool:
# Running a thorough audit to detect drift across all target servers
ansible-playbook -i inventory/production site.yml --check --diff
Let’s look at an output illustration produced when Ansible detects an unauthorized manual modification to the /etc/ssh/sshd_config file on your production server:
TASK [Deploy the SSH hardening configuration] *******************************************
--- before: /etc/ssh/sshd_config
+++ after: /tmp/ansible-tmp-171861273-sshd_config
@@ -12,4 +12,4 @@
Port 22
-PermitRootLogin yes # ← Unauthorized manual modification by an admin via SSH
+PermitRootLogin no # ← The secure configuration that should be (desired state)
PasswordAuthentication no
From the diff output above, you immediately know that someone changed the PermitRootLogin security policy to yes manually on the target server. Ansible identifies this deviation without changing the server state during the check audit.
Scheduled Drift Detection #
To make sure your infrastructure stays free of deviations continuously, you shouldn’t only rely on manual checks. You must build an automatic periodic drift detection system (scheduled audit).
You can write a special playbook that evaluates the most critical parts of your servers (like /etc/ configuration files, sudoers user membership, and service statuses), then sends notifications to your team’s communication channels if deviations are detected.
Here’s an example drift audit playbook (playbooks/audit-drift.yml):
# playbooks/audit-drift.yml
---
- name: Periodic Configuration Drift Audit
hosts: production
gather_facts: true
vars:
critical_configs:
- { src: "templates/sshd_config.j2", dest: "/etc/ssh/sshd_config" }
- { src: "templates/nginx.conf.j2", dest: "/etc/nginx/nginx.conf" }
- { src: "templates/sudoers.j2", dest: "/etc/sudoers.d/deployer" }
tasks:
- name: Check the alignment of critical configuration files
template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
check_mode: true # Forces this task to ALWAYS run in check mode
diff: true # Make sure diff output is collected
register: config_audit
loop: "{{ critical_configs }}"
loop_control:
label: "{{ item.dest }}"
- name: Evaluate the audit results and collect findings
set_fact:
drifted_items: "{{ config_audit.results | selectattr('changed') | map(attribute='item.dest') | list }}"
- name: Display a warning if deviations occur
debug:
msg: |
[WARNING] Configuration drift detected on host {{ inventory_hostname }}!
The following files were modified outside of Ansible:
{{ drifted_items | to_nice_yaml }}
when: drifted_items | length > 0
In the playbook above, we use the check_mode: true option at the task level. This means even if you run this playbook normally (without the --check flag), the template testing task still runs in dry-run mode. This is very useful for automatic monitoring scripts running in the background.
Automating Drift Mitigation with AWX, Ansible Tower, and GitOps #
After you have a drift detection system, the next step is establishing a mitigation or remediation strategy. There are two main ways to do mitigation:
- Manual Remediation (Ad-hoc): An administrator reviews the audit diff output, and if the changes are safe to overwrite, the administrator re-runs the normal playbook without
--checkto return the server state to the desired state. - Automatic Remediation (Continuous Enforcement): An automatic system periodically forces server configurations back to the state defined in your git repository.
To automate this enforcement process, you can use orchestration platforms like AWX (the open-source version) or Red Hat Ansible Automation Platform (Ansible Tower). These platforms let you:
- Create execution schedules (Scheduler) to run remediation playbooks every hour or every day.
- Apply the GitOps paradigm, where every time there’s a code change on the
mainbranch of your Git repository, a Webhook triggers AWX to immediately deploy the latest configuration across the entire server cluster.
Here’s a flow diagram of how AWX/Tower manages drift mitigation continuously:
flowchart TD
Git["Git Repository"] -- "Webhook on Commit" --> AWX["AWX / Ansible Tower"]
Git --> DS["Desired State (v3)"]
AWX -- "Scheduled Job Run" --> RP["Run Playbook to Target"]
RP --> DS
RP --> MS["Managed Servers<br/>(Drift Overwritten & Enforced)"]If you don’t use AWX or Tower, you can apply a simple approach using the built-in ansible-pull tool. ansible-pull is a utility installed on managed nodes (via a cron job) that does a git pull of the playbook repository locally, then executes ansible-playbook against the server itself in a pull-based manner:
# Example cron job entry (/etc/cron.d/ansible-pull) on the managed node
# Run code synchronization from git and run the playbook every hour
0 * * * * root ansible-pull -U https://git.company.com/infra/playbooks.git site.yml -i localhost,
Long-Term Mitigation Strategy: Immutable Infrastructure #
Although Ansible is very reliable at fixing drift, the best long-term approach is minimizing the chance of drift occurring in the first place. You can do this by switching from the Mutable Infrastructure model (infrastructure modified in place) to Immutable Infrastructure (infrastructure never changed after being deployed).
In the Immutable Infrastructure model:
- You never do direct configuration updates on running servers.
- If there’s a configuration change or software update, you build a new server machine (virtual machine image or container) from scratch using tools like HashiCorp Packer combined with Ansible as its provisioner.
- The new server is deployed to the cluster, traffic is switched using a load balancer, and the old server is destroyed.
However, if your infrastructure isn’t ready for a full immutable model, you can apply the following mutability safeguard measures using Ansible:
1. Limit Direct SSH Access #
Configure the SSH daemon to only allow login through a Bastion Host and disable password access to restrict people from entering production servers ad-hoc.
2. Use Auditd to Monitor Critical Files #
Install the Linux security monitoring daemon auditd to record every time a process outside Ansible modifies main configuration files.
Let’s look at the Ansible task to configure auditd monitoring:
- name: Set up file monitoring using auditd
block:
- name: Ensure auditd is installed
package:
name: auditd
state: present
- name: Add monitoring rules for critical configuration files
blockinfile:
path: /etc/audit/rules.d/audit-drift.rules
create: true
mode: '0640'
block: |
# Watch for changes to the SSH configuration
-w /etc/ssh/sshd_config -p wa -k ssh_drift_detection
# Watch for changes to sudoers files
-w /etc/sudoers -p wa -k sudoers_drift_detection
-w /etc/sudoers.d/ -p wa -k sudoers_d_drift_detection
notify: Restart auditd
handlers:
- name: Restart auditd
service:
name: auditd
state: restarted
use_backend: systemd # Force the use of the systemd backend
The -w /etc/ssh/sshd_config -p wa -k ssh_drift_detection rule tells the Linux kernel to record every write (w) access and attribute change (a) on that file, tagging them with the ssh_drift_detection key in the /var/log/audit/audit.log log file.
Case Study: Implementing an Automatic Drift Audit System #
Let’s build an integrated drift audit scenario. We’ll create a playbook that checks web server configuration file integrity, collects findings, sends a status email, and if the force_remediation parameter is enabled by the administrator, directly overwrites the drifted files to restore system stability.
Here’s the content of our complete audit and remediation playbook (playbooks/enforce-drift.yml):
# playbooks/enforce-drift.yml
---
- name: Integrated Drift Detection and Remediation System
hosts: webservers
become: true
vars:
# Set to true to directly fix found drift
force_remediation: false
monitored_files:
- { src: "files/nginx/nginx.conf", dest: "/etc/nginx/nginx.conf", validate: "nginx -t -c %s" }
- { src: "files/nginx/vhost.conf", dest: "/etc/nginx/sites-available/default", validate: "nginx -t -c %s" }
tasks:
# 1. Detection Phase (Audit Run)
- name: Run the configuration file integrity audit
copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
validate: "{{ item.validate | default(omit) }}"
check_mode: true # Always run dry-run in this phase
diff: true
register: audit_results
loop: "{{ monitored_files }}"
loop_control:
label: "{{ item.dest }}"
# 2. Findings Evaluation Phase
- name: Identify files with manual deviations
set_fact:
drifted_files: "{{ audit_results.results | selectattr('changed') | map(attribute='item.dest') | list }}"
# 3. Reporting Phase
- name: Display the system audit status
debug:
msg: >
[AUDIT RESULT] Host {{ inventory_hostname }} {{ 'HAS DRIFT!' if drifted_files | length > 0 else 'IS FREE OF DRIFT.' }}
List of problematic files: {{ drifted_files }}
# 4. Conditional Remediation Phase (Only runs if force_remediation is set to true)
- name: Run automatic configuration recovery (Remediation)
copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
owner: root
group: root
mode: '0644'
backup: true # Save a backup of the drifted file before overwriting
validate: "{{ item.validate | default(omit) }}"
loop: "{{ monitored_files }}"
loop_control:
label: "{{ item.dest }}"
when:
- drifted_files | length > 0
- force_remediation | bool
notify: Reload Nginx
handlers:
- name: Reload Nginx
systemd:
name: nginx
state: reloaded
This integrated playbook provides very high operational flexibility for your infrastructure team. By default, when you run the daily monitoring job, you leave force_remediation: false. The playbook only acts as a watchdog, detecting illegal changes and reporting them.
However, if an incident occurs where many servers experience manual configuration chaos due to operational panic, the administrator just triggers the same job with the variable parameter -e "force_remediation=true". Ansible immediately takes over control, overwrites all manual deviations with the official configuration stored in your git repository, saves the broken versions as .bak backup files for forensic purposes, and smoothly reloads the web servers without downtime.
Summary #
- Drift Definition — Recognize configuration drift as the deviation of actual server conditions from the ideal state (desired state) due to manual intervention, external scripts, or automatic updates.
- Idempotency Principle — Understand that Ansible’s idempotent nature ensures tasks only take modification actions if the actual server state differs from the desired state.
- Audit Without Changing — Use the combination of
--checkand--diffparameters on theansible-playbookcommand line to see configuration differences before applying them.- Task-Level Audit — Leverage the
check_mode: trueoption at the task level to force audit tasks to always run in dry-run mode in the background.- AWX Automatic Remediation — Build periodic remediation schedules using AWX or Ansible Tower to keep servers consistent with your Git repository.
- ansible-pull Method — Use
ansible-pullas an alternative pull-based architecture running through local cron schedulers on managed nodes.- Long-Term Strategy — Reduce drift risk by implementing Immutable Infrastructure using Packer to create ready-to-use VM images.
- Kernel Auditd Monitoring — Install and configure the
auditdtool to detect and record every manual modification access on protected system configuration files.