Lessons Learned #

This final article in the practical Ansible guide series isn’t meant to discuss new syntax features or complex technical modules. Instead, this section is dedicated as a reflection and summary of the valuable lessons we gained directly from the field while operating large-scale infrastructure with Ansible. Many of the architectural decisions and operational patterns in official documentation feel very sensible on paper, but when faced with the reality of a production server incident at midnight, reality tells a different story.

The lessons summarized here were born from real failures, repeated configuration errors, and maintenance decisions proven to have both fatal and system-saving impacts. By studying this retrospective, we can avoid the common traps DevOps teams often face when adopting Infrastructure as Code (IaC) and build a mature, sustainable configuration management system.

1. Dry Run (–check) Is Not a Substitute for a Staging Environment #

The most fundamental mistake teams repeatedly make is relying on simulation or dry run mode (--check) as a guarantee that our playbook is 100% safe to run on production servers. We’re often tempted to think that if the --check execution shows no red errors in the terminal, we can directly apply it to production servers safely without going through an intermediate (staging) testing environment.

However, in reality, --check mode has significant technical limitations:

Runtime Conditions That --check Mode Fails to Detect:
1. Active Service Status: Ansible doesn't test whether services can actually respond to connections after being configured.
2. Storage Capacity: Full-disk failures right when new configuration file writing is in progress go undetected.
3. Race Conditions: Simultaneous resource contention between several async tasks isn't detected in simulation.
4. Library Version Differences: Minor system library version inconsistencies between the local environment and production systems.

Let’s look at the mindset comparison between the dangerous anti-pattern and the safe solution:

# ANTI-PATTERN: Relying on --check then deploying directly to production
# We assume local simulation sufficiently represents production conditions
ansible-playbook -i inventory/production site.yml --check --diff
# "The check result is clean, let's deploy straight to production!" -> Runtime disasters often happen here.

# CORRECT: Forcing the environment promotion flow from Staging to Production
# Step 1: Deploy and fully test functionality in the staging environment
ansible-playbook -i inventory/staging site.yml --diff

# Step 2: Run functionality tests (smoke tests) in staging
ansible-playbook -i inventory/staging playbooks/verify-post-change.yml

# Step 3: Deploy to production after staging proves stable
ansible-playbook -i inventory/production site.yml --diff

Key Lesson: --check mode is very useful for verifying syntax correctness and seeing whether there are unexpected file changes. However, it can never replace real functional verification in a representative staging environment.

2. Broken Idempotency Is a Time Bomb in Production #

Idempotency — a playbook’s ability to be run many times with always-consistent end results without making unnecessary changes — is the main foundation of Ansible reliability. Unfortunately, idempotency is often sacrificed for code writing speed. We often let our playbooks produce a changed status every time they run, with the excuse “the important thing is the task gets done”.

Broken idempotency almost always triggers hard-to-detect side effects that damage server stability later.

# ANTI-PATTERN: Using shell without idempotency validation
# Every time the playbook runs, configuration lines keep being added to the file
- name: Add the upstream DNS configuration
  shell: echo "nameserver 8.8.8.8" >> /etc/resolv.conf
  # Run 1: changed=1 (resolv.conf contains 1 nameserver line)
  # Run 2: changed=1 (resolv.conf contains 2 duplicate nameserver lines)
  # Run 100: resolv.conf is full of 100 identical duplicate lines!

# CORRECT: Using a declarative module guaranteeing idempotency by default
- name: Ensure the upstream DNS configuration is properly set
  lineinfile:
    path: /etc/resolv.conf
    line: "nameserver 8.8.8.8"
    state: present
  # Run 1: changed=1 (line added)
  # Run 2: changed=0 (no change because the line already exists)

Key Lesson: Always verify our playbook’s idempotency level by executing it twice in a row in the staging environment. The second run must produce a zero change status (changed=0). If any task still triggers changes on the second run, fix it immediately using built-in modules or add the right changed_when parameter.

3. Losing Vault Password Access Is an Operational Disaster #

Encrypting credentials with Ansible Vault is a best practice for keeping our data confidential in Git repositories. However, Vault key management itself often becomes the operational weak point. A classic problem we often encounter is the Vault key stored unstructured: only on one senior engineer’s laptop, saved in private chat messages, or placed in a local text file without backups.

When that engineer leaves the company or their laptop breaks, the team instantly loses access to all production secret files, crippling our deployment capability immediately.

Safe Vault Key Management Pattern:

flowchart LR
    A["Vault Key on a Local Laptop"] --> B["Synced to the Team Password Manager"]
    B --> C["Access Given to Min. 2 On-Call Members"]
    C --> D["CI/CD Secrets Integration"]

To avoid single-person dependency, we must apply the following practices:

  • Use a Company Password Manager: Store the Vault master password in a secure centralized credential storage (like 1Password, Vaultwarden, or AWS Secrets Manager) accessible by at least two engineers on the on-call rotation.
  • Key Integration via Environment Variables: In automated integration systems (CI/CD), avoid storing Vault password files on disk. Use secret injection mechanisms through environment variables:
    # Run the playbook by reading the vault password from a CI environment variable
ansible-playbook -i inventory/production site.yml \
  --vault-password-file <(echo "$ANSIBLE_VAULT_PASSWORD")

Key Lesson: The Vault password is the master key to our infrastructure gateway. Manage that key with the same level of security and availability discipline as other sensitive data.

4. The “We’ll Clean Up This Code Later” Myth #

Pressure to release features or solve incidents often forces us to cut corners in code writing. We create giant playbooks in a single file, hardcode server IP addresses, copy-paste task blocks between folders, or skip writing Molecule tests with a promise to ourselves: “We’ll clean it up later once this task is done”.

But in reality, free time for that refactoring almost never comes. That temporary code stays in our repository for years, piling up into enormous technical debt, until eventually no team member dares to touch it because of its complexity.

Here’s a visualization of technical debt accumulation from postponing refactoring:

flowchart TD
    subgraph Dependencies
        A["Fast Release Pressure"] --> B["Corner Cutting"]
        B --> C["Technical Debt Grows"]
        C --> D["Playbooks Become Monolithic & Fragile"]
        D --> E{"Applying the Boy Scout Rule?"}
        E -->|"No: Workload Grows"| A
        E -- "Yes" --> F["Gradual & Continuous Refactoring"]
        F --> G["Clean & Stable Ansible Codebase"]
    end

Key Lesson: We must apply the Boy Scout Rule in our infrastructure codebase: “Always leave code in a cleaner condition than when we first found it”. Don’t allow new Pull Requests in if they add new technical debt. Do small, gradual refactoring in every PR we create.

5. Communication Channels Matter More Than Isolated Technical Reliability #

An engineer often focuses too much on the beauty of their playbook architecture until forgetting the human element operating it. We could design a very sophisticated, downtime-free deployment automation system. But if the automation execution process runs mysteriously without other team members knowing, our system is judged a collaborative failure.

When deployments run without notifications, the support team will be confused finding the cause if error fluctuations appear on monitoring dashboards, and operations managers panic thinking there’s a cyber attack on the servers.

Therefore, we must integrate automatic status notifications on every important deployment cycle using our team’s communication channel webhooks:

# playbooks/roles/common/tasks/notify-slack.yml
---
- name: Send the deployment status notification to Slack
  community.general.slack:
    token: "{{ slack_webhook_token }}"
    channel: "#deploy-notifications"
    msg: |
      *Ansible Deployment Status*
      • Host: `{{ inventory_hostname }}`
      • Executor: `{{ lookup('env', 'USER') }}`
      • Status: *{{ deployment_status | default('SUCCESS') }}*
      • Note: System configuration changes have been applied.      
    color: "{{ 'good' if (deployment_status | default('SUCCESS') == 'SUCCESS') else 'danger' }}"
  delegate_to: localhost
  become: false
  ignore_errors: yes  # Don't fail the main deployment just because the Slack API is having issues

Key Lesson: Keep all stakeholders clearly informed about what’s being changed, when the change happens, and who is responsible for executing the change. Transparent communication can dampen team panic during incidents.

6. Test Restore Feasibility, Not Just Backups #

Many DevOps teams feel very safe because they’ve configured automatic backup scripts using Ansible cron jobs running smoothly every night to cloud storage. The backup dashboard shows green indicators meaning backup files successfully uploaded. However, this sense of safety is often false.

The real problem only surfaces when disaster strikes the main server:

  • The backup file turns out to be 0 KB because the database dumping process was interrupted midway.
  • The backup file encryption format has changed and no engineer knows how to decrypt it.
  • The restore procedure takes 12 hours, while our business RTO target is at most 1 hour.
✓ CORRECT: Regularly schedule a "Chaos Day" to test restore playbooks
✗ DON'T: Assume backups are valid just because the upload process succeeded

Key Lesson: A backup never tested for restore isn’t a valid backup. We must regularly hold disaster recovery drills at least once every three months. Use our recovery playbooks to restore backup data to a newly provisioned empty server and thoroughly verify application functionality.

7. Parallel Execution Scale Can Cripple the Network #

When we start managing hundreds of servers with Ansible, we face network scalability problems. By default, Ansible tries to run tasks in parallel on a number of hosts determined by the forks parameter in our configuration. If we run tasks requiring large bandwidth (like downloading 500 MB packages or cloning large Git repositories) to 500 servers simultaneously, we’ll cripple our own infrastructure.

That execution traffic can flood our network gateway, trigger rate limiting protection on external package repository servers, or exhaust the connection pool on central database servers.

To overcome this scalability constraint, we must get used to controlling task execution rates using the serial and throttle parameters:

# playbooks/deploy-app-scale.yml
---
- name: Deploy application updates on a large cluster
  hosts: webservers
  become: true
  # Run in stages: 10% of the total servers in the first stage,
  # then increase to the next 20% if there are no errors.
  serial:
    - "10%"
    - "20%"
    - "100%"

  tasks:
    - name: Download the new release package from the local repository
      get_url:
        url: "http://internal-nexus.company.internal/repository/myapp.tar.gz"
        dest: /tmp/myapp.tar.gz
      # Limit a maximum of 5 servers downloading simultaneously
      # to protect our internal Nexus server bandwidth
      throttle: 5

    - name: Restart the application service
      systemd:
        name: myapp
        state: restarted

Key Lesson: Always consider the impact of our playbook execution load on network and supporting infrastructure capacity when running at the scale of hundreds of servers. Use gradual release strategies (serial and throttle) to spread the execution load evenly.

8. Use Ansible as a State-Enforcement Tool, Not a Script Runner #

Many teams migrating from traditional Bash scripts to Ansible still keep the imperative mindset: “Write step-by-step instructions on how to make something”. As a result, their playbooks are full of shell and command modules executing raw commands in sequence. This is a mistaken way of using Ansible.

Ansible is designed as a declarative (state-enforcement) tool where we simply write the desired state we want on the system, and let Ansible figure out how to achieve that state.

Let’s compare the difference between these two mindsets:

# ANTI-PATTERN: Imperative mindset (using Ansible only as a script runner)
- name: Delete the old nginx configuration and create a new one
  shell: |
    rm -f /etc/nginx/sites-enabled/default
    echo "server { listen 80; }" > /etc/nginx/sites-enabled/myapp.conf
    systemctl restart nginx    
  # This approach is error-prone if folders don't exist, isn't idempotent, and breaks status logging.

# CORRECT: Declarative mindset (defining the desired system state)
- name: Ensure the default nginx configuration is disabled
  file:
    path: /etc/nginx/sites-enabled/default
    state: absent

- name: Ensure the application configuration is active using a template
  template:
    src: myapp.conf.j2
    dest: /etc/nginx/sites-enabled/myapp.conf
    mode: '0644'
  notify: Reload nginx

Key Lesson: Get rid of imperative scripts in our playbooks. Start thinking declaratively by leveraging the power of built-in Ansible modules that automatically handle idempotency, error handling, and target OS distribution differences safely.

Principles That Stand the Test of Time #

After going through various incident cycles and infrastructure architecture evolution, our team agrees on five main principles proven to always save our production systems:

  1. Simplicity Beats Elegance: A simple playbook readable and understandable by all junior team members is far more valuable than a complex playbook with branching loop logic only maintainable by its single author.
  2. Measure Before Optimizing: Don’t guess where our playbook execution bottlenecks are. Enable the ansible.posix.profile_tasks plugin in our ansible.cfg configuration to get accurate execution duration data from each task.
  3. Incremental Automation: Avoid the ambition of doing comprehensive all-at-once automation (big bang automation). Start by automating small, frequently repeated maintenance tasks consistently, secure them, then expand to the next components.
  4. Code Validity Equals Server Validity: An outdated, never-updated playbook is as bad as having no automation at all. Maintain our Ansible repository with code writing discipline standards equivalent to our business application code.
  5. Humans Above Tools: Ansible is just a work tool. Real success lies in how our team collaborates, communicates openly during incidents, and shares knowledge to grow together.

Summary #

  • Simulation Is Not Staging — Remember that --check mode can’t detect runtime system constraints. Always run deployment promotion tests to a representative staging environment before touching production.
  • Guarantee Idempotency — Avoid tasks triggering repeated change statuses. Consistently run double-run testing to guarantee zero change status at the end of execution.
  • Distribute Key Access — Eliminate single-person dependency on Vault password management. Secure team credentials in a centralized password manager accessible by at least two engineers.
  • Adopt the Boy Scout Rule — Pay off our technical debt early, gradually in every Pull Request. Don’t let messy temporary playbooks become long-term maintenance burdens.
  • Integrate Notifications — Use Slack or Discord webhooks to give the whole team visibility into deployment activity to speed up incident handling analysis.
  • Simulate Disaster Recovery — Don’t assume backup files are safe before their restore functionality is really tested on a clean server through periodic disaster simulations.
  • Control Parallelism — Protect internal network bandwidth capacity from mass execution traffic load spikes using serial and throttle limits.
  • Use the Declarative Approach — Leverage built-in Ansible modules to cleanly define our system’s desired state and avoid fragile custom imperative scripts.
← Previous: Production Readiness
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact