Testing Strategy #

Testing Infrastructure as Code has very different challenges compared to conventional software application testing. We’re not testing deterministic programming functions that take pure inputs and return pure outputs. Instead, we’re testing side effects on real operating systems and servers: Is the network port open? Is the database service running and able to accept connections? Is the server configuration safe from exploitation gaps? Running deployments without a mature testing strategy is a high-risk speculative action. We need a structured testing approach to ensure system stability without burdening the team with complicated test suite maintenance processes.

1. Understanding the Ansible Testing Pyramid #

To build an efficient testing strategy, we must adopt a testing pyramid model adapted for Ansible needs. The main principle is maximizing fast, cheap tests at the bottom, and limiting slow, expensive tests at the top.

Here’s a visualization of the Ansible testing pyramid structure we apply:

flowchart TD
    Integration["Integration<br/>(Few, slow, high cost, very close to production)"]
    Molecule["Molecule Test<br/>(Per-role functional testing in local containers/VMs)"]
    Syntax["Syntax Check<br/>(Playbook YAML file structure validation testing)"]
    Lint["ansible-lint<br/>(Many, very fast (seconds), run per commit)"]

    Integration --- Molecule
    Molecule --- Syntax
    Syntax --- Lint

With this structure, we filter out most writing errors, style non-compliance, and basic logic errors at the bottom layer (ansible-lint & syntax-check) within seconds. We only run time-consuming tests (like creating virtual machines in the cloud) when our code has proven to pass that initial selection.

2. Syntax Checking and Check Mode Simulation #

The first testing layer after linting is syntax verification and execution simulation without changing the real system state (dry-run).

A. Syntax Checking #

We can use Ansible’s built-in utilities to scan all playbooks and ensure there are no indentation errors or invalid module parameter calls:

# Run syntax checks on all playbooks in the playbooks/ directory
for playbook in playbooks/*.yml; do
  ansible-playbook "$playbook" --syntax-check
done

B. Check Mode and Diff Mode #

Before applying changes to production servers, we must simulate changes using the --check flag (for dry-run) and --diff (to visually compare configuration file changes).

# Run a deployment simulation to the staging environment
ansible-playbook -i inventory/staging/ playbooks/site.yml --check --diff

Important Check Mode Limitations: #

We must realize that --check has limitations. If our playbook has tasks depending on output from previous tasks that should have been created (e.g. downloading a binary file then extracting it), the second task fails in check mode because the binary file doesn’t actually exist on the system yet. To overcome this, we must set conditional options properly on dynamic tasks.

3. Standalone Role Testing Using Molecule #

Molecule is the de-facto standard for unit testing Ansible roles. Molecule lets us automatically create isolated test environments (using Docker or VMs), apply our role there, verify the results match expectations, then destroy the test environment.

Here’s the automated test lifecycle run by Molecule:

flowchart TD
    A["Start Molecule Test"] --> B["Create (Create Container/VM)"]
    B --> C["Prepare (Initial Host Setup)"]
    C --> D["Converge (Run the Playbook the First Time)"]
    D --> E["Idempotence (Run the Playbook the Second Time)"]
    E --> F{"Any changed?"}
    F -->|"Yes"| G["Failed: Idempotency Bug"]
    F -->|"No"| H["Verify (Run Assertion Tests)"]
    H --> I{"Did all asserts pass?"}
    I -->|"No"| J["Failed: Wrong System Verification"]
    I -->|"Yes"| K["Destroy (Delete Container/VM)"]
    K --> L["Success: Valid Role"]

A. Molecule File Structure #

Inside the role folder (e.g. roles/postgresql/), we create the following molecule folder structure:

roles/postgresql/
├── tasks/
│   └── main.yml
└── molecule/
    └── default/
        ├── molecule.yml    # Driver, platform, and verifier configuration
        ├── converge.yml    # Playbook to apply the role
        └── verify.yml      # Playbook to verify the role's side effects

B. molecule.yml Configuration File #

Here’s our standard molecule.yml configuration using the Docker driver:

# roles/postgresql/molecule/default/molecule.yml
---
dependency:
  name: galaxy
driver:
  name: docker
platforms:
  # Using an Ubuntu container with active systemd
  - name: test-postgres-ubuntu
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    command: ""
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
    cgroupns_mode: host
    privileged: true
    pre_build_image: true
provisioner:
  name: ansible
  config_options:
    defaults:
      callbacks_enabled: profile_tasks
verifier:
  name: ansible

C. verify.yml Verification File #

Don’t rely on the assumption that “if the playbook finishes without errors, the system is correct”. We must do concrete tests (assertions) on the target server to verify the system’s final state in the verify.yml file.

# roles/postgresql/molecule/default/verify.yml
# Deeply verifying PostgreSQL installation side effects
---
- name: Verify the PostgreSQL Installation and Configuration
  hosts: all
  gather_facts: true

  tasks:
    - name: Ensure the postgresql binary is installed on the system
      command: psql --version
      register: pg_version_check
      changed_when: false
      failed_when: pg_version_check.rc != 0

    - name: Get the postgresql service status from systemd
      systemd:
        name: postgresql
      register: pg_service_status

    - name: Verify the postgresql service is active and enabled in systemd
      assert:
        that:
          - pg_service_status.status.ActiveState == 'active'
          - pg_service_status.status.UnitFileState == 'enabled'
        fail_msg: "PostgreSQL is not active or not configured to auto-start!"

    - name: Check whether database port 5432 is open and listening for connections
      wait_for:
        port: 5432
        timeout: 5
        state: started

    - name: Test authentication and database query connection directly
      command: >
        psql -U postgres -c "SELECT 1;"        
      become: true
      become_user: postgres
      register: pg_query_test
      changed_when: false
      failed_when:
        - pg_query_test.rc != 0
        - "'1' not in pg_query_test.stdout"

We run these tests locally on a developer machine by typing the command:

# Run the full Molecule lifecycle from start to finish
molecule test

4. Guaranteeing Code Idempotency #

Idempotency is a fundamental principle in Ansible: running a playbook many times on the same server must produce an identical system state, and must not trigger new changes after the first run finishes (the second run must report changed=0 status).

Why Is Idempotency Often Broken? #

The main cause of broken idempotency is careless use of command or shell modules, because those modules always report a changed status by default even when there’s no real change on the server.

How to Fix Idempotency Bugs: #

# ANTI-PATTERN: Running a bash script execution without idempotency limits
- name: Download and run the database initialization script
  shell: /usr/local/bin/init-db.sh

# CORRECT: Setting changed_when or creates limits so the task is idempotent
- name: Run the database initialization script safely and idempotently
  shell: /usr/local/bin/init-db.sh
  register: init_db_result
  # The task only has 'changed' status if the script output contains the text 'database created'
  changed_when: "'database created' in init_db_result.stdout"
  # The task is fully skipped if the lock file /var/lib/mysql/db_created already exists
  creates: /var/lib/mysql/db_created

Molecule automatically detects this problem during the molecule test process. If on the second iteration (idempotence phase) any task reports a changed status, Molecule stops the test process and reports an idempotency test failure.

5. Integration Testing Strategies #

After verifying each role independently at the Molecule level, we need integration testing to ensure the entire chain of roles (load balancer, app server, and database) can work together forming one functional application stack.

We create a separate integration test file simulating an end-to-end application deployment.

Example integration test file tests/integration/test-stack.yml:

# tests/integration/test-stack.yml
# Ensuring the load balancer, web app, and database are perfectly integrated
---
- name: Application Stack Integration Testing Scenario
  hosts: test_servers
  become: true

  roles:
    - role: common
    - role: postgresql
    - role: nodejs
    - role: myapp
    - role: nginx

  post_tasks:
    - name: Wait for the application HTTP port initialization time in Nginx
      wait_for:
        port: 80
        timeout: 15
        state: started

    - name: Verify the web server and application integration (HTTP Status Code)
      uri:
        url: "http://localhost/"
        status_code: 200
        return_content: true
      register: http_response
      failed_when: "'Welcome to MyApp' not in http_response.content"

    - name: Verify the application successfully communicates with the database
      uri:
        url: "http://localhost/api/users"
        status_code: 200
        return_content: true
      register: api_response
      failed_when: 
        - api_response.status != 200
        - "'db_connection_status: connected' not in api_response.content"

    - name: Ensure system logs don't contain critical error messages
      shell: "grep -i 'error' /var/log/myapp/error.log || true"
      register: log_audit
      changed_when: false
      failed_when: log_audit.stdout_lines | length > 5

This integration test ensures our nginx configuration truly points to the right nodejs application port, and that the nodejs application successfully authenticates to the postgresql database prepared by the database role.

6. Choosing a Driver for the Test Environment (Docker vs Vagrant vs Cloud) #

Choosing the platform where we run automated tests is a crucial decision. We must balance execution speed and production similarity level.

Here’s a detailed comparison table to help us choose the right testing driver:

Our Recommended Flow: #

1. Use Docker on local developer machines to write new tasks and test idempotency because the feedback loop is very instant.
2. Use Vagrant/VM if our roles manipulate operating system kernels, modify disk partition tables, or configure complex network interfaces.
3. Use the Cloud Driver (EC2/GCP) on nightly pipeline builds to test server upgrade scenarios before major maintenance.

7. Determining What Should and Shouldn’t Be Tested #

Writing tests for every tiny parameter will exhaust our team maintaining the test suite every time there’s a minor configuration update. We must be pragmatic in determining testing focus.

WHAT WE MUST TEST:
  ✓ Service Availability: Is the port open and correctly responding to queries/requests.
  ✓ End-to-End Functionality: Does nginx successfully reverse-proxy to the backend application socket.
  ✓ Configuration Syntax Correctness: Are generated configuration files free of syntax errors (e.g. nginx -t passes).
  ✓ Idempotency: Does the second playbook run return changed=0 status.
  ✓ Edge Case Handling: Is the database still safe if the role is re-run on a server already containing old data.

WHAT WE DON'T NEED TO TEST:
  ✗ Internal Ansible Module Logic: Don't test whether the apt module successfully installs nginx (that's Ansible QA's job).
  ✗ Default Value Consistency: Don't test whether default variables hold their default values if we don't change them.
  ✗ Every Text File Line: Don't assert regexps to verify every line of template-generated configuration files.

Anti-Patterns in Testing Strategies #

Here are wrong testing patterns often found and we must avoid:

1. Letting Automated Tests Always Fail #

Leaving test status red (fail) in the repository and assuming “oh, that’s a built-in error, just ignore it” lowers team alertness.

# ANTI-PATTERN: Leaving tests failing in CI/CD
[Job Status: FAILED] - "Just ignore it, the Docker test server often has problems with cron jobs."

# CORRECT: Fix the error or cleanly disable the specific task in the test environment
- name: Configure the cron job for database backups
  cron:
    name: "db_backup"
    minute: "0"
    hour: "2"
    job: "/usr/local/bin/backup.sh"
  # Avoid failures in testing containers if the cron daemon isn't installed
  when: ansible_virtualization_type != 'docker'

2. Skipping Idempotency Verification #

Ignoring idempotency checks with the argument “the important thing is the app runs” causes server configuration to change uncontrollably on every run, risking operational stability.

Testing Strategy Checklist #

We use the following checklist to validate the readiness of our Ansible project’s testing strategy:

TESTING STANDARDS:
  □ Main playbook files are equipped with automatic syntax-check pipelines in the Git repository.
  □ Every internal custom role has at least one default Molecule testing scenario.
  □ Molecule testing uses base OS images matching our production target operating systems.
  □ Molecule testing includes idempotency testing (the second run returns changed=0).

ASSERTIONS (REAL VERIFICATION):
  □ The verify.yml file checks for listening network ports.
  □ The verify.yml file explicitly verifies systemd service activity (active/enabled).
  □ Real functionality testing (like SQL query execution or HTTP requests) is done on applications.
  □ Template-generated configuration files are validated for syntax correctness (e.g. apachectl configtest).

CI/CD PIPELINE MANAGEMENT:
  □ Linting and syntax check stages run automatically on every Pull Request.
  □ Test build results are automatically cleaned up (destroyed) on both success and failure.
  □ Functional test failures automatically block the merge process to the main branch.

Summary #

• Apply the Testing Pyramid — We must allocate the largest testing investment to linting (ansible-lint) and syntax checks because they provide the fastest feedback at the lowest cost.
• Mandate Unit Tests via Molecule — Every internal role must have a default Molecule scenario to isolate functional testing in local containers before trying staging servers.
• Guarantee Idempotency Status — Ensure all tasks in playbooks are idempotent. Use creates or changed_when parameters on command/shell modules to guarantee the second run produces changed=0.
• Focus on Real Side Effects — Test systems based on concrete functionality indicators (like active service status and HTTP 200 responses) instead of testing Ansible module internal syntax implementations.
• Separate Integration Testing Strategies — Use separate cross-role integration testing to verify the full application stack functionality (E2E).
• Choose Test Drivers Wisely — Use the Docker driver for fast daily developer test iteration, and choose VMs (Vagrant/Cloud) for tests modifying the OS kernel or network adapters.
• Avoid Over-Testing — Be pragmatic by avoiding excessive testing of built-in Ansible modules or line-by-line non-critical configuration files so the test suite maintenance burden doesn’t grow.

← Previous: Code Quality Next: Team Workflow →