CI/CD Integration #

Running Ansible locally from an administrator’s laptop is a good first step for experimentation. However, when working in large teams in production environments, executing playbooks manually from personal machines carries high risks like Ansible version differences, Python dependency library mismatches, and a lack of clear execution audit history. The best solution is integrating Ansible into continuous integration and delivery (CI/CD) pipelines. By moving execution control to standardized CI/CD runners, we can ensure every configuration change is first validated through automated testing processes, documented in the version control system, and deployed to production consistently through strict approval gates.

Designing a Safe Progressive Pipeline #

When designing a CI/CD workflow for infrastructure automation with Ansible, we must apply the progressive delivery principle. We must not let new playbook code be directly applied to production servers without going through a series of syntax tests, style compliance analysis (linting), functional testing in the staging environment (trial), and manual approval from senior reviewers.

Here’s the ideal process flow of a safe Ansible CI/CD pipeline:

flowchart TD
      A["Developer Pushes Code"] --> B["Validation stage: syntax check & ansible-lint"]
      B --> C{"Does validation pass?"}
      C -- "No" --> D["Fail the Pipeline & Send Alert"]
      C -- "Yes" --> E["Deploy to Staging Environment (Automatic)"]
      E --> F["Run Integration Tests in Staging"]
      F --> G{"Do staging tests pass?"}
      G -- "No" --> D
      G -- "Yes" --> H["Wait for Manual Approval Gate (Required Reviewer)"]
      H --> I{"Approved?"}
      I -- "Rejected" --> D
      I -- "Approved" --> J["Deploy to Production Environment"]
      J --> K["Run Production Smoke Tests"]
      K --> L["Done (Success)"]

Each stage in the diagram above has a crucial role in filtering out errors as early as possible before they impact real production systems. The static validation stage ensures there are no YAML typos or logic structure errors, while the staging environment acts as a trial replica to prove the correctness of our playbook implementation.

Integration with GitHub Actions #

GitHub Actions is one of the most popular CI/CD platforms that integrates directly with our Git repository. We can configure GitHub Actions workflows using YAML files in the .github/workflows/ directory.

The biggest challenge when running Ansible inside GitHub Actions is handling secure credentials like SSH private keys and Ansible Vault passwords. We must leverage GitHub’s built-in Encrypted Secrets feature and ensure that after execution completes, all secret keys that were copied to the runner are cleanly removed, even if the pipeline fails midway.

A Robust GitHub Actions Pipeline Playbook #

Here’s an example of a complete GitHub Actions workflow configuration for validation, automatic staging deployment, and production deployment with an approval gate:

# .github/workflows/deploy.yml
name: Ansible CI-CD Pipeline

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  validate:
    name: Static Playbook Validation
    runs-on: ubuntu-latest
    steps:
      - name: Fetch Source Code
        uses: actions/checkout@v4

      - name: Setup Python Runtime
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install Ansible and Linter
        run: |
          python -m pip install --upgrade pip
          pip install ansible ansible-lint          

      - name: Download Ansible Galaxy Dependencies
        run: |
          if [ -f requirements.yml ]; then
            ansible-galaxy install -r requirements.yml
          fi          

      - name: Test Playbook Syntax
        run: |
          ansible-playbook site.yml --syntax-check -i inventory/staging/          

      - name: Run Code Style Scan (Ansible Lint)
        run: |
          ansible-lint site.yml          

  deploy-staging:
    name: Deploy to the Staging Environment
    needs: validate
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment: staging
    steps:
      - name: Fetch Source Code
        uses: actions/checkout@v4

      - name: Setup Ansible
        run: |
          pip install ansible
          ansible-galaxy install -r requirements.yml          

      - name: Configure SSH Key for Staging Server Access
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.STAGING_SSH_PRIVATE_KEY }}" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          ssh-keyscan -H ${{ secrets.STAGING_HOST_IP }} >> ~/.ssh/known_hosts          

      - name: Write the Ansible Vault Encryption Key
        run: |
          echo "${{ secrets.STAGING_VAULT_PASSWORD }}" > .vault_pass
          chmod 600 .vault_pass          

      - name: Execute the Playbook to Staging
        run: |
          ansible-playbook -i inventory/staging/ site.yml \
            --vault-password-file .vault_pass \
            --private-key ~/.ssh/id_ed25519          

      - name: Clean Up Credentials (Cleanup)
        if: always()
        run: |
          rm -f ~/.ssh/id_ed25519 .vault_pass          

  deploy-production:
    name: Deploy to the Production Environment
    needs: deploy-staging
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment: production # Enabling Required Reviewers in GitHub Settings
    steps:
      - name: Fetch Source Code
        uses: actions/checkout@v4

      - name: Setup Ansible
        run: |
          pip install ansible
          ansible-galaxy install -r requirements.yml          

      - name: Configure SSH Key for Production Server Access
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.PROD_SSH_PRIVATE_KEY }}" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          ssh-keyscan -H ${{ secrets.PROD_HOST_IP }} >> ~/.ssh/known_hosts          

      - name: Write the Production Ansible Vault Encryption Key
        run: |
          echo "${{ secrets.PROD_VAULT_PASSWORD }}" > .vault_pass
          chmod 600 .vault_pass          

      - name: Execute the Playbook to Production
        run: |
          ansible-playbook -i inventory/production/ site.yml \
            --vault-password-file .vault_pass \
            --private-key ~/.ssh/id_ed25519          

      - name: Clean Up Production Credentials (Cleanup)
        if: always()
        run: |
          rm -f ~/.ssh/id_ed25519 .vault_pass          

Using the Approval Gate and GitHub Environments #

In the configuration above, on the deploy-production job, we include the line environment: production. This is a built-in GitHub Enterprise/Team feature that lets us lock specific environments.

When the pipeline finishes running the staging deployment stage, it pauses when entering the production stage. GitHub sends a notification to the designated review team (Required Reviewers). The pipeline only continues after reviewers give manual approval through the GitHub web interface. This is a crucial risk control technique so no accidental updates directly impact production systems.

Integration with GitLab CI #

GitLab CI is GitLab’s built-in integration system using distributed runner agents and configured through one centralized .gitlab-ci.yml file. The advantage of using GitLab CI is native support for using Docker containers as the execution environment and a more flexible SSH agent system.

GitLab CI Configuration with Caching and Reusable Templates #

Let’s compose a .gitlab-ci.yml configuration that leverages Python dependency caching techniques to minimize pipeline execution time, and uses YAML anchors (anchors) to share similar credential configuration tasks between jobs.

# .gitlab-ci.yml
stages:
  - validate
  - deploy-staging
  - deploy-production

variables:
  ANSIBLE_FORCE_COLOR: "true"
  ANSIBLE_HOST_KEY_CHECKING: "false" # Disable host key verification if servers are dynamic
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  key: ansible-deps-$CI_COMMIT_REF_SLUG
  paths:
    - .cache/pip
    - .ansible/roles

.prepare_ansible_env: &prepare_ansible_env
  before_script:
    - apt-get update -y && apt-get install -y openssh-client python3-pip
    - pip install ansible
    - if [ -f requirements.yml ]; then ansible-galaxy install -r requirements.yml; fi
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$VAULT_PASSWORD" > .vault_pass
    - chmod 600 .vault_pass
  after_script:
    - rm -f .vault_pass

lint-testing:
  stage: validate
  image: python:3.11-slim
  script:
    - pip install ansible ansible-lint
    - ansible-playbook site.yml --syntax-check -i inventory/staging/
    - ansible-lint site.yml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-to-staging:
  stage: deploy-staging
  image: python:3.11-slim
  <<: *prepare_ansible_env
  variables:
    SSH_PRIVATE_KEY: $STAGING_SSH_PRIVATE_KEY
    VAULT_PASSWORD: $STAGING_VAULT_PASSWORD
  script:
    - ansible-playbook -i inventory/staging/ site.yml \
        --vault-password-file .vault_pass
  environment:
    name: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-to-production:
  stage: deploy-production
  image: python:3.11-slim
  <<: *prepare_ansible_env
  variables:
    SSH_PRIVATE_KEY: $PROD_SSH_PRIVATE_KEY
    VAULT_PASSWORD: $PROD_VAULT_PASSWORD
  script:
    - ansible-playbook -i inventory/production/ site.yml \
        --vault-password-file .vault_pass
  environment:
    name: production
  when: manual # Requiring manual triggering from the GitLab UI (Approval Gate)
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Credential Security and Vault Password Handling #

In the GitLab CI implementation above, we use the &prepare_ansible_env anchor that encapsulates the Ansible installation tasks, SSH agent initialization (ssh-agent), encrypted private key addition (ssh-add), and the .vault_pass vault password container file creation.

In GitLab CI, we must set sensitive variables like $PROD_SSH_PRIVATE_KEY and $PROD_VAULT_PASSWORD in the Settings > CI/CD > Variables menu by checking the “Mask variable” and “Protect variable” options. The mask option ensures secret key values never leak into runner output logs even if a debug command accidentally displays them, while the protect option limits those variables’ availability to only protected Git branches like the main branch.

Execution Environment Management #

One classic problem in team-level infrastructure automation implementation is “works on my machine” (works on my laptop but fails on the CI/CD server). This happens because of version differences in Ansible Collections or Python libraries installed on the executing host. For example, Ansible’s Kubernetes modules require the kubernetes library installed on the host operating system. If the library version on the runner host differs from the developer’s laptop, the workflow errors out.

To solve this problem, the modern Ansible ecosystem introduced the Ansible Execution Environment (EE) concept. An Execution Environment is a ready-to-use container image that wraps all the runtime dependencies Ansible needs into one self-contained container.

How Ansible Builder Works #

We can create our own custom Execution Environment using a tool called ansible-builder. This tool uses a simple configuration file to define the base operating system, Ansible version, Galaxy collection list, and mandatory Python libraries.

Here’s an example configuration file for creating an Execution Environment:

# execution-environment.yml
version: 3
images:
  base_image:
    name: "registry.redhat.io/ansible-automation-platform-24/ee-minimal-rhel8:latest"

dependencies:
  galaxy:
    collections:
      - name: kubernetes.core
      - name: community.general
  python:
    - kubernetes>=28.0.0
    - pyyaml>=6.0
  system:
    - git-core
    - openssh-client

additional_build_steps:
  prepend_galaxy:
    - RUN pip3 install --upgrade pip

By running the ansible-builder build command, this tool generates a Dockerfile we can build (docker build) and upload to our company’s internal container registry. In our CI/CD pipeline on GitHub Actions or GitLab CI, we just specify that container image we built as the main execution image (for example image: myregistry.local/ansible-ee:v1.0), so we guarantee 100% runtime execution consistency from the testing stage through production.

SSH Key and Ansible Vault Handling Patterns #

To maintain ongoing credential security in CI/CD pipelines, here’s a comparison of credential management methods we can apply along with usage recommendations:

Summary #

• Design a Progressive Pipeline — Arrange pipeline stages gradually from static validation (syntax check and linting), automatic staging environment deployment, to smoke testing before stepping into production.
• Apply an Approval Gate — Use the Environments feature in GitHub or the when: manual option in GitLab CI to delay production deployment until manually approved by senior reviewers.
• Defensive Credential Cleanup — Always include secret file cleanup commands in cleanup blocks like if: always() in GitHub Actions to prevent private key leaks if a job fails.
• Restrict Key Access Rights — Use different SSH keys for staging and production environments, and limit those SSH execution permissions to only their respective target hosts.
• Secure Secret Variables — Check the “Mask” and “Protect” options on your platform’s CI/CD variable configuration to prevent secret keys from being accidentally exposed in runner execution logs.
• Use an Execution Environment — Build a custom container image containing Ansible and standardized dependency libraries to guarantee execution consistency and avoid “works on my machine” problems.

← Previous: Rolling Update Next: Scheduled Task →