CI/CD Anti Pattern #

Integrating infrastructure automation with Continuous Integration and Continuous Delivery (CI/CD) pipelines is the peak of operational maturity for DevOps teams. By combining Ansible with tools like GitLab CI, GitHub Actions, or Jenkins, we can fully realize the Infrastructure as Code (IaC) model, where every server configuration change can be automatically deployed as soon as code is merged into the main repository. However, automating deployments without proper guardrails can actually increase the risk of system failures.

A poorly designed CI/CD pipeline acts like an error amplifier. If a slow manual deployment process gives us a chance to detect errors mid-way, an automated pipeline without validation will spread configuration errors or security bugs across all production servers within seconds. Without clear environment separation, automatic syntax validation, isolated testing, and reliable failure handling mechanisms, we’re just building a highway for incident propagation. In this article, we’ll deeply review various anti-patterns in integrating Ansible with CI/CD and learn how to overcome them.

1. Deploying Changes Directly to Production Without Going Through Staging #

One of the most common CI/CD workflow design mistakes is triggering a direct deployment to the production environment as soon as there’s a new commit on the main branch (like main or master), without passing through a verification process in an intermediate (staging) environment.

Why Is This Dangerous? #

Every new playbook code or template configuration risks carrying bugs, like variable name typos, conflicting package dependencies, or incompatible system configuration changes. If these changes are directly applied to production servers without first being tested in a similar staging environment, user services can be immediately crippled. We lose the last chance to do functional testing and smoke testing on a real running system.

# ANTI-PATTERN: A pipeline deploying directly to production without safeguards
# File: .github/workflows/deploy.yml
on:
  push:
    branches:
      - main  # Triggers execution as soon as something merges to the main branch

jobs:
  deploy-production:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

      - name: Run the deployment directly to the main servers
        run: |
          ansible-playbook -i inventory/production/site.yml site.yml
          # ✗ DON'T: No staging environment, no smoke tests, directly touching production!

The solution to this anti-pattern is building a multi-stage pipeline. Changes must first be automatically deployed to the staging environment. After the staging deploy succeeds, we must run automated test scripts (smoke tests). Finally, to release to production, we must apply a manual approval gate mechanism to ensure the team has full control over release timing.

# CORRECT: A structured pipeline flow with Staging, Smoke Tests, and Approval
# File: .github/workflows/deploy.yml
on:
  push:
    branches:
      - main

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

      - name: Deploy to the Staging Environment
        run: |
          ansible-playbook -i inventory/staging/site.yml site.yml          

  verify-staging:
    runs-on: ubuntu-latest
    needs: deploy-staging
    steps:
      - name: Run the Smoke Test on Staging
        run: |
          # Query the application health endpoint on staging
          curl -f --retry 5 --retry-delay 5 https://staging-app.company.internal/health
          # ✓ CORRECT: Ensures the application starts and responds properly before moving further          

  deploy-production:
    runs-on: ubuntu-latest
    needs: verify-staging
    environment:
      name: production  # ✓ CORRECT: Enables the Manual Approval option in GitHub Actions settings
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

      - name: Deploy to the Production Environment
        run: |
          ansible-playbook -i inventory/production/site.yml site.yml

Here’s a visualization of the safe pipeline workflow from commit to production release:

flowchart TD
    Trigger["Developer Pushes / Merges to the main branch"] --> Validate["Linting & Syntax Check: ansible-lint"]
    Validate -- "Failed" --> NotifyFail["Send Failure Notification & Cancel the Pipeline"]
    Validate -- "Success" --> Molecule["Molecule Test (Local Testing/Ephemeral VM)"]
    Molecule -- "Failed" --> NotifyFail
    Molecule -- "Success" --> DeployStaging["Deploy to the Staging Environment"]
    DeployStaging --> SmokeTest["Smoke Test & Integration Test on Staging"]
    SmokeTest -- "Failed" --> RollbackStaging["Rollback Staging & Investigate Logs"]
    RollbackStaging --> NotifyFail
    SmokeTest -- "Success" --> ManualApproval{"Manual Approval (Prod Gate)?"}
    ManualApproval -- "Rejected / Timeout" --> Cancel["Cancel the Production Release"]
    ManualApproval -- "Approved" --> DeployProd["Deploy to the Production Environment (Rolling Update)"]
    DeployProd --> PostCheck["Post-Deployment Health Check in Production"]
    PostCheck -- "Failed" --> RollbackProd["Automatic Rollback in Production"]
    RollbackProd --> NotifyFail
    PostCheck -- "Success" --> Complete["Deployment Finished & Successful!"]

2. Hardcoding Credentials in Pipeline Definitions #

To run Ansible playbooks on remote targets from CI/CD runners, we need various credentials, like SSH private keys, Ansible Vault encryption passwords, API tokens for dynamic inventory modules, or cloud provider credentials.

Why Is This Dangerous? #

Writing these credential values plaintext inside pipeline definition files (like .gitlab-ci.yml or .github/workflows/deploy.yml) is a very serious security violation. Anyone with read access to the repository can see those credentials. Additionally, those credentials are permanently recorded in Git history logs. Another fatal mistake is letting credentials print in CI/CD runner console logs because we forgot to hide the temporary file writing command output.

# ANTI-PATTERN: Writing sensitive credentials plaintext in pipeline files
# File: .gitlab-ci.yml
deploy-job:
  stage: deploy
  script:
    - echo "my-super-secret-vault-pass" > /tmp/.vault_pass  # ✗ DON'T: Vault password exposed in Git!
    - ansible-playbook site.yml --vault-password-file /tmp/.vault_pass \
        -e "ssh_key_content=-----BEGIN RSA PRIVATE KEY-----..."  # ✗ DON'T: SSH key passed as an inline string

To solve this, we must use the secret variable features (Secret Variables / Masked Variables) provided by our CI/CD platform. Credentials must not be written in the repository, but stored in the CI/CD admin configuration panel, set as Masked (so they’re automatically censored as stars *** if accidentally printed in logs), and accessed in the pipeline as environment variables.

# CORRECT: Using Masked CI/CD Variables and securely cleaning up credential files
# File: .gitlab-ci.yml
variables:
  # The options below reference secret variables configured in the GitLab panel
  # VAULT_PASSWORD_PROD is set as the "Variable" type (Masked + Protected)
  # PROD_SSH_PRIVATE_KEY is set as the "File" type (Protected)

deploy-job:
  stage: deploy
  script:
    # Create the vault password file from the secret variable with strict permissions
    - echo "$VAULT_PASSWORD_PROD" > "${CI_PROJECT_DIR}/.vault_pass"
    - chmod 600 "${CI_PROJECT_DIR}/.vault_pass"
    
    # Secure the permissions of the SSH private key file passed as a File-type variable
    - chmod 600 "$PROD_SSH_PRIVATE_KEY"
    
    # Run the playbook with secure credential file references
    - ansible-playbook -i inventory/production/site.yml site.yml \
        --vault-password-file "${CI_PROJECT_DIR}/.vault_pass" \
        --private-key "$PROD_SSH_PRIVATE_KEY"
        
  after_script:
    # ✓ CORRECT: Always remove the local vault password file from the runner after execution
    # to prevent leftover data leaks on shared runners
    - rm -f "${CI_PROJECT_DIR}/.vault_pass"

3. Designing Pipelines That Aren’t Safe to Retry (Low Idempotency) #

CI/CD runners can fail mid-execution for various unexpected reasons, like network connection drops, running out of runner memory resources, or crashes on target host systems during the deployment process.

Why Is This Dangerous? #

If our Ansible playbooks aren’t designed to be idempotent (i.e. safe to run repeatedly with the same input), then pressing the Retry button on a failed pipeline can worsen system damage. For example, a task adding a new configuration line using the shell module with a plain append command (echo "config" >> file.conf) will duplicate that line every time the pipeline is re-run. This corrupts the configuration file and the application fails to reload data.

# ANTI-PATTERN: A task that isn't idempotent and is dangerous to re-run
- name: Set Up the Web Application Configuration
  hosts: appservers
  tasks:
    - name: Download the new source code version (delete the old folder first)
      shell: "rm -rf /var/www/html/* && wget -O /tmp/src.tar.gz https://api.com/src.tar.gz && tar -xzf /tmp/src.tar.gz -C /var/www/html"
      # ✗ DON'T: This task isn't idempotent. If the wget connection drops mid-way,
      # the remote server is left with an empty /var/www/html folder (total downtime)!

To guarantee re-execution safety, we must make sure every task in our playbook is idempotent — only making changes if the current system state doesn’t match the declared target. Additionally, we must use error handling blocks (block and rescue) to do automatic rollbacks if a failure happens mid-way.

# CORRECT: Using idempotent declarative modules and implementing automatic rollback
- name: Set Up the Web Application Configuration Safely and Idempotently
  hosts: appservers
  serial: 2                    # Run the deployment on at most 2 servers at once
  max_fail_percentage: 0       # Immediately cancel the whole play if even 1 server fails
  tasks:
    - block:
        - name: Download the new source code version safely
          git:
            repo: "https://git.company.internal/app/web.git"
            dest: "/var/www/html"
            version: "{{ deploy_version }}"
          # ✓ CORRECT: The git module is idempotent by default. It compares commit hashes
          # and only downloads data differences, without roughly deleting the target folder.

        - name: Run the local health check (smoke check)
          uri:
            url: "http://localhost:8080/health"
            status_code: 200
          retries: 5
          delay: 5
          register: health_check
          # Verify application health before continuing to the next server batch

      rescue:
        - name: Roll back to the stable version if the deployment errors
          git:
            repo: "https://git.company.internal/app/web.git"
            dest: "/var/www/html"
            version: "{{ rollback_version }}"
          # ✓ CORRECT: Automatically restores the code to the previous stable commit if the block above fails

        - name: Stop the pipeline with a clear error message
          fail:
            msg: "Deployment failed on host {{ inventory_hostname }}. Automatic rollback to version {{ rollback_version }} has been completed."

4. Ignoring Automated Testing in the Pipeline #

Many DevOps teams configure their CI/CD pipelines only as plain deployment command executors. As soon as there’s a code change, the pipeline directly runs the ansible-playbook command.

Why Is This Dangerous? #

Ignoring automated testing before deployment is like letting syntax errors or indentation errors sabotage our production servers. A trivial YAML indentation mistake can make Ansible misinterpret parameters and break execution logic. Additionally, without automated tests validating coding style compliance, our playbooks will accumulate technical debt that complicates team collaboration.

We must include a test stage at the start of the pipeline before the playbook is allowed to touch any environment. This stage must at minimum do syntax checks (--syntax-check) and static analysis with ansible-lint. For more reliable advanced testing, we must integrate unit testing using Molecule.

# CORRECT: Running syntax validation and linting automatically at the start of the pipeline
# File: .github/workflows/deploy.yml
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.10'

      - name: Install Ansible and Ansible Lint
        run: |
          pip install ansible ansible-lint          

      - name: Run the Playbook Syntax Check
        run: |
          ansible-playbook site.yml --syntax-check -i inventory/staging/site.yml
          # ✓ CORRECT: Ensures the YAML file can be parsed properly by the Ansible engine          

      - name: Run the Static Code Analysis (Linter)
        run: |
          ansible-lint site.yml
          # ✓ CORRECT: Automatically checks code compliance with Ansible best practice rules

5. Excessive Pipeline Complexity Hard to Document #

In an effort to make the automation system very dynamic, it’s not uncommon for teams to build very complex pipeline architectures. They use dozens of interdependent jobs, convoluted conditional logic, dynamic file generators inside the pipeline, and layered sub-pipeline calls.

Why Is This Dangerous? #

Overly complicated, over-engineered pipelines become very hard for other team members to understand. When an execution failure occurs, debugging takes a very long time because we must trace the complex job dependency chain. Additionally, this complexity becomes scary technical debt — no team member dares to modify or update the pipeline configuration file because they’re afraid of breaking the whole poorly documented workflow.

The main principle in designing CI/CD pipelines for Ansible is simplicity and flow clarity. Every step in the pipeline must be understandable by a new engineer within 5 minutes without needing thick external documentation.

# CORRECT: A simple, clean, easy-to-understand pipeline flow
# Every job has a clear purpose, runs on explicit conditions,
# and doesn't have convoluted dependency chains.
# File: .github/workflows/deploy.yml
jobs:
  # Job 1: Static code testing (Always runs on Pull Requests)
  test-and-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run the linter
        run: pip install ansible-lint && ansible-lint site.yml

  # Job 2: Deploy to Staging (Only runs after merging to the main branch and Job 1 succeeds)
  deploy-staging:
    runs-on: ubuntu-latest
    needs: test-and-lint
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Staging
        run: ansible-playbook -i inventory/staging/site.yml site.yml

  # Job 3: Deploy to Production (Only runs after the Staging deploy succeeds & needs approval)
  deploy-production:
    runs-on: ubuntu-latest
    needs: deploy-staging
    environment: production  # Triggers manual approval in the production environment
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Production
        run: ansible-playbook -i inventory/production/site.yml site.yml

6. Disorganized Pipeline Artifact Storage Management #

Ansible produces various valuable execution outputs, like change report files (deployment-report.txt), security audit status, compiled configuration files, or temporary cache database files.

Why Is This Dangerous? #

There are two extreme mistakes in managing pipeline artifacts:

  1. Not storing artifacts at all: If the pipeline fails or problems occur after the deployment finishes, we have no detailed log records or system state at the time of failure to investigate. We lose important audit trails.
  2. Storing all artifacts forever: Permanently storing large build folders or complete execution logs without expiry limits makes CI/CD server storage or cloud bucket capacity balloon quickly. This produces unnecessary storage cost bills.

We must configure artifact storage with clear expiry limits (expire_in or retention-days). Store only the important files helping with debugging, and set a reasonable retention time (e.g. 7 to 30 days).

# CORRECT: Configuring artifact storage with proper retention
# File: .gitlab-ci.yml
build-assets:
  stage: build
  script:
    - mkdir -p build/
    - ansible-playbook -i inventory/staging/site.yml build-assets.yml
    - echo "Build finished at $(date)" > build/build-report.txt
  artifacts:
    paths:
      - build/
      - .fact_cache/
    expire_in: 14 days    # ✓ CORRECT: Artifacts are automatically deleted after 14 days to save storage
    when: always          # ✓ CORRECT: Store artifacts even if the job fails to ease debugging

Summary #

  • Mandate a Staging Test Path — Never let playbook changes be directly applied to production servers without first being validated in a structurally identical staging environment.
  • Secure Credentials via Secret Variables — Remove all forms of hardcoded credentials in Git repositories. Use the masked/file-type secret variables provided by the CI/CD platform, and clean up container files after use.
  • Apply the Idempotency Principle for the Retry Button — Make sure every task is safe to run repeatedly without damaging system state. Use rescue blocks to handle failures and trigger automatic rollbacks.
  • Integrate Automated Testing — Make syntax checks (--syntax-check) and static analysis with ansible-lint a main prerequisite that must pass before deployments are allowed to run.
  • Simplify the Pipeline Workflow — Avoid excessive complexity in pipeline definitions. Create linear, intuitive workflows quickly understandable by all DevOps team members.
  • Set Artifact Retention Limits — Configure expiry options on pipeline artifact storage (like expire_in: 14 days) so debugging logs stay available when needed without exhausting server storage quotas.

← Previous: Performance Anti Pattern Next: Project Structure →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact