Pipeline Design #

A good CI/CD pipeline isn’t just automation of steps previously done manually. It’s a system that gives fast feedback when there’s a problem, ensures only validated code reaches production, and makes deployments a boring event — not a tense moment. This article discusses pipeline design principles integrating Ansible as the deployment engine, and serves as the foundation for the next two articles discussing concrete implementations in GitHub Actions and GitLab CI.

Pipeline Design Principles #

These five principles must exist in every pipeline we design. Ignoring any of them will create problems that are more expensive to fix later:

1. Fail Fast
   Put the fastest checks at the start of the pipeline.
   Don't run a staging deployment if lint already failed.
   A slow pipeline makes developers lazy about waiting and eventually
   merge code that hasn't been properly tested.

2. Environment Promotion, not Re-Build
   The same artifact (Docker image, package) must be promoted
   from staging to production — not rebuilt for each environment.
   Rebuilding means the code reaching production
   isn't necessarily the same as what was tested in staging.

3. Idempotent at Every Stage
   Every stage must be safe to run repeatedly without side
   effects. Re-running a failed pipeline must not break state.
   Ansible is idempotent by design — leverage this.

4. Immutable Artifacts
   After the build, the artifact must not change.
   Tag Docker images with the git SHA or semantic version, not
   'latest'. The same image must be exactly the same in all envs.

5. Separation of Concern
   Build pipeline ≠ Deployment pipeline.
   Build produces artifacts; deploy distributes them.
   Mix both and we can't promote without rebuilding.

The “fail fast” principle doesn’t mean all steps must be parallel. What matters is execution order: cheap, fast steps (lint, syntax check) at the front; expensive, slow steps (integration tests, multi-node deploys) at the back. If a step can finish in 30 seconds and catch 80% of problems, it must be the first gate.

The Fail Fast Flow in a Pipeline #

stateDiagram-v2
    [*] --> Lint
    Lint --> UnitTest: lint pass
    Lint --> Failed: lint failed
    UnitTest --> Build: test pass
    UnitTest --> Failed: test failed
    Build --> Scan: image built
    Build --> Failed: build error
    Scan --> PushRegistry: scan clean
    Scan --> Failed: high vulnerability
    PushRegistry --> DeployStaging: immutable tag
    DeployStaging --> IntegrationTest: deployed
    IntegrationTest --> Gate: smoke test pass
    IntegrationTest --> RollbackStaging: test failed
    Gate --> DeployProduction: approved
    Gate --> Hold: waiting for approval
    Hold --> DeployProduction: reviewer approved
    Hold --> RollbackStaging: reviewer rejected
    DeployProduction --> VerifyProd: deployed
    VerifyProd --> [*]: success
    VerifyProd --> RollbackProd: health check failed
    RollbackStaging --> Failed
    RollbackProd --> Failed
    Failed --> [*]

The state diagram above shows the decisions made at every stage. Notice that every failure path (Failed) can be rolled back before it ever damages production. This is the essence of fail-fast: detect problems as close to their source as possible.

CI and CD Pipeline Anatomy #

Modern pipelines are usually divided into two big parts with different triggers, outputs, and lifecycles. Mixing them in one workflow is one of the most common sources of CI/CD pipeline problems.

flowchart LR
    subgraph CI["CI Pipeline — trigger: push/PR"]
        A["Push code"] --> B["Lint"]
        B --> C["Unit Test"]
        C --> D["Build Image"]
        D --> E["Scan Vulnerability"]
        E --> F["Push to Registry"]
    end

    F -->|image:2.1.0-abc123| G[("Container Registry")]

    subgraph CD["CD Pipeline — trigger: CI success / merge main"]
        G --> H["Deploy Staging"]
        H --> I["Integration Test"]
        I --> J{"Gate Approval"}
        J -->|approve| K["Deploy Production"]
        J -->|reject| L["Hold"]
        K --> M["Verify Prod"]
    end

The difference is significant. The CI pipeline is triggered by every code change and focuses on verification: is this code correct? Is it safe to build? Does it pass tests? The CI output is a quality-assured artifact — usually an immutable-tagged container image. The CD pipeline is triggered after CI succeeds and focuses on distribution: the same artifact is installed in staging, tested, then promoted to production. Ansible takes the dominant role on the CD side: from community.docker.docker_image to pull images, to deployment tasks on target servers.

The main advantage of this separation: one build, many deploys. The exact same image is tested in staging and deployed to production. No more “it works in staging, why does it error in production?” drama.

Never rebuild images per environment. If we have a build job running in staging and a build job running in production with the same code, we lose the guarantee that production == staging. Build once, tag immutable (git SHA or semver), promote across all environments.

Branching Strategy and Pipeline Triggers #

The branching strategy determines when and what the pipeline triggers. The wrong choice makes our pipeline slow, unreliable, or confusing for developers. The following table compares the three most common strategies:

For most teams running Ansible as the deployment engine, trunk-based or GitHub Flow is the best choice. The pipeline just triggers on pushes to main and pull requests; no need to handle many different branches with their own configurations.

Decision Tree for Choosing a Branching Strategy #

flowchart TD
    A{"Do we need<br/>long-lived versions<br/>for clients?"} -- Yes --> B["GitFlow"]
    A -- No --> C{"Deploy at least<br/>1x per day?"}
    C -- Yes --> D["Trunk-Based Development"]
    C -- No --> E{"Is this a public<br/>library/API with<br/>strict SemVer?"}
    E -- Yes --> F["GitFlow variant:<br/>release branch only"]
    E -- No --> G["GitHub Flow"]
    B --> H["Per-branch pipeline<br/>configuration"]
    D --> I["One pipeline,<br/>trigger main + PR"]
    F --> J["Pipeline focused<br/>on the release branch"]
    G --> K["PR = preview deploy,<br/>merge = prod deploy"]

If we’re just starting out, choose trunk-based development. The main branch is always deployable, feature branches are short-lived (≤1 day), and the pipeline only needs two triggers: pull requests (for validation) and pushes to main (for deployment). Pipeline configuration stays simple and developers don’t need to think about which branch to merge first.

Separating CI and CD Pipelines #

Now let’s look at a concrete implementation. The example below separates two workflows in GitHub Actions — one for build, one for deploy with Ansible. The same pattern can be adopted in GitLab CI with a different stages structure.

# .github/workflows/ci.yml — Build and test
name: CI

on:
  push:
    branches: ['**']
  pull_request:

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: |
          pip install -r requirements-dev.txt
          pytest tests/
          ansible-lint          

  build-image:
    needs: lint-and-test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    outputs:
      image_tag: ${{ steps.meta.outputs.tags }}
      image_digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Generate image metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: registry.company.com/myapp
          tags: |
            type=sha,prefix=,format=short
            type=semver,pattern={{version}}            

      - name: Build and push the image
        id: build
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

# .github/workflows/cd.yml — Deploy with Ansible
name: CD

on:
  workflow_run:
    workflows: [CI]
    types: [completed]
    branches: [main]

jobs:
  deploy-staging:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4

      - name: Setup Ansible
        run: |
          pip install ansible
          ansible-galaxy install -r requirements.yml          

      - name: Get the image tag from the CI run
        id: get_tag
        run: |
          # Get the tag from the CI workflow output
          echo "IMAGE_TAG=${{ github.event.workflow_run.head_sha | truncate(7) }}" >> $GITHUB_ENV          

      - name: Deploy to staging with Ansible
        run: |
          echo "${{ secrets.STAGING_SSH_KEY }}" > /tmp/id_ed25519
          echo "${{ secrets.VAULT_PASS_STAGING }}" > /tmp/.vault_pass
          chmod 600 /tmp/id_ed25519 /tmp/.vault_pass
          ansible-playbook -i inventory/staging/ deploy.yml \
            -e "app_version=${{ env.IMAGE_TAG }}" \
            --vault-password-file /tmp/.vault_pass \
            --private-key /tmp/id_ed25519          

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment:
      name: production    # Required reviewers are set in GitHub environment settings
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to production
        run: |
          pip install ansible
          ansible-galaxy install -r requirements.yml
          echo "${{ secrets.PROD_SSH_KEY }}" > /tmp/id_ed25519
          echo "${{ secrets.VAULT_PASS_PROD }}" > /tmp/.vault_pass
          chmod 600 /tmp/id_ed25519 /tmp/.vault_pass
          ansible-playbook -i inventory/production/ deploy.yml \
            -e "app_version=${{ env.IMAGE_TAG }}" \
            --vault-password-file /tmp/.vault_pass \
            --private-key /tmp/id_ed25519
          rm -f /tmp/id_ed25519 /tmp/.vault_pass          

Notice how the CI workflow knows nothing about deployments. It only produces an image and writes its tag to the output. The CD workflow waits for CI success, then takes the exact same image from the registry to deploy to staging, and finally to production. The workflow_run trigger in CD is GitHub’s official pattern for chaining workflows without direct coupling.

A Deploy Playbook That Receives the Version from the Pipeline #

A good pipeline will never run a playbook without an explicit version. Ansible provides the mandatory filter we can use to force certain variables to always exist:

# playbooks/deploy.yml
---
- name: Deploy the application
  hosts: appservers
  vars:
    # app_version MUST be passed from the pipeline: -e "app_version=abc1234"
    app_image: "registry.company.com/myapp:{{ app_version | mandatory }}"

  pre_tasks:
    - name: Verify the image exists in the registry
      command: "docker manifest inspect {{ app_image }}"
      changed_when: false
      delegate_to: localhost

  tasks:
    - name: Pull the image to every server
      community.docker.docker_image:
        name: "{{ app_image }}"
        source: pull
        force_source: true

    - name: Deploy the new container
      community.docker.docker_container:
        name: myapp
        image: "{{ app_image }}"
        state: started
        restart_policy: unless-stopped
        recreate: true
        pull: false           # Already pulled above

  post_tasks:
    - name: Verify the deployment succeeded
      uri:
        url: "http://localhost:{{ app_port }}/health"
        status_code: 200
      retries: 10
      delay: 6

Three important things in this playbook: first, app_version | mandatory forces the pipeline to send an explicit version — if absent, the playbook fails immediately with a clear error. Second, pre_tasks verifies the image actually exists in the registry before trying to pull, so we don’t deploy a placeholder. Third, post_tasks runs a health check with retries — if the container needs time to start, the deployment is only considered successful after the health endpoint actually responds 200.

Never hardcode app_image: "registry.company.com/myapp:latest" in a playbook. The latest tag is mutable — it can change between build time and deploy time. If someone rebuilds an image in the registry with the same tag (e.g. for a hotfix), production suddenly runs code that was never tested. Always tag with the git SHA or semver version.

Pipeline Gates: Check Before Continuing #

A gate is a point in the pipeline that must be passed before continuing to the next stage. Without gates, production deployments happen automatically every time someone merges to main — a recipe for disaster.

# Add a gate between staging and production
  validate-staging:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Run smoke tests against staging
        run: |
          # Test the main endpoints
          curl -f https://staging.company.com/health
          curl -f https://staging.company.com/api/version          

      - name: Check the staging error rate in Prometheus
        run: |
          ERROR_RATE=$(curl -s \
            "https://prometheus.company.com/api/v1/query?query=rate(http_requests_total{status=~'5..',env='staging'}[5m])" \
            | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['data']['result'][0]['value'][1] if d['data']['result'] else '0')")
          if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
            echo "Staging error rate too high: $ERROR_RATE"
            exit 1
          fi          

The gate above has two layers: a smoke test to ensure the endpoint is truly alive, and a Prometheus query to ensure the error rate hasn’t spiked. If either fails, the pipeline dies here and doesn’t continue to production. We can add stricter gates as needed — security scans, performance tests, or manual reviewer approval.

A good gate is a gate that can be automated and repeated. If the gate is just “manual check in Slack”, it will be ignored when people are in a hurry. Combine automated checks (Prometheus, tests) with environment protection rules in GitHub or GitLab for manual approval. What’s automatic must be automatic; what’s manual must be explicit and recorded.

Pipeline Observability: Knowing What’s Happening #

A pipeline that runs successfully without showing its process is confusing when it errors. We need to know: how long each stage takes, where failures often happen, and which deployment just completed. This section cross-links to the observability section for complete details on the tooling.

sequenceDiagram
    participant Dev as Developer
    participant GHA as GitHub Actions
    participant Reg as Container Registry
    participant Ans as Ansible
    participant Mon as Monitoring
    participant Slack as Slack/Alert

    Dev->>GHA: push to main
    GHA->>GHA: lint + test (stage 1)
    GHA->>GHA: build image (stage 2)
    GHA->>Reg: push image tag=abc1234
    GHA->>Mon: emit metric: ci_build_duration_seconds
    GHA->>Mon: emit event: ci_build_success (image: abc1234)

    GHA->>Ans: trigger deploy-staging
    Ans->>Reg: pull image abc1234
    Ans->>Mon: emit metric: deploy_duration_seconds (env: staging)
    Ans->>Mon: emit event: deploy_complete (env: staging, version: abc1234)

    Mon->>Slack: alert if error_rate > threshold
    Ans->>Mon: smoke test result
    Mon-->>Dev: deployment success/failure notification

The sequence diagram above shows what metadata should be emitted at every critical point. At minimum we need:

• Metric ci_build_duration_seconds — how long each CI stage takes. If it suddenly rises, there’s a bottleneck.
• Metric deploy_duration_seconds{env,version} — how long the deploy to each environment takes. Anomalies here can indicate network or configuration problems.
• Event deploy_complete{env,version,timestamp} — an audit record. Can be used for compliance (“who deployed what when”) and for correlating with production incidents.
• Automatic alerts when the staging error rate spikes after a deploy — the pipeline should have died before reaching production, but if it slips through, monitoring is the last safety net.

Details on setting up metric collection, dashboards, and alerts are in the observability section — especially the Metric Collection, Dashboard, and Alerting articles. What we need to remember at the pipeline design stage: the pipeline is a system that can be observed too, not a black box that runs and finishes.

Anti-Patterns to Avoid #

Here are the three most common CI/CD pipeline anti-patterns, complete with the correct versions:

1. Rebuilding Images per Environment #

# ANTI-PATTERN: image rebuilt per environment
# This job builds the same image in staging and production

stages:
  - build-staging
  - build-production
  - deploy-staging
  - deploy-production

build-staging:
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push myapp:$CI_COMMIT_SHA-staging
  stage: build-staging

build-production:
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push myapp:$CI_COMMIT_SHA-prod
  stage: build-production

# ANTI-PATTERN: two separate build jobs, two different tags for images
# with the same contents. Staging and production never truly run
# the exact same code.

# CORRECT: build once, promote to all environments
stages:
  - build
  - deploy-staging
  - deploy-production

build:
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push myapp:$CI_COMMIT_SHA
  stage: build

deploy-staging:
  script:
    - ansible-playbook -i inventory/staging/ deploy.yml
        -e "app_version=$CI_COMMIT_SHA"
  stage: deploy-staging

deploy-production:
  script:
    - ansible-playbook -i inventory/production/ deploy.yml
        -e "app_version=$CI_COMMIT_SHA"
  stage: deploy-production

# CORRECT: one image with one immutable tag promoted to
# staging and production. What's tested in staging = what runs in
# production. No possibility of drift.

2. Long-Running Pipelines That Can’t Resume #

# ANTI-PATTERN: one long workflow that must restart from scratch
# if any step fails

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: ./build.sh          # 10 minutes
      - run: ./test.sh           # 5 minutes
      - run: ./deploy-staging.sh # 5 minutes — fails here
      - run: ./deploy-prod.sh    # Never runs, must re-run everything

# ANTI-PATTERN: failing at step 3 wastes 15 minutes on
# build + test that actually succeeded. Developers must wait
# or spend runner costs twice.

# CORRECT: separate into standalone jobs with caching

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.txt') }}
      - run: ./build.sh
      - run: ./test.sh

  deploy-staging:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy-staging.sh

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy-prod.sh

# CORRECT: if deploy-staging fails, just re-run that one job.
# The already-successful test job doesn't need to be repeated. The pip
# cache speeds up iteration.

3. Secrets Hardcoded in Pipeline Files #

# ANTI-PATTERN: secrets written directly in the workflow

jobs:
  deploy:
    steps:
      - run: |
          echo "ssh-rsa AAAAB3NzaC1yc2EAAAA..." > /tmp/deploy_key
          echo "vaultpassword123" > /tmp/.vault_pass
          ansible-playbook -i inventory/prod/ deploy.yml          

# ANTI-PATTERN: secrets committed to git. Anyone with read
# access to the repository can see production credentials. Rotating
# credentials also requires re-committing, which is usually postponed.

# CORRECT: secrets fetched from the secret manager and auto-cleaned

jobs:
  deploy:
    steps:
      - name: Fetch the SSH key from the secret
        env:
          SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
        run: |
          echo "$SSH_KEY" > /tmp/deploy_key
          chmod 600 /tmp/deploy_key          

      - name: Fetch the vault password from the secret
        env:
          VAULT: ${{ secrets.VAULT_PASS_PROD }}
        run: |
          echo "$VAULT" > /tmp/.vault_pass
          chmod 600 /tmp/.vault_pass          

      - name: Deploy
        run: ansible-playbook -i inventory/prod/ deploy.yml \
              --vault-password-file /tmp/.vault_pass \
              --private-key /tmp/deploy_key

      - name: Cleanup
        if: always()
        run: rm -f /tmp/deploy_key /tmp/.vault_pass

# CORRECT: secrets only live in the runner's memory while the job runs.
# After finishing (success or failure), the files are deleted immediately.
# The repository stays clean, the audit trail is clear.

Summary #

• Separate CI and CD — CI produces verified artifacts, CD distributes them. Deployment isn’t part of the build process.
• Immutable artifacts: tag Docker images with the git SHA or semantic version, not latest. The same image is promoted from staging to production.
• Environment promotion: the same artifact is used in all environments — this proves that what’s tested in staging is exactly what’s deployed to production.
• app_version | mandatory in the playbook ensures the version is always passed from the pipeline — deployments can’t run without an explicit version.
• Gates between staging and production: smoke tests and metric checks before promotion — automatically stop the deployment if there’s a problem in staging.
• A simple branching strategy (trunk-based or GitHub Flow) makes the pipeline simpler and more reliable — many branches = many configurations = many bugs.
• The pipeline emits metrics and events to the monitoring system — not a black box. At minimum track stage durations, success rates, and deploy events per environment.
• Delete credentials with rm -f in every step after finishing, use if: always() for cleanup that runs even if the pipeline fails.

← Previous: AWX & Tower Next: GitHub Actions →