Best Practice #

After discussing all CI/CD components — pipeline design, GitHub Actions, GitLab CI, environment management, rollback, artifacts, and notifications — this article summarizes the principles and patterns distinguishing a good pipeline from one that merely functions. Many teams build pipelines that can deploy, but fewer build pipelines that are trusted and relied upon by the team every day. Here are eight foundational principles, plus a list of the anti-patterns most often blocking CI/CD maturity, and a review checklist we can use directly before declaring a pipeline “production-ready”.

stateDiagram-v2
    [*] --> Manual
    Manual: "Manual deploy<br/>by developers"
    Scripted: "Bash scripts<br/>run on laptops"
    CI: "CI server<br/>build + test"
    PipelineAsCode: "Pipeline in Git<br/>+ testing"
    GitOps: "Git as the<br/>source of truth"
    AI: "AI-assisted<br/>pipeline tuning"
    Manual --> Scripted: "Write scripts"
    Scripted --> CI: "Move to a CI server"
    CI --> PipelineAsCode: "Versioned in Git"
    PipelineAsCode --> GitOps: "Auto-sync to the cluster"
    GitOps --> AI: "Optimize test & deploy"
    AI --> [*]: "Continuous improvement"

The lifecycle above shows five maturity levels commonly encountered. Manual (deploy via SSH one by one), Scripted (using bash scripts but still from a laptop), CI (a CI server triggering automatically), PipelineAsCode (the pipeline itself version-controlled in Git), and GitOps (Git as the single source of truth for the desired state). The best practices discussed in this article help us move to at least the PipelineAsCode level — where everyone trusts that the pipeline will produce the same result every time it runs.

1. Pipelines Must Be Idempotent and Retryable #

The same pipeline must be runnable twice, three times, or twenty times producing an identical end state. This isn’t a luxury — it’s the foundation of any serious automation. Without idempotency, every retry can produce side effects (deploying twice, running database migrations twice, or inconsistently overwriting configuration files).

# ANTI-PATTERN: a non-idempotent task
- name: Add a line to the crontab
  shell: (crontab -l; echo "*/5 * * * * /opt/app/sync.sh") | crontab -
  # If the playbook is retried, the same line gets added AGAIN
  # After 5 retries, the crontab has 5 identical lines
  # → 5x executions per 5 minutes, not 1x

# CORRECT: use an idempotent Ansible module
- name: Add the sync cron job
  ansible.builtin.cron:
    name: "app-sync"
    minute: "*/5"
    job: "/opt/app/sync.sh"
    user: app
  # Ansible detects a cron job with the same name → only one entry
  # Retry is safe, no duplication

Idempotency isn’t just about which module is used — it’s also about logic. If a playbook creates a new configuration file, run a template: task (which cleanly overwrites the file), not lineinfile: with patterns that might duplicate. If a playbook deploys a Docker image, use the docker_container module with an explicit version tag (not latest which can change at any time). If a playbook restarts a service, use service: with state: restarted (idempotent), not command: systemctl restart (which can error if the service isn’t running).

Retry safety must also be explicitly configured. Add retries and delay parameters to tasks prone to transient failures (HTTP calls, git clones, or Docker pulls). However, make sure the retried task is truly safe to repeat — if not, retrying can itself become a source of new bugs.

2. Build the Artifact Once, Promote to All Environments #

A frequently wrong pattern: every environment (dev, staging, production) rebuilds from source code. The problem: the binary reaching production is never truly identical to what was tested in staging — dependency versions can differ, compiler configurations can differ, or embedded timestamps can differ. As a result, bugs passing staging tests appear in production not because the code changed, but because the build process differed.

# ANTI-PATTERN: build per environment
# job build-staging:
- name: Build the image for staging
  docker_image:
    name: myapp
    tag: staging-{{ ansible_date_time.epoch }}
    build:
      path: .
      dockerfile: Dockerfile
# job build-production:
- name: Build the image for production
  docker_image:
    name: myapp
    tag: production-{{ ansible_date_time.epoch }}
    build:
      path: .
      dockerfile: Dockerfile
# Problem: two builds, two binaries, two dependency sets
# → no guarantee that the staging image == the production image

# CORRECT: build once, promote to all envs
# job build (once):
- name: Build the image once
  docker_image:
    name: myapp
    tag: "{{ git_sha }}"    # Immutable tag
    build:
      path: .
      dockerfile: Dockerfile
    push: yes
# job deploy-staging (uses the same artifact):
- name: Deploy the built image to staging
  command: kubectl set image deployment/myapp myapp=myapp:{{ git_sha }}
# job deploy-production (PROMOTES the same artifact):
- name: Deploy the same image to production
  command: kubectl set image deployment/myapp myapp=myapp:{{ git_sha }}
# The myapp:{{ git_sha }} image is exactly the same in staging and production
# Only the runtime configuration differs (env vars, secrets)

Image tags must be immutable — use git_sha or semver, never latest or timestamps. The latest tag in different environments can resolve to different images (if a new push happens mid-way). Timestamp tags are tempting because they look unique, but make auditing harder (“when was this image built?” must be cross-referenced to CI logs).

Environment-specific configuration (database URLs, API keys, feature flags) must be injected at runtime via environment variables, Kubernetes ConfigMaps, or Vault — not permanently baked into the image. This allows one image to be promoted to staging with configuration A, then to production with configuration B, without needing a rebuild.

3. Test at Every Stage — Unit, Integration, and End-to-End #

A pipeline without testing at every stage is a pipeline that merely tries to deploy, not one that validates deployments. Bugs that should have been caught in unit tests slip through to staging, then to production. Every stage must have the testing type matching the confidence level needed.

flowchart LR
    A["Commit"] --> B["Stage 1:<br/>Unit Test<br/>(< 2 minutes)"]
    B --> C["Stage 2:<br/>Lint + Static Analysis<br/>(< 1 minute)"]
    C --> D["Stage 3:<br/>Build Artifact<br/>(< 3 minutes)"]
    D --> E["Stage 4:<br/>Integration Test<br/>(< 5 minutes)"]
    E --> F["Stage 5:<br/>Deploy to Staging"]
    F --> G["Stage 6:<br/>Smoke + E2E Test<br/>(< 10 minutes)"]
    G --> H["Stage 7:<br/>Deploy to Production"]
    H --> I["Stage 8:<br/>Post-deploy<br/>health check"]
    B -. "fail" .-> X["Pipeline stops<br/>fix first"]
    E -. "fail" .-> X
    G -. "fail" .-> X
    I -. "fail" .-> Y["Trigger<br/>rollback"]

The flow above illustrates testing progression — the further right, the more expensive the testing (time, resources) but the higher the confidence it provides. Unit tests are cheap and fast, while e2e tests are expensive and slow. Ideally, most bugs are caught in early stages (unit + integration), and late stages (e2e) only validate the happy path.

# ANTI-PATTERN: a pipeline without layered testing
- name: Deploy to production
  command: kubectl apply -f deployment.yml
  # If there are no previous tests, we're deploying "hopes" not "validation"
  # Bugs are only discovered when users complain

# CORRECT: every stage has appropriate validation
- name: Unit test (stage 1)
  command: pytest tests/unit/
  # must finish < 2 minutes, otherwise split into smaller pieces

- name: Integration test (stage 4)
  command: pytest tests/integration/
  # needs service dependencies (database, redis), use a docker-compose test harness

- name: End-to-end test (stage 6) on staging
  uri:
    url: "https://staging.example.com/health"
    status_code: 200
  retries: 3
  delay: 5
  register: health_check
  failed_when: false
  # If the health check fails, the pipeline stops at stage 6
  # Production won't be deployed

- name: Post-deploy health check (stage 8)
  uri:
    url: "https://example.com/health"
    status_code: 200
  register: prod_health
  failed_when: false
  # If it fails, trigger an automatic rollback

The fast feedback principle: a bug found in unit tests costs little (1 minute of developer time), a bug found in staging costs medium (rollback + redeploy), while a bug found in production costs the most (incidents, post-mortems, and reputation). A good testing pyramid shifts detection to the left — as many bugs as possible caught at the earliest stages.

4. Secrets Injected at Runtime, Not Baked into Images #

One of the most common and most dangerous security mistakes: secrets baked into Docker images or Ansible artifacts. An image pushed to public Docker Hub (or a leaking internal registry) will contain database passwords, API keys, or certificates. Once the image spreads, the secrets spread too — and their rotation becomes a nightmare.

# ANTI-PATTERN: secrets baked into the image
# Dockerfile:
# FROM python:3.11
# ENV DB_PASSWORD=hunter2
# ENV API_KEY=sk-abc123
# COPY app.py /app/
# → The image pushed to the registry = leaked secrets

# CORRECT: secrets injected at runtime
# Dockerfile:
# FROM python:3.11
# COPY app.py /app/
# ENV DB_PASSWORD_FILE=/run/secrets/db_password
# ENV API_KEY_FILE=/run/secrets/api_key
# → The image has no secrets, only references to files/envs injected at deploy time

# Deployment:
- name: Deploy with secrets from Vault
  kubernetes.core.k8s:
    definition:
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: myapp
      spec:
        template:
          spec:
            containers:
              - name: myapp
                image: "registry.company.com/myapp:{{ git_sha }}"
                env:
                  - name: DB_PASSWORD
                    valueFrom:
                      secretKeyRef:
                        name: myapp-secrets
                        key: db_password
                  - name: API_KEY
                    valueFrom:
                      secretKeyRef:
                        name: myapp-secrets
                        key: api_key

For Ansible itself, secrets are managed via Ansible Vault (encrypted files) or external secret managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). When the playbook runs, secrets are fetched from the source and passed to tasks as variables. Make sure to use the no_log: true parameter on every task using secrets, and run ansible-vault encrypt on files containing credentials.

Never echo secret values in commands or debug messages, even for testing purposes. Once a secret enters CI logs or terminal history, consider it compromised. Use lookup('community.general.vault', ...) or environment variables with no_log: true for all secret access.

5. Fast Feedback — Pipelines Under 10 Minutes #

Slow pipelines tend to be bypassed. This isn’t a hypothesis, it’s a consistent pattern across CI/CD surveys: pipelines taking > 15 minutes make developers start looking for shortcuts. They’ll commit directly to main (skipping CI), rebase to re-run failed stages (instead of fixing them), or deploy manually with the excuse “don’t want to wait for the pipeline”. All of these behaviors destroy CI/CD discipline.

# ANTI-PATTERN: a slow pipeline because it's not parallel
stages:
  - test-unit-python
  - test-unit-javascript
  - test-integration
  - build
  - deploy
# Total: 5 + 3 + 8 + 4 + 6 = 26 minutes (sequential)
# Developers get tired of waiting → skip the pipeline

# CORRECT: parallel in early stages, sequential at the end
stages:
  # Parallel: three jobs run simultaneously
  - name: test-unit-python
  - name: test-unit-javascript
  - name: lint
  # Sequential: needs the artifact from the previous job
  - name: build
    needs: [test-unit-python, test-unit-javascript, lint]
  - name: deploy-staging
    needs: [build]
  - name: e2e-test
    needs: [deploy-staging]
  - name: deploy-production
    needs: [e2e-test]
# Parallel total: 5 minutes (max of the jobs)
# Sequential total: 5 + 4 + 6 + 3 = 18 minutes (5 + build + staging + e2e + prod)
# → 18 minutes total, but "did the tests pass" feedback within 5 minutes

Strategies to keep pipelines fast:

Parallel in early stages — unit tests, lint, and static analysis can run simultaneously. These tasks are independent and don’t need shared state.
Cache dependencies — pip cache, npm cache, or maven cache can be reused across runs. Downloading dependencies often takes >30% of total build time.
Incremental builds — build only the changed layers, not full rebuilds. Use Docker layer caching, Bazel incremental builds, or make -j for parallel compilation.
Fail fast — put the most frequently failing tests at the very beginning. A pipeline failing at stage 1 finishes in 2 minutes; one failing at stage 5 finishes in 12.
Conditional stages — e2e tests in production only run on release tags, not every commit.

6. The Pipeline Itself Must Be Observed #

A frequent irony: teams carefully monitor their production applications, but the pipeline deploying those applications is never monitored. Pipelines fail without alerting, retries happen without clear reasons, or stuck jobs hang on inventory semaphores — all happening without anyone knowing. During incidents, the team has no forensic data.

flowchart TD
    A["Pipeline run"] --> B["CI Server<br/>(GitLab/GitHub Actions)"]
    B --> C{"Logging"}
    C --> D["Loki/ELK<br/>(log pipeline runs)"]
    B --> E{"Metrics"}
    E --> F["Prometheus<br/>(duration, success rate,<br/>queue time)"]
    B --> G{"Alerting"}
    G --> H["Alertmanager<br/>(pipeline failed 3x in a row,<br/>duration > 15 minutes)"]
    B --> I{"Tracing"}
    I --> J["OpenTelemetry<br/>(trace job-to-job<br/>in multi-stage)"]
    B --> K{"Dashboard"}
    K --> L["Grafana<br/>(7-day success rate,<br/>p95 duration, flakiness)"]

Four things to observe about the pipeline itself:

Success rate per stage — which stage fails most often? Is there a regression (e.g. success rate dropping sharply this week)?
p50 and p95 duration — is the pipeline getting slower over time? This is an early warning for dependency or resource problems.
Flakiness — tests/jobs with intermittent failures (passing once, failing once without code changes) must be flagged and fixed, not silently retried until forgotten.
Queue time — how long do jobs wait in the queue? Long queue times indicate insufficient CI resources.

The minimum alerts to prepare: pipeline failed 3x in a row (possibly a structural problem), pipeline duration > 2x the average (performance regression), queue time > 10 minutes (insufficient capacity), and a specific job stuck > 30 minutes (hang).

7. A Rollback Plan MUST Exist Before Deploying #

The pattern distinguishing senior teams from junior teams: the rollback plan is written and tested before the deployment, not after. Junior teams only think “if it fails we’ll roll back” when the deployment is already running and production is down. At that point, rollback is done in panic, with uncertainty, and often adds new problems.

# ANTI-PATTERN: a rollback plan that was never prepared
- name: Deploy to production
  command: kubectl apply -f deployment.yml
  # Deployment finishes, then:
  #   1. The application is down
  #   2. "Let's roll back!"
  #   3. "To which version? Check git log first..."
  #   4. "What was the previous image tag?"
  #   5. "kubectl rollout undo? Or apply the old file?"
  #   6. 15 minutes pass, production is still down

# CORRECT: a structured, tested, automatic rollback plan
- name: Save the pre-deployment state
  set_fact:
    previous_image_tag: "{{ lookup('pipe', 'kubectl get deployment myapp -o jsonpath=\"{.spec.template.spec.containers[0].image}\"') }}"
    previous_replicas: "{{ lookup('pipe', 'kubectl get deployment myapp -o jsonpath=\"{.spec.replicas}\"') }}"

- name: Deploy the new image
  command: kubectl set image deployment/myapp myapp=myapp:{{ git_sha }}

- name: Wait for the rollout to finish
  kubernetes.core.k8s_info:
    kind: Deployment
    name: myapp
    wait_condition:
      type: Complete
      status: "True"
    wait_timeout: 300
  register: rollout_status

- name: Verify post-deploy
  uri:
    url: "https://example.com/health"
    status_code: 200
  register: health_check
  until: health_check.status == 200
  retries: 5
  delay: 10

- name: Automatic rollback if the health check fails
  command: >
    kubectl set image deployment/myapp myapp={{ previous_image_tag }}
    && kubectl scale deployment/myapp --replicas={{ previous_replicas }}    
  when: health_check.failed
  # Rollback within 30 seconds, without human intervention
  # previous_image_tag was saved before the deploy

The three components a rollback plan must have:

Previous state saved — image tag, replica count, and configuration. Without this, rollback goes to “approximately the previous version”.
Automatic verification — a health check or smoke test determining whether the release succeeded or failed. Without this, we don’t know when to roll back.
An automatic rollback procedure — a script or playbook runnable without human intervention. Manual rollback during incidents is prone to additional errors.

The rollback plan must also have been tested, not just written on paper. Run a rollback simulation in staging, measure how long it takes, and identify obstacles. Update the plan based on real findings.

8. Pipeline as Code, Versioned in Git #

Pipeline configuration is code — it must be reviewed, tested, and version-controlled like application code. No pipeline configuration may be changed via the CI server UI without leaving a trace. No actions like “we added a new step in GitHub Actions directly from the web, but forgot to commit”.

# CI/CD as code directory structure
cicd/
├── pipelines/
│   ├── main.yml              # Pipeline for the main branch
│   ├── pr.yml                # Pipeline for pull requests
│   └── release.yml           # Pipeline for release tags
├── tasks/
│   ├── build.yml             # Image build task
│   ├── test.yml              # Unit test task
│   ├── deploy.yml            # Deploy-to-env task
│   ├── rollback.yml          # Rollback task
│   └── notify.yml            # Notification task
├── vars/
│   ├── common.yml            # Shared variables
│   ├── prod.yml              # Production variables
│   └── staging.yml           # Staging variables
├── templates/
│   ├── deployment.yml.j2     # Kubernetes manifest template
│   └── notification.json.j2  # Slack payload template
└── tests/
    ├── test-pipeline-syntax.yml  # YAML validation
    └── test-deploy.yml           # Test deploy to an ephemeral environment

The important pattern in the structure above: pipeline tasks are reusable (not copy-pasted). The deploy.yml file is used by the main, pr, and release pipelines with different parameters. This step prevents drift — if we update the deploy method in the main pipeline but forget to update the release pipeline, their behaviors diverge.

# ANTI-PATTERN: pipeline configuration changed via the CI UI
# Real story:
#   1. Developer A adds a step in GitHub Actions via the web
#   2. The pipeline runs for 3 weeks without anyone noticing the step wasn't committed
#   3. Developer B deletes the branch, the configuration disappears
#   4. The pipeline suddenly fails without any code changes
#   5. "How is that possible? Nothing changed!"

# CORRECT: all changes via PRs, reviewed, tested
# .github/workflows/deploy.yml (or .gitlab-ci.yml)
#   ↑ committed on a feature branch
#   ↑ PR reviewed by at least 1 person
#   ↑ CI runs pipeline syntax tests + deploys to staging
#   ↑ after the merge, the new pipeline becomes active

The consequence of adopting pipeline as code: the pipeline itself can have bugs. A build.yml task can be in the wrong order, or deploy.yml can call the wrong environment. Therefore, pipeline configuration needs the same testing as application code. Do YAML linting (yamllint), schema validation, and test deployments to ephemeral environments (run a small cluster, execute the pipeline, then verify the results).

CI vs CD: Separate or Combined? #

The term “CI/CD” is often used as one whole concept, but it actually consists of two different things with different trade-offs. CI (Continuous Integration) focuses on code integration — combining changes from many developers, running tests, and ensuring no regressions. CD can mean two things: Continuous Delivery (automatically builds and tests, but production deployment requires human approval) or Continuous Deployment (automatically deploys to production after all tests pass).

flowchart LR
    A["CI<br/>(Continuous Integration)"] --> A1["Merge code"]
    A1 --> A2["Build"]
    A2 --> A3["Unit + Integration tests"]
    A3 --> A4["Artifact published"]

    B["CD - Continuous Delivery"] --> A4
    B1["Automatic staging deploy"] --> B2["Manual approval"]
    B2 --> B3["Deploy to production"]

    C["CD - Continuous Deployment"] --> A4
    C1["Deploy to staging"] --> C2["E2E tests"]
    C2 --> C3["Automatic production<br/>deploy"]

    style A stroke:#4caf50,stroke-width:2px
    style B stroke:#ff9800,stroke-width:2px
    style C stroke:#f44336,stroke-width:2px

When to choose each approach:

CI only — for internal applications with infrequent deployments, very strict compliance rules, or teams just starting to learn. This is the realistic first step.
Continuous Delivery — for teams with solid CI, wanting to deploy more often, but where production deployments still need human control. This is the sweet spot for most teams.
Continuous Deployment — for mature applications with high test coverage, sufficient error budgets, and an always-ready on-call team. Only recommended for teams that have been running Delivery smoothly for months.

For Ansible and infrastructure-as-code, Continuous Delivery is the sweet spot — every change merged to main, artifact built, automatic staging deploy, then production deploy with manual approval. Manual approval in production isn’t bureaucracy, it’s recognition that there are times when we need humans to make strategic decisions (e.g. postponing a deploy because a high-traffic event is ongoing).

flowchart TD
    Start{"Pipeline or<br/>manual workflow?"} -- "Pipeline" --> P1{"Deployment frequency<br/>> 1x per week?"}
    P1 -- "Yes" --> P2{"Need automatic<br/>rollback?"}
    P2 -- "Yes" --> Pipeline["Pipeline as code"]
    P2 -- "No" --> Workflow["Workflow with approval"]
    P1 -- "No" --> Workflow
    Start -- "Workflow" --> W1{"Does compliance require<br/>human review?"}
    W1 -- "Yes" --> Workflow
    W1 -- "No" --> Pipeline

This decision tree helps decide whether a process should be fully automated into a pipeline or keep a workflow flow with approvals. The trade-off: automatic pipelines provide high speed but less flexibility, while workflows are slower but put humans in the supervision loop. For routine changes (like application deploys or service restarts), choose the pipeline option. For major one-time changes (like large database migrations), an approval flow is more recommended.

Anti-Patterns to Avoid #

Here are the anti-patterns most often found in CI/CD pipelines on Ansible projects. Each anti-pattern comes with a concise solution we can adopt immediately.

# ✗ Anti-pattern 1: A bypassable pipeline
# There's a "Skip CI" or "Force merge" button that's frequently used
# Situation: developers skip CI because "the test is flaky, later"
# Consequence: bugs slip through, quality drops, trust in CI is lost
# ✓ Solution: remove the skip button, or use it only for documented
#           situations (automatic dependency updates). Flaky tests
#           must be fixed, not skipped.

# ✗ Anti-pattern 2: A pipeline that only builds, doesn't validate
# The pipeline finishes successfully but nobody knows whether the artifact
# actually works
# ✓ Solution: add a verification stage — smoke tests, contract tests,
#           or deployment to an ephemeral environment + run e2e

# ✗ Anti-pattern 3: Deploying to production from a laptop
# "Just push manually to production, it's fine"
# ✓ Solution: production deploys ALWAYS go through the pipeline. No
#           shortcuts. If the pipeline is too slow, fix the pipeline
#           (don't bypass it).

# ✗ Anti-pattern 4: Secrets hardcoded in pipeline files
# .github/workflows/deploy.yml:
- name: Deploy
  run: ansible-playbook -e "db_pass=hunter2" site.yml
  # db_pass enters CI logs = leaked
# ✓ Solution: use a secret manager (Vault, GitHub Secrets, GitLab CI
#           variables). Pass to the playbook via environment variables
#           with no_log: true

# ✗ Anti-pattern 5: Tests always skipped or commented out
# "This test sometimes fails, skip it for now"
# The problem: "temporarily" = forever. Skipped tests become
#              a source of slipping bugs.
# ✓ Solution: fix the test within 1 sprint. If it can't be fixed,
#           delete the test. A skipped test is worse than
#           no test, because it gives a false illusion of validation.

# ✗ Anti-pattern 6: Pipelines without timeouts or resource limits
# A hung job hogs CI resources forever
# Unbounded concurrency can exhaust runners
# ✓ Solution: set a timeout per job (e.g. 30 minutes). Set resource limits
#           (memory, CPU) for heavy jobs. Use cancel-in-progress
#           for superseded jobs.
# .github/workflows/deploy.yml:
#   jobs:
#     deploy:
#       timeout-minutes: 30
#       steps: ...
#       resources:
#         limits:
#           memory: 4Gi
#           cpu: "2"

# ✗ Anti-pattern 7: Non-idempotent pipelines
# Re-running produces side effects (duplicate data, duplicate files)
# ✓ Solution: audit every task. Use idempotent Ansible modules
#           (template, copy, lineinfile with state: present).
#           Test: run the pipeline 2x and compare the results.

CI/CD Review Checklist #

Use this checklist every time you set up a new pipeline, or every time you review an already-running pipeline. Make sure all criteria are met before declaring the pipeline “production-ready” and the team can trust that deploying through the pipeline is as safe as manual deployment (even far safer).

BUILD
  □ Artifact built once and promoted (build-once-promote-everywhere)
  □ Immutable artifact tags (git_sha / semver), not latest / timestamps
  □ Reproducible builds — building on another machine produces an identical binary
  □ Dependencies pinned to exact versions (not ranges), lock files committed
  □ Build caching enabled (pip/npm/maven cache) for speed
  □ Image scanning for vulnerabilities (Trivy, Snyk, Grype)
  □ Image size minimized (multi-stage builds, alpine base images)

TEST
  □ Unit tests run < 2 minutes per module
  □ Integration tests run < 5 minutes per service
  □ End-to-end tests run on staging before production deploys
  □ Test coverage measured and a minimum threshold defined
  □ Flaky tests monitored and fixed (not silently retried)
  □ Linting (yaml, ansible-lint, shellcheck) in the pipeline
  □ Secret scanning (gitleaks, trufflehog) in the pipeline
  □ Security static analysis (bandit for Python, etc.)

DEPLOY
  □ Automatic staging deploy after a successful build
  □ Production deploys through approval (Continuous Delivery)
    or automatic (Continuous Deployment)
  □ Deployment strategy: rolling / blue-green / canary, not recreate
  □ Post-deploy health checks (not just "pod running")
  □ Post-deploy smoke tests in production
  □ Deployment to ephemeral envs for pipeline testing (not just staging)
  □ Environment-specific configuration via env vars / ConfigMap / Vault

ROLLBACK
  □ Pre-deployment state saved (image tag, config version)
  □ Automatic rollback if health checks fail (timeout: 5-10 minutes)
  □ Rollback tested on staging (not just written)
  □ Database migrations are backward-compatible (expand-contract pattern)
  □ Feature flags used to separate code deploys from feature releases
  □ Manual rollback runbook available (if automatic fails)

SECRETS
  □ No secrets in the repository (Vault, .env.example, .gitignore)
  □ Secrets injected at runtime, not baked into images
  □ Ansible Vault or an external secret manager for all credentials
  □ no_log: true on all tasks accessing secrets
  □ CI secrets rotated periodically (at least quarterly)
  □ Service accounts have least privilege (not admin)
  □ Audit logs for secret access (who accessed what when)

OBSERVABILITY
  □ Pipeline runs logged to a central log (Loki/ELK)
  □ Metrics: success rate, p50/p95 duration, queue time
  □ Alert: pipeline failed 3x in a row
  □ Alert: pipeline duration > 2x the average
  □ Alert: a job stuck > 30 minutes
  □ Pipeline dashboard available in Grafana
  □ Success/failure notifications to the right channels (see
    the Notification & Reporting article)
  □ A runbook for every pipeline alert

SECURITY
  □ SBOM (Software Bill of Materials) generated per release
  □ Image signature verification (cosign, Notary)
  □ Dependency vulnerability scanning on every build
  □ Compliance checks (CIS benchmarks) in the pipeline for infra changes
  □ Approval gates for sensitive production actions (data deletion, etc.)
  □ Audit trail of who approved what when

DOCUMENTATION
  □ README explains how to trigger the pipeline (manual + automatic)
  □ Every pipeline variable documented (description, default, range)
  □ Runbooks for "pipeline failed because of X" available
  □ Pipeline architecture diagram updated on changes
  □ On-call knows how to cancel a stuck pipeline
  □ Pipeline configuration reviewed like application code
  □ Post-mortems for every pipeline-caused incident

Building a Healthy CI/CD Culture #

A good pipeline won’t produce optimal results without a supporting team culture. Three cultural pillars to build:

"Deploy often, deploy small, deploy with confidence"

Deploy often:
  Infrequent deployments = large changes = high risk.
  Deploying daily (or more often) forces us to make
  every deployment small and easy to roll back.

Deploy small:
  The smaller the change per deployment, the easier it is to find
  the cause of problems if something goes wrong. Feature flags allow
  deploying code without enabling features.

Deploy with confidence:
  Confidence comes from:
  - Good test coverage
  - Staging mirroring production
  - Adequate observability
  - A rollback that has been tried and proven to work

The “deploy often” culture isn’t about chasing speed alone — it’s about reducing risk. Small changes are easier to debug, easier to roll back, and easier to review. Teams afraid of deploying end up deploying once a month with huge change bundles that can’t be safely rolled back.

Feature flags are the main enabler of the “deploy small” pattern. With feature flags, we can deploy code containing new features, but those features stay inactive until the flag is enabled. This separates when code is deployed (often, small) from when features are released to users (less often, coordinated). Rolling back a feature is then just disabling the flag — without a redeploy.

Summary #

Idempotency and retry safety are the foundation — pipelines must be retryable without side effects. Use idempotent Ansible modules, immutable artifact tags, and define retries/delay on transient tasks.
Build-once-promote-everywhere: the artifact reaching production must be identical to what was tested in staging. Tag with git_sha/semver, then inject per-environment configuration at runtime.
Layered testing: unit tests (< 2 minutes), integration tests (< 5 minutes), e2e tests (on staging), and post-deploy health checks. The further right, the more expensive; the further left, the more bugs must be caught.
Secrets at runtime, not baked in: images must be free of secrets. Use Vault or Ansible Vault as the source, and environment variables or ConfigMaps for runtime injection. Set no_log: true on all secret tasks.
Fast feedback: keep pipeline duration under 10 minutes. Run early stages in parallel, leverage dependency caching, and apply fail-fast. Slow pipelines tend to be bypassed by developers.
Observable pipelines: monitor success rates, p95 durations, flakiness, and queue times. An unmonitored pipeline breeds silent failures.
Rollback plans before deploying: save pre-deployment state, do automatic verification, and create automatic rollback procedures. Rollback must have been tested, not just written in documentation.
Pipeline as code: store pipeline configuration in Git, review in PRs, and test like application code. Avoid making configuration changes via the CI server UI.
CI vs CD: start with CI, then Continuous Delivery (production deploys with approval), and Continuous Deployment (automatic deploys) only for very mature teams.
The “deploy often, small, with confidence” culture: use feature flags, expand test coverage, implement observability, and test rollback mechanisms. This lets teams deploy daily without drama.

← Previous: Notification & Reporting Next: Playbook Anti Pattern →