Environment Management #

Almost all production systems run in several environments: development for experiments, staging for validation, production for real users. The challenge is ensuring the right configuration is applied to the right environment — without duplication, without unintentional variation, and without the risk of staging configuration leaking into production or vice versa. Ansible provides a structure enabling this to be done cleanly and safely. This article discusses proven environment management patterns for Ansible deployment pipelines — from inventory structure, variable hierarchy, promotion flows, to secret management with Ansible Vault.

Per-Environment Inventory Structure #

The most scalable pattern is one inventory directory per environment. Each environment stands alone with its own hosts, variables, and vault:

inventory/
  ├── development/
  │   ├── hosts.ini
  │   └── group_vars/
  │       ├── all.yml           # Variables for all dev hosts
  │       ├── webservers.yml
  │       └── databases.yml
  │
  ├── staging/
  │   ├── hosts.ini
  │   └── group_vars/
  │       ├── all.yml
  │       ├── webservers.yml
  │       └── databases.yml
  │
  └── production/
      ├── hosts.ini
      └── group_vars/
          ├── all.yml
          ├── webservers.yml
          └── vault.yml         # Sensitive variables — encrypted

Each environment has its own all.yml defining values specific to that environment:

# inventory/development/group_vars/all.yml
env: development
app_replicas: 1
app_debug: true
log_level: debug
db_pool_size: 5
enable_https: false

# inventory/staging/group_vars/all.yml
env: staging
app_replicas: 2
app_debug: false
log_level: info
db_pool_size: 10
enable_https: true

# inventory/production/group_vars/all.yml
env: production
app_replicas: 4
app_debug: false
log_level: warning
db_pool_size: 50
enable_https: true

With this structure, running a playbook to a specific environment only needs one argument: -i inventory/production/. There’s no risk of picking the wrong group, no variables unintentionally carried over from another environment. Our CI/CD pipeline just needs three jobs — deploy-dev, deploy-staging, deploy-prod — each pointing at a different inventory.

The Promotion Flow Between Environments #

Three environments aren’t just different in configuration, but also different in terms of who triggers the deploy, when, and what gets promoted. A healthy promotion flow looks like this:

flowchart LR
    A["Push to feature/*"] -->|Auto-deploy| B["Development"]
    B -->|"PR merged to main"| C["Staging"]
    C -->|"Manual approval"| D["Production"]
    B -. "Tag v*.*.*" .-> E["Container Registry"]
    C -. "Tag v*.*.*" .-> E
    D -. "Tag v*.*.*" .-> E
    E -. "Pull image" .-> B
    E -. "Pull image" .-> C
    E -. "Pull image" .-> D

Notice the crucial point: the same artifact (Docker image tagged v2.1.0-abc123) is promoted from one environment to the next. The image that passed testing in staging is the exact same image deployed to production. Never rebuild the image per environment — that removes the guarantee that staging represents production.

Our pipeline can be represented as a sequence diagram like this:

sequenceDiagram
    participant Dev as Developer
    participant Git as Git Repository
    participant CI as CI Pipeline
    participant Reg as Container Registry
    participant Dev2 as Dev Environment
    participant Stg as Staging
    participant Prd as Production
    Dev->>Git: push commit to main
    Git->>CI: trigger pipeline
    CI->>CI: build & test image
    CI->>Reg: push image v2.1.0-abc123
    CI->>Dev2: automatic deploy
    Dev2-->>CI: health check OK
    CI->>Stg: automatic deploy
    Stg-->>CI: integration test OK
    CI->>Prd: needs approval
    Prd-->>CI: approved by oncall
    CI->>Prd: deploy
    Prd-->>CI: health check OK

This sequence diagram clarifies the promotion gates: development auto-deploys, staging auto-deploys after the integration test, production needs human approval. We can automate as much as we want, but the production gate must always stay manual until we’re truly confident.

Shared vs Environment-Specific Variables #

Many variables are the same across all environments — no need to duplicate them. Ansible has an elegant variable precedence hierarchy: variables in more specific locations override variables in more general locations. Leverage this to prevent duplication.

group_vars/         ← Outside the inventory directory (shared)
  └── all.yml       ← Applies to all environments

inventory/
  ├── development/
  │   └── group_vars/
  │       └── all.yml     ← Override shared variables for dev
  ├── staging/
  └── production/

# group_vars/all.yml (shared — applies to all environments)
app_port: 8080
app_name: myapp
app_user: deployer
app_dir: /opt/myapp
backup_retention_days: 30

# Default values to be overridden per environment:
app_replicas: 1
log_level: info

# inventory/production/group_vars/all.yml (override for production)
app_replicas: 4
log_level: warning
# No need to redefine app_port, app_name, etc.
# They're inherited from the shared group_vars

Ansible determines which variable is used by precedence order: host_vars/<host> > group_vars/<group>/ in the inventory > shared group_vars/ > role defaults. Variables in more specific locations automatically win. This means: put everything universal in the shared group_vars/, override only what differs per environment.

The rule of thumb when debugging variables that don’t behave as expected: the closer a variable definition is to the host, the higher its priority. host_vars/hostname.yml overrides group_vars/all.yml overrides group_vars/shared/all.yml overrides role defaults. To see a variable’s effective value, use ansible -i inventory/production all -m debug -a "var=app_replicas" — the output shows the final value that host will use, accounting for the entire precedence chain.

ANTI-PATTERN: Hard-Coded Environment vs Externalized Config #

One of the most common traps is writing environment configuration directly inside playbooks or templates. This makes the configuration non-portable, hard to test, and almost impossible to promote to another environment.

# ANTI-PATTERN: configuration hardcoded inside the playbook
# playbooks/deploy-app.yml
- name: Deploy the application
  hosts: appservers
  vars:
    db_host: "prod-db-01.internal"      # Hard-coded!
    db_password: "s3cr3t!"              # Hard-coded AND leaked!
    redis_url: "redis://prod-redis.internal:6379"  # Hard-coded!
    app_replicas: 4                     # Hard-coded!
    enable_debug: false                 # Hard-coded!

  tasks:
    - name: Deploy
      # ... code using all the variables above

The problem: this playbook can only be used for production. For staging, we’d have to copy-paste and change all the values. For development, copy-paste again. Three playbooks that are 95% identical but with different values. When we want to update one line of logic, we must remember to change it in three places — and sooner or later we’ll forget.

# CORRECT: configuration externalized, read from the per-environment inventory
# playbooks/deploy-app.yml
- name: Deploy the application
  hosts: appservers
  # No hardcoded vars — everything is taken from the inventory

  tasks:
    - name: Write the application configuration
      template:
        src: app-config.j2
        dest: "{{ app_dir }}/config.yml"
        owner: "{{ app_user }}"
        mode: '0644'
      vars:
        db_host: "{{ db_host }}"        # From group_vars
        db_password: "{{ vault_db_password }}"  # From vault
        redis_url: "{{ redis_url }}"    # From group_vars
        app_replicas: "{{ app_replicas }}"  # From group_vars
        enable_debug: "{{ app_debug | bool }}"  # From group_vars

# inventory/development/group_vars/all.yml
db_host: "dev-db.internal"
redis_url: "redis://dev-redis.internal:6379"
app_replicas: 1
app_debug: true

# inventory/production/group_vars/all.yml
db_host: "prod-db.internal"
redis_url: "redis://prod-redis.internal:6379"
app_replicas: 4
app_debug: false

Now one playbook works in all environments. The pipeline just swaps the inventory directory: ansible-playbook -i inventory/production/ deploy-app.yml. No duplication, no risk of forgetting to update one place.

Environment Promotion with Git Tags #

A common pattern: each environment is deployed from a different Git branch or tag. This provides strong traceability — from the running application version, we can immediately know which commit produced it.

Git Flow for Deployment:
  feature/* → main (auto-deploy to development)
  main      → release candidate (manual promote to staging)
  tag v*.*.*→ production (after staging is verified)

# playbooks/deploy.yml
---
- name: Deploy the application
  hosts: appservers
  vars:
    # Overridden from the pipeline based on environment and Git tag
    deploy_version: "{{ version | mandatory }}"
    deploy_env: "{{ env }}"   # From the inventory group_vars

  pre_tasks:
    - name: Verify we're deploying to the correct environment
      assert:
        that:
          - deploy_env == expected_env | default(deploy_env)
        fail_msg: >
          Environment mismatch! The inventory points to '{{ deploy_env }}'
          but the pipeline expects '{{ expected_env }}'.          

    - name: Verify the production version only comes from official tags
      assert:
        that:
          - deploy_version is match('^v?[0-9]+\.[0-9]+\.[0-9]+$')
        fail_msg: >
          Production deployments may only use semver tags.
          Version '{{ deploy_version }}' is invalid for production.          
      when: deploy_env == 'production'

    - name: Record deployment metadata on the server
      copy:
        content: |
          version={{ deploy_version }}
          env={{ deploy_env }}
          git_sha={{ git_sha | default('unknown') }}
          deployed_at={{ ansible_date_time.iso8601 }}
          deployed_by={{ lookup('env', 'CI_JOB_URL') | default(lookup('env', 'USER') | default('manual')) }}          
        dest: "{{ app_dir }}/DEPLOYMENT_INFO"
        mode: '0644'

The assert in pre_tasks is the last safety net. If the pipeline passes the wrong environment (e.g. a production job but the env variable points to staging), the playbook fails before any changes happen. This differs from just logging a warning — the Ansible assert is a hard stop.

Don’t trust the job label in the pipeline as the only environment marker. Always read the environment from the inventory file, and validate that the environment targeted by the inventory matches what the pipeline expects. Most “why did production get hit?” incidents start from the wrong inventory being selected or an incorrect environment variable being passed.

Different Configuration but the Same Playbook #

The main strength of this pattern: one playbook working in all environments based on variables. All decisions — SSL on or off, worker count, log level — are determined by variables Ansible resolves from the inventory.

# playbooks/setup-nginx.yml
---
- name: Set up nginx for all environments
  hosts: webservers
  become: true
  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present

    - name: Deploy the nginx configuration
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify: Reload nginx

    # SSL configuration — only if enable_https: true
    - name: Deploy the SSL certificate
      copy:
        src: "files/ssl/{{ inventory_hostname }}.crt"
        dest: /etc/ssl/certs/app.crt
      when: enable_https | bool

    # Worker processes — different per environment
    - name: Set the nginx worker processes
      lineinfile:
        path: /etc/nginx/nginx.conf
        regexp: '^worker_processes'
        line: "worker_processes {{ ansible_processor_vcpus if env == 'production' else 1 }};"
      notify: Reload nginx

{# templates/nginx.conf.j2 — an environment-aware template #}
worker_processes {{ ansible_processor_vcpus if env == 'production' else 1 }};

events {
    worker_connections {{ 1024 if env == 'production' else 256 }};
}

http {
    {% if enable_https | bool %}
    server {
        listen 443 ssl;
        ssl_certificate /etc/ssl/certs/app.crt;
        # ... SSL config
    }
    {% else %}
    server {
        listen 80;
        # Dev/staging: no SSL
    }
    {% endif %}
}

This nginx.conf.j2 template is interesting because it doesn’t care which environment runs it — all decisions are in the variables. Production gets as many workers as CPU cores and SSL; development gets 1 worker and plain HTTP. The same playbook, different results, based only on the env and enable_https values.

Comparing Environment Strategies #

There’s no one universal environment strategy. The right choice depends on team size, deployment frequency, and our tolerance for specific trade-offs.

For small teams just starting, classic multi-env (dev/staging/prod) is the most pragmatic starting point. As the number of contributors and PR frequency grows, evaluate per-PR ephemeral envs. For consumer products with low traffic and high tolerance for specific trade-offs, single-env + feature flags can be enough. Choose based on context, not on “absolute best practices” read in a blog.

ANTI-PATTERN: Shared Staging Cluster vs Per-PR Ephemeral Envs #

For teams often experiencing “it works in staging, why does it error in production?”, the problem is often that staging is shared by many PRs. Engineer A deploys to staging, then engineer B also deploys to staging before engineer A finishes testing. Result: engineer A isn’t sure whether the bug they see is from their code, or contaminated by engineer B.

# ANTI-PATTERN: shared staging cluster
# inventory/staging/hosts.ini
[appservers]
staging-01.internal
staging-02.internal

# playbooks/deploy-staging.yml
- hosts: appservers
  tasks:
    - name: Deploy the latest PR
      docker_container:
        name: myapp
        image: "registry.company.com/myapp:{{ pr_number }}"

All PRs are deployed to the same servers. When two PRs deploy at close times, there’s no isolation. Engineer A testing the checkout flow can be disrupted by engineer B deploying changes to the payment service. Worse, a race condition during deploys can cause brief staging downtime confusing both engineers — “why the error, the code didn’t change?”

# CORRECT: per-PR ephemeral environment
# playbooks/create-pr-env.yml
- name: Create an ephemeral environment for the PR
  hosts: localhost
  vars:
    pr_number: "{{ pr_number | mandatory }}"

  tasks:
    - name: Provision the Kubernetes namespace for the PR
      kubernetes.core.k8s:
        name: "pr-{{ pr_number }}"
        api_version: v1
        kind: Namespace
        state: present

    - name: Deploy the application to the PR namespace
      kubernetes.core.k8s:
        definition:
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: myapp
            namespace: "pr-{{ pr_number }}"
          spec:
            template:
              spec:
                containers:
                  - name: myapp
                    image: "registry.company.com/myapp:pr-{{ pr_number }}"
        state: present

    - name: Write the preview URL to the PR comment
      uri:
        url: "https://api.github.com/repos/{{ repo }}/issues/{{ pr_number }}/comments"
        method: POST
        body_format: json
        body:
          body: "Preview environment: https://pr-{{ pr_number }}.preview.company.com"
        headers:
          Authorization: "token {{ vault_github_token }}"

Now each PR has its own environment with a unique URL like https://pr-1234.preview.company.com. Testers and engineers can review the PR in a fully isolated environment. After the PR is merged or closed, the environment is automatically destroyed. No more “why is staging erroring after the deploy?” because every deploy has its own home.

Environment Lock: Preventing Deploys to the Wrong Environment #

Defense-in-depth beyond pipeline validation: write a file on the server marking which environment should run on that server. A playbook targeting the wrong environment will fail before making any changes.

# roles/common/tasks/env-lock.yml
# Run this at the start of every deployment playbook

- name: Check the environment lock file
  stat:
    path: /etc/ansible-env-lock
  register: env_lock

- name: Verify the environment matches if the lock file exists
  block:
    - name: Read the expected environment from the lock file
      slurp:
        src: /etc/ansible-env-lock
      register: lock_content

    - name: Validate the configured environment
      assert:
        that:
          - lock_content.content | b64decode | trim == env
        fail_msg: >
          ENVIRONMENT MISMATCH!
          This server is configured for: {{ lock_content.content | b64decode | trim }}
          The inventory used: {{ env }}
          Stop the deployment to prevent incorrect configuration!          
  when: env_lock.stat.exists

- name: Write the environment lock file if it doesn't exist
  copy:
    content: "{{ env }}\n"
    dest: /etc/ansible-env-lock
    mode: '0444'   # Read-only
  when: not env_lock.stat.exists

A lock file on the server is the last safety net, not a replacement for pipeline validation. The best combination: (1) the pipeline ensures the targeted environment is correct before execution, (2) the inventory only contains hosts for that environment, (3) the server verifies which environment is plausible for itself. These three defense layers make “why did production get a staging deploy?” almost impossible.

Secret Management with Ansible Vault #

Environment management without secret management is a time bomb. Each environment has its own secrets (database passwords, API keys, TLS private keys) that must be kept from leaking, but still usable by playbooks.

Ansible Vault encrypts YAML files with AES256. Encrypted files can still be included in playbooks normally, but their contents can’t be read in the Git repository without the password.

# Create a new vault — Ansible will prompt for the password
ansible-vault create inventory/production/group_vars/vault.yml

# Edit an existing vault
ansible-vault edit inventory/production/group_vars/vault.yml

# Encrypt an existing file
ansible-vault encrypt inventory/production/group_vars/vault.yml

# Decrypt to view (only for local debugging, don't commit!)
ansible-vault decrypt inventory/production/group_vars/vault.yml

# View the contents without decrypting to a file
ansible-vault view inventory/production/group_vars/vault.yml

# inventory/production/group_vars/vault.yml (encrypted, contents unreadable)
$ANSIBLE_VAULT;1.1;AES256
35383532316537623534346434643731303636303037303762653965333235646564623935383032
6430346231653137633334336238303130383134323362610a643866393537303332636162326238
# ... encrypted lines

To run a playbook with vault:

# Prompt for the password at execution
ansible-playbook -i inventory/production/ deploy.yml --ask-vault-pass

# Or use a password file (for CI/CD)
ansible-playbook -i inventory/production/ deploy.yml --vault-password-file ~/.vault-pass

# Or via an environment variable
ANSIBLE_VAULT_PASSWORD_FILE=/run/secrets/vault-pass ansible-playbook ...

The .vault-pass file must NEVER be committed to Git. Store it somewhere safe: a secrets manager (HashiCorp Vault, AWS Secrets Manager), CI/CD secrets (GitHub Actions secrets, GitLab CI variables), or a local file on developer workstations ignored by .gitignore. For CI/CD, inject the password as an environment variable or mount it as a file in /run/secrets/ that isn’t persistent.

Per-Environment Vault Passwords #

For teams managing many environments, each environment can have its own vault password. This enables granular access control: junior developers have access to the development vault, senior engineers to staging, and the SRE team to production.

# Set up multiple vault passwords
ansible-vault create inventory/development/group_vars/vault.yml
# Prompt: enter the password for the development vault

ansible-vault create inventory/staging/group_vars/vault.yml
# Prompt: enter the password for the staging vault

ansible-vault create inventory/production/group_vars/vault.yml
# Prompt: enter the password for the production vault

# Run with a specific vault password
ansible-playbook -i inventory/production/ \
  --vault-id production@~/.vault-pass-production \
  deploy.yml

Alternatively, for small teams, one vault password for all environments is pragmatic. The trade-off: slightly less security (whoever has the password has access to all vaults), but much simpler operations. Choose based on team needs.

Summary #

• One inventory directory per environment is the most scalable pattern — each environment has its own separated hosts, variables, and vault.
• Shared group_vars/ outside the inventory directory for variables that are the same across all environments — override in inventory/<env>/group_vars/ only for what differs.
• Promote the same artifact between environments (don’t rebuild) — the image that passed testing in staging must be identical to what’s deployed to production.
• Validate in pre_tasks: make sure we’re deploying to the right environment and a valid version before making any changes.
• An environment lock file at /etc/ansible-env-lock prevents the fatal accident of running a production inventory against staging servers or vice versa.
• Production deployments only from official semver tags — enforce this in the playbook with assert so nobody can bypass it.
• Externalized configuration: no configuration hardcoded in playbooks — all values come from the inventory and vault.
• Ansible Vault encrypts per-environment secrets — combine with a secrets manager for CI/CD.
• Choose the environment strategy (single-env, multi-env, per-PR ephemeral) based on team size and deploy frequency, not dogma.

← Previous: GitLab CI Next: Rollback Strategy →