Workflow #

Vault and encryption protect sensitive data. But automation security isn’t just about encryption — it’s also about who can run what, from where, and when. An insecure workflow can make even the strongest encryption meaningless: if everyone can run playbooks against production anytime without review, the security value provided by Vault becomes minimal. This article discusses workflows that ensure Ansible automation runs with the right controls, from who executes playbooks, how deployment pipelines are built, to how every change to production can be traced.

The Least Privilege Principle for Service Users #

The user Ansible uses to connect to managed nodes must have the minimum rights needed — no more. This principle is often called least privilege and is the foundation of the entire Ansible security workflow. Without least privilege, all other controls above it become fragile: a strong SSH key means nothing if the user can do anything after logging in, and a well-encrypted Vault protects nothing if the vault password can be accessed by a user with excessive privileges.

# roles/common/tasks/ansible-user.yml
# Create a dedicated Ansible user with restricted rights

- name: Create the ansible-deploy user
  user:
    name: ansible-deploy
    shell: /bin/bash
    system: false
    state: present

- name: Add the SSH key for the ansible-deploy user
  authorized_key:
    user: ansible-deploy
    key: "{{ lookup('file', 'files/ansible_deploy.pub') }}"
    exclusive: true     # Only this key is allowed, remove others

- name: Configure sudoers with minimum rights
  template:
    src: ansible-sudoers.j2
    dest: /etc/sudoers.d/ansible-deploy
    mode: '0440'
    validate: 'visudo -cf %s'

{# templates/ansible-sudoers.j2 #}
# Ansible deploy user — minimum rights for deployment operations
ansible-deploy ALL=(ALL) NOPASSWD: /bin/systemctl restart myapp
ansible-deploy ALL=(ALL) NOPASSWD: /bin/systemctl start myapp
ansible-deploy ALL=(ALL) NOPASSWD: /bin/systemctl stop myapp
ansible-deploy ALL=(ALL) NOPASSWD: /usr/bin/rsync
ansible-deploy ALL=(ALL) NOPASSWD: /usr/bin/pip3 install *
# Deny access to unneeded operations
ansible-deploy ALL=(ALL) !NOPASSWD: /bin/su
ansible-deploy ALL=(ALL) !NOPASSWD: /bin/bash

validate: 'visudo -cf %s' is very important — Ansible refuses to save a sudoers file with wrong syntax, so there’s no risk of locking yourself out of the server because of a broken template.

Choosing the Privilege Escalation Model #

Ansible provides several ways to escalate privileges, and choosing the right model affects the security posture of your entire infrastructure. The following table compares the three commonly considered models:

For Ansible on modern Linux, sudo with per-command specific configuration is the most common combination — Ansible already packages sudo logic well through the become module, and sudoers configuration gives you tighter OS-level control. doas is interesting for new deployments wanting concise configuration and better default security. become_user is very useful for tasks that must run as a specific application user (for example deployer, postgres) rather than as root.

flowchart TD
    A["Need privilege escalation?"] -- "No" --> B["Run the task without become"]
    A -- "Yes" --> C{"Escalate to whom?"}
    C -- "root" --> D["sudo with specific commands"]
    C -- "service user" --> E["become_user: deployer/postgres/..."]
    C -- "BSD/OpenBSD" --> F["doas with permit rules"]
    D --> G{"Log to syslog?"}
    G -- "Yes" --> H["sudo provides an OS audit trail"]
    F --> I["doas with 5-minute persist"]
    E --> J["Ansible log + clear become_user"]

The decision tree above isn’t a sequence of steps to follow, but a tool for thinking about which model best fits your context. Many production teams use a combination — sudo for package installation and service configuration tasks, become_user for application deploys, and no escalation for read-only tasks.

Access Separation by Environment #

One team shouldn’t be able to access all environments with the same credentials. Per-environment access separation is a fundamental control: if staging credentials leak through phishing, accidental commits, or a lost laptop, the impact is limited to staging — production stays safe.

Recommended Access Model:

Developers:
  ✓ Can run playbooks to development
  ✓ Can run playbooks to staging (with approval)
  ✗ Cannot directly run to production

SRE/Ops:
  ✓ Can run playbooks to all environments
  ✓ Has the production vault password
  ✓ Can approve production deployments

CI/CD Pipeline:
  ✓ Automatically deploys to staging after a PR merge
  ✓ Deploys to production only after manual approval
  ✗ The pipeline has no direct access to the production vault password

The implementation uses different SSH keys per environment — currently a simple but effective defense in depth control. A leaked staging SSH key doesn’t give direct access to production because the key is different.

# inventory/production/group_vars/all.yml
ansible_ssh_private_key_file: ~/.ssh/ansible_production
ansible_user: ansible-prod-deploy

# inventory/staging/group_vars/all.yml
ansible_ssh_private_key_file: ~/.ssh/ansible_staging
ansible_user: ansible-staging-deploy

For more mature production setups, also separate the vault passwords per environment. You can learn more about vault password management in the Secret Management article.

# Different vault IDs per environment
ansible-playbook site.yml \
  --vault-id production@~/.vault_pass_prod \
  --vault-id staging@~/.vault_pass_staging

Never use the same SSH key for staging and production. If a CI runner uses the wrong key (for example forgets to switch inventory), the consequences can be fatal. Separate keys make this mistake fail loud — Ansible fails with Permission denied instead of silently changing production servers.

Deployment Pipeline with Security Gates #

A secure Ansible workflow isn’t just about access control — it’s also about a structured change flow. Every code change must pass through several gates before reaching production, and each gate has clear criteria. The following diagram illustrates the recommended pipeline:

flowchart LR
    A["Developer"] -->|"push branch"| B["CI: Lint & Syntax"]
    B --> C["CI: ansible-lint"]
    C --> D["CI: --check --diff to staging"]
    D --> E["Merge Request"]
    E --> F{"Peer Review"}
    F -- "Rejected" --> A
    F -- "Approved" --> G["Merge to main"]
    G --> H["Auto deploy staging"]
    H --> I["Integration test"]
    I --> J{"Health check"}
    J -- "Failed" --> K["Rollback + alert"]
    J -- "Passed" --> L["Manual approval"]
    L --> M["Deploy production"]
    M --> N["Prod smoke test"]
    N -- "Failed" --> O["Auto rollback"]
    N -- "Passed" --> P["Audit log + notifications"]

At least four security gates are visible in the diagram: peer review before merge, dry-run in CI, integration test after the staging deploy, and manual approval before the production deploy. Each gate catches a different class of problems — lint catches syntax issues, review catches logic and security issues, dry-run catches change impact, and integration tests catch runtime regressions.

Pipeline Configuration #

# .gitlab-ci.yml — automatic dry run on every MR
ansible-check:
  stage: validate
  script:
    - ansible-lint
    - ansible-playbook -i inventory/staging/ site.yml --check --diff
  only:
    - merge_requests

ansible-deploy-staging:
  stage: deploy-staging
  script:
    - ansible-playbook -i inventory/staging/ site.yml
  only:
    - main

ansible-deploy-production:
  stage: deploy-production
  script:
    - ansible-playbook -i inventory/production/ site.yml
  when: manual        # Must be triggered manually, not automatic
  only:
    - main
  environment:
    name: production
    url: https://app.example.com

when: manual is a critical control — the pipeline must not deploy to production automatically, even after merging to main. A human must explicitly click the deploy button in the pipeline UI. This gives an opportunity for a final review, checking monitoring notifications, and making sure nothing suspicious happens before changes reach production.

You can also integrate this pipeline with external change management systems like ServiceNow, Jira Service Management, or internal approval tools. For a deeper discussion of CI/CD integration, see the CICD Integration article.

Dry Run as a Mandatory Gate #

Require --check --diff before every deployment to production. Dry run tells you what will change without actually changing anything — very important for Ansible deployments because many Ansible tasks are idempotent, and the diff from a dry run is often more informative than the actual task log.

# Step 1: Always run the dry run first
ansible-playbook -i inventory/production/ site.yml --check --diff

# Review the output — make sure only expected changes appear
# Only then run for real

# Step 2: Run for real after reviewing the dry run
ansible-playbook -i inventory/production/ site.yml

Check mode in Ansible works by reporting what tasks would do without actually doing them. For fully idempotent modules (like apt, copy, template, service), this is very accurate. For the command or shell modules, check mode can’t predict the effect of arbitrary commands — for these modules, you need creates/removes arguments or explicit changed_when.

# Task with a command that dry run can't predict
- name: Restart the application after deploy
  command: /opt/app/bin/restart.sh
  # ANTI-PATTERN: dry run reports "changed" but it's unclear what will happen

# CORRECT: use the service module or explicit changed_when
- name: Restart the application after deploy
  service:
    name: myapp
    state: restarted
  # Dry run clearly reports "changed" — the service will be restarted

Integrate --check --diff into your CI/CD pipeline. Every merge request must run a dry run against staging before being allowed to merge. This gives reviewers a baseline of what will change in production — reviewers can directly compare the diff with their expectations, instead of having to run the playbook locally to see the impact.

You can see the changed_when pattern in more depth in the Drift Handling article, which discusses idempotency and how to detect unexpected changes.

Code Review as a Security Control #

Code review is often considered a code quality control, but in the Ansible context, it’s also an important security control. A careful reviewer catches things that escape automated tools: overly loose directory permissions, secrets accidentally entering variables, tasks running on the wrong host, and become: true in places that don’t need it.

ANTI-PATTERN: Skipping Review for “Quick Fixes” #

# ANTI-PATTERN: merge to main without review because it's a "small change"
# .github/workflows/ansible-deploy.yml
on:
  push:
    branches: [main]    # Deploys directly on push, no review

“Quick fix” is the most common excuse for skipping review, and statistically one of the main sources of security incidents. Changes that look small (changing one sudoers line, adding one environment variable) can have non-small impacts.

CORRECT: Mandatory Review with a Checklist #

# CORRECT: require review from at least one peer
# .github/workflows/ansible-deploy.yml
on:
  pull_request:
    branches: [main]
    required_reviewers: 1
    required_status_checks:
      - ansible-lint
      - dry-run-staging

Add a review checklist specific to security:

PLAYBOOK REVIEW CHECKLIST:

Access and privilege:
  □ Does the user used for this task have the minimum required rights?
  □ Are there any `become: true` that could be replaced with `become_user` or removed?
  □ Are the added sudoers specific per command (not ALL)?

Secrets and sensitive data:
  □ Are there any sensitive values not going through Vault?
  □ Do tasks handling secrets have `no_log: true`?
  □ Do deployed files have the correct permissions (mode 0600/0640)?

Idempotency:
  □ Will the task produce the same diff every time it runs?
  □ Are there commands that dry run can't predict?

Target and blast radius:
  □ Is the host pattern in the play header correct?
  □ Was this playbook tested in staging first?

This checklist can be stored as a PULL_REQUEST_TEMPLATE.md in the repository so it automatically appears when developers open merge requests.

Sequence Diagram: Production Approval Flow #

The approval process for production deployments involves several roles and systems. The following sequence diagram illustrates the complete flow from merge to deployment, with the relevant controls:

sequenceDiagram
    participant Dev as "Developer"
    participant Git as "Repository"
    participant CI as "CI/CD Pipeline"
    participant Rev as "Reviewer"
    participant Mgr as "SRE/Ops"
    participant Prod as "Production Server"

    Dev->>Git: "push branch + open MR"
    Git->>CI: "trigger pipeline"
    CI->>CI: "ansible-lint + --check staging"
    CI-->>Git: "status check passed"
    Dev->>Rev: "request review"
    Rev->>Git: "review diff"
    Rev-->>Dev: "approve"
    Dev->>Git: "merge to main"
    Git->>CI: "trigger deploy pipeline"
    CI->>CI: "auto deploy to staging"
    CI->>Prod: "health check + integration test"
    CI->>Mgr: "notification: ready for prod"
    Mgr->>Mgr: "review monitoring + change request"
    Mgr->>CI: "manual approval click"
    CI->>Prod: "deploy production"
    Prod-->>CI: "deployment success"
    CI->>Mgr: "notification: deployed"
    CI->>Git: "tag release + write audit log"

Notice who has control at each stage: developers can only push and merge after approval, reviewers only approve code, SRE/Ops is the only one who can click the production deploy button. This role separation ensures no single person can push changes to production alone — a simple but effective two-person control.

Credential Rotation #

Credentials that are never rotated are a security risk of their own — if they leak, the attacker has access forever. Create a credential rotation playbook that can be run on a schedule, ideally from a CI/CD pipeline with a documented schedule:

# playbooks/rotate-credentials.yml
---
- name: Rotate the SSH key for the deployment user
  hosts: all
  become: true
  vars_prompt:
    - name: "new_ssh_key"
      prompt: "Enter the new SSH public key"
      private: false

  tasks:
    - name: Add the new SSH key
      authorized_key:
        user: ansible-deploy
        key: "{{ new_ssh_key }}"
        state: present

    - name: Verify the connection with the new key succeeds
      ping:
      vars:
        ansible_ssh_private_key_file: /path/to/new_key

    - name: Remove the old SSH key after successful verification
      authorized_key:
        user: ansible-deploy
        key: "{{ old_ssh_key }}"
        state: absent

The SSH key rotation order is important: add the new key, verify the connection succeeds, only then remove the old key. If this order is reversed (remove first, add later), there’s a time window where no valid key exists for Ansible — which means the next playbook will fail, and if that playbook is the rotation playbook, you’ll be locked out.

Rotation isn’t just about SSH keys. Also consider rotation for:

Before removing the old SSH key, ALWAYS verify the connection with the new key succeeds. The safe pattern is: add the new key → run a verification playbook with the new key → if successful, remove the old key. Reversing this order can lock Ansible out of servers. For TLS certificates, always deploy the new certificate and make sure the service has reloaded before the old certificate expires.

Logging and Audit Trail #

Every playbook execution against production must be recorded. The audit trail isn’t just for forensics during incidents, but also for early detection: unusual execution patterns (for example deploys outside working hours, or deploys from a user who never deployed before) can indicate credential compromise.

# ansible.cfg
[defaults]
log_path = /var/log/ansible/ansible.log

# Add an audit task at the start of every production playbook
- name: Log the deployment to the audit trail
  local_action:
    module: lineinfile
    path: /var/log/deployments.log
    line: "{{ ansible_date_time.iso8601 }} | {{ lookup('env','USER') }} | {{ inventory_dir | basename }} | {{ playbook_dir | basename }}"
    create: true
  run_once: true

This log captures four important pieces of information: when the deployment happened, who ran it, which inventory was used, and which playbook was executed. From these four pieces you can reconstruct an incident timeline and limit the investigation scope.

Sending Logs to a Centralized System #

Logs stored on only one server are hard to audit and easily lost. Send Ansible logs to a centralized log system (Loki, ELK, Splunk) so the audit trail survives even if a server has problems:

# tasks/post-log.yml — run at the end of every playbook
- name: Send deployment metadata to Loki
  uri:
    url: "https://loki.internal/loki/api/v1/push"
    method: POST
    body_format: json
    body:
      streams:
        - stream:
            job: ansible-deploy
            env: production
          values:
            - - "{{ ansible_date_time.iso8601 }}000000000"
              - "{{ lookup('env','USER') }} ran {{ playbook_dir | basename }}"
  delegate_to: localhost
  run_once: true

For deeper integration with the observability stack, see the Logging and Monitoring articles discussing centralized log patterns and alerting based on infrastructure events.

Summary #

• Least privilege: Ansible users should only have the rights truly needed — create per-command specific sudoers, not ALL=(ALL) NOPASSWD: ALL. Choose the privilege escalation model that fits the context: sudo for OS-level granularity, doas for concise configuration, become_user for tasks as a specific service user.
• Separate access per environment with different SSH keys and vault passwords — a leaked staging credential doesn’t endanger production.
• Pipeline with security gates: lint → dry-run → review → staging → integration test → manual approval → production. Each gate catches a different class of problems.
• Mandatory dry run before every production deployment — integrate --check --diff into the CI/CD pipeline as an automatic, unbypassable gate.
• Code review is a security control, not just a quality control — use a review checklist specific to things that escape automated tools (privilege, secrets, blast radius).
• Manual approval for production deployments in CI/CD pipelines — automation is fine, but a human must approve changes to production. The deploy button in the pipeline UI is a simple but effective control.
• Periodic credential rotation with a safe order — add the new one, verify, then remove the old one. SSH keys, vault passwords, service accounts, and TLS certificates all need scheduled rotation.
• Audit trail through Ansible logs, deployment logs, and integration with centralized log systems — every change to production must be traceable: who, what, when, and from where.

← Previous: Encryption Next: Secret Management →