Secret Management #

When managing modern infrastructure, we face the big challenge of keeping sensitive data confidential, such as database passwords, API keys, SSL certificates, and private keys. Ansible Vault provides a good initial solution by encrypting files inside the Git repository. However, as teams grow and infrastructure complexity increases, full reliance on Ansible Vault often creates new problems. We must share the same vault password with all developers, re-commit every time we rotate credentials, and we have no audit log recording who accessed the secret.

To overcome these challenges, we need a more dynamic and centralized solution through external secret managers like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. By integrating Ansible with an external secret manager, we can do runtime injection (injecting secrets directly into memory while the playbook runs) without ever storing them on disk, apply dynamic secrets with short Time-to-Live (TTL), and protect CI/CD logs from sensitive data leaks.

Secret Lifecycle: From Create to Revoke #

Before choosing and configuring the tools we’ll use, we must understand the ideal secret lifecycle in secure infrastructure management. Many operations teams manage secrets reactively — manually creating passwords when needed, storing them anywhere, and leaving them active forever without ever changing them. A mature secret lifecycle must have at least five main phases running continuously:

flowchart LR
    A["1. Create<br/>generate secret<br/>with high entropy"] --> B["2. Store<br/>save in secret manager<br/>with at-rest encryption"]
    B --> C["3. Use<br/>inject at runtime<br/>no permanent env"]
    C --> D["4. Rotate<br/>change periodically<br/>or automatically"]
    D --> E["5. Revoke<br/>remove access<br/>and audit trail"]
    E --> A

    style A stroke:#4a90e2,stroke-width:2px
    style B stroke:#7b68ee,stroke-width:2px
    style C stroke:#50c878,stroke-width:2px
    style D stroke:#f5a623,stroke-width:2px
    style E stroke:#d0021b,stroke-width:2px

These phases must be applied to every credential in our production environment. For example, when creating credentials for a new database: we generate a random password with high entropy (phase 1), store it in a secure encrypted storage system (phase 2), fetch and inject it directly into Ansible memory during application deployment (phase 3), rotate the password value periodically (phase 4), and finally revoke access rights and delete the data when the application is decommissioned (phase 5). If any of these phases is neglected, the attack surface on our systems increases significantly.

Ansible Vault vs External Secret Manager #

Before deciding to switch from or combine Ansible Vault with an external secret manager, we need to analyze the fundamental differences between the two. The following table compares various important dimensions we must consider:

We need to realize that the main difference here isn’t encryption strength. Both Ansible Vault and external secret managers use very strong encryption standards. The real differentiator lies in audit control, granular access management (RBAC), token lifetime (TTL), and the ability to rotate secrets automatically without manual intervention. For production systems subject to strict compliance rules (like SOC2, PCI-DSS, or HIPAA), integration with an external secret manager is an absolute requirement.

HashiCorp Vault with Ansible #

HashiCorp Vault is one of the most popular choices for storing secrets in multi-cloud and hybrid environments. Vault provides advanced functionality like AppRole for system-to-system authentication, dynamic secrets, and very detailed audit logging.

Installation and Dependency Setup #

To start using HashiCorp Vault with Ansible, we need to install the community collection providing the lookup plugin plus the hvac Python library on our control node:

# Installing the Ansible Galaxy collection for HashiCorp Vault
ansible-galaxy collection install community.hashi_vault

# Installing the hvac Python module required by the lookup plugin
pip install hvac

Token Request and Secret Retrieval Flow #

It’s very important to understand how Ansible communicates with Vault to fetch secrets behind the scenes. The following sequence diagram illustrates the token exchange and data retrieval flow:

sequenceDiagram
    participant Play as "Ansible Playbook"
    participant LP as "Lookup Plugin<br/>community.hashi_vault"
    participant Auth as "Vault Auth Method<br/>(AppRole/Kubernetes/AWS)"
    participant Core as "Vault Core"
    participant SecEng as "Secret Engine<br/>(KV v2/Database/AWS)"

    Play->>LP: "lookup('hashi_vault', secret=..., url=..., auth_method=approle)"
    LP->>Auth: "POST /v1/auth/approle/login<br/>(role_id + secret_id)"
    Auth-->>LP: "client_token (with TTL)"
    LP->>Core: "SET X-Vault-Token: <token>"
    LP->>SecEng: "GET /v1/secret/data/myapp/database"
    SecEng->>SecEng: "Check policy — can this<br/>token read this path?"
    SecEng-->>LP: "{ data: { username, password } }"
    LP-->>Play: "Return secret value"
    Note over Play,SecEng: "Secret only exists in memory<br/>while the task runs"
    Play->>Play: "Use the secret for template/configure"
    Note over Play: "Secret variables are scoped<br/>to the task block, cleared after"

From the diagram above, we can draw several important points:

1. Secure Authentication: Ansible uses AppRole (Role ID and Secret ID) for the initial login. Vault then returns a short-lived access token.
2. Runtime Memory: The secret returned by Vault is only stored in Ansible’s execution memory and is never written to permanent storage media on either the control node or managed nodes.
3. Policy Checking: Every secret access is validated based on the access rights attached to that token, limiting illegal access even if the token is somehow leaked.

Playbook Implementation Example with AppRole #

Below is an example of how to write a playbook that dynamically fetches database credentials from the KV (Key-Value) v2 engine in HashiCorp Vault:

- name: Deploy the application with credentials from HashiCorp Vault
  hosts: appservers
  gather_facts: false
  vars:
    # Reading authentication info from environment variables on the control node
    vault_role_id: "{{ lookup('env', 'VAULT_ROLE_ID') }}"
    vault_secret_id: "{{ lookup('env', 'VAULT_SECRET_ID') }}"
    vault_addr: "https://vault.company.internal:8200"

  tasks:
    - name: Fetch the secret data from Vault
      set_fact:
        db_secrets: "{{ lookup('community.hashi_vault.hashi_vault',
          'secret=secret/data/myapp/database:data',
          url=vault_addr,
          auth_method='approle',
          role_id=vault_role_id,
          secret_id=vault_secret_id) }}"
      no_log: true  # Prevent secret values from being written to stdout and log files

    - name: Configure the database.conf file
      template:
        src: templates/database.conf.j2
        dest: /etc/app/database.conf
        owner: appuser
        group: appuser
        mode: '0600'
      vars:
        db_username: "{{ db_secrets.username }}"
        db_password: "{{ db_secrets.password }}"
      no_log: true

We must always include the no_log: true attribute on every task that processes or displays secrets. Without this flag, if an error or execution failure occurs, Ansible prints the object contents to the standard log, so our production secrets could be visible to anyone with access to the CI/CD system.

Dynamic Secrets: Credentials with Self-Destruct #

HashiCorp Vault’s most advanced feature is its ability to create dynamic secrets. Instead of using the same static database password forever, Vault can generate a unique database user for Ansible with limited access rights and a short TTL. When the TTL expires, Vault automatically removes that user from the database.

- name: Fetch dynamic database credentials from Vault
  set_fact:
    db_dynamic_creds: "{{ lookup('community.hashi_vault.hashi_vault',
      'database/creds/myapp-writer',
      url=vault_addr,
      auth_method='approle',
      role_id=vault_role_id,
      secret_id=vault_secret_id) }}"
  no_log: true
  vars:
    # Requesting a special 30-minute TTL for this deployment
    vault_secret_ttl: "30m"

With dynamic secrets, we permanently eliminate the risk of long-term credential leaks. Even if those credentials are accidentally exposed, their very short active lifetime minimizes the impact of outside exploitation.

AWS Secrets Manager with Ansible #

For those of us operating infrastructure exclusively in AWS, AWS Secrets Manager is a highly integrated solution. We don’t need to manage a backend server ourselves and can leverage native integration with AWS IAM.

AWS Setup and Dependency Installation #

We need the amazon.aws collection plus the boto3 and botocore Python modules to communicate with the AWS API:

# Installing the AWS collection for Ansible
ansible-galaxy collection install amazon.aws

# Installing the AWS SDK library for Python
pip install boto3 botocore

Fetching Secrets Using an IAM Role (Passwordless Control) #

Instead of hardcoding AWS access keys into variables, the best practice we should apply is using an IAM Instance Profile on the EC2 Instance acting as the Ansible control node. This way, the amazon.aws.aws_secret lookup plugin automatically detects temporary credentials.

Here’s an example playbook for fetching RDS secrets:

- name: Fetch RDS credentials from AWS Secrets Manager
  hosts: appservers
  gather_facts: false
  tasks:
    - name: Fetch the secret JSON from Secrets Manager
      set_fact:
        rds_raw_secret: "{{ lookup('amazon.aws.aws_secret',
          'prod/myapp/rds',
          region='ap-southeast-1',
          on_missing='error') | from_json }}"
      no_log: true

    - name: Render the database configuration file on the target host
      template:
        src: templates/app_config.yml.j2
        dest: /var/www/app/config.yml
        owner: www-data
        group: www-data
        mode: '0600'
      vars:
        db_host: "{{ rds_raw_secret.host }}"
        db_user: "{{ rds_raw_secret.username }}"
        db_pass: "{{ rds_raw_secret.password }}"
      no_log: true

We’re advised to group secrets in JSON format in AWS Secrets Manager so we can fetch all the key-value pairs we need in a single API call, saving on AWS Secrets Manager API call operational costs.

SOPS: File Encryption for GitOps #

Mozilla SOPS (Secrets OPerationS) offers a different approach from centralized secret managers. SOPS is an encrypted file editor supporting YAML, JSON, ENV, and binary formats. SOPS’s main advantage is that it only encrypts the value of a YAML file, while the file structure keys remain in plaintext. This makes code review much easier because the configuration file structure stays visually readable without leaking the secret values.

Additionally, SOPS doesn’t store encryption keys in the repository. Those encryption keys are delegated to external providers like AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault, PGP, or age.

SOPS Encrypted YAML File Illustration #

Below is an example of the file appearance difference before and after we encrypt using SOPS:

# File: group_vars/production/secrets.sops.yml
db_host: "prod-db.internal"           # still readable (plaintext)
db_user: "app_prod"                   # still readable (plaintext)
db_password: ENC[AES256_GCM,data:3XqBf8==,tag:1Xz...,type:str] # encrypted

SOPS Integration into Ansible Playbooks #

To integrate it with Ansible, we can use the community.sops lookup plugin:

- name: Read sensitive data from a SOPS encrypted file
  hosts: appservers
  gather_facts: false
  tasks:
    - name: Load SOPS encrypted variables into memory
      set_fact:
        decrypted_secrets: "{{ lookup('community.sops.sops', 'group_vars/production/secrets.sops.yml') | from_yaml }}"
      no_log: true

    - name: Run application configuration with secrets
      template:
        src: templates/app.env.j2
        dest: /opt/myapp/.env
        mode: '0600'
      vars:
        app_db_password: "{{ decrypted_secrets.db_password }}"
      no_log: true

This pattern is ideal for GitOps workflows, where we store all configuration files in a Git repository, but the values remain securely protected using tightly controlled KMS encryption keys.

Pattern: Runtime Injection vs Permanent Environment #

One of the most fatal architecture mistakes we often find is leaving secrets permanently stored in target OS environment variables (for example via /etc/environment or .bashrc files).

# ANTI-PATTERN: Storing secrets permanently in server environment variables
# File: /etc/environment (✗ HIGHLY NOT RECOMMENDED)
DATABASE_PASSWORD="VerySecretPassword123!"
THIRD_PARTY_API_TOKEN="api_token_val_abc_xyz"

# Why is this dangerous?
# 1. All child processes inherit these variables, including text editors and schedulers.
# 2. Env var values can be easily read via the '/proc/<pid>/environ' file by non-root users.
# 3. System monitoring logs (monitoring agents) often record all env vars when detecting crashes.
# 4. It makes quick credential rotation very difficult.

The correct solution is applying Runtime Injection. We store secrets in a centralized secret manager, fetch them on-demand using Ansible lookup plugins when the playbook runs, inject them into the application’s local configuration file with restricted access rights (0600), then let Ansible’s memory be cleared when the playbook execution process finishes.

# CORRECT: Runtime injection into a tightly permissioned local configuration file
- name: Apply runtime injection for the database configuration
  hosts: db_clients
  gather_facts: false
  tasks:
    - name: Fetch the password dynamically
      set_fact:
        fetched_pass: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/data/db:password', ...) }}"
      no_log: true

    - name: Write to the application configuration file with a safe mode
      template:
        src: templates/local_conf.j2
        dest: /etc/myapp/local.conf
        owner: myapp_runner
        group: myapp_runner
        mode: '0600'  # Only the owner has read and write permissions
      vars:
        secret_db_pass: "{{ fetched_pass }}"
      no_log: true

With the runtime injection model, we ensure those sensitive variables don’t linger in the OS’s global environment, which is vulnerable to information leakage.

Pattern: Automatic Rotation vs No Rotation #

Storing secrets safely in a secret manager isn’t enough if we leave those credentials active forever without ever changing them. Many teams experience security breaches not because their encryption algorithm was broken, but because their static credentials leaked unnoticed and stayed active for years.

# ANTI-PATTERN: Static secrets without a rotation scheme
# The prod/db-password database credential was created in 2022
# The password value is still the same in 2026
# No reminders, no rotation automation
# The credential has been stored on a former engineer's laptop, old logs, and tape backups

We must design automation to replace secrets periodically (for example every 30 days). In AWS environments, we can configure AWS Secrets Manager to trigger an AWS Lambda function that updates the password on the target database while atomically updating the secret value in Secrets Manager:

# Example AWS Lambda script for automatic RDS password rotation (rotation_lambda.py)
import boto3
import json
import secrets
import string

def generate_strong_password(length=32):
    alphabet = string.ascii_letters + string.digits + "!@#$%^&*()"
    return "".join(secrets.choice(alphabet) for _ in range(length))

def lambda_handler(event, context):
    secrets_client = boto3.client('secretsmanager')
    rds_client = boto3.client('rds')
    
    secret_arn = event['SecretId']
    token = event['ClientRequestToken']
    
    # 1. Generate a new password with high entropy
    new_password = generate_strong_password()
    
    # 2. Update the password on the target database instance
    rds_client.modify_db_instance(
        DBInstanceIdentifier='myapp-prod-db',
        MasterUserPassword=new_password,
        ApplyImmediately=True
    )
    
    # 3. Update the secret value in AWS Secrets Manager
    secrets_client.put_secret_value(
        SecretId=secret_arn,
        ClientRequestToken=token,
        SecretString=json.dumps({
            'username': 'db_admin',
            'password': new_password
        }),
        VersionStages=['AWSCURRENT']
    )
    
    return {'statusCode': 200, 'body': 'Rotation successful'}

By integrating this script with AWS Secrets Manager, we ensure our database passwords rotate automatically without requiring manual intervention from infrastructure administrators.

RBAC: The Least Privilege Principle for Secrets #

External secret managers offer far more advanced role-based access control (RBAC) than simply encrypting files with one password. We can restrict access very granularly so each entity only has the minimum permissions they truly need to perform their function.

For example, we can define a policy in HashiCorp Vault using the HCL format like below:

# File: policies/ansible-app-deploy.hcl
# Allow Ansible to read secrets for prod web application deployment
path "secret/data/myapp/production/*" {
  capabilities = ["read"]
}

# Allow Ansible to fully read secrets for staging (read and list)
path "secret/data/myapp/staging/*" {
  capabilities = ["read", "list"]
}

# Restrict access to sensitive database root data
path "secret/data/myapp/production/db-root" {
  capabilities = [] # Explicit deny
}

On the AWS side, we can restrict the Ansible service account’s access rights using an IAM Policy with strict resource condition limits:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAnsibleAppSecretsOnly",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:prod/myapp/*"
    },
    {
      "Sid": "DenyRootDbAccess",
      "Effect": "Deny",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:prod/myapp/db-root-*"
    }
  ]
}

Applying policies like this diligently minimizes security risk if one of our credentials or control nodes is compromised. Attackers only get limited access to a specific infrastructure segment, not our entire server fleet.

Secret Scanning: Detecting Leaks Early #

Even after migrating to a centralized secret manager, the possibility of accidentally committing secrets to the Git repository remains. Therefore, we need defense in depth by applying automatic scanning tools (secret scanning) like Gitleaks or Trufflehog.

We can add this scanning step directly into our CI/CD pipeline to analyze every push commit:

# File: .github/workflows/secret-scanning.yml
name: Secret Scanning Guard
on: [push, pull_request]

jobs:
  gitleaks_scan:
    name: Run Gitleaks Scan
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Fetch the entire commit history for analysis

      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

For local work, we can also force pre-commit scanning on developer laptops before the commit is successfully created:

# Running a manual scan in our local project directory
gitleaks detect --source . --verbose

# Analyzing the entire git commit history to detect old leaks
gitleaks detect --source . --log-opts="--all"

If a secret has already made it into our Git history, the first step we must take isn’t deleting the file and making a new commit (because the secret still exists in the old commit history). We must immediately rotate those credentials at the target backend, then use BFG Repo-Cleaner or the git filter-repo command to permanently clean those credentials from our entire Git history.

When to Keep Using Ansible Vault #

Although external secret manager systems offer high reliability, for certain cases, standalone Ansible Vault usage can still be justified. The following table summarizes guidelines to help us determine the best choice:

A hybrid pattern is also very common in industry: we use Ansible Vault to store initial bootstrap secrets (like API keys or access tokens Ansible uses to connect to AWS Secrets Manager or HashiCorp Vault), then use the external secret manager to fetch all application runtime secrets dynamically. This pattern limits credential exposure risk while keeping our operational management easy.

Integration with Configuration Management #

Let’s review a real implementation example combining all the best practices above into our web application deployment playbook. Here we use structured blocks, handle exceptions (rescue), set strict file ownership, and disable sensitive log recording.

# File: site-deploy.yml
- name: Deploy the web application with secure secret management
  hosts: production_servers
  gather_facts: false
  vars:
    vault_addr: "https://vault.company.internal:8200"

  tasks:
    - name: Secure the deployment using a control block
      block:
        - name: Load the role ID and secret ID from the control node environment
          set_fact:
            role_id: "{{ lookup('env', 'VAULT_ROLE_ID') }}"
            secret_id: "{{ lookup('env', 'VAULT_SECRET_ID') }}"
          no_log: true

        - name: Fetch the database secrets and API key from Vault
          set_fact:
            vault_data: "{{ lookup('community.hashi_vault.hashi_vault',
              'secret=secret/data/myapp/production:data',
              url=vault_addr,
              auth_method='approle',
              role_id=role_id,
              secret_id=secret_id) }}"
          no_log: true

        - name: Create the .env configuration file with strict access permissions
          template:
            src: templates/env.j2
            dest: /var/www/app/.env
            owner: app_runner
            group: app_runner
            mode: '0600'  # Owner read and write only
          vars:
            app_db_password: "{{ vault_data.db_password }}"
            app_api_token: "{{ vault_data.api_token }}"
          no_log: true

        - name: Ensure the application service is running
          service:
            name: myapp
            state: restarted
          become: true

      rescue:
        - name: Handle deployment errors without leaking secrets
          debug:
            msg: "Deployment to host {{ inventory_hostname }} failed. Please check the application logs or connectivity to the secret manager."

In the example above, if the credential fetching process fails or the configuration file can’t be created, the playbook execution is redirected to the rescue block which displays a generic error message without exposing internal error codes that could potentially leak our technical configuration.

Summary #

• Ansible Vault Limitations — Although reliable for local Git file encryption, Ansible Vault doesn’t support dynamic secrets, call audit logs, granular RBAC restrictions, or automatic rotation.
• Secret Lifecycle — We must ensure every credential passes through the five lifecycle stages: Create, Store, Use, Rotate, and Revoke diligently.
• HashiCorp Vault & AppRole — We’re advised to use AppRole authentication for system-to-system automation, which provides access tokens with safe time limits (TTL).
• AWS Secrets Manager & IAM — For AWS-native deployments, leverage an IAM Instance Profile on the control node so we can look up secrets without storing static AWS access credentials.
• SOPS Encrypted Files — Mozilla SOPS is highly favored in GitOps architectures because it allows encrypting only variable values while the keys stay readable in Git.
• Runtime Injection — Apply secret injection directly into memory while the playbook runs, and avoid storing secrets in permanent target OS environment variables.
• Automatic Rotation — Prevent long-term leak threats by enabling periodic automatic rotation using Lambda or secret manager cron engines.
• Least Privilege Policy — Configure minimal access policies (RBAC) on the secret manager so each playbook can only read data relevant to its task scope.
• Secret Scanning — Install pre-commit hooks and pipeline checks using Gitleaks to prevent accidentally committing plaintext credentials into our Git history.
• The no_log: true Attribute — Always include the no_log: true flag on every Ansible task that reads or touches secret data to protect our CI/CD system log files.

← Previous: Workflow Next: SSH Security →