Dynamic Inventory #

Static inventory in the form of hosts.ini or YAML files works well for small, stable infrastructures. But once we manage a cloud environment with hundreds of instances spinning up and terminating every day, or a hybrid infrastructure where some hosts are in AWS, some on-premise, some in a CMDB — maintaining inventory files manually becomes an impossible, error-prone job. Dynamic inventory solves this problem: Ansible takes the host list from authoritative sources — the AWS API, GCP Compute, Azure ARM, vCenter, or an internal CMDB — every time a playbook runs, so the inventory always reflects the current infrastructure state.

This article discusses three forms of dynamic inventory in Ansible: inventory plugins (the modern, YAML-based, most recommended way), inventory scripts (legacy, executables returning JSON), and how to combine multiple inventory sources into one coordinated ecosystem.

Why Dynamic Inventory Matters #

Imagine this scenario: the SRE team deploys 50 new EC2 instances for an auto-scaling group, deletes 30 old instances, and changes the Environment tag on 20 existing instances. With static inventory, we must manually update the hosts.ini file — a non-scalable, error-prone job. With dynamic inventory, Ansible calls the AWS API every time a playbook runs and gets the latest state in real-time.

flowchart LR
    A["Playbook: ansible-playbook -i aws_ec2.yml"] --> B["Ansible reads the inventory file"]
    B --> C{"YAML or executable?"}
    C -->|YAML| D["Inventory plugin"]
    C -->|Executable| E["Inventory script"]
    D --> F["AWS API / GCP API / Azure API"]
    E --> F
    F --> G["JSON host data"]
    G --> H["Ansible populates the inventory"]
    H --> I["keyed_groups & compose"]
    I --> J["Groups: role_webserver, env_production, etc."]
    J --> K["The playbook runs with the right hosts"]

The advantages of dynamic inventory over static:

Always up-to-date — Ansible sees the infrastructure state when the playbook runs, not when the file was last edited.
Self-service — developers can deploy new instances without having to tell SRE to update the inventory.
Tag-based grouping — groups are created automatically from cloud tags (e.g., tag:Role=webserver → group role_webserver).
Large scale — hundreds to thousands of hosts handled without manual intervention.
Hybrid-ready — combine inventories from AWS, GCP, and CMDB in one directory.

Inventory Plugins: The Modern Way #

Inventory plugins are the most recommended way to do dynamic inventory. Plugins are YAML-based, installed via collections, and have built-in features like caching, keyed_groups, compose, and filters. Ansible automatically calls the right plugin based on the file name and the plugin: configuration in the first line.

AWS EC2 with `amazon.aws.aws_ec2` #

The amazon.aws collection provides the aws_ec2 plugin that fetches instances directly from the AWS API:

# Install dependencies
ansible-galaxy collection install amazon.aws
pip install boto3

# inventory/aws_ec2.yml
# The file name MUST end with aws_ec2.yml or aws_ec2.yaml

plugin: amazon.aws.aws_ec2

# Regions to scan
regions:
  - ap-southeast-1      # Singapore
  - ap-southeast-3      # Jakarta
  - us-east-1           # Virginia

# Filter the instances to fetch
filters:
  instance-state-name: running
  "tag:Environment": production

# Fields used as the Ansible hostname
hostnames:
  - private-ip-address

# Automatic groups based on tags
keyed_groups:
  - prefix: role
    key: tags.Role          # Groups: role_webserver, role_database, etc.
  - prefix: env
    key: tags.Environment   # Groups: env_production, env_staging
  - prefix: az
    key: placement.availability_zone

# Groups based on custom conditions
groups:
  singapore: "'ap-southeast-1' in placement.region"
  jakarta: "'ap-southeast-3' in placement.region"
  high_memory: "instance_type.startswith('r5') or instance_type.startswith('r6')"

# Map AWS fields to Ansible variables
compose:
  ansible_host: private_ip_address
  ansible_user: "'ec2-user' if 'Windows' not in image_id else 'Administrator'"
  instance_type: instance_type
  availability_zone: placement.availability_zone
  vpc_id: vpc_id

# Variables for all hosts
groups:
  aws_production:
    vars:
      ansible_ssh_private_key_file: /opt/keys/aws-prod.pem

# Test the inventory before using it
ansible-inventory -i inventory/aws_ec2.yml --list
ansible-inventory -i inventory/aws_ec2.yml --graph

# --graph output:
# @all:
#   |--@aws_production:
#   |   |--@env_production:
#   |   |   |--@ap_southeast_1:
#   |   |   |   |--ip-10-0-1-10.ap-southeast-1.compute.internal
#   |   |   |   |--ip-10-0-1-11.ap-southeast-1.compute.internal
#   |   |   |--@ap_southeast_3:
#   |   |   |   |--ip-10-0-2-5.ap-southeast-3.compute.internal
#   |   |--@role_webserver:
#   |   |   |--ip-10-0-1-10.ap-southeast-1.compute.internal

# Use in a playbook
ansible-playbook -i inventory/aws_ec2.yml site.yml

GCP Compute with `google.cloud.gcp_compute` #

# inventory/gcp_compute.yml
plugin: google.cloud.gcp_compute

projects:
  - my-production-project
  - my-shared-services-project

zones:
  - asia-southeast2-a    # Jakarta
  - asia-southeast2-b
  - asia-southeast2-c

# Filter instances — use GCP JMESPath syntax
filters:
  - status = RUNNING
  - labels.environment = production
  - labels.managed_by = ansible

# Automatic groups
keyed_groups:
  - prefix: role
    key: labels.role
  - prefix: zone
    key: zone
  - prefix: env
    key: labels.environment

# Map GCP fields to Ansible variables
compose:
  ansible_host: networkInterfaces[0].networkIP
  ansible_user: "'ubuntu' if 'debian' in labels.base_os else 'centos'"
  machine_type: machineType
  project: project

Azure with `azure.azcollection.azure_rm` #

# inventory/azure_rm.yml
plugin: azure.azcollection.azure_rm

include_vm_resource_groups:
  - production-rg
  - staging-rg

# Filters
filters:
  - power_state == 'running'
  - tags.environment == 'production'

# Groups
keyed_groups:
  - prefix: role
    key: tags.role
  - prefix: env
    key: tags.environment

# Compose
compose:
  ansible_host: properties.networkProfile.networkInterfaces[0].properties.ipConfigurations[0].properties.privateIPAddress

Inventory Source Comparison Table #

Aspect	Inventory Plugin (AWS/GCP/Azure)	Inventory Script (Python)	Static File (YAML/INI)
Format	YAML	Executable returning JSON	YAML or INI
Caching	Built-in, configurable	Manual, must implement yourself	Not needed
keyed_groups	Native	Must be generated manually	Manual
compose	Native	Must be handled manually	Manual
Filters	Native (JMESPath/YAML)	Full Python logic	Not applicable
Performance (1000 hosts)	1 API call (cached)	1 API call per run	Instant
Reusability	Sharing via collection	Sharing via internal repo	One project
Testing	`ansible-inventory --list`	Manual script execution	Look at the file
Recommended for	Cloud, hybrid, modern	Custom CMDB, legacy systems	< 50 stable hosts

The `ansible-inventory` Workflow #

Every time we run ansible-playbook -i inventory/, Ansible calls ansible-inventory behind the scenes. Our plugin or script is executed to produce a JSON representation of the inventory. The following diagram shows the complete flow:

sequenceDiagram
    participant User
    participant AP as "ansible-playbook"
    participant AI as "ansible-inventory"
    participant Plugin as "aws_ec2 plugin"
    participant AWS as "AWS EC2 API"
    participant Cache as "JSON Cache"

    User->>AP: "ansible-playbook -i aws_ec2.yml site.yml"
    AP->>AI: "Load inventory"
    AI->>Plugin: "parse(config)"
    Plugin->>Cache: "Is the cache valid?"
    alt Cache valid
        Cache-->>Plugin: "Return cached data"
    else Cache expired/missing
        Plugin->>AWS: "DescribeInstances(filters)"
        AWS-->>Plugin: "List of instances"
        Plugin->>Cache: "Write cache"
    end
    Plugin->>Plugin: "Apply keyed_groups"
    Plugin->>Plugin: "Apply compose"
    Plugin-->>AI: "Inventory dict"
    AI-->>AP: "Populated inventory"
    AP->>AP: "Run tasks on matched hosts"

Caching is an important feature for inventory plugins: without caching, every ansible-playbook call triggers an API call to AWS, which can hit rate limits if the playbook runs from CI on every push. Enable caching in ansible.cfg:

# ansible.cfg
[inventory]
cache = true
cache_plugin = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout = 300   # Cache is valid for 5 minutes

Writing Your Own Inventory Script #

For inventory sources that don’t have a built-in plugin — internal CMDBs, legacy databases, monitoring systems exposing host data — we can write a Python inventory script returning JSON in the format Ansible expects. The script accepts the --list flag (all hosts) and --host <hostname> (variables for one host).

Script Anatomy #

#!/usr/bin/env python3
# inventory/cmdb_inventory.py
# Fetch the inventory from the company's internal CMDB

import json
import sys
import argparse
import os

try:
    import requests
except ImportError:
    print(json.dumps({"_meta": {"hostvars": {}}}))
    sys.exit(0)

CMDB_URL = os.environ.get(
    'CMDB_URL', 'https://cmdb.company.internal/api'
)
CMDB_TOKEN = os.environ.get('CMDB_API_TOKEN', '')


def get_inventory():
    """Fetch all servers from the CMDB and group them."""
    if not CMDB_TOKEN:
        sys.stderr.write(
            "ERROR: The environment variable CMDB_API_TOKEN has not been set.\n"
        )
        return {"_meta": {"hostvars": {}}}

    headers = {"Authorization": f"Bearer {CMDB_TOKEN}"}

    try:
        response = requests.get(
            f"{CMDB_URL}/servers",
            params={"status": "active"},
            headers=headers,
            timeout=15
        )
        response.raise_for_status()
        servers = response.json()
    except requests.exceptions.RequestException as e:
        sys.stderr.write(f"Error fetching the inventory from the CMDB: {e}\n")
        return {"_meta": {"hostvars": {}}}

    inventory = {
        "_meta": {"hostvars": {}},
        "all": {"children": []},
    }

    for server in servers:
        hostname = server["hostname"]
        env = server.get("environment", "unknown")
        role = server.get("role", "unknown")
        datacenter = server.get("datacenter", "unknown")

        # Host variables
        inventory["_meta"]["hostvars"][hostname] = {
            "ansible_host": server.get("ip_address", hostname),
            "server_id": server.get("id"),
            "datacenter": datacenter,
            "os": server.get("os"),
            "ansible_user": "ubuntu" if "ubuntu" in server.get("os", "") else "centos",
        }

        # Group by environment
        env_group = f"env_{env}"
        if env_group not in inventory:
            inventory[env_group] = {
                "hosts": [],
                "vars": {"env": env},
            }
            inventory["all"]["children"].append(env_group)
        inventory[env_group]["hosts"].append(hostname)

        # Group by role
        role_group = f"role_{role}"
        if role_group not in inventory:
            inventory[role_group] = {"hosts": []}
            inventory["all"]["children"].append(role_group)
        inventory[role_group]["hosts"].append(hostname)

        # Group by datacenter
        dc_group = f"dc_{datacenter}"
        if dc_group not in inventory:
            inventory[dc_group] = {"hosts": []}
            inventory["all"]["children"].append(dc_group)
        inventory[dc_group]["hosts"].append(hostname)

    return inventory


def get_host(hostname):
    """Fetch the variables for one host."""
    headers = {"Authorization": f"Bearer {CMDB_TOKEN}"}
    try:
        response = requests.get(
            f"{CMDB_URL}/servers/{hostname}",
            headers=headers,
            timeout=10
        )
        response.raise_for_status()
        server = response.json()
        return {
            "ansible_host": server.get("ip_address", hostname),
            "server_id": server.get("id"),
            "datacenter": server.get("datacenter"),
            "os": server.get("os"),
        }
    except requests.exceptions.RequestException:
        return {}


def main():
    parser = argparse.ArgumentParser(description='CMDB Inventory Script')
    parser.add_argument('--list', action='store_true',
                        help='Display the full inventory')
    parser.add_argument('--host', type=str,
                        help='Display variables for a specific host')
    args = parser.parse_args()

    if args.list:
        print(json.dumps(get_inventory(), indent=2))
    elif args.host:
        print(json.dumps(get_host(args.host), indent=2))
    else:
        print(json.dumps({}))


if __name__ == '__main__':
    main()

# Make it executable
chmod +x inventory/cmdb_inventory.py

# Manual test
./inventory/cmdb_inventory.py --list
./inventory/cmdb_inventory.py --host web-01.company.com

# Use in a playbook
ansible-playbook -i inventory/cmdb_inventory.py site.yml

Tip — Always exit with valid JSON, even when an error occurs. If our script crashes with a Python stack trace, Ansible shows a cryptic error. Better to return an empty {"_meta": {"hostvars": {}}} and write the error to stderr so the playbook can decide whether to continue or stop.

Anti-Pattern: Inventory Scripts Returning All Hosts Without Filters #

# ANTI-PATTERN: fetch all hosts, let the playbook filter
def get_inventory():
    response = requests.get(f"{CMDB_URL}/servers")  # 5000 hosts!
    servers = response.json()
    inventory = {"_meta": {"hostvars": {}}, "all": {"hosts": []}}
    for s in servers:
        inventory["all"]["hosts"].append(s["hostname"])
    return inventory

# CORRECT: filter at the source, expose only relevant hosts
def get_inventory():
    params = {"status": "active", "managed_by": "ansible"}
    response = requests.get(f"{CMDB_URL}/servers", params=params)
    servers = response.json()
    # ... build the inventory with meaningful groups
    return inventory

# Usage: use --limit for additional filtering
ansible-playbook -i inventory/cmdb_inventory.py site.yml --limit env_production

The consequence of the anti-pattern above: the playbook loads 5000 hosts into memory only to run tasks on 50 hosts in production. The all group balloons, the --list output becomes uninformative, and filtering must be done at the playbook layer (which should have happened earlier).

Warning — An inventory script returning all hosts without filters can make the playbook hang or run out of memory on large infrastructures. Always filter at the source (CMDB/API) and expose already-relevant groups. Use --limit for additional filtering at run time, but don’t rely on it as the main strategy.

Anti-Pattern: Hard-Coding Inventory in Playbooks #

# ANTI-PATTERN: hard-code hosts in the playbook
- name: "Deploy the web app"
  hosts: web-01.company.com,web-02.company.com,web-03.company.com
  tasks: [...]

# CORRECT: use a group from dynamic inventory
- name: "Deploy the web app"
  hosts: role_webserver
  tasks: [...]

Hard-coding hosts in a playbook binds the playbook to the current infrastructure state. When new instances are added or removed, the playbook must be re-edited — a non-scalable pattern. Dynamic inventory + group-based selection makes playbooks agnostic to the host count: run for 3 hosts today, 30 hosts tomorrow, without editing a single line.

Combining Multiple Inventory Sources #

Ansible can combine inventories from several sources at once. Just point -i at a directory, and Ansible automatically loads all files matching the *.yml, *.yaml, *.ini patterns, or executables ending without an extension:

inventory/
├── aws_ec2.yml              # Servers in AWS
├── gcp_compute.yml          # Servers in GCP
├── cmdb_inventory.py        # On-premise servers from the CMDB
├── static_hosts.ini         # Special servers not in the cloud/CMDB
└── group_vars/
    ├── all.yml              # Variables for all hosts
    ├── role_webserver.yml   # Variables for the role_webserver group
    ├── role_database.yml    # Variables for the role_database group
    └── env_production.yml   # Variables for the env_production group

# Ansible automatically combines all sources
ansible-playbook -i inventory/ site.yml

# View the combined inventory
ansible-inventory -i inventory/ --graph

# Output:
# @all:
#   |--@env_production:
#   |   |--@role_database:
#   |   |   |--db-01.aws.internal
#   |   |   |--db-gcp-01.c.internal
#   |   |   |--db-onprem.cmp.internal
#   |   |--@role_webserver:
#   |   |   |--web-01.aws.internal
#   |   |   |--web-02.aws.internal

Loading Order and Precedence #

flowchart TD
    A["Ansible scans the inventory/ directory"] --> B["Load *.yml files"]
    A --> C["Load *.yaml files"]
    A --> D["Load *.ini files"]
    A --> E["Load executable files"]
    B --> F["Merge into the global inventory"]
    C --> F
    D --> F
    E --> F
    F --> G["Apply group_vars/*.yml per group"]
    G --> H["Apply host_vars/*.yml per host"]
    H --> I["Final inventory ready to use"]

Rule precedence (from highest to lowest):

host_vars/<hostname>.yml — per-host variables
group_vars/<groupname>.yml — per-group variables
group_vars/all.yml — variables for all hosts
compose: in the inventory plugin — dynamic variables
hostvars: in the inventory script — dynamic variables
Inline variables in the playbook (vars:)

Info — Files in group_vars/ are automatically associated with a group based on the file name. group_vars/role_webserver.yml is automatically applied to the role_webserver group without any additional configuration. This is a powerful pattern keeping inventory + configuration tidy in one place.

Caching for Large Inventories #

For infrastructures with hundreds to thousands of hosts, calling the AWS API every time a playbook runs is slow and can hit rate limits. Enable caching in ansible.cfg:

# ansible.cfg
[inventory]
cache = true
cache_plugin = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout = 300   # Cache is valid for 5 minutes

# Optional: cache only for specific plugins
cache_plugin_omit = ['hostfile']  # Skip caching for static files

# Manually refresh the cache when needed
ansible-inventory -i inventory/ --list --flush-cache

# View the cache contents
ls -la /tmp/ansible_inventory_cache/
cat /tmp/ansible_inventory_cache/aws_ec2_*.json | python -m json.tool | head -30

The Right Caching Strategy #

Scenario	Cache TTL	Reason
Dev environment, changes often	60 seconds	Want fast changes visible
Staging, deploys every hour	300 seconds (5 minutes)	Balance between freshness and API calls
Production, rarely changes	3600 seconds (1 hour)	Minimize API calls
On-call incident, need fresh data	0 (off) or manual flush	Don’t rely on cache while troubleshooting
CI/CD pipeline	Disabled	Every run needs fresh data

Tip — For CI/CD, disable caching or set a very low TTL. An expired cache can cause the playbook to deploy to already-terminated hosts, or skip newly provisioned hosts. Always flush-cache at the start of critical CI jobs.

Advanced Pattern: Multi-Region and Multi-Account #

For organizations with several AWS accounts or regions, the following inventory pattern helps:

# inventory/aws_production.yml
plugin: amazon.aws.aws_ec2
regions:
  - ap-southeast-1
  - ap-southeast-3
  - us-east-1
filters:
  instance-state-name: running
  "tag:Environment": production
  "tag:ManagedBy": ansible
keyed_groups:
  - prefix: role
    key: tags.Role
  - prefix: env
    key: tags.Environment
  - prefix: account
    key: tags.Account
compose:
  ansible_host: private_ip_address
  aws_account_id: tags.Account

# inventory/aws_staging.yml
plugin: amazon.aws.aws_ec2
regions:
  - ap-southeast-1
filters:
  instance-state-name: running
  "tag:Environment": staging
keyed_groups:
  - prefix: role
    key: tags.Role
groups:
  staging: true

# Choose the environment with -i
ansible-playbook -i inventory/aws_production.yml site.yml   # Production
ansible-playbook -i inventory/aws_staging.yml site.yml      # Staging

Or combine everything in one directory and filter with --limit:

ansible-playbook -i inventory/ site.yml --limit env_production
ansible-playbook -i inventory/ site.yml --limit env_staging

Advanced Pattern: Inventory for Kubernetes #

For Kubernetes clusters, the kubernetes.core.k8s inventory plugin generates inventory from K8s objects (pods, services, nodes):

# inventory/k8s.yml
plugin: kubernetes.core.k8s

connections:
  - kubeconfig: ~/.kube/config
    name: production-cluster

# Fetch nodes from the cluster
# Filter based on labels

A common use: orchestrating workloads on K8s and VMs simultaneously in one playbook.

Testing and Validating Inventory #

Before using inventory in production, always test it first:

# View the inventory as JSON
ansible-inventory -i inventory/aws_ec2.yml --list | jq 'keys'

# View hosts in a specific group
ansible-inventory -i inventory/aws_ec2.yml --list | \
  jq '._meta.hostvars | keys'

# View hosts in the env_production group
ansible-inventory -i inventory/aws_ec2.yml --list | \
  jq '.env_production.hosts'

# View variables for one host
ansible-inventory -i inventory/aws_ec2.yml --host web-01.internal

# Test connectivity to all hosts in a group
ansible all -i inventory/aws_ec2.yml -m ping --limit env_production

# Dry-run the playbook (--check) for validation without execution
ansible-playbook -i inventory/aws_ec2.yml site.yml --check --diff

# Save the inventory to a file for offline inspection
ansible-inventory -i inventory/aws_ec2.yml --list > /tmp/inv_dump.json

# Compare the inventory between two points in time
diff <(jq -S . /tmp/inv_dump.json) <(ansible-inventory -i inventory/aws_ec2.yml --list | jq -S .)

When to Use Static vs Dynamic #

Less than 20 hosts and rarely changing?
  → Static inventory (YAML/INI) is enough
    Example: lab environment, 5 personal VPS

20-100 hosts, changing a few times a month?
  → Semi-dynamic: a simple inventory script or hybrid

100+ hosts, auto-scaling, multi-cloud?
  → Dynamic inventory plugin with caching
    Example: production AWS + GCP + on-premise

Hosts from many sources (cloud + CMDB + files)?
  → An inventory directory with multiple files

Need testing/development without real infrastructure?
  → Static inventory with mock hosts
    Example: localhost, Vagrant, Docker

Summary #

Inventory plugins (AWS EC2, GCP Compute, Azure ARM) are the most recommended way to do dynamic inventory — YAML-based, with native support for caching, keyed_groups, and compose.
Use keyed_groups to automatically create groups from cloud tags — tag:Role=webserver → group role_webserver without additional per-host configuration.
compose maps API fields to Ansible variables — ansible_host: private_ip_address ensures Ansible uses the private IP, not the public one.
Python inventory scripts must support --list (all hosts) and --host <hostname> (host variables) — this is the interface Ansible expects from executables.
Combine multiple sources with -i inventory/dir/ — Ansible automatically loads all files in that directory, then merges them into one inventory.
Enable caching for large inventories — without cache, every ansible-playbook call triggers an API call to the cloud, slow and prone to rate limits.
Filter at the source, not in the playbook — an inventory script returning all hosts without filters balloons memory and makes the --list output uninformative.
Don’t hard-code hosts in playbooks — use groups (hosts: role_webserver) so playbooks are agnostic to the host count.
Always test inventory with ansible-inventory -i <inv> --list and ansible all -m ping --limit <group> before running state-changing playbooks.
For multi-environment setups, use an inventory directory + filtering with --limit or separate inventory files per environment.

← Previous: Custom Plugin Next: Collection →