Cloud #

When you manage infrastructure in a traditional static data center, a physical inventory file (hosts.ini) is a very reliable method. Physical servers rarely scale up or down, and IP addresses tend to stay the same for years. However, this landscape changes dramatically in modern cloud computing environments. With autoscaling features, rapid virtual machine recycling, and on-demand provisioning, virtual machines can be created, stopped, and deleted within minutes. Manually maintaining an IP list in a static inventory file under these dynamic conditions isn’t just tedious — it’s impossible to do without errors. This is where Dynamic Inventory comes in as the standard solution. This article breaks down the need for dynamic inventory, the cloud API communication workflow, AWS, GCP, and Azure integration, and caching optimization techniques to prevent API call rate limit bottlenecks.

The Need for Dynamic Inventory #

Modern cloud infrastructure demands an adaptive automation approach. Imagine an e-commerce system experiencing a traffic spike. The cloud autoscaling module automatically multiplies the number of web server instances from 5 to 20 virtual machines within seconds to distribute the workload.

If you stick with the static inventory method, your automation hits operational dead ends:

  1. Blind to Targets: Ansible won’t know about the 15 new web servers just launched because their IPs aren’t registered in hosts.ini.
  2. Lost Connectivity: Ansible wastes time trying to reach old servers already deleted by autoscaling, triggering spurious failed task statuses.
  3. Release Speed Bottleneck: You’re forced to write extra manual scripts to sync IPs every time a playbook runs.

With dynamic inventory, Ansible no longer reads a passive IP file. Instead, Ansible acts proactively by querying the Cloud Provider API directly just before the playbook runs to ask: “What server instances are currently active?”. The metadata response from that API is then assembled into an instant in-memory inventory.

Cloud API Communication Workflow #

To understand how server data is dynamically identified, the sequence diagram below illustrates the API call lifecycle from the control node to the cloud provider, up to when playbook execution begins:

sequenceDiagram
    participant ControlNode as "Control Node (Ansible Engine)"
    participant Plugin as "aws_ec2 Inventory Plugin"
    participant Cache as "Local Cache (jsonfile)"
    participant CloudAPI as "AWS EC2 Web API"
    participant Targets as "Managed Nodes (EC2 Instances)"

    ControlNode->>Plugin: Run Playbook (Start Inventory Loading)
    Plugin->>Cache: Check Local Cache
    alt "Cache Valid (< 300 seconds)"
        Cache-->>Plugin: Return Cached Instance Data
    else "Cache Expired / Missing"
        Plugin->>CloudAPI: "Send API Request (DescribeInstances)"
        CloudAPI-->>Plugin: "Return JSON Response (VM Metadata)"
        Plugin->>Cache: Save New Response to Local Cache
    end
    Plugin->>Plugin: "Parse Metadata (Filter 'running' Status, Map Tags)"
    Plugin-->>ControlNode: "Return Grouped Inventory (role_webserver, env_production)"
    ControlNode->>Targets: "Run Playbook Automation via SSH"

Through this flow, you no longer need to worry about IP address changes because Ansible always gets the updated server list in real time.

Plugin vs Script Comparison #

In Ansible’s development history, there are two methods for implementing dynamic inventory:

  1. Inventory Scripts (Legacy Method): Standalone executable scripts written in a programming language (like Python or Bash) that Ansible calls to produce standardized JSON-format text output. This method is self-contained but hard to configure, requires complex script code maintenance, and doesn’t natively support Ansible’s built-in caching integration.
  2. Inventory Plugins (Modern / Recommended Method): Internal Ansible plugins (part of the Core ecosystem or Collections) configured using simple YAML files. This method is far safer, more efficient, natively integrated with Ansible’s cache system, and uses standard declarative parameters.

You are strongly recommended to always use Inventory Plugins for every new project because they’re Ansible’s future standard.

AWS EC2 Integration #

To connect Ansible with Amazon Web Services (AWS) dynamically, you use the official amazon.aws.aws_ec2 plugin.

1. Dependency Preparation #

Install the AWS collection from Ansible Galaxy and the official AWS Python SDK (boto3 and botocore) on your control node:

# Activate your virtual environment first if you have one
# source ~/ansible-env/bin/activate

# Install the AWS collection
ansible-galaxy collection install amazon.aws

# Install the AWS SDK library for Python
pip install boto3 botocore

2. AWS Credential Configuration #

Make sure the control node has authentication access to the AWS API. The safest way is using standard AWS environment variables or the AWS CLI credential file (~/.aws/credentials):

export AWS_ACCESS_KEY_ID="AKIAEXAMPLE1234567890"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_DEFAULT_REGION="ap-southeast-1"

3. Creating the Plugin Configuration File #

The configuration file name for the AWS dynamic inventory plugin must end with the special extension aws_ec2.yml or aws_ec2.yaml so it’s recognized automatically by Ansible’s parser engine.

# File: inventory/production/aws_ec2.yml
---
# Declare the plugin being used
plugin: amazon.aws.aws_ec2

# Limit the instance search to specific regions (saves query time)
regions:
  - ap-southeast-1

# Instance filter: only take VMs that are running
filters:
  instance-state-name:
    - running

# Auto-create dynamic groups based on AWS instance tags (Keyed Groups)
keyed_groups:
  # Create a group based on the "Role" tag value.
  # If an instance has Role="webserver", the generated group is "role_webserver".
  - key: tags.Role
    prefix: role
    separator: "_"
  # Create a group based on the "Environment" tag value.
  # If an instance has Environment="production", the generated group is "env_production".
  - key: tags.Environment
    prefix: env
    separator: "_"
  # Create a group based on the AWS Availability Zone.
  # If an instance is in zone "ap-southeast-1a", the generated group is "zone_ap_southeast_1a".
  - key: placement.availability_zone
    prefix: zone
    separator: "_"

# Determine the priority mapping of target host identifier addresses.
# Ansible tries to read the fields below in order:
hostnames:
  - private-ip-address # Primary choice for internal VPC servers without Public IPs
  - ip-address         # Fallback if the private IP is unavailable (uses the Public IP)
  - dns-name           # Fallback using the AWS public DNS address

# Define additional connection variables dynamically using Jinja2 expressions
compose:
  # Map the SSH host address using the private IP address from the AWS API
  ansible_host: private_ip_address
  # Determine the default SSH user dynamically. If the OS is Ubuntu, use user 'ubuntu'.
  # For other OSes, use 'ec2-user'.
  ansible_user: "tags.OS | default('ubuntu')"
  # Provide the environment tag to host vars
  server_environment: tags.Environment

To verify the results of that dynamic AWS query mapping, run the following graph command in your terminal:

ansible-inventory -i inventory/production/aws_ec2.yml --graph

Google Cloud Platform Integration #

If you run infrastructure on Google Cloud Platform (GCP), you use the google.cloud.gcp_compute plugin.

1. Dependency Preparation #

# Install the GCP collection
ansible-galaxy collection install google.cloud

# Install Google authentication libraries
pip install google-auth requests

2. GCP Inventory File Configuration #

Prepare a Service Account JSON Key file from the GCP console, then create a configuration file ending with the name gcp_compute.yml:

# File: inventory/production/gcp_compute.yml
---
plugin: google.cloud.gcp_compute

# Our GCP Project ID
projects:
  - my-corporate-gcp-project-123

# Zones to scan
zones:
  - asia-southeast1-a
  - asia-southeast1-b

# Instance search filter based on GCP labels
filters:
  - status = RUNNING
  - labels.environment = production

# Automatic dynamic group creation based on GCP VM labels
keyed_groups:
  - key: labels.role
    prefix: role
    separator: "_"
  - key: zone
    prefix: zone
    separator: "_"

# GCP API authentication method configuration
auth_kind: serviceaccount
service_account_file: /opt/ansible/secrets/gcp-service-account-key.json

# Target host IP address mapping
compose:
  ansible_host: networkInterfaces[0].networkIP # GCP internal Private IP
  ansible_user: "gcp-deployer"

Microsoft Azure Integration #

For Microsoft Azure users, the azure.azcollection.azure_rm plugin is the standard to use.

1. Dependency Preparation #

# Install the Azure collection
ansible-galaxy collection install azure.azcollection

# Install Azure Python dependencies
pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

2. Azure Inventory File Configuration #

Create a configuration file named azure_rm.yml (or ending with azure_rm.yaml):

# File: inventory/production/azure_rm.yml
---
plugin: azure.azcollection.azure_rm

# Limit the query to our production Resource Group
include_vm_resource_groups:
  - production-rg

# Use Managed Service Identity (MSI) or Service Principal for authentication
auth_source: auto

# Create groups based on Azure VM Tags
keyed_groups:
  - key: tags.role
    prefix: role
    separator: "_"
  - key: tags.environment
    prefix: env
    separator: "_"

# SSH connectivity via Private IP
use_private_ip: true

Hybrid Merging and Constructed Grouping #

In the real world, you often manage hybrid infrastructure: some physical servers in a local data center (managed statically) while hundreds of application VMs run on AWS EC2 (managed dynamically).

Ansible facilitates this merging very easily. Just point the inventory at a folder and place the static file side by side with the dynamic plugin configuration file:

inventory/production/
  ├── hosts.ini            # Local servers (static)
  ├── aws_ec2.yml          # AWS EC2 VMs (dynamic)
  └── group_vars/          # Shared variables
      ├── all.yml
      └── role_web.yml

Using the Constructed Plugin for Advanced Grouping #

When you merge inventory sources, you may need more complex new group classifications based on combinations of dynamic tags. You can use the built-in constructed plugin:

# File: inventory/production/constructed.yml
---
plugin: ansible.builtin.constructed

# Create new groups based on logical condition expressions
groups:
  # Create the "secure_webservers" group if the host is in the "role_web" group
  # AND in the Southeast Asia zone 'zone_ap_southeast_1a'
  secure_webservers: "'role_web' in group_names and 'zone_ap_southeast_1a' in group_names"

  # Create a group based on the target operating system
  ubuntu_nodes: "'ubuntu' in ansible_host"

Caching and Rate Limit Management #

Querying the cloud provider API directly (like AWS, GCP, Azure) every time you run a playbook or ad-hoc command is one of the biggest mistakes to avoid.

Why Is Caching Mandatory? #

  1. Rate Limiting: Cloud providers impose API query limits (rate limits) per second to prevent abuse. If your CI/CD pipeline runs playbooks repeatedly in a short time, the cloud API will throttle your connections.
  2. Speed Degradation: API calls take between 3 and 10 seconds depending on network latency. Waiting for an API query to finish on every playbook execution significantly slows down your deployment workflow.

Enabling Caching in ansible.cfg #

You can enable local cache storage on your control node machine to store API query response results for a certain period. Add the following configuration to your project’s ansible.cfg file:

# File: ansible.cfg
[defaults]
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_fact_cache
fact_caching_timeout = 86400

[inventory]
# Enable cache storage for dynamic inventory
cache = True

# Use the local JSON file storage plugin
cache_plugin = jsonfile

# Cache file storage location on disk
cache_connection = /tmp/ansible_inventory_cache

# Cache validity period in seconds (300 seconds = 5 minutes)
cache_timeout = 300

With the configuration above, Ansible only makes an API call to AWS/GCP/Azure once every 5 minutes. The remaining playbook executions within that period read the local cache database loaded instantly (0.1 seconds).

If you just launched a new server in the cloud and want to force Ansible to ignore the old cache to detect that new server instantly, run the following command with the cache refresh parameter:

ansible-inventory -i inventory/production/ --list --refresh-cache

Summary #

  • Cloud Scalability Solution — Dynamic inventory solves the manual IP maintenance problem on elastic infrastructure (like autoscaling) by querying the API in real time.
  • Modern Plugin Standard — Always use Inventory Plugins based on declarative YAML configuration instead of complex custom JSON scripts for new projects.
  • Keyed Groups Utilization — Instance tags in the cloud console (AWS, GCP, Azure) are automatically converted by the plugin into structured Ansible group names (like role_webserver).
  • Hybrid Integration — You can combine local static servers (hosts.ini) with dynamic cloud servers (aws_ec2.yml) in the same inventory directory.
  • Logical Group Construction — Use the constructed plugin to build complex new group classifications based on combined group name and target zone region logic.
  • API Throttling Prevention — Enabling inventory cache configuration in ansible.cfg is mandatory to speed up playbook execution and avoid rate limit blocks from cloud providers.
  • Manual Cache Clearing — Run the --refresh-cache command to force instant inventory data updates when there are urgent changes to virtual infrastructure.

← Previous: Variable Next: Anti-Pattern →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact