Custom Module #

Ansible provides hundreds of built-in modules covering most automation needs — from package, file, and service management to cloud integrations. But in the real world, there are situations where no module is quite right: internal company APIs, legacy systems without client libraries, or specific business logic that must be wrapped so it can be called from many playbooks. That’s when we need a custom module — Python code we write ourselves, but treated by Ansible exactly like a built-in module. This article discusses how to build modules that are idempotent, easy to test, and ready to distribute.

When to Create a Custom Module #

Before writing code, first evaluate whether we truly need a new module. Ansible has three execution layers on remote nodes: command/shell for quick commands, script for logic that doesn’t need to be reusable, and Python modules for logic that must be idempotent and distributed. Many engineers immediately use command for everything, then end up with playbooks that can’t be tested, don’t support --check, and aren’t portable.

Use command/shell if:
  ✓ The logic is simple, one or two commands
  ✓ Idempotency doesn't matter (run once, done)
  ✓ It won't be called from many playbooks

Use script/executable if:
  ✓ The logic is complex but doesn't need to be reusable
  ✓ We're more comfortable with Bash/Go/Rust
  ✓ The script output doesn't need to be processed by Ansible

Create a custom module if:
  ✓ It must be idempotent and support check mode
  ✓ It will be called from many roles or playbooks
  ✓ It interacts with APIs or services that don't have a module
  ✓ The output needs to be parsed and looped (register, with_items)

The decision tree below helps us choose the right layer:

flowchart TD
    A["Need execution on remote?"] -->|No| B["Filter/Lookup plugin"]
    A -->|Yes| C{"Idempotent & reusable?"}
    C -->|No| D{"Complex logic?"}
    D -->|No| E["command/shell module"]
    D -->|Yes| F["script module"]
    C -->|Yes| G{"Output needs parsing?"}
    G -->|No| H["command + creates/removes"]
    G -->|Yes| I["Custom Python module"]

Custom Module Anatomy #

An Ansible module is basically a Python script that receives arguments from Ansible (via stdin in JSON format), executes logic, then writes JSON to stdout. There’s no special framework — what we need is the AnsibleModule helper from ansible.module_utils.basic which handles the boilerplate of argument parsing, output formatting, and error handling.

Here’s the module execution flow diagram when called from a playbook:

sequenceDiagram
    participant PB as "Playbook"
    participant EX as "ansible-executor"
    participant TR as "Transfer (SFTP)"
    participant MN as "Managed Node"
    participant MOD as "Custom Module"

    PB->>EX: "task: my_module name=foo value=bar"
    EX->>TR: "Send the module file to the managed node"
    TR->>MN: "SCP/SFTP module.py to /tmp/"
    EX->>MN: "Execute: python module.py"
    MN->>MOD: "Run the module with args (stdin JSON)"
    MOD->>MOD: "Validate arguments via AnsibleModule"
    MOD->>MOD: "Run the business logic"
    alt Success
        MOD-->>MN: "exit_json({changed: true, ...})"
    else Failure
        MOD-->>MN: "fail_json(msg='...')"
    end
    MN-->>EX: "JSON output"
    EX->>EX: "Parse the result, record changed/failed"
    EX->>PB: "result registered to a variable"

Module Documentation (DOCUMENTATION, EXAMPLES, RETURN) #

The three special strings at the start of a module — DOCUMENTATION, EXAMPLES, RETURN — aren’t just ordinary documentation. These strings are read by ansible-doc, rendered into documentation pages, and used by some plugins for validation. If we skip all three, our module becomes a “black box” that can’t be introspected.

#!/usr/bin/python
# -*- coding: utf-8 -*-
# library/my_module.py

DOCUMENTATION = r'''
---
module: my_module
short_description: Manage application configuration via an internal REST API
description:
  - This module creates, updates, or deletes application configuration
    by calling the company's internal REST API.
  - Designed for full idempotency: running the module twice
    with the same arguments will not change state on the second run.
version_added: "1.0.0"
options:
  name:
    description: The name of the configuration to manage.
    required: true
    type: str
  value:
    description: The configuration value. Required when state=present.
    required: false
    type: str
  state:
    description: The desired state for the configuration.
    choices: [present, absent]
    default: present
    type: str
  api_url:
    description: The internal API base URL, without a trailing slash.
    required: true
    type: str
  api_token:
    description: The authentication token for the API. Recommended to store in Ansible Vault.
    required: true
    type: str
    no_log: true
author:
  - Infrastructure Team
'''

EXAMPLES = r'''
- name: Set the max_connections configuration
  my_module:
    name: "max_connections"
    value: "100"
    api_url: "https://api.internal.com"
    api_token: "{{ vault_api_token }}"
    state: present

- name: Remove a deprecated configuration
  my_module:
    name: "deprecated_setting"
    api_url: "https://api.internal.com"
    api_token: "{{ vault_api_token }}"
    state: absent
'''

RETURN = r'''
config:
  description: Details of the successfully created or updated configuration.
  returned: when state is present and changed
  type: dict
  sample:
    name: max_connections
    value: "100"
    created_at: "2024-03-15T14:30:00Z"
'''

Info — The three strings above must be written in YAML format, wrapped with r'''...''' (a raw string) so backslashes and special characters aren’t escaped by Python. ansible-doc -t module my_module will render these strings into a documentation page other developers can read.

Full Implementation with AnsibleModule #

This section shows a module managing configuration via a REST API. Note several important things: AnsibleModule accepts an argument_spec for argument declaration, supports_check_mode=True so --check works, and every execution branch returns a result with an accurate changed flag.

# library/my_module.py
from ansible.module_utils.basic import AnsibleModule
import json

try:
    import requests
    HAS_REQUESTS = True
except ImportError:
    HAS_REQUESTS = False


def get_config(api_url, api_token, name):
    """Fetch the existing configuration. Return None if not found."""
    headers = {"Authorization": f"Bearer {api_token}"}
    response = requests.get(
        f"{api_url}/api/config/{name}",
        headers=headers,
        timeout=10
    )
    if response.status_code == 404:
        return None
    response.raise_for_status()
    return response.json()


def create_or_update_config(api_url, api_token, name, value):
    """Create or update the configuration. Return (data, changed)."""
    headers = {
        "Authorization": f"Bearer {api_token}",
        "Content-Type": "application/json"
    }
    payload = {"name": name, "value": value}

    existing = get_config(api_url, api_token, name)

    # Idempotency: if the value is already the same, no change
    if existing and existing.get("value") == value:
        return existing, False

    if existing:
        response = requests.put(
            f"{api_url}/api/config/{name}",
            headers=headers, json=payload, timeout=10
        )
    else:
        response = requests.post(
            f"{api_url}/api/config",
            headers=headers, json=payload, timeout=10
        )

    response.raise_for_status()
    return response.json(), True


def delete_config(api_url, api_token, name):
    """Delete the configuration. Return changed (bool)."""
    existing = get_config(api_url, api_token, name)
    if not existing:
        return False  # Already absent, no change

    headers = {"Authorization": f"Bearer {api_token}"}
    response = requests.delete(
        f"{api_url}/api/config/{name}",
        headers=headers, timeout=10
    )
    response.raise_for_status()
    return True


def main():
    module_args = dict(
        name=dict(type='str', required=True),
        value=dict(type='str', required=False),
        state=dict(type='str', default='present', choices=['present', 'absent']),
        api_url=dict(type='str', required=True),
        api_token=dict(type='str', required=True, no_log=True),
    )

    module = AnsibleModule(
        argument_spec=module_args,
        supports_check_mode=True,
        required_if=[
            ('state', 'present', ['value']),
        ]
    )

    if not HAS_REQUESTS:
        module.fail_json(
            msg="The 'requests' library is required. Install it with: pip install requests"
        )

    name = module.params['name']
    value = module.params.get('value')
    state = module.params['state']
    api_url = module.params['api_url'].rstrip('/')
    api_token = module.params['api_token']

    result = dict(changed=False)

    try:
        if state == 'present':
            if module.check_mode:
                # Simulate: check whether it would change without actually changing
                existing = get_config(api_url, api_token, name)
                result['changed'] = (
                    not existing or existing.get('value') != value
                )
                module.exit_json(**result)

            data, changed = create_or_update_config(
                api_url, api_token, name, value
            )
            result['changed'] = changed
            result['config'] = data

        elif state == 'absent':
            if module.check_mode:
                existing = get_config(api_url, api_token, name)
                result['changed'] = existing is not None
                module.exit_json(**result)

            result['changed'] = delete_config(api_url, api_token, name)

    except requests.exceptions.ConnectionError as e:
        module.fail_json(msg=f"Cannot connect to the API: {e}")
    except requests.exceptions.HTTPError as e:
        module.fail_json(
            msg=f"The API returned error {e.response.status_code}: {e.response.text}"
        )
    except Exception as e:
        module.fail_json(msg=f"Unexpected error: {e}")

    module.exit_json(**result)


if __name__ == '__main__':
    main()

The Role of the `AnsibleModule` Helper #

AnsibleModule handles all the details that would be tedious to write ourselves:

Parsing arguments from JSON stdin (Ansible sends arguments as JSON, not environment variables).
Type validation according to argument_spec — if name is declared type='str' and the user sends an integer, the module automatically fails with a clear message.
check_mode and diff mode — when the user runs ansible-playbook --check, our module must not actually change state; just report “would change” via result['changed'] = True then module.exit_json(**result).
Output formatting — module.exit_json(**result) writes JSON to stdout in the format Ansible expects. module.fail_json(msg="...") writes JSON with failed: true to stdout (not stderr), with a non-zero exit code.
Automatic no_log for parameters marked no_log=True — we don’t need to filter ourselves when logging.

Anti-Patterns and Correct Solutions #

Here are three patterns most frequently appearing when engineers first write custom modules. Each has consequences that aren’t immediately felt, but will bite when the playbook is used in production.

1. ANTI-PATTERN: command/shell for Everything #

# ANTI-PATTERN: use shell to "manage" resources
- name: "Create config via API"
  shell: |
    curl -X POST https://api.internal.com/api/config \
      -H "Authorization: Bearer *** vault_api_token }}" \
      -H "Content-Type: application/json" \
      -d '{"name":"{{ item.name }}","value":"{{ item.value }}"}'    
  loop: "{{ configs }}"
  changed_when: false   # Ansible doesn't know if this changed or not!

# CORRECT: use a proper custom module
- name: "Create config via API"
  my_module:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    api_url: "https://api.internal.com"
    api_token: "{{ vault_api_token }}"
    state: present
  loop: "{{ configs }}"

The consequence difference: the shell above doesn’t know whether state changed, doesn’t support check_mode, and will run curl every time the playbook runs — no idempotency. The custom module only changes when the value actually differs, and ansible-playbook --check can detect upcoming changes without actually calling the API.

2. ANTI-PATTERN: Hard-Coded Paths and Logic Inside Tasks #

# ANTI-PATTERN: hard-code paths and commands in the playbook
- name: "Deploy app config"
  hosts: appservers
  tasks:
    - name: "Write the config file"
      copy:
        dest: "/etc/myapp/config.yaml"
        content: |
          database:
            host: db.internal.com
            port: 5432
            max_connections: 100          
    - name: "Restart the service if changed"
      shell: "systemctl restart myapp"

The “check whether max_connections needs changing” logic lives in the playbook, scattered across many places, hard to test. If the config format changes, all playbooks must be updated.

# CORRECT: logic moved into a reusable module
# myapp_config.py
def main():
    module_args = dict(
        path=dict(type='str', default='/etc/myapp/config.yaml'),
        database_host=dict(type='str', required=True),
        max_connections=dict(type='int', default=100),
    )
    # ... read the file, parse YAML, compare with the desired state,
    # write only if there's a change, return changed=True/False

# Usage becomes simple, the logic is hidden in the module
- name: "Deploy app config"
  myapp_config:
    path: "/etc/myapp/config.yaml"
    database_host: "db.internal.com"
    max_connections: 100
  notify: restart myapp

3. ANTI-PATTERN: Modules That Don’t Support Check Mode #

# ANTI-PATTERN: no check_mode support
def main():
    module = AnsibleModule(argument_spec=module_args)  # without supports_check_mode
    # Run immediately, doesn't care whether it's --check or not
    response = requests.post(url, json=payload)
    module.exit_json(changed=True)

This module will actually create the resource every time it runs, including when the user only wants ansible-playbook --check for a dry-run. This defeats the entire purpose of check mode.

# CORRECT: honor check_mode
def main():
    module = AnsibleModule(
        argument_spec=module_args,
        supports_check_mode=True   # REQUIRED
    )

    if module.check_mode:
        # Check what would happen, but don't execute
        result['changed'] = will_change()
        module.exit_json(**result)

    # Normal execution
    result['changed'] = do_change()
    module.exit_json(**result)

Warning — A module that doesn’t honor check_mode will execute real changes when the user runs ansible-playbook --check or --diff. This is a serious bug that can cause outages. Always implement check mode for every state-changing action.

Module Type Comparison Table #

Aspect	Action Plugin	New-Style Module	Script Module
Language	Python	Python	Any (Bash, Ruby, etc.)
Execution location	Controller (Python on the Ansible node)	Managed node	Managed node
File transfer	Not needed	Automatic via SFTP	Automatic via SFTP
Arg communication	Native Python	JSON via stdin	JSON via stdin
Idempotency	Must implement manually	Must implement manually	None (runs once)
Check mode	Manual	Manual (but easier)	Not applicable
Complexity	High	Medium	Low
Use case	Inventory/variable manipulation, controller-side side effects	Modules for remote configuration	Ad-hoc scripts, one-time use

New-style modules (Python, run on the remote) are the most common and most recommended type for daily automation. Action plugins are used when we need to run code on the controller — for example, creating dynamic inventory files or processing variables before passing them to another module.

Module Locations: Project, Role, or Collection #

Custom modules can be placed in three locations, each with different trade-offs:

project_root/                    # Applies to this project only
├── library/
│   ├── my_module.py
│   └── another_module.py
├── module_utils/                # Helpers used by many modules
│   └── api_client.py
└── playbooks/
    └── site.yml

roles/
└── my_role/
    ├── library/                # Modules specific to this role
    │   └── my_role_module.py
    ├── tasks/
    │   └── main.yml
    └── module_utils/           # Helpers specific to the role
        └── role_helper.py

my_namespace/
└── my_collection/               # Cross-project distribution
    └── plugins/
        ├── modules/
        │   └── my_module.py
        ├── module_utils/
        │   └── api_client.py
        └── ...

Tip — For modules used in more than two projects, package them in a Collection from the start. Migrating from library/ in the project root to a Collection later is far more troublesome because all playbooks referencing the module must be updated to the FQCN.

A Healthy Project Directory Structure #

infrastructure-ansible/
├── ansible.cfg
├── requirements.yml             # Collection dependencies
├── inventory/
│   ├── production/
│   └── staging/
├── playbooks/
│   ├── site.yml
│   └── deploy_app.yml
├── roles/
│   ├── common/
│   ├── webserver/
│   └── database/
├── library/                    # Project-specific custom modules
│   ├── company_user.py
│   └── internal_api.py
├── module_utils/               # Helpers for the modules above
│   └── company_client.py
└── tests/
    └── test_modules.py

Testing Custom Modules #

An untested module is a module that will fail in production. The minimum testing that should be done:

Unit Tests with pytest #

# tests/test_my_module.py
import json
import sys
from unittest.mock import patch, MagicMock

# Mock AnsibleModule before importing the module
sys.modules['ansible'] = MagicMock()
sys.modules['ansible.module_utils'] = MagicMock()
sys.modules['ansible.module_utils.basic'] = MagicMock()

import library.my_module as my_module


def test_create_config_success(monkeypatch):
    """Test creating a new config."""
    fake_response = MagicMock()
    fake_response.status_code = 201
    fake_response.json.return_value = {"name": "foo", "value": "bar"}

    with patch.object(my_module, 'requests') as mock_requests:
        mock_requests.get.return_value = MagicMock(status_code=404)
        mock_requests.post.return_value = fake_response

        result = my_module.create_or_update_config(
            "https://api.test", "token", "foo", "bar"
        )

    data, changed = result
    assert changed is True
    assert data["name"] == "foo"


def test_create_config_idempotent():
    """Second test: no change if the value is already the same."""
    with patch.object(my_module, 'get_config') as mock_get:
        mock_get.return_value = {"name": "foo", "value": "bar"}

        data, changed = my_module.create_or_update_config(
            "https://api.test", "token", "foo", "bar"
        )

    assert changed is False
    assert data["value"] == "bar"

Integration Tests with Ansible Directly #

# Run the module manually with arguments from a file
echo '{
  "ANSIBLE_MODULE_ARGS": {
    "name": "test_setting",
    "value": "42",
    "state": "present",
    "api_url": "https://api.test",
    "api_token": "fake-token"
  }
}' | python library/my_module.py

# Expected output (success):
# {"changed": true, "config": {...}}

# Output in check mode:
ANSIBLE_CHECK_MODE=true echo '...' | python library/my_module.py
# {"changed": true}   # without actually changing state

Integration with Roles #

After the module is distributed, integrate it into a role so teams can call it with familiar syntax:

# roles/webserver/tasks/main.yml
- name: Deploy the app configuration
  my_module:
    name: "max_connections"
    value: "{{ webserver_max_connections | default(100) }}"
    api_url: "{{ app_api_url }}"
    api_token: "{{ vault_api_token }}"
    state: present
  no_log: true

- name: Ensure the deprecated config is absent
  my_module:
    name: "old_setting"
    api_url: "{{ app_api_url }}"
    api_token: "{{ vault_api_token }}"
    state: absent
  no_log: true

Danger — Always add no_log: true when our module accepts sensitive parameters like api_token, password, or private_key. Without no_log, the token value appears in playbook logs and can be exposed to centralized log systems. This is a serious security risk often ignored.

Distribution via Collections #

For modules used in many projects, distribute them as an Ansible Collection. This gives us versioning, dependency management, and standardized installation:

# my_company/infrastructure/galaxy.yml
namespace: my_company
name: infrastructure
version: 1.2.0
readme: README.md
description: >
  A collection of internal modules and plugins for My Company infrastructure.
  Includes modules for application configuration, CMDB integration, and
  wrappers for internal APIs.  
authors:
  - SRE Team <[email protected]>
license:
  - GPL-2.0-or-later
tags:
  - infrastructure
  - internal
dependencies:
  community.general: ">=7.0.0"

Modules in a collection are accessed with the FQCN (Fully Qualified Collection Name):

- name: "Set config via the collection module"
  my_company.infrastructure.my_module:
    name: "max_connections"
    value: "100"
    api_url: "{{ app_api_url }}"
    api_token: "{{ vault_api_token }}"
    state: present
  no_log: true

Full details about collections, versioning, and publishing to a Private Automation Hub are covered in the Collection article, which continues the module and plugin topology.

Summary #

A custom module is a Python script receiving arguments via JSON stdin and returning JSON to stdout — Ansible treats it exactly like a built-in module.
Use AnsibleModule from ansible.module_utils.basic — this helper handles argument parsing, type validation, output formatting, and check_mode boilerplate.
Always implement supports_check_mode=True and honor module.check_mode — a module changing state without check mode makes dry-runs useless.
no_log=True on sensitive parameters like tokens, passwords, and API keys — prevents credential leakage into logs.
Write DOCUMENTATION, EXAMPLES, and RETURN in the module — these strings are rendered by ansible-doc and used for module introspection.
Idempotency is the life of a module — run it twice with the same arguments, the second run must return changed=False.
Choose the location by scope: project library/ for one project, role-scoped for one role, Collection for cross-project distribution.
Write unit tests with pytest and integration tests calling the module directly via stdin — an untested module will fail in production.
Choose between custom modules, script, or command/shell based on three things: idempotency requirements, reusability, and output parsing needs.

← Previous: Best Practice Next: Custom Plugin →