Collection #

Roles are a way to package and share Ansible configuration. But roles have limitations — they can only contain tasks, handlers, templates, and variables. They can’t package custom modules, filter plugins, or inventory plugins together. Ansible Collections exist to solve this limitation: one distribution unit that can contain roles, modules, plugins, playbooks, and documentation — all with clear versions and declared dependencies. In the earlier Dynamic Inventory article, we saw inventory plugins from the amazon.aws collection. In the Custom Plugin article, we also wrote lookup and filter plugins. A collection is the structured way to package all of that into one bundle that can be shared with other teams or even the open source community.

Why a Collection, Not Just a Role? #

Before diving into the technical side, understand what distinguishes a collection from a role. Both look similar — both are directories with a certain structure, both can be shared. But collections are far more expressive and are the official distribution unit used by the Ansible ecosystem since version 2.9. The following short comparison helps us decide when to stop at a role and when to move up to a collection:

If our team only needs “a set of tasks to set up a web server”, a role is enough. But once we want to package Python modules for interacting with internal APIs, Jinja2 filters for company data transformations, and two or three roles at once — a collection is the answer. Many large companies like Red Hat, Cisco, and Microsoft release their products as collections, not separate roles.

When to Create a New Collection? #

Not all Ansible code needs to be wrapped in a collection. There’s a point where the structural overhead isn’t worth the benefit. The following decision tree helps us decide:

flowchart TD
    A["Need to distribute Ansible code?"] --> B{"Are there custom modules or plugins?"}
    B -- No --> C{"More than 1 shared role?"}
    C -- No --> D["A role is enough"]
    C -- Yes --> E["Consider a collection"]
    B -- Yes --> F{"Used together with other roles and plugins?"}
    F -- No --> G["Standalone module/plugin might be enough"]
    F -- Yes --> H{"Shared with other teams/organizations?"}
    H -- No --> I["Small internal collection"]
    H -- Yes --> J["Collection with namespace, publish to Hub"]

Real-world usage usually looks like this: a DevOps team in a company has several modules for talking to internal APIs (deployment, monitoring, secret rotation), several filters for data transformation (account number formatting, employee name parsing), and one or two roles for bootstrapping new servers. All of that is wrapped in one my_company.platform collection — installed in every playbook with requirements.yml, tested with Molecule, and published to the Private Automation Hub on every release.

Collection Structure #

The collection directory layout follows a very specific convention. Ansible doesn’t look for components through configuration — it looks for them in agreed-upon locations. This differs from roles, where file names inside the tasks/ directory are free-form. In a collection, every component type has its own directory under plugins/. Following this convention isn’t just best practice — it’s mandatory, because if we place a module outside plugins/modules/, Ansible won’t find it.

my_namespace/
└── my_collection/
    ├── galaxy.yml              # Collection metadata (required)
    ├── README.md
    ├── CHANGELOG.rst
    ├── LICENSES/               # License file per component (REUSE compliant)
    │   └── GPL-2.0-or-later.txt
    ├── docs/                   # Documentation (sphinx/rst)
    │   ├── index.rst
    │   └── modules/
    │       └── app_config.rst
    ├── plugins/
    │   ├── modules/            # Custom modules — appear as modules
    │   │   └── app_config.py
    │   ├── module_utils/       # Shared utilities for modules
    │   │   └── api_client.py
    │   ├── lookup/             # Lookup plugins
    │   │   └── company_cmdb.py
    │   ├── filter/             # Filter plugins
    │   │   └── company_filters.py
    │   ├── callback/           # Callback plugins (logging, notifications)
    │   │   └── deployment_notifier.py
    │   ├── inventory/          # Inventory plugins
    │   │   └── cmdb.py
    │   ├── connection/         # Connection plugins (rare, for custom transports)
    │   └── strategy/           # Strategy plugins (very rare)
    ├── roles/                  # Roles bundled in the collection
    │   ├── webserver/
    │   └── database/
    ├── playbooks/              # Example playbooks
    │   └── deploy.yml
    ├── tests/                  # Unit tests for modules/plugins
    │   └── unit/
    │       └── plugins/
    │           └── modules/
    │               └── test_app_config.py
    └── changelogs/             # Changelog per release (fragment-style)
        └── fragments/
            └── add-app-config-module.yml

What we need to notice in the structure above: plugins/module_utils/ is a crucial directory often overlooked. If two of our modules use the same function (e.g. an HTTP call helper for an internal API), put that function in module_utils/ and import it from both modules. This avoids code duplication and ensures changes in one place are automatically reflected in other modules. The module_utils + module pattern is exactly what official modules like those in community.general use.

Creating a New Collection #

The Ansible Galaxy CLI provides the init command that creates a collection directory skeleton complete with empty files ready to fill in:

# Create the collection structure from the template
ansible-galaxy collection init my_namespace.my_collection

# Move to the collection directory
cd my_namespace/my_collection

# View the created structure
ls -la
# drwxr-xr-x  galaxy.yml
# drwxr-xr-x  plugins/
# drwxr-xr-x  roles/
# drwxr-xr-x  playbooks/
# drwxr-xr-x  tests/
# drwxr-xr-x  docs/

Notice that init uses the namespace.collection convention — two segments separated by a dot. This isn’t just a naming style, it’s the Fully Qualified Collection Name (FQCN) we’ll use in playbooks. Choosing a consistent namespace is important because once published, it can’t be changed. Some guidelines:

• Namespace for individuals/teams: a short unique name, e.g. unisbadri.tools or badricreativetech.platform
• Namespace for companies: the company or division name, e.g. my_company.cloud
• Avoid generic namespaces like common or utilities — they clash with other people’s namespaces on Galaxy

galaxy.yml: Collection Metadata #

The galaxy.yml file is the heart of a collection — it defines identity, version, and dependencies. One of the most common mistakes is leaving fields empty or having overly short descriptions. Galaxy uses this metadata for detail pages, so investing 10 minutes at the start will save time explaining to other users later.

# galaxy.yml
namespace: my_company
name: infrastructure
version: 2.1.0
readme: README.md
description: >
  A collection for My Company's internal infrastructure — contains roles, modules,
  and plugins used across all SRE teams. Supports deployment to AWS,
  secret management via Vault, and integration with internal monitoring.  

authors:
  - SRE Team <[email protected]>

license:
  - GPL-2.0-or-later

tags:
  - infrastructure
  - deployment
  - monitoring
  - aws

repository: https://github.com/mycompany/ansible-infrastructure
documentation: https://docs.mycompany.internal/ansible
issues: https://github.com/mycompany/ansible-infrastructure/issues
build_ignore:
  - .git
  - .github
  - changelogs/*.fragment
  - tests/output

# Dependencies on other collections — minimum version or range
dependencies:
  community.general: ">=7.0.0"
  amazon.aws: ">=7.0.0"
  community.docker: ">=3.4.0"

ANTI-PATTERN vs CORRECT in galaxy.yml #

# ANTI-PATTERN: dependencies without version pins — can break at any time
dependencies:
  community.general:
  amazon.aws:

# CORRECT: pin to a tested minimum version, with a reasonable range
dependencies:
  community.general: ">=7.0.0,<9.0.0"
  amazon.aws: ">=7.0.0,<8.0.0"
  community.docker: "3.4.6"   # Exact pin in conservative production

Pinning dependencies to a tested range isn’t paranoia — it’s the only way to ensure our playbooks don’t suddenly fail mid-deployment because an upstream collection released a breaking change. The CI pipeline should regularly test our collection against the latest dependency versions, and once confident they’re compatible, raise the range in galaxy.yml.

The build_ignore field is also often ignored but important. When we run ansible-galaxy collection build, Ansible creates a tarball. The .git files, .github/ directory, and test output can significantly increase tarball size and — more importantly — leak internal information publicly. Always exclude unnecessary directories.

Installing and Using Collections #

Before modules and plugins from a collection can be used, the collection must be installed. There are several installation methods, each suitable for a different scenario:

# Build the collection into a .tar.gz archive
ansible-galaxy collection build

# Output: my_company-infrastructure-2.1.0.tar.gz

# Install from a local file — suitable for internal testing
ansible-galaxy collection install my_company-infrastructure-2.1.0.tar.gz -p ./collections

# Install from public Galaxy
ansible-galaxy collection install my_company.infrastructure

# Install with a specific version
ansible-galaxy collection install my_company.infrastructure:==2.1.0

# Install with a version range
ansible-galaxy collection install 'my_company.infrastructure:>=2.0.0,<3.0.0'

Once installed, modules and plugins are accessed using the Fully Qualified Collection Name (FQCN). The FQCN is the namespace.collection.<component> format that ensures no ambiguity when two different collections have modules with the same name:

# FQCN: namespace.collection.module_name
- name: Set the application configuration
  my_company.infrastructure.app_config:
    name: max_connections
    value: "100"
    api_url: "{{ api_url }}"
    api_token: "{{ vault_api_token }}"
    state: present

# A role inside a collection
- name: Set up the web server
  import_role:
    name: my_company.infrastructure.webserver
  vars:
    nginx_port: 443

# A filter from a collection
- name: Generate the report
  debug:
    msg: "{{ user_list | my_company.infrastructure.format_employee_id }}"

# A lookup from a collection
- name: Fetch data from the CMDB
  debug:
    msg: "{{ lookup('my_company.infrastructure.company_cmdb', 'server-001') }}"

ANTI-PATTERN vs CORRECT in Module Usage #

# ANTI-PATTERN: use the short name without the FQCN — fragile and ambiguous
- name: Set the configuration
  app_config:
    name: max_connections
    value: "100"

# CORRECT: always use the FQCN — explicit and name-collision-proof
- name: Set the configuration
  my_company.infrastructure.app_config:
    name: max_connections
    value: "100"

Why is the FQCN mandatory even in internal playbooks? Imagine we use app_config in 30 playbooks. One day our team imports a new vendor collection that happens to also have an app_config module (not impossible — generic names often clash). Without the FQCN, Ansible prioritizes one based on installation order — and worse, this can differ between dev machines, CI, and production. With the FQCN, the behavior is deterministic: what we write is what gets used.

The Build, Publish, and Install Flow #

Before a collection can be used by other teams or on production servers, there’s a flow to go through. Understanding this sequence is important because each stage has different error modes:

sequenceDiagram
    participant Dev as "Developer"
    participant Local as "Local Repo"
    participant Build as "ansible-galaxy build"
    participant Tarball as ".tar.gz"
    participant Hub as "Galaxy/Hub"
    participant CI as "CI Pipeline"
    participant Server as "Target Server"

    Dev->>Local: "Edit module/role/plugin"
    Dev->>Local: "Update galaxy.yml (version bump)"
    Dev->>Local: "Add a changelog fragment"
    Dev->>CI: "Push & open a PR"
    CI->>CI: "Run sanity tests (ansible-test sanity)"
    CI->>CI: "Run unit tests (pytest)"
    CI->>CI: "Run integration tests (Molecule)"
    CI->>Build: "Trigger a build on the merge to main"
    Build->>Tarball: "Generate the tarball with the version"
    Build->>Hub: "Publish to Galaxy/Hub"
    Hub->>Server: "ansible-galaxy install pulls the new version"
    Server->>Server: "ansible-playbook runs with the new collection"

The sequence diagram above shows that this flow isn’t just “build and publish” — there are many validation points to pass through. Sanity tests, for example, reject modules lacking documentation, with inconsistent return values, or importing forbidden Python modules (os.system, subprocess.Popen without an argument list, etc.). Skip one stage and our tarball will be rejected by Galaxy or, worse, become a time bomb in production.

requirements.yml for Dependency Management #

Projects using several collections should define all of them in one file. Without requirements.yml, everyone who clones the repo must remember to ansible-galaxy collection install the right collections with the right versions. This is a recipe for bugs that only appear on certain developer laptops but not in CI.

# requirements.yml
---
collections:
  # Collections from Ansible Galaxy
  - name: community.general
    version: ">=7.0.0,<8.0.0"

  - name: community.docker
    version: "3.4.6"          # Pin to an exact version in production

  - name: amazon.aws
    version: ">=7.0.0"

  - name: kubernetes.core
    version: "2.4.0"

  # Private collections from the internal Automation Hub
  - name: my_company.infrastructure
    version: "2.1.0"
    source: https://automation-hub.mycompany.internal/api/galaxy/

  # Collections from direct source control (for not-yet-published collections)
  - name: my_company.experimental
    source: https://github.com/mycompany/ansible-experimental.git
    type: git
    version: main

roles:
  - name: geerlingguy.nginx
    version: "3.2.0"
    src: https://github.com/geerlingguy/ansible-role-nginx

# Install all dependencies at once
ansible-galaxy install -r requirements.yml
ansible-galaxy collection install -r requirements.yml

# Or both at once (Ansible >= 2.10)
ansible-galaxy install -r requirements.yml

ANTI-PATTERN vs CORRECT in requirements.yml #

# ANTI-PATTERN: no version pins — can break every time dependencies update
collections:
  - name: community.general
  - name: amazon.aws

# CORRECT: pin a version or range already tested in CI
collections:
  - name: community.general
    version: ">=7.0.0,<8.0.0"
  - name: amazon.aws
    version: "7.5.0"

A bad scenario that often happens: a developer clones the repo, forgets to run ansible-galaxy install, and directly runs the playbook. The playbook fails because the community.general.parted module doesn’t exist. Or more subtly — the playbook works on the developer’s laptop because a compatible community.general version happens to be installed, but fails in CI which builds from scratch with the latest version. Pinning versions in requirements.yml eliminates this variable.

Pin dependencies in production, but use loose ranges in development. A healthy pattern: use the >=X,<Y range on the development branch so the team always tests the latest versions, and pin the exact version (X.Y.Z) on the tagged production branch. Before bumping versions in production, CI must have already run integration tests with the new version.

Private Automation Hub #

For internal collections that must not be published to public Galaxy, use a Private Automation Hub or serve via a simple HTTP server. Ansible Galaxy itself is a relatively simple HTTP protocol — many tools can provide it without installing a full Automation Controller.

# ansible.cfg
[galaxy]
server_list = automation_hub, galaxy

[galaxy_server.automation_hub]
url = https://automation-hub.mycompany.internal/api/galaxy/
auth_url = https://automation-hub.mycompany.internal/auth/token/
token = {{ lookup('env', 'AUTOMATION_HUB_TOKEN') }}

[galaxy_server.galaxy]
url = https://galaxy.ansible.com/

# Publish the collection to the private hub
ansible-galaxy collection publish \
  my_company-infrastructure-2.1.0.tar.gz \
  --server automation_hub

# Other teams install from the hub
ansible-galaxy collection install my_company.infrastructure \
  --server automation_hub

A lighter alternative: some teams host collection tarballs on an internal S3 bucket or a simple HTTP server, then reference them directly in requirements.yml:

collections:
  - name: my_company.infrastructure
    source: https://artifacts.mycompany.internal/ansible-collections/
    type: url
    version: "2.1.0"

This approach suits small teams that don’t want to operate a full Automation Hub. The trade-off is no UI for browsing collections, no automatic update notifications, and no RBAC.

Automation Hub vs simple Galaxy: The Private Automation Hub (from Red Hat) provides a UI, RBAC, approval workflows, and SSO integration — suitable for organizations with 50+ engineers. For small teams, a plain HTTP server is enough and far lighter to operate.

Testing Collections #

A collection’s quality is determined by its testing. Without tests, we can’t be sure a new module doesn’t break old modules, or that a role still works on newer dependency versions. Collection testing patterns have three layers, each with a different purpose:

# molecule/default/molecule.yml for testing the collection
---
dependency:
  name: galaxy
  options:
    requirements-file: requirements.yml

driver:
  name: docker

platforms:
  - name: ubuntu22
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
  - name: rocky9
    image: geerlingguy/docker-rocky9-ansible:latest
    pre_build_image: true

provisioner:
  name: ansible
  playbooks:
    converge: converge.yml
  inventory:
    links:
      group_vars: ../../../inventory/group_vars

verifier:
  name: ansible

# Sanity tests — check code style, documentation, and standards compliance
ansible-test sanity --docker default

# Unit tests for Python modules
ansible-test units --docker default

# Integration tests with Molecule
molecule test

These three test layers have different functions. Sanity tests are cheap (minutes) and run on every commit — checking that every module has documentation, imports are allowed, and return values are consistent. Unit tests are for complex Python modules (e.g. modules interacting with APIs) — run on every PR. Integration tests with Molecule — run nightly or before release, because they need Docker images and are slower.

Run sanity tests in a pre-commit hook. ansible-test sanity can run in seconds for a single module. Add it to a pre-commit hook and we get instant feedback every time we edit a module, without waiting for CI. This greatly reduces the cycle time for documentation or code style fixes.

Decision Tree for When to Create a Collection #

The last topic in this article is the most frequently asked decision: does this code deserve to be wrapped in a collection? The answer isn’t always clear, and creating a collection too early (for one role used by two playbooks) is just as wasteful as not creating one when it’s time (five modules + three roles + two plugins copied into every project).

flowchart TD
    Start["Start: Have Ansible code"] --> Q1{"Are there custom modules/plugins?"}
    Q1 -- No --> Q2{"More than 1 role?"}
    Q2 -- No --> Single["A role is enough"]
    Q2 -- Yes --> Q3{"Used in more than 1 project?"}
    Q3 -- No --> Q4["Consider symlinks or copying"]
    Q3 -- Yes --> Coll1["Internal collection"]
    Q1 -- Yes --> Q5{"Are there modules + roles + plugins?"}
    Q5 -- No --> Single2["Standalone module/plugin + role"]
    Q5 -- Yes --> Q6{"Shared with other teams?"}
    Q6 -- No --> Coll2["Internal collection with meta"]
    Q6 -- Yes --> Q7{"Published publicly?"}
    Q7 -- No --> Coll3["Collection to the Private Hub"]
    Q7 -- Yes --> Coll4["Collection to public Galaxy"]

This decision tree isn’t a hard rule — it’s a heuristic we can adjust. Some companies create a collection from the first role because they already know they’ll have many roles. Other companies wait until they have at least three custom modules because the collection setup overhead (galaxy.yml, CI, testing) isn’t worth it for one module.

Summary #

• A Collection is the complete Ansible distribution unit — containing roles, modules, plugins, and playbooks in one package with clear versions. It’s the evolution of roles, not their replacement.
• The collection directory structure follows strict conventions: plugins/modules/, plugins/filter/, roles/, and so on — Ansible finds components by location, not configuration. plugins/module_utils/ is for shared code used by many modules.
• galaxy.yml defines the collection’s identity, version, and dependencies. Always pin dependencies to a minimum version or tested range, and exclude unnecessary files in build_ignore.
• Use the FQCN (Fully Qualified Collection Name) when using modules from a collection — my_company.infrastructure.app_config, not just app_config. This ensures no ambiguity when two collections have modules with the same name.
• requirements.yml to define all collection dependencies in one place. Pin versions for reproducibility, and run ansible-galaxy collection install -r requirements.yml in CI/CD so environments stay consistent.
• For internal collections, use a Private Automation Hub or an internal repository — don’t publish internal code to public Galaxy. For small teams, a plain HTTP server is enough.
• Layered testing: ansible-test sanity for code style (fast, runs in pre-commit), unit tests for Python module logic, and Molecule for integration tests. All three complement each other, not replace.
• Choose a namespace once and consistently — a namespace can’t be changed after being published to Galaxy.

← Previous: Dynamic Inventory Next: Advanced Jinja2 →