AWX & Tower #

Running Ansible from the command line works well for one person. But as the team grows, new challenges appear: who may run which playbook to which environment? How is the history of all executions recorded? How can non-engineers trigger deployments without terminal access? AWX (the open source version) and Ansible Tower (now called Red Hat Ansible Automation Platform / Automation Controller) answer all of this — they are web platforms for managing, scheduling, and controlling access to Ansible. This article discusses when we need this platform, its architecture, its core concepts, and the patterns to avoid when implementing it.

AWX vs Ansible Tower — Feature Comparison #

AWX is the open source upstream of Ansible Tower. The same code, two different distributions — the choice between them is usually determined by support needs, licensing model, and organizational scale. The following table summarizes the differences most relevant to a production decision:

Important point: AWX and Tower share the core codebase. Features we see in AWX usually appear in Tower 6-12 months later with additional hardening. But stability and support are what we pay for in Tower — not features.

Decision Tree: Choosing AWX or Tower #

flowchart TD
    A{"Need official<br/>Red Hat support?"} -- "Yes" --> B{"Subscription budget<br/>available?"}
    A -- "No" --> C{"Can our team<br/>maintain AWX<br/>upgrades itself?"}
    B -- "Yes" --> D["Tower / Automation Controller"]
    B -- "No" --> E["Evaluate the trade-off:<br/>support vs budget"]
    C -- "Yes" --> F{"Node count<br/>> 100?"}
    C -- "No" --> G["Consider a managed<br/>service or consultant"]
    F -- "No" --> H["AWX"]
    F -- "Yes" --> I{"Need a private<br/>Automation Hub?"}
    I -- "Yes" --> D
    I -- "No" --> J{"Strict compliance<br/>requirements?"}
    J -- "Yes" --> D
    J -- "No" --> H

Use this decision tree as a starting point, not an absolute rule. Many large teams still choose AWX because they have engineers whose maintenance fits into the team routine. And many small teams actually choose Tower because they don’t want to deal with upgrade compatibility headaches.

AWX version 24.x (as of early 2026) is already stable for light-medium production needs. New major versions usually bring improvements to the UI, performance, and Kubernetes integration — but major version migrations sometimes need special attention because of API changes. Always read the release notes before upgrading.

AWX Architecture: Control Plane and Execution Plane #

AWX is split into two planes with different responsibilities. Understanding this separation is important when scaling AWX to thousands of nodes or when troubleshooting why a particular job is slow.

flowchart LR
    subgraph CP["Control Plane — manages, does not execute"]
        API["REST API<br/>:8043 / :443"]
        WEB["Web UI<br/>nginx + Django"]
        TASKD[("Task System<br/>awx-manage")]
        SCHED[("Scheduler<br/>awx-manage schedule")]
        PG[("PostgreSQL<br/>state + audit log")]
    end

    subgraph EP["Execution Plane — runs playbooks"]
        EX1["Execution Node 1<br/>awx-execution-env"]
        EX2["Execution Node 2<br/>awx-execution-env"]
        EX3["Execution Node 3<br/>awx-execution-env"]
    end

    subgraph EXT["External Systems"]
        GIT[("Git Repository<br/>playbooks + roles")]
        CMDB[("Inventory Source<br/>AWS / GCP / Azure")]
        CRED[("Credential Store<br/>Vault / cloud KMS")]
        SLACK["Slack / Email<br/>callback notifier"]
    end

    USER(["User / Engineer"]) --> WEB
    CI(["CI/CD System"]) --> API
    GIT -->|"scm update"| TASKD
    CMDB -->|"sync inventory"| TASKD
    CRED -->|"fetch secret at runtime"| EX1
    CRED -->|"fetch secret at runtime"| EX2

    WEB --> API
    API --> TASKD
    TASKD --> PG
    SCHED --> TASKD
    TASKD -->|"distribute job"| EX1
    TASKD -->|"distribute job"| EX2
    TASKD -->|"distribute job"| EX3

    EX1 -->|"callback event"| TASKD
    TASKD --> SLACK

The control plane contains the API server, web UI, the task system scheduling jobs, the scheduler, and the PostgreSQL database. All state — inventory, projects, job templates, execution history — is stored in the database. The control plane never executes playbooks directly; it only distributes jobs to the execution plane.

The execution plane contains execution nodes running awx-execution-env (a container image with Ansible + collections + dependencies). Each job runs on an execution node, then its result (stdout, return code, artifacts) is returned to the control plane to be recorded in the database.

This separation has important consequences. First, horizontal scaling: add execution nodes when the job queue is long, without having to grow the control plane. Second, environment isolation: the execution plane can be on a network with access to managed nodes, while the control plane just needs user access. Third, easier observability: all events are recorded in the control plane, so we can build dashboards reading from the AWX API.

Never run the execution plane on the same node as production managed nodes. AWX jobs need privileges (root or sudo to managed nodes) and credential access — consolidating everything on one host means one compromised credential could directly touch the entire fleet.

Core AWX Concepts #

Before writing code, we need to understand AWX’s resource hierarchy. All objects in AWX live under an Organization — a logical container for grouping resources by team, business unit, or environment.

flowchart TD
    ORG["Organization"]
    ORG --> INV["Inventory"]
    ORG --> PROJ["Project"]
    ORG --> CRED["Credential"]
    ORG --> JT["Job Template"]
    ORG --> WF["Workflow Template"]
    ORG --> ROLE["Role / Team"]

    INV --> HOST["Hosts + Groups"]
    PROJ --> SCMDET[("Git Repo<br/>playbooks")]
    CRED --> CREDT["Credential Type<br/>SSH, Vault, AWS, etc."]

    JT -. "combination" .-> INV
    JT -. "combination" .-> PROJ
    JT -. "combination" .-> CRED
    JT -. "combination" .-> CREDT
    JT --> SURVEY["Survey<br/>input form"]

    WF --> JTA["Job Template A"]
    WF --> JTB["Job Template B"]
    WF --> JTC["Job Template C"]
    WF -->|"success/failure link"| JTA
    WF -->|"success/failure link"| JTB
    WF -->|"success/failure link"| JTC

Inventory is the collection of hosts targeted by playbooks. It can be static (YAML file) or dynamic (fetched from AWS EC2, GCP, Azure, VMware, or a CMDB). Inventory can have groups that work like the hosts: webservers pattern in playbooks.

Project is a pointer to a Git repository (GitHub, GitLab, Bitbucket) containing playbooks. AWX will git pull this repository every time a job runs (if scm_update_on_launch: true), so we always use the latest playbook version.

Credential is an object storing authentication information — SSH keys, vault passwords, AWS access keys, and so on. Credentials are never stored in the database as plaintext; AWX encrypts them with a key from the SECRET_KEY file, and only decrypts them when a job runs.

Job Template is the main execution unit. It’s a combination: Inventory + Project + Playbook + Credential + (optional) Survey. Every time a Job Template runs, AWX creates a Job object recording the input, output, return code, and duration.

Workflow Template connects several Job Templates with branching logic. Workflows are AWX’s way of implementing pipelines — we can define “run Job A; if successful run Job B; if failed run Job C” without writing a single line of code.

ANTI-PATTERN: Running Playbooks Directly from the CLI for Production #

A pattern often appearing in teams newly migrated to AWX: developers keep running ansible-playbook from their laptops to deploy to production because it’s “faster” and “already memorized”. This is a serious anti-pattern we must eliminate from day one.

# ANTI-PATTERN: production deploy from an engineer's laptop
# No audit trail, no approval, no automatic rollback
ssh ops@jump-host "cd /opt/ansible && ansible-playbook -i production deploy.yml --extra-vars 'version=2.1.0'"

# Problems:
# 1. Who deployed? -- only "whoever has ssh access"
# 2. When was it deployed? -- not recorded
# 3. What changed? -- diff manually calculated from Git
# 4. If it fails? -- the engineer must SSH manually to roll back
# 5. Compliance? -- auditors can't answer "who deployed what where"

# CORRECT: production deploy through an AWX Job Template with approval
# playbooks/configure-awx.yml — define a Job Template with ask_scm_diff and ask_limit_on_launch
---
- name: Configure the production Job Template with approval
  hosts: localhost
  vars:
    awx_host: "https://awx.company.internal"
    awx_oauthtoken: "{{ vault_awx_token }}"

  tasks:
    - name: Create the production Job Template with an approval workflow
      awx.awx.job_template:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Deploy — Production"
        organization: "Platform Engineering"
        job_type: run
        inventory: "Production Inventory"
        project: "Infrastructure Playbooks"
        playbook: "playbooks/deploy.yml"
        credentials:
          - "Production SSH Key"
          - "Ansible Vault Password"
        # Features lost when using the CLI directly:
        ask_scm_diff_on_launch: true     # Show the Git diff before running
        ask_limit_on_launch: true        # Can limit to a host subset
        ask_variables_on_launch: true    # Can override variables
        ask_inventory_on_launch: false   # Production inventory stays fixed
        survey_enabled: true
        become_enabled: true
        # Schedule for the deployment window
        verbosity: 1                     # Default verbosity, can be raised at run time
      no_log: true

With a Job Template like the one above, every production execution:

1. Is recorded in the AWX audit log — who triggered it, when, with what parameters.
2. Shows the Git diff before running — engineers can review what changes will be deployed.
3. Can be limited to a host subset (--limit webservers-01,webservers-02) — useful for canary deploys.
4. Is part of a Workflow with an approval gate — no production deploy without a reviewer approving.

Job Lifecycle: From Click to Result #

Understanding the states a job goes through helps us debug when a job is “stuck” or never finishes. The following state diagram shows the possible transitions:

stateDiagram-v2
    [*] --> New: "launch"
    New --> Pending: "enters the queue"
    Pending --> Waiting: "needs input<br/>(survey/approval)"
    Waiting --> Pending: "input given"
    Pending --> Running: "execution node<br/>available"
    Running --> Successful: "exit code 0"
    Running --> Failed: "non-zero exit code"
    Running --> Error: "execution error<br/>(connection lost, etc.)"
    Running --> Canceled: "user cancel"
    Waiting --> Canceled: "user cancel"
    Pending --> Canceled: "user cancel"
    Successful --> [*]
    Failed --> [*]
    Error --> [*]
    Canceled --> [*]

The Pending state is the most confusing: the job was triggered but hasn’t run yet. Common causes: no execution node available (all busy), nodes in maintenance, or the control plane running out of resources. Check Instances → Jobs in the AWX UI to see which jobs are running and which execution node handles them.

The Waiting state means the job needs further input — usually because a Workflow Template has an approval node, or a Job Template has a survey that hasn’t been filled in.

The Error state differs from Failed. Failed means the playbook exited with a non-zero code (a task failed). Error means a problem at the execution level — connection to the managed node lost, the execution environment crashed, or OOM on the execution node.

Configuring AWX with Ansible: As Code for the Automation Platform #

AWX itself can be configured with Ansible using the awx.awx collection and the tower_* modules behind it. This isn’t recursive irony — it’s actually best practice. Platform automation must be declared in Git, reviewed via PRs, and rolled back if there are problems, exactly like the managed nodes AWX configures.

# Install dependencies
ansible-galaxy collection install awx.awx
pip install awxkit

# playbooks/configure-awx.yml
---
- name: Configure AWX for the infrastructure project
  hosts: localhost
  vars:
    awx_host: "https://awx.company.internal"
    awx_oauthtoken: "{{ vault_awx_token }}"

  tasks:
    # 1. Create the Organization
    - name: Create the organization
      awx.awx.organization:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Platform Engineering"
        description: "Platform Engineering team — owns production infrastructure"
        state: present

    # 2. Create the SSH Credential
    - name: Create the SSH credential for production
      awx.awx.credential:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Production SSH Key"
        organization: "Platform Engineering"
        credential_type: "Machine"
        inputs:
          ssh_key_data: "{{ vault_production_ssh_key }}"
          username: ansible-deploy
          become_method: sudo
          become_username: root
        state: present
      no_log: true
      # no_log: true prevents the private key from leaking into Ansible logs

    # 3. Create the Vault Credential
    - name: Create the Ansible Vault credential
      awx.awx.credential:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Ansible Vault Password"
        organization: "Platform Engineering"
        credential_type: "Vault"
        inputs:
          vault_password: "{{ vault_ansible_vault_password }}"
        state: present
      no_log: true

    # 4. Create the AWS Credential (for dynamic inventory)
    - name: Create the AWS credential for inventory sync
      awx.awx.credential:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "AWS Read-Only for Inventory"
        organization: "Platform Engineering"
        credential_type: "Amazon Web Services"
        inputs:
          username: "{{ vault_aws_access_key }}"
          password: "{{ vault_aws_secret_key }}"
        state: present
      no_log: true

    # 5. Create the Project (link to the Git repository)
    - name: Create the project from the Git repository
      awx.awx.project:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Infrastructure Playbooks"
        organization: "Platform Engineering"
        scm_type: git
        scm_url: "https://github.com/company/ansible-infra.git"
        scm_branch: main
        scm_update_on_launch: true    # Always pull the latest before running
        scm_clean: true              # Remove local changes before updating
        scm_delete_on_update: false  # Keep credentials in the working copy
        timeout: 60
        state: present

    # 6. Create a dynamic inventory source from AWS EC2
    - name: Create the production inventory
      awx.awx.inventory:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Production Inventory"
        organization: "Platform Engineering"
        state: present

    - name: Create the inventory source from AWS EC2
      awx.awx.inventory_source:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "AWS EC2 Production"
        inventory: "Production Inventory"
        source: ec2
        credential: "AWS Read-Only for Inventory"
        region: "ap-southeast-1"
        # Cache the inventory for 15 minutes to reduce API calls
        cache_timeout: 900
        # Automatic groups from EC2 tags
        keyed_groups:
          - prefix: role
            key: tags.Role
          - prefix: env
            key: tags.Environment
        update_on_launch: true
        state: present

    # 7. Create the Job Template for deployment
    - name: Create the Job Template for deployment
      awx.awx.job_template:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Deploy — Production"
        organization: "Platform Engineering"
        job_type: run
        inventory: "Production Inventory"
        project: "Infrastructure Playbooks"
        playbook: "playbooks/deploy.yml"
        credentials:
          - "Production SSH Key"
          - "Ansible Vault Password"
        # Survey: input form before running
        survey_enabled: true
        survey_spec:
          description: "Deployment parameters to production"
          name: "Deploy Parameters"
          spec:
            - variable: version
              question_name: "Version to deploy"
              question_description: "Example: 2.1.0 — make sure the Git tag exists"
              required: true
              type: text
              min: 1
              max: 32
            - variable: confirm
              question_name: "Confirm the deploy to production"
              question_description: "Production deploys affect users — make sure on-call is aware"
              required: true
              type: multiplechoice
              choices: "yes\nno"
              default: "no"
        # Approval
        ask_scm_diff_on_launch: true     # Show the Git diff
        ask_limit_on_launch: true        # Can limit the host subset
        ask_variables_on_launch: true    # Can override variables
        state: present

Notice several important things in the playbook above. First, all tasks handling secrets have no_log: true — this prevents private keys and passwords from leaking into Ansible output (which might enter AWX logs and become searchable). Second, scm_clean: true ensures the AWX working copy is clean before pulling, but scm_delete_on_update: false prevents credentials in the working copy from being deleted. Third, the inventory source is configured with cache_timeout: 900 (15 minutes) — preventing API throttling to AWS and speeding up job launches.

ANTI-PATTERN: Storing Credentials as Plaintext #

One of the most dangerous mistakes often appearing in AWX deployments: developers copy-paste private keys or AWS secrets directly into input fields in the AWX UI, or even write them in the survey spec. AWX encrypts credentials in the database, but user input in surveys or extra variables isn’t always protected.

# ANTI-PATTERN: plaintext credentials in a Job Template survey
- name: Job Template with credentials in the survey
  awx.awx.job_template:
    name: "Deploy — Production"
    survey_enabled: true
    survey_spec:
      spec:
        - variable: ssh_private_key
          question_name: "SSH Private Key"
          type: textarea        # ANTI-PATTERN: this gets stored in the launch history
        - variable: db_password
          question_name: "Database Password"
          type: password        # ANTI-PATTERN: still stored in the launch history

# CORRECT: use external credential types (Vault, cloud secret manager)
---
- name: Create a credential type for HashiCorp Vault
  awx.awx.credential_type:
    controller_host: "{{ awx_host }}"
    controller_oauthtoken: "{{ awx_oauthtoken }}"
    name: "HashiCorp Vault SSH Key"
    kind: cloud
    inputs:
      fields:
        - id: vault_url
          type: string
          label: Vault URL
        - id: vault_token
          type: string
          label: Vault Token
          secret: true
        - id: ssh_key_path
          type: string
          label: SSH Key Path in Vault
      required:
        - vault_url
        - vault_token
        - ssh_key_path
    injectors:
      ssh_private_key: "{{ lookup('hashi_vault', 'secret=' + ssh_key_path, token=vault_token, url=vault_url)['data']['private_key'] }}"
    state: present

- name: Create an external credential referencing Vault
  awx.awx.credential:
    controller_host: "{{ awx_host }}"
    controller_oauthtoken: "{{ awx_oauthtoken }}"
    name: "Production SSH Key (from Vault)"
    organization: "Platform Engineering"
    credential_type: "HashiCorp Vault SSH Key"
    inputs:
      vault_url: "https://vault.company.internal:8200"
      vault_token: "{{ vault_operator_token }}"
      ssh_key_path: "secret/data/infrastructure/prod-ssh-key"
    state: present
  no_log: true
  # The token is decrypted when the job runs, then injected as ssh_private_key
  # used by the 'Machine' credential module at runtime

Table of Available Credential Types #

AWX provides many built-in credential types, and we can create custom credential types for internal systems:

The principle we must hold: credentials are never written in playbooks, surveys, or extra variables. Always use external credential types fetching secrets at runtime from Vault, cloud KMS, or another secret management system. The AWX audit log records who used which credential, but never records the credential contents themselves.

If we see an AWX playbook with extra_vars containing a string that looks like -----BEGIN RSA PRIVATE KEY-----, stop. That’s a private key, and it has already leaked everywhere: Git history, AWX launch history, Ansible logs, and maybe monitoring logs. Rotate the key, audit its access, and refactor to an external credential type.

ANTI-PATTERN: Inventory Hardcoded in the Project Repository #

A pattern often appearing in teams newly adopting AWX: store the inventory/production.yml file in the playbook Git repository, and use that inventory directly from the Job Template. This works, but it denies one of AWX’s main strengths — dynamic inventory.

# ANTI-PATTERN: inventory hardcoded in the playbook repository
# inventory/production.yml — stored in Git, manually updated when new servers appear
---
all:
  hosts:
    web-01:
      ansible_host: 10.0.1.10
    web-02:
      ansible_host: 10.0.1.11
    db-01:
      ansible_host: 10.0.2.10
  children:
    webservers:
      hosts:
        web-01:
        web-02:
    databases:
      hosts:
        db-01:

# CORRECT: dynamic inventory source in AWX
---
- name: Create the inventory source from AWS EC2
  awx.awx.inventory_source:
    controller_host: "{{ awx_host }}"
    controller_oauthtoken: "{{ awx_oauthtoken }}"
    name: "AWS EC2 Production"
    inventory: "Production Inventory"
    source: ec2
    credential: "AWS Read-Only for Inventory"
    region: "ap-southeast-1"
    # Filter: only running instances with specific tags
    instance_filters:
      tag:Environment: production
      instance-state-name: running
    # Automatic groups from EC2 tags
    keyed_groups:
      - prefix: role
        key: tags.Role
      - prefix: env
        key: tags.Environment
    # Override variables per host
    hostvars:
      ansible_user: ansible-deploy
      ansible_python_interpreter: /usr/bin/python3
    # Cache 15 minutes, refresh on job launch
    cache_timeout: 900
    update_on_launch: true
    state: present

With dynamic inventory, new servers tagged Environment=production and Role=webserver in AWS automatically enter the role_webserver and env_production groups in AWX — no Git commit needed. Terminated servers automatically disappear from the inventory when the cache expires.

Table of Inventory Sources Supported by AWX #

The simplest way to migrate from static to dynamic: create a new inventory with the same source as our cloud provider, then point the Job Template to the new inventory. The old inventory can be archived, and dynamic inventory becomes the primary source.

Workflow Templates: Deployment Pipelines with Approval #

Workflow Templates are AWX’s way of implementing pipelines. Instead of writing shell or Python scripts calling several ansible-playbook commands sequentially, we define a graph of Job Templates in the AWX UI — more visual, easier to review, and integrated with RBAC.

flowchart TD
    START(["Start"]) --> A["JT: Lint & Validate"]
    A -->|"success"| B{"JT: Deploy Staging"}
    A -->|"failure"| NOTIFY1["End: Notify Failure"]
    B -->|"success"| C{"JT: Integration Test"}
    B -->|"failure"| NOTIFY2["End: Notify Failure"]
    C -->|"success"| APPROVAL{"Gate: Approval"}
    C -->|"failure"| NOTIFY3["End: Notify Failure"]
    APPROVAL -->|"approved"| D["JT: Deploy Production"]
    APPROVAL -->|"rejected"| HOLD["End: Hold"]
    D -->|"success"| E["JT: Verify Health"]
    D -->|"failure"| ROLLBACK["JT: Rollback Production"]
    ROLLBACK --> NOTIFY4["End: Notify Failure"]
    E -->|"healthy"| END(["End: Success"])
    E -->|"unhealthy"| ROLLBACK

This workflow implements a complete production deployment pipeline: lint → deploy staging → integration test → approval gate → deploy production → verify health. If any step fails, the failure path leads to a notification or rollback.

# playbooks/configure-workflow.yml
---
- name: Create the full deployment workflow
  hosts: localhost
  vars:
    awx_host: "https://awx.company.internal"
    awx_oauthtoken: "{{ vault_awx_token }}"

  tasks:
    - name: Create the Workflow Job Template
      awx.awx.workflow_job_template:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        name: "Full Deployment Pipeline"
        organization: "Platform Engineering"
        description: "Full deployment pipeline: validate, staging, integration, approval, production, verify"
        # Survey at the workflow level
        survey_enabled: true
        survey_spec:
          description: "Deployment pipeline parameters"
          name: "Pipeline Parameters"
          spec:
            - variable: version
              question_name: "Version to deploy"
              required: true
              type: text
        state: present
      register: workflow

    - name: Add the validate node to the workflow
      awx.awx.workflow_job_template_node:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        workflow_job_template: "Full Deployment Pipeline"
        identifier: "validate"
        unified_job_template: "Validate Playbook"
        state: present

    - name: Add the approval node after the integration test
      awx.awx.workflow_job_template_node:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        workflow_job_template: "Full Deployment Pipeline"
        identifier: "production_approval"
        all_parents_must_converge: false
        # The approval node pauses until a reviewer approves
        state: present
      register: approval_node

    - name: Set the approval to 'always' — the workflow pauses at this node
      awx.awx.workflow_job_template_node_approval:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        workflow_job_template: "Full Deployment Pipeline"
        node_identifier: "production_approval"
        timeout: 3600  # Auto-cancel after 1 hour if no approval
        state: present

    - name: Connect the nodes with success/failure links
      awx.awx.workflow_job_template_node:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        workflow_job_template: "Full Deployment Pipeline"
        identifier: "validate"
        related:
          success_nodes:
            - identifier: "deploy_staging"
          failure_nodes:
            - identifier: "notify_failure"
        state: present

    - name: Connect all nodes to the production deploy and rollback paths
      awx.awx.workflow_job_template_node:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        workflow_job_template: "Full Deployment Pipeline"
        identifier: "deploy_production"
        related:
          success_nodes:
            - identifier: "verify_health"
          failure_nodes:
            - identifier: "rollback_production"
        state: present

The approval node is a frequently underestimated feature. When the workflow reaches this node, it stops and waits — no further execution until a reviewer approves. Reviewers receive a notification (configurable per workflow) and can approve or reject from the AWX UI or via the REST API. This implements the four-eyes principle for production deployments without needing extra scripts.

This workflow can be triggered from many sources: manual launch from the UI, REST API from CI/CD, webhooks from GitHub/GitLab, or a schedule. For integration with more complex CI/CD pipelines, see the Pipeline Design article — that article discusses how CI produces artifacts and CD (including AWX workflows) distributes them.

RBAC: Role-Based Access Control #

AWX has a detailed RBAC system. We can give permissions to users or teams at the Organization, Project, Inventory, Job Template, or Workflow Template level. The common pattern: give only the needed access, nothing more.

# playbooks/configure-rbac.yml
---
- name: Configure RBAC for the development and SRE teams
  hosts: localhost
  vars:
    awx_host: "https://awx.company.internal"
    awx_oauthtoken: "{{ vault_awx_token }}"

  tasks:
    # Developer team: can execute staging, but NOT production
    - name: Give the execute role to the developer team for staging
      awx.awx.role:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        team: "Developers"
        role: execute
        job_templates:
          - "Deploy — Staging"
          - "Restart App — Staging"
        state: present

    - name: Give the read-only role to the developer team for the production inventory
      awx.awx.role:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        team: "Developers"
        role: read
        inventories:
          - "Production Inventory"
        state: present
        # Developers can see the production inventory contents (for debugging)
        # but CANNOT execute Job Templates targeting production

    # SRE team: full admin
    - name: Give the admin role to the SRE team
      awx.awx.role:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        team: "SRE"
        role: admin
        organizations:
          - "Platform Engineering"
        state: present
        # SRE can modify Job Templates, manage credentials, etc.

    # Auditors: read-only at all levels
    - name: Give the read-only role to the auditors
      awx.awx.role:
        controller_host: "{{ awx_host }}"
        controller_oauthtoken: "{{ awx_oauthtoken }}"
        team: "Auditors"
        role: read
        organizations:
          - "Platform Engineering"
        state: present
        # Auditors can see job history, configuration, logs
        # but cannot trigger or modify anything

Good RBAC pattern: least privilege. Developers don’t need access to production credentials; they only need staging Job Templates. SRE needs full access because they’re the ones on-call. Auditors need read-only for compliance checks. Every role has a clear reason.

Webhooks: Triggering Jobs from External Systems #

AWX supports webhooks for GitHub, GitLab, and generic HTTP POST. This allows jobs to be triggered automatically on repository pushes, or when CI pipelines finish:

# Enable the webhook on a Job Template
- name: Enable the webhook on the Job Template
  awx.awx.job_template:
    controller_host: "{{ awx_host }}"
    controller_oauthtoken: "{{ awx_oauthtoken }}"
    name: "Deploy — Staging"
    webhook_service: github     # or gitlab, or 'none' for generic
    webhook_credential: "GitHub Webhook Token"  # optional, for HMAC validation
    state: present

# In GitHub Actions — trigger AWX on pushes to main
- name: Trigger the AWX deployment
  if: github.ref == 'refs/heads/main'
  run: |
    curl -X POST \
      -H "Content-Type: application/json" \
      -H "X-GitHub-Event: push" \
      -H "X-GitHub-Delivery: ${{ github.event_id }}" \
      "https://awx.company.internal/api/v2/job_templates/42/github/" \
      -d '{"ref": "refs/heads/main", "version": "${{ github.sha }}"}'    

# In GitLab CI — trigger AWX via webhook
trigger-awx:
  stage: deploy
  script:
    - |
      curl -X POST \
        -H "X-Gitlab-Event: Push Hook" \
        -H "X-Gitlab-Token: $AWX_WEBHOOK_SECRET" \
        "https://awx.company.internal/api/v2/job_templates/42/gitlab/" \
        -d '{"ref": "refs/heads/main"}'      
  only:
    - main

When using webhooks with HMAC credentials, AWX verifies the signature from GitHub/GitLab before running the job. This prevents unauthorized people from triggering jobs just by knowing the webhook URL. For generic webhooks (without GitHub/GitLab), we can implement custom authentication via a header token.

When integrating AWX with CI/CD pipelines, role separation is important: CI (build, test, scan) doesn’t need access to AWX production credentials. CI just triggers the AWX workflow via webhook; AWX itself handles the deployment with its own credentials. If CI is compromised, the attacker doesn’t immediately have production access — they can only trigger jobs that still go through the AWX approval gate.

Observability: Dashboards and Alerts from AWX #

AWX exposes a REST API that can be scraped for monitoring. The most useful endpoints for observability:

# Total running jobs (for a dashboard gauge)
GET /api/v2/unified_job_templates/?page_size=1

# Jobs that failed in the last hour
GET /api/v2/jobs/?status=failed&started__gte=2026-06-08T04:00:00Z

# Jobs still pending (long queue = problem)
GET /api/v2/jobs/?status=pending

# Inventory sources that failed to sync
GET /api/v2/inventory_sources/?last_job_failed=true

This data can be scraped into Prometheus via json_exporter, then visualized in Grafana. The Dashboard article discusses patterns for building dashboards from REST APIs like this one. For alerting, we can set Prometheus alerts based on metrics: awx_jobs_failed_total > 5 means 5 jobs failed in the last 5 minutes, possibly indicating an environment problem.

Summary #

• AWX is open source (Apache 2.0), suitable for labs, homelabs, and teams able to maintain their own upgrades. Tower / Automation Controller is paid with Red Hat support, suitable for companies with compliance requirements. Both share the core codebase.
• AWX is split into a control plane (API, UI, database, scheduler) and an execution plane (execution nodes running playbooks). Scale horizontally by adding execution nodes; the control plane just needs enough for UI and API traffic.
• Core concepts: Organization → Project (Git repo) + Inventory (target hosts) + Credential (secrets) + Job Template (combination of the three) + Workflow Template (a graph of several Job Templates with approvals).
• ANTI-PATTERN: running playbooks directly from the CLI for production — no audit, approval, or rollback. Use Job Templates with ask_scm_diff_on_launch and approval gates.
• ANTI-PATTERN: plaintext credentials in surveys or extra variables — always use external credential types (Vault, cloud KMS). The audit log records who accessed credentials, not their contents.
• ANTI-PATTERN: hardcoded inventory in the repository — use dynamic inventory sources (AWS EC2, GCP, Azure) with keyed_groups for auto-grouping from cloud tags/labels.
• Workflow Templates implement deployment pipelines (lint → staging → integration → approval → production → verify) without code. Approval nodes enforce the four-eyes principle.
• RBAC must use least privilege: developers have execute on staging but not production; SRE has admin on production; auditors have read-only.
• Webhooks let CI/CD trigger AWX jobs via HTTP POST. Use HMAC credentials for signature validation, and separate CI secrets from AWX credentials.
• Observability through the AWX REST API — scrape into Prometheus, visualize in Grafana, set alerts for failed jobs, long queues, and inventory sync failures.

← Previous: Strategy & Serial Next: Pipeline Design →