Delegation & Local Action #

When writing Ansible playbooks, we often assume every task in a play must be executed directly on the managed node currently being processed. This is Ansible’s default behavior, which is very efficient for most administrative tasks like installing packages, copying configuration files, or starting services. However, in complex modern infrastructure orchestration, we often need to coordinate between servers. For example, before upgrading a web server, we must disable it on the load balancer server first, create a new DNS entry on the main DNS server, or record the deployment log on our local control node. For multi-host scenarios like these, we need the Delegation and Local Action features.

Basic Concepts of Execution Delegation #

Execution delegation in Ansible is managed using the delegate_to directive. This directive tells Ansible to run a specific task on another host we designate, instead of on the target host currently being processed by the play queue.

Although the task executes on another host (the delegated host), the very important thing to understand is that the variable and facts context still refers to the original target host. That means variables like inventory_hostname, ansible_default_ipv4, or host-specific variables defined in host_vars still carry data belonging to the web server we’re upgrading, not the load balancer server where the delegated command executes.

Here’s a flow diagram comparing the execution flow without delegation vs the execution flow with delegation:

flowchart TD
    subgraph "Normal Flow ("Without Delegation")"
        C1["Control Node"] -->|"Send & Execute Module"| W1["Web Server 1"]
        C1 -->|"Send & Execute Module"| W2["Web Server 2"]
    end

    subgraph "Delegation Flow ("With delegate_to: Load Balancer")"
        C2["Control Node"] -->|"Send Instruction (Web Server 1 Variables)"| LB["Load Balancer Server"]
        LB -->|"Execute Action for Web Server 1"| LB
        C2 -->|"Send Instruction (Web Server 2 Variables)"| LB
        LB -->|"Execute Action for Web Server 2"| LB
    end

With delegation, we can easily create playbooks that bridge inter-server communication dynamically during the deployment process.

local_action and delegate_to: localhost #

In many automation scenarios, we need the control node (the machine where we run the ansible-playbook command) to perform an action. Examples include:

  • Sending HTTP POST notifications to Slack, Discord, or Microsoft Teams webhooks.
  • Downloading sensitive configuration files from an external Vault to the control node before distribution.
  • Waiting for specific TCP ports to be up on managed nodes from the control node’s perspective (ensuring no firewall blocking).
  • Recording deployment history to a local log file on the control node.

Ansible provides two ways to direct execution to the control node: delegate_to: localhost and local_action. Both are functionally identical, but have different syntax writing formats.

delegate_to: localhost vs local_action Syntax #

The delegate_to: localhost syntax is the modern recommended approach because it keeps the task writing structure consistent with regular modules. Meanwhile, local_action is a legacy syntax where the module name is placed as the first argument of the local_action directive.

Let’s look at the implementation difference between both through an anti-pattern and solution writing example:

# ANTI-PATTERN: Using a complicated local curl shell script delegated unnecessarily
- name: Send a slack notification manually
  hosts: app_servers
  tasks:
    - name: Send a slack message via local shell
      ansible.builtin.shell: "curl -X POST -H 'Content-type: application/json' --data '{\"text\":\"Deploying to {{ inventory_hostname }}\"}' https://hooks.slack.com/services/XXX"
      delegate_to: localhost
      # DON'T: Using raw shell commands breaks idempotency and makes error handling difficult.

# CORRECT: Using the clean 'uri' module with delegate_to: localhost
- name: Send the slack notification optimally
  hosts: app_servers
  tasks:
    - name: Send a slack message via the uri module
      ansible.builtin.uri:
        url: https://hooks.slack.com/services/XXX
        method: POST
        body_format: json
        body:
          text: "Ansible successfully configured the server: {{ inventory_hostname }}"
      delegate_to: localhost # ✓ Executing the uri module locally from our control node

If we want to write the same thing using local_action, the format looks like this:

# CORRECT: Using the alternative local_action syntax
- name: Record a local deployment log
  hosts: app_servers
  tasks:
    - name: Write to the local log file
      local_action:
        module: ansible.builtin.lineinfile
        path: /var/log/ansible_runs.log
        line: "Host {{ inventory_hostname }} successfully configured at {{ ansible_date_time.iso8601 }}"
        create: true
      # ✓ The lineinfile module runs on localhost

Although local_action is still fully supported, we’re advised to use delegate_to: localhost for code readability by other team members.

delegate_facts for Distributing Information Between Hosts #

By default, if a task is delegated to another host using delegate_to, the facts collected by that task (for example if we run the setup module or a module registering new variables) are still stored under the original target host name in the hostvars memory variable.

However, sometimes we deliberately delegate a task to fetch specific information from the delegated host and we want that information stored as facts belonging to that delegated host so other hosts can use it later. To achieve this scenario, we use the delegate_facts: true directive.

delegate_facts Use Case #

Imagine we’re configuring a group of web servers (web_servers) that need the internal IP address of a database master located in another server group (db_servers). We can delegate the IP lookup task to that database server, store its facts directly on that database server, then access them from our web servers.

Here’s the practical implementation:

# CORRECT: Collecting the IP from the database server and distributing it to web servers using delegate_facts
---
- name: Connect the Web Servers with the Database Cluster
  hosts: web_servers
  vars:
    db_host_target: "db-master.our.internal"
  tasks:
    - name: Collect facts from the database server in a delegated manner
      ansible.builtin.setup:
        filter: ansible_default_ipv4
      delegate_to: "{{ db_host_target }}"
      delegate_facts: true # ✓ Storing the default_ipv4 facts to db-master.our.internal, not to web_servers
      run_once: true        # Just run it once, no need to repeat for every web server

    - name: Write the database configuration file on all web servers
      ansible.builtin.template:
        src: db_config.j2
        dest: /var/www/html/config.php
        mode: '0600'
      vars:
        # We access the facts belonging to db-master.our.internal filled by the previous task
        database_ip: "{{ hostvars[db_host_target]['ansible_default_ipv4']['address'] }}"

Without delegate_facts: true, the database IP facts above would be stored under the first web host processing that task, which confuses other developers when re-reading our template code because the variables are disorganized.

Production Case Study: Load Balancer Cordon and Uncordon #

Let’s discuss a real-world case study very often encountered in DevOps production environments: Zero-Downtime Rolling Upgrade.

We have a web application cluster with 3 servers behind an HAProxy load balancer. If we stop the web service and deploy to all servers simultaneously, our application users experience downtime. To prevent this, we must use the rolling upgrade technique:

  1. Cordon: Contact the load balancer to disable new traffic to Server A.
  2. Connection Draining: Wait a few seconds so active HTTP connections on Server A finish processing naturally.
  3. Upgrade: Perform application code updates, migrations, and service restarts on Server A.
  4. Health Check: Do a local verification that the application port on Server A responds correctly.
  5. Uncordon: Re-enable Server A on the load balancer to start accepting new traffic.
  6. Repeat steps 1-5 for Server B and Server C in rotation.

This rolling upgrade process is visualized through the following sequence diagram:

sequenceDiagram
    participant CN as Control Node
    participant LB as HAProxy Load Balancer
    participant WS as Web Server (Target)

    CN->>LB: Cordon (Disable traffic to the Web Server via delegate_to)
    Note over LB: HAProxy changes the target server status to MAINT
    CN->>CN: Connection Draining (Wait 10 seconds)
    CN->>WS: Deploy & Upgrade the Application (New code & Service Restart)
    CN->>WS: Local Health Check (Ensure the webapp port responds 200 OK)
    WS-->>CN: Healthy (200 OK)
    CN->>LB: Uncordon (Re-enable the Web Server via delegate_to)
    Note over LB: HAProxy changes the target server status to READY

Here’s a complete, functional production Ansible playbook example for running the scenario above by manipulating the HAProxy statistics API in a delegated manner:

# CORRECT: Rolling upgrade playbook with HAProxy load balancer cordon/uncordon using delegate_to
---
- name: Zero-Downtime Application Rolling Upgrade
  hosts: web_servers
  serial: 1  # ✓ Very important! Executing servers one by one in rotation
  vars:
    app_port: 8080
    lb_control_host: "lb-prod.our.internal"
    lb_backend_name: "be_webapp"
    app_dir: "/var/www/webapp"

  tasks:
    - name: 1. CORDON - Disable the server on the HAProxy Load Balancer
      ansible.builtin.uri:
        url: "http://{{ lb_control_host }}:9000/stats"
        method: POST
        user: admin
        password: securepassword123
        force_basic_auth: true
        body_format: form-urlencoded
        body:
          # We send the backend/server command to enter maintenance mode (MAINT)
          s: "{{ lb_backend_name }}/{{ inventory_hostname }}"
          action: disable
      delegate_to: "{{ lb_control_host }}" # ✓ The API action is called from the load balancer server itself
      changed_when: true

    - name: 2. DRAINING - Give active connections time to finish
      ansible.builtin.pause:
        seconds: 15
      # This pause step runs locally on the control node to pause the transition process

    - name: 3. UPGRADE - Pull the latest application code from the repository
      ansible.builtin.git:
        repo: "[email protected]:us/webapp.git"
        dest: "{{ app_dir }}"
        version: "tags/v2.1.5"

    - name: 4. UPGRADE - Run internal dependency installation
      ansible.builtin.command: npm install --production
      args:
        chdir: "{{ app_dir }}"

    - name: 5. UPGRADE - Restart the webapp application service daemon
      ansible.builtin.systemd:
        name: webapp
        state: restarted

    - name: 6. HEALTH CHECK - Ensure the local application is up and running correctly
      ansible.builtin.uri:
        url: "http://127.0.0.1:{{ app_port }}/health"
        status_code: 200
      register: local_health
      until: local_health.status == 200
      retries: 6
      delay: 5
      # This task ensures we don't put a broken server back into the load balancer

    - name: 7. UNCORDON - Re-enable the server on the HAProxy Load Balancer
      ansible.builtin.uri:
        url: "http://{{ lb_control_host }}:9000/stats"
        method: POST
        user: admin
        password: securepassword123
        force_basic_auth: true
        body_format: form-urlencoded
        body:
          # Re-enabling the server to exit MAINT mode into READY mode
          s: "{{ lb_backend_name }}/{{ inventory_hostname }}"
          action: enable
      delegate_to: "{{ lb_control_host }}" # ✓ Sent back to the load balancer
      changed_when: true

The playbook above shows the power of delegate_to combined with serial: 1. Every time the loop processes one server, it pulls itself out of the global traffic circulation, performs safe installation, verifies its own health locally, and re-enters the load balancer cluster without ever disturbing active users.

run_once for Execution Efficiency #

By default, if a play covers 20 servers, every task in that play runs 20 times (once for each server). However, some operations are global and only need to execute exactly once for the entire play cycle. Classic examples are:

  • Running database migration scripts (for example django-admin migrate or flyway migrate). We only need to run it from one application server, not from 20 servers simultaneously which would actually corrupt the database schema integrity.
  • Creating a global release folder on centralized storage systems (S3 or NAS).
  • Sending one email or Slack notification at the start of the deployment process to inform the team that the automation process has begun.

To save execution time and prevent crashes from duplicate actions, we can attach the run_once: true directive to that task.

How Does run_once Choose the Host? #

When run_once: true is declared without additional parameters, Ansible by default picks the first host registered in the active inventory to execute that task. We can also combine run_once with delegate_to to force that single execution to happen on a specific host, for example on the master database server or on the localhost control node.

Here’s a comparison of the anti-pattern of writing global tasks without run_once vs the correct solution:

# ANTI-PATTERN: Running the database migration on all hosts.
# This causes write races (race condition) and risks corrupting the database cluster!
- name: Web server deployment
  hosts: app_servers
  tasks:
    - name: Run the database schema migration
      ansible.builtin.command: /var/www/webapp/bin/db-migrate
      # DON'T: Without run_once, this task is called by every server in app_servers in parallel.

# CORRECT: Using run_once to limit the migration to a single execution
- name: Optimal web server deployment
  hosts: app_servers
  tasks:
    - name: Run the database schema migration only once
      ansible.builtin.command: /var/www/webapp/bin/db-migrate
      run_once: true # ✓ Guarantees it's only run by the first host in app_servers
      # We can also add delegate_to if we want it processed from a specific node

Below is an example of combining run_once and delegate_to to globally record the release start time on the central monitoring server:

# CORRECT: Combining run_once with delegate_to
- name: Large system deployment
  hosts: web_servers
  tasks:
    - name: Notify the monitoring server that the deployment started
      ansible.builtin.uri:
        url: "http://monitoring.our.internal/api/events"
        method: POST
        body_format: json
        body:
          event: "Deployment started"
          cluster: "web-prod"
      run_once: true                  # ✓ Only send 1 notification event, not 50 events!
      delegate_to: localhost          # ✓ Sent directly from the control node

Summary #

  • delegate_to directs task execution to another specified host, but still carries and maintains the variable and facts context belonging to the original target host (inventory_hostname).
  • local_action and delegate_to: localhost are used to execute tasks locally on the control node, like sending Slack notifications, writing local audit logs, or waiting for external ports to be up.
  • delegate_facts: true forces facts collected from delegated tasks to be stored under the delegating host’s name in the hostvars memory, not under the play’s target host.
  • The combination of serial: 1 with delegate_to: load_balancer is the industry-standard architecture pattern for building zero-downtime deployments (Cordon, Upgrade, Health Check, Uncordon).
  • run_once: true limits task execution to just once for all hosts in the play, crucial for preventing race conditions on global tasks like database schema migrations.

← Previous: Error Handling Next: Performance →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact