Tracing #

Distributed tracing is the industry standard for mapping how a request flows across dozens of microservices in production environments. However, this observability technique isn’t only useful for tracking our application code. In the world of infrastructure automation, a deployment process using Ansible Playbooks often consists of hundreds of tasks running on dozens of target servers in parallel or sequentially. When the deployment process suddenly slows down, plain text logs in the terminal aren’t enough to help us identify which task is the bottleneck or which host is experiencing high latency. This article discusses distributed tracing integration using OpenTelemetry and Jaeger to visually map Ansible Playbook task execution, configure collector agents, and analyze traces to optimize our automation.

OpenTelemetry and Jaeger Integration #

To build an execution tracking system, we need to understand the two main components in the modern distributed tracing ecosystem: OpenTelemetry (OTel) and Jaeger.

Comparison with Traditional APM Solutions #

Before OpenTelemetry became the industry standard, developers and operations teams had to install proprietary APM (Application Performance Monitoring) agents specific to each vendor (like New Relic, Dynatrace, or Datadog). These old agents were closed-source, consumed high system resources because they aggressively did bytecode instrumentation, and locked us into a single vendor (vendor lock-in).

OpenTelemetry breaks those limitations by separating code instrumentation from the backend storage system. Using openly standardized OTel APIs and SDKs, we only need to instrument our applications once. The generated data can be sent to a local collector and forwarded to whatever visualization backends we want without modifying the source code again.

Basic Distributed Tracing Concepts #

At the basic level, distributed tracing works by connecting event chains through trace metadata:

• Trace ID: A unique 128-bit identifier representing one complete transaction flow. All components involved in this transaction must attach the same Trace ID to their logs and telemetry.
• Span: A representation of a single unit of work. Spans have a start time, end time, and a series of custom attributes (tags) providing additional context.
• Span ID: A unique 64-bit identifier to distinguish one unit of work from another within the same trace.
• Parent ID: An identifier pointing to the Span ID that triggered the current unit of work. This parent-child relationship is what allows visualization systems to compose precise tree diagrams (waterfall diagrams).

In our automation ecosystem, we integrate both so Ansible can export execution details of every module directly to the trace backend without disrupting the playbook flow:

flowchart TD
    subgraph PlaybookRun["Playbook Execution Process"]
        A["Ansible CLI (Playbook Run)"] -->|"Start Task"| B["Callback Plugin (OTel)"]
        B -->|"Create root span"| C["Play Span"]
        C -->|"Create child span"| D["Task Span (e.g. Apt Install)"]
    end
    subgraph TracingPipeline["Tracing Data Path"]
        B -->|"Send OTLP (gRPC)"| E["OpenTelemetry Collector Agent"]
        E -->|"Batch & Export"| F["Jaeger Backend"]
        F -->|"Visualization"| G["Jaeger UI / Grafana"]
    end

The Ansible controller acts as a data producer. Through the callback plugin, it sends a span every time a play starts, a task runs, or a handler is triggered. The OpenTelemetry Collector agent receives this data locally, then forwards it to the central Jaeger server for indexing.

Setting Up Collector Agents with Ansible #

The first step in this implementation is deploying the Jaeger backend and the OpenTelemetry Collector agent on our monitoring servers using Ansible automation.

1. Deploy the Jaeger Backend #

We’ll run Jaeger All-in-One using Docker so the installation process and local in-memory storage database configuration happen quickly.

# playbooks/deploy_jaeger.yml
---
- name: Deploy the Jaeger Tracing Backend
  hosts: monitoring_servers
  become: true
  tasks:
    - name: Ensure Docker is installed
      apt:
        name: docker.io
        state: present
        update_cache: true

    - name: Run the Jaeger All-in-One container
      docker_container:
        name: jaeger
        image: jaegertracing/all-in-one:1.57
        state: started
        restart_policy: always
        published_ports:
          - "16686:16686" # HTTP port for the Jaeger Web UI
          - "4317:4317"   # OTLP gRPC receiver port
          - "4318:4318"   # OTLP HTTP receiver port
        env:
          COLLECTOR_OTLP_ENABLED: "true"

2. Deploy the OpenTelemetry Collector Agent #

We install the OpenTelemetry Collector on the controller server side or network gateway to collect, process, and forward execution spans to Jaeger.

# roles/otel_collector/tasks/main.yml
---
- name: Download the OTEL Collector deb package
  get_url:
    url: "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.98.0/otelcol-contrib_0.98.0_linux_amd64.deb"
    dest: "/tmp/otelcol.deb"
    mode: '0640'

- name: Install the OTEL Collector deb package
  apt:
    deb: "/tmp/otelcol.deb"
    state: present

- name: Deploy the otelcol configuration file
  template:
    src: otel-collector-config.yaml.j2
    dest: /etc/otelcol-contrib/config.yaml
    owner: otelcol-contrib
    group: otelcol-contrib
    mode: '0640'
  notify: Restart otelcol-contrib

- name: Ensure the otelcol service is running and enabled
  systemd:
    name: otelcol-contrib
    state: started
    enabled: true

Here’s the /etc/otelcol-contrib/config.yaml configuration template file we use to route data to Jaeger. We include several industry-standard processors like memory_limiter to prevent the agent from running out of memory (out-of-memory) when handling high loads, and batch to group data before sending it over the network:

{# roles/otel_collector/templates/otel-collector-config.yaml.j2 #}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    timeout: 1s
    send_batch_size: 256
    send_batch_max_size: 512

exporters:
  otlp/jaeger:
    endpoint: "{{ jaeger_internal_url }}:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]

Tracing Playbook Tasks: Tracing the Ansible Cycle #

The most important part of this guide is configuring Ansible to actively send tracing data to our collector. Since version 2.14+, Ansible provides a native OpenTelemetry-based callback plugin we can enable easily.

1. ansible.cfg Configuration #

We must enable that callback plugin in our project’s configuration file so Ansible loads the telemetry module before executing the server inventory. Edit our project’s ansible.cfg file:

# ansible.cfg
[defaults]
# Enable the otel callback for trace delivery
callbacks_enabled = community.general.opentelemetry
stdout_callback = yaml

2. Environment Variable Configuration for the OTel Callback #

Ansible’s OpenTelemetry callback plugin reads data delivery configuration through standard W3C environment variables. Before running the playbook, we must export those variables to route spans to the OTEL Collector endpoint:

# Set our local collector endpoint (using the gRPC protocol)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

# Give a service name that reflects our automation task
export OTEL_SERVICE_NAME="ansible-infrastructure-deploy"

# Add custom metadata to all traces
export OTEL_RESOURCE_ATTRIBUTES="environment=production,executor=jenkins-agent-01"

3. CI/CD Pipeline Integration (Jenkinsfile) #

In modern workflows, Ansible Playbook execution is almost always triggered from a centralized CI/CD system. We can integrate these OpenTelemetry environment variables into our Jenkins pipeline, capture the Trace ID from stdout, and display a direct link to the Jaeger dashboard so developer teams can self-serve when deployments are slow.

Here’s an example snippet of the execution stage inside a Jenkinsfile:

stage('Ansible Deploy') {
    environment {
        OTEL_EXPORTER_OTLP_ENDPOINT = 'http://otel-collector.internal:4317'
        OTEL_SERVICE_NAME = "ansible-pipeline-${env.JOB_NAME}"
        OTEL_RESOURCE_ATTRIBUTES = "jenkins.build_number=${env.BUILD_NUMBER},env=production"
    }
    steps {
        script {
            // Run the playbook and record its output
            sh 'ansible-playbook -i inventories/production playbooks/deploy_app.yml'
            
            // Record a help message for the operations team
            echo "----------------------------------------------------------------------"
            echo "Deployment trace sent to the OpenTelemetry Collector."
            echo "To see the visual task execution duration analysis, open the Jaeger UI:"
            echo "http://jaeger.internal.zone:16686/search?service=ansible-pipeline-${env.JOB_NAME}"
            echo "----------------------------------------------------------------------"
        }
    }
}

Mapping Execution Traces and Analyzing Bottlenecks #

With traces stored in Jaeger, we can start mapping executions and analyzing why a task runs slowly. The biggest advantage of timeline visualization is being able to see the overlap of task executions running in parallel when we configure the forks parameter in Ansible.

Here’s a sequence diagram illustrating how trace spans are created and propagated when the Ansible Controller executes a playbook on two target hosts in parallel:

sequenceDiagram
    participant CLI as "Ansible Controller CLI"
    participant CB as "OTel Callback Plugin"
    participant COL as "OTel Collector"
    participant H1 as "Target Host 1 (Web)"
    participant H2 as "Target Host 2 (DB)"

    CLI->>CB: "Playbook start: deploy.yml"
    CB->>COL: "Start root span (deploy.yml)"
    CLI->>H1: "Run Task 1 (Setup Nginx)"
    CB->>COL: "Start child span (Setup Nginx - Host 1)"
    H1-->>CLI: "Task 1 completed"
    CB->>COL: "End child span (Setup Nginx - Host 1)"

    CLI->>H2: "Run Task 2 (Setup PostgreSQL)"
    CB->>COL: "Start child span (Setup DB - Host 2)"
    H2-->>CLI: "Task 2 completed"
    CB->>COL: "End child span (Setup DB - Host 2)"

    CLI->>CB: "Playbook finished"
    CB->>COL: "End root span (deploy.yml)"

Span Attributes Generated by Ansible #

Every span sent by the Ansible callback has a series of standard metadata very valuable for query filter analysis in Jaeger. Here’s the list of key attributes we can search:

Analyzing Hidden Bottlenecks #

When examining trace visualizations in the Jaeger UI, we must look for the following patterns to find bottlenecks:

1. Blank Gaps: If there’s a long empty time gap between one task finishing and the next task starting, this indicates internal processing latency on the Ansible Controller (for example, the controller is slow at compiling very complex local Jinja2 templates).
2. Extreme Single Module Duration: If a task with the apt module takes up to 3 minutes, we can check the span tags to see whether the target server is waiting on a package manager lock (dpkg lock) or has a slow repository mirror connection.
3. Inefficient Serial Patterns: If we see tasks running sequentially (step-by-step) on 20 servers, we can consider raising the default forks value in ansible.cfg (for example, from 5 to 20) so tasks run in parallel and save up to 75 percent of deployment time.

By visually tracking playbook executions, we no longer guess which part is blocking our automation.

Tracing Playbook Anti-Patterns and Solutions #

Implementing distributed tracing on automation tools like Ansible has its own challenges different from microservice application tracing.

1. Enabling Tracing for Every Ad-Hoc Execution #

• Anti-Pattern: Permanently configuring OTel environment variables in /etc/environment on the operations team’s local machines. Every time the team runs small ad-hoc commands like ansible all -m ping, garbage trace data is sent to Jaeger, crowding the storage database and obscuring important deployment traces.
• Solution: Enable tracing conditionally. We should only export OpenTelemetry environment variables inside CI/CD pipeline scripts (like Jenkins, GitLab CI, or GitHub Actions) when running official release playbooks.

2. Hardcoding the Collector Endpoint in ansible.cfg #

• Anti-Pattern: Writing the OTEL Collector server IP address statically inside the Git project’s ansible.cfg file. This makes the configuration inflexible when we have to run playbooks in different network environments (for example, from a local laptop via VPN vs from an internal CI/CD runner).
• Solution: Let the ansible.cfg file only define the callback plugin loading, while the collector endpoint address is dynamically injected using the OTEL_EXPORTER_OTLP_ENDPOINT environment variable when the execution process starts.

3. Sending Data Without Encryption (mTLS) Across Public Networks #

• Anti-Pattern: Sending OTLP traces directly over the public WAN without encryption (using plain HTTP or insecure gRPC). Attackers can sniff data in transit and see the entire internal server structure, task names, and our system metadata.
• Solution: Configure TLS and mTLS authentication on the OpenTelemetry Collector and Jaeger. Make sure client certificates are deployed to the Ansible controller server and referenced in the transport configuration parameters.

Here’s an example of a secure exporter configuration using TLS encryption in the OpenTelemetry Collector file:

# Example of an encrypted exporter configuration in otel-collector-config.yaml
exporters:
  otlp/secure-jaeger:
    endpoint: "jaeger-prod.internal.zone:4317"
    tls:
      insecure: false
      ca_file: "/etc/otelcol-contrib/certs/ca.crt"
      cert_file: "/etc/otelcol-contrib/certs/client.crt"
      key_file: "/etc/otelcol-contrib/certs/client.key"

4. Ignoring Network Overhead on Task Loops #

• Anti-Pattern: Using Ansible task loops (with_items or loop) to process hundreds of small items, where each iteration triggers creating a new span. This causes a span explosion that overloads the controller network.
• Solution: As much as possible use Ansible modules that support bulk operations, like the apt module with a direct package list, instead of looping to call the apt module repeatedly. This not only speeds up execution but also keeps trace sizes clean and concise.

Here’s an example comparison between the bad and optimal task loop implementations:

# ANTI-PATTERN: Loop triggers creating 3 separate slow spans
- name: Install utility packages one by one
  apt:
    name: "{{ item }}"
    state: present
  loop:
    - curl
    - htop
    - git
  # ✗ Triggers 3 separate OTLP connections from the controller to the collector

# CORRECT: Bulk operation produces 1 concise, fast span
- name: Install utility packages in bulk
  apt:
    name:
      - curl
      - htop
      - git
    state: present
  # ✓ Only triggers 1 single span for the entire package installation

By optimizing task writing, we keep our telemetry system stable without sacrificing the depth of information we collect.

Summary #

• Distributed tracing on Ansible Playbooks gives visual visibility into our infrastructure automation execution flow, making bottleneck tracking easier.
• OpenTelemetry provides the standard protocol (OTLP) bridging trace data delivery from the Ansible Controller to various visualization backends.
• Jaeger All-in-One can be run quickly using Docker on monitoring servers to store and analyze trace spans.
• The community.general.opentelemetry Callback Plugin is the native Ansible module tasked with recording the execution lifecycle and exporting it as spans.
• Root Spans and Child Spans map hierarchical relationships in Jaeger, separating execution durations from the playbook level, play level, down to individual tasks.
• Forks optimization and identifying blank gaps in the Jaeger UI help us drastically cut deployment time through task parallelization.
• Selective activation of OTel environment variables in CI/CD pipelines prevents the trace database from being crowded by daily ad-hoc testing activity.
• Using bulk modules to replace task loops is very important to prevent span explosions that overload the controller memory.

← Previous: Metric Collection Next: Health Check →