Advanced Jinja2 #

Most Ansible users master basic Jinja2 — variables {{ var }}, conditionals {% if %}, loops {% for %}. But Jinja2 is far more expressive than that. Reusable macros, namespaces for mutating variables inside loops, elegant filter chaining — these techniques turn complex templates into readable, maintainable code. This article discusses Jinja2 features that are often overlooked but very useful in production environments, especially when dealing with non-uniform configuration data. Before reading this article, make sure you’re comfortable with basic loops, filters, and conditionals — everything here is built on the foundations discussed in the previous section.

The Template Rendering Pipeline #

To understand why these advanced techniques matter, it’s worth knowing what happens when Ansible processes a template. Jinja2 isn’t just a “variable inserter” — it’s a complete template engine with a lexer, parser, compiler, and runtime. Understanding this pipeline helps explain several behaviors that initially look strange, like why set inside a loop doesn’t work as expected.

flowchart LR
    A["Template .j2"] --> B["Lexer"]
    B --> C["Parser"]
    C --> D["AST"]
    D --> E["Compiler"]
    E --> F["Python code"]
    F --> G["Execution context"]
    G --> H["Final string output"]
    Var["Ansible Variables"] --> G
    Filt["Filter plugins"] --> G
    Test["Test plugins"] --> G

What we need to notice from this diagram: Jinja2 compiles the template to Python code, then executes that code. This explains two important things. First, variables set inside a loop actually create a new variable in the loop’s local scope, not modifying the variable in the outer scope — because in Python, variables defined inside a for loop aren’t visible outside the loop. Second, every time Ansible renders the same template, it recompiles (unless cached) — very large templates can add unnecessary latency.

Whitespace control with -: Jinja2 produces output exactly as written, including spaces and newlines. For whitespace-sensitive configurations (nginx, systemd units, YAML itself), use {{- expr }} (remove whitespace to the left) or {%- block -%} (remove on both sides). This trick is very useful but easy to miss until our first output is full of weird empty lines.

Macros: Reusable Callable Functions #

Macros are the way to define template blocks that can be called repeatedly like functions. Without macros, the only way to repeat a template block is copy-paste — and we know how fragile that approach is when configuration needs to change. Macros introduce much-needed abstraction.

{# templates/nginx.conf.j2 #}

{# Define a macro for the server block #}
{% macro server_block(server_name, port, root, ssl=false) %}
server {
    listen {{ port }}{% if ssl %} ssl{% endif %};
    server_name {{ server_name }};
    root {{ root }};

    {% if ssl %}
    ssl_certificate /etc/ssl/certs/{{ server_name }}.crt;
    ssl_certificate_key /etc/ssl/private/{{ server_name }}.key;
    {% endif %}

    location / {
        try_files $uri $uri/ =404;
    }
}
{% endmacro %}

{# Use the macro for every vhost #}
{% for vhost in nginx_vhosts %}
{{ server_block(vhost.name, vhost.port | default(80), vhost.root) }}
{% if vhost.ssl | default(false) %}
{{ server_block(vhost.name, 443, vhost.root, ssl=true) }}
{% endif %}
{% endfor %}

What makes macros powerful is parameters with default values (ssl=false), and the ability to pass filters into parameters (vhost.port | default(80)). As our vhost data grows — say additional proxy_pass, client_max_body_size, or add_header — we just add a parameter to the macro, and all calls automatically get the new parameter with a sensible default. No copy-paste needed.

ANTI-PATTERN vs CORRECT: Macro vs Copy-Paste #

{# ANTI-PATTERN: copy-paste the server block for every vhost #}
{% for vhost in nginx_vhosts %}
server {
    listen {{ vhost.port | default(80) }};
    server_name {{ vhost.name }};
    root {{ vhost.root }};
    location / {
        try_files $uri $uri/ =404;
    }
}
{% if vhost.ssl | default(false) %}
server {
    listen 443 ssl;
    server_name {{ vhost.name }};
    root {{ vhost.root }};
    ssl_certificate /etc/ssl/certs/{{ vhost.name }}.crt;
    ssl_certificate_key /etc/ssl/private/{{ vhost.name }}.key;
    location / {
        try_files $uri $uri/ =404;
    }
}
{% endif %}
{% endfor %}

{# CORRECT: define the macro once, call with parameters #}
{% macro server_block(server_name, port, root, ssl=false) %}
server {
    listen {{ port }}{% if ssl %} ssl{% endif %};
    ...
}
{% endmacro %}

{% for vhost in nginx_vhosts %}
{{ server_block(vhost.name, vhost.port | default(80), vhost.root) }}
{% if vhost.ssl | default(false) %}
{{ server_block(vhost.name, 443, vhost.root, ssl=true) }}
{% endif %}
{% endfor %}

A real-world scenario where macros are very valuable: HAProxy templates with dozens of backends, Prometheus alert rule templates with dozens of alerts, systemd unit templates with Environment variations. Every time we find ourselves writing the same block more than twice, macros are the answer.

Namespaces: Mutable Variables in Loops #

The most frustrating problem for Jinja2 beginners: we need an accumulator in a loop, but a regular set inside the loop can’t be accessed outside the loop. namespace solves this. But before jumping to the solution, let’s understand why the problem exists. Jinja2 inherits Python’s scope behavior — variables defined inside a for block don’t escape that block. This is a language safety feature, but it feels odd for those coming from languages with global variables.

{# ANTI-PATTERN: variables in a loop can't be mutated and read outside #}
{% set found = false %}
{% for server in servers %}
  {% if server.role == 'primary' %}
    {% set found = true %}   {# This does NOT change 'found' outside the loop! #}
  {% endif %}
{% endfor %}
{{ found }}  {# Still false! #}

{# CORRECT: use a namespace #}
{% set ns = namespace(found=false, primary_server='') %}
{% for server in servers %}
  {% if server.role == 'primary' %}
    {% set ns.found = true %}
    {% set ns.primary_server = server.hostname %}
  {% endif %}
{% endfor %}
{# Now ns.found and ns.primary_server hold the correct values #}
Primary server: {{ ns.primary_server }}
Found: {{ ns.found }}

The namespace object is a mutable container. Unlike regular set which creates a new variable in the local scope, ns.found = true modifies the attribute of the same object declared in the outer scope. Because the object is the same, changes inside the loop are visible outside the loop. This trick also works for deeper scopes — a namespace inside a for inside an if inside a for all access the same object.

A more relevant practical example — generate an nginx upstream config with total weight:

{# templates/upstream.conf.j2 #}
{% set ns = namespace(total_weight=0) %}
{% for server in upstream_servers %}
  {% set ns.total_weight = ns.total_weight + server.weight | default(1) %}
{% endfor %}

# Total weight: {{ ns.total_weight }}
upstream {{ upstream_name }} {
    {% for server in upstream_servers %}
    server {{ server.host }}:{{ server.port }} weight={{ server.weight | default(1) }};
    {% endfor %}
}

Variable Lookup Sequence Diagram #

To understand why namespace works but regular set doesn’t, see how Jinja2 looks up variables:

sequenceDiagram
    participant T as "Template"
    participant L as "Local Scope (for loop)"
    participant E as "Enclosing Scope"
    participant G as "Global Scope"

    T->>L: Look up the 'found' variable
    L->>L: Is there a local 'found'? (for set inside a loop, YES but scoped locally)
    L-->>T: Return the local value
    Note over T,L: But after the loop ends, the local scope is discarded

    T->>E: Look up the 'ns' variable
    E->>G: Look up in the parent scope
    G-->>E: 'ns' is here (a namespace object)
    E-->>T: Return the ns object
    T->>T: ns.found = true → mutates the object
    Note over T,G: Mutation on the same object, visible in all scopes

This visualization explains one important thing: set creates a new variable in the scope, while ns.attr = value modifies the attribute of an existing object. Because the ns object is a reference (not a copy), the modification is visible in every scope holding the same reference.

Expressive Filter Chaining #

Filters are Jinja2’s main weapon for data transformation. Ansible adds many filters on top of Jinja2’s built-ins — to_yaml, to_json, b64encode, combine, dict2items, and dozens more. The real power comes from the ability to chain filters — one filter’s output becomes the next filter’s input.

vars:
  servers:
    - {name: web-01, role: webserver, env: production, ip: 10.0.1.1}
    - {name: web-02, role: webserver, env: production, ip: 10.0.1.2}
    - {name: db-01,  role: database,  env: production, ip: 10.0.2.1}
    - {name: web-03, role: webserver, env: staging,    ip: 10.1.1.1}

tasks:
  # Fetch only production webservers, extract IPs, sort
  - debug:
      msg: >-
        {{ servers
           | selectattr('env', 'equalto', 'production')
           | selectattr('role', 'equalto', 'webserver')
           | map(attribute='ip')
           | sort
           | list }}        
  # Output: ['10.0.1.1', '10.0.1.2']

  # Group by role, list the group names
  - debug:
      msg: "{{ servers | groupby('role') | map('first') | list }}"

  # Build a dictionary from the list: hostname → ip
  - debug:
      msg: >-
        {{ servers
           | items2dict(key_name='name', value_name='ip') }}        
  # Output: {web-01: 10.0.1.1, web-02: 10.0.1.2, ...}

  # Flatten and deduplicate
  - debug:
      msg: >-
        {{ servers
           | map(attribute='role')
           | unique
           | sort
           | list }}        
  # Output: ['database', 'webserver']

Table of the Most Useful Filters #

ANTI-PATTERN vs CORRECT: Filter Chain vs Manual Loop #

# ANTI-PATTERN: write a manual loop for a transformation that fits on one line
tasks:
  - name: Fetch production webserver IPs
    set_fact:
      prod_web_ips: "{{ prod_web_ips | default([]) + [item.ip] }}"
    loop: "{{ servers }}"
    when:
      - item.env == 'production'
      - item.role == 'webserver'
  - name: Sort the result
    set_fact:
      prod_web_ips: "{{ prod_web_ips | sort }}"

# CORRECT: one filter chain, easier to read and faster
tasks:
  - name: Fetch production webserver IPs
    debug:
      msg: >-
        {{ servers
           | selectattr('env', 'equalto', 'production')
           | selectattr('role', 'equalto', 'webserver')
           | map(attribute='ip')
           | sort
           | list }}        

It’s not just an aesthetic issue — a filter chain executes in a single pass on the Ansible controller side, while a loop with set_fact creates a new variable in every iteration written to the fact cache. For small lists the difference isn’t felt, but for large inventories with hundreds of hosts and complex lists, filter chains can be 10x faster.

Be careful with mandatory in different environments. The mandatory filter raises an error if a variable is undefined — useful for preventing bugs with missing configuration. But don’t use mandatory on values we expect to be empty in some environments (e.g. optional_feature: ""). Use default('') or default(none) for those cases.

Filter Chain State Diagram #

A filter chain is actually a pipeline — each filter receives input, transforms it, and sends it to the next filter. This visualization helps when debugging “why is this filter output weird” — we can isolate which filter introduces the problem:

stateDiagram-v2
    [*] --> InputList: "list servers"
    InputList --> AfterSelect1: "selectattr env=production"
    AfterSelect1 --> AfterSelect2: "selectattr role=webserver"
    AfterSelect2 --> AfterMap: "map attribute=ip"
    AfterMap --> AfterSort: "sort"
    AfterSort --> FinalList: "list"
    FinalList --> [*]: "['10.0.1.1', '10.0.1.2']"

    note right of AfterSelect1
      3 of 4 servers pass
    end note
    note right of AfterMap
      List of dicts → list of strings
    end note

When debugging, insert a length filter in the middle to see how many items pass at each stage. {{ servers | selectattr('env', 'equalto', 'production') | length }} will show 3 (if 3 of 4 servers are production). This way we can pinpoint at which filter the expected transformation didn’t happen.

Custom Tests: Reusable Boolean Logic #

In addition to filters, Jinja2 supports tests — boolean functions used with is. This differs from filters: filters transform data, tests only return True/False. Tests are used with is, while filters are used with |. This separation isn’t just cosmetic — tests can execute faster because they can short-circuit ({% if X is test %} doesn’t need to process all the data).

# filter_plugins/custom_tests.py
# Ansible looks for tests in plugins/test/ in a collection, or in filter_plugins/

import ipaddress
import re


def is_private_ip(ip):
    """Test whether an IP is a private address."""
    try:
        addr = ipaddress.ip_address(ip)
        return addr.is_private
    except (ValueError, TypeError):
        return False


def is_valid_hostname(hostname):
    """Test whether a hostname is valid (letters, numbers, and hyphens)."""
    if not isinstance(hostname, str):
        return False
    pattern = r'^[a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?)*$'
    return bool(re.match(pattern, hostname))


def is_in_maintenance_window(current_hour, start, end):
    """Test whether the current hour is within a maintenance window."""
    if start <= end:
        return start <= current_hour < end
    else:
        # A window crossing midnight
        return current_hour >= start or current_hour < end


class FilterModule:
    def filters(self):
        return {}

    def tests(self):
        return {
            'private_ip': is_private_ip,
            'valid_hostname': is_valid_hostname,
            'in_maintenance_window': is_in_maintenance_window,
        }

{# Use a custom test in a template #}
{% for server in servers %}
  {% if server.ip is private_ip %}
  # Internal server: {{ server.name }}
  {% endif %}
{% endfor %}

{# In an Ansible task #}
- name: Validate the hostname
  assert:
    that:
      - inventory_hostname is valid_hostname
    fail_msg: "Hostname '{{ inventory_hostname }}' is invalid!"

- name: Skip the deployment if in the maintenance window
  debug:
    msg: "Maintenance in progress, skipping the batch"
  when:
    - ansible_date_time.hour is in_maintenance_window(2, 4)

ANTI-PATTERN vs CORRECT: Test vs Filter or Hardcode #

{# ANTI-PATTERN: hardcode complex logic in the template, not reusable #}
{% if server.ip.startswith('10.') or server.ip.startswith('192.168.') or server.ip.startswith('172.') %}
# Internal server
{% endif %}

{# CORRECT: use a test defined once and reusable #}
{% if server.ip is private_ip %}
# Internal server
{% endif %}

The ANTI-PATTERN version is fragile: it doesn’t cover all private IP ranges (there’s 172.16.0.0/12, IPv6 ULA, CGNAT 100.64.0.0/10), and if the logic needs to change (e.g. add IPv6 ULA support), we have to change every template using it. The CORRECT version — private IP logic is handled by the comprehensive, always up-to-date Python ipaddress module.

Handling Hierarchical Data #

Templates for configurations with complex nested data structures are one of Jinja2’s most challenging use cases. On one hand we want the template to stay readable. On the other, data can be missing fields (requiring default), have varying levels, or be wrong. Here’s an HAProxy template example for multi-service with dynamic backend lists:

{# templates/haproxy.cfg.j2 #}
{# Expected input data:
   services:
     - name: api
       port: 80
       backends:
         - host: 10.0.1.1
           port: 8080
           weight: 2
         - host: 10.0.1.2
           port: 8080
           weight: 1
       health_check:
         path: /health
         interval: 5s
#}

{% for service in services %}
frontend {{ service.name }}_front
    bind *:{{ service.port | default(80) }}
    default_backend {{ service.name }}_back

backend {{ service.name }}_back
    balance {{ service.balance | default('roundrobin') }}
    {% if service.health_check is defined %}
    option httpchk GET {{ service.health_check.path | default('/health') }}
    {% endif %}

    {% for backend in service.backends %}
    server {{ service.name }}_{{ loop.index }}
        {{- ' ' + backend.host + ':' + backend.port | string }}
        {{- ' weight=' + backend.weight | default(1) | string }}
        {%- if service.health_check is defined %}
        {{- ' check inter ' + service.health_check.interval | default('10s') }}
        {%- endif %}
    {% endfor %}

{% endfor %}

What to notice in this template: there are three uses of is defined to check whether a key exists before accessing it. This is important because Ansible won’t error when the template renders a missing key in the data — but it will write an empty string, which is usually worse than an explicit error. The “check first, then use” pattern makes templates robust against incomplete data.

Table of Patterns for Hierarchical Data #

Recursive loops for tree data: Jinja2 supports recursive loops with the recursive modifier, useful for tree structures like recursive DNS zone configs or nested ACL groups. The pattern {% for item in items recursive %}{{ loop( item.children ) }}{% endfor %} is very useful but rarely used — learn it when you encounter such a case.

Jinja2 in Variables (Not Only Templates) #

Jinja2 isn’t only for template files — it can also be used in variable values. This is very useful for building dynamic strings from multiple variables, or for making conditional values without writing a separate task.

# group_vars/all.yml
app_log_dir: "/var/log/{{ app_name }}"
db_url: "postgresql://{{ db_user }}:{{ vault_db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}"
backup_filename: "backup-{{ inventory_hostname }}-{{ ansible_date_time.date }}.tar.gz"

# Conditional expressions in variables
nginx_worker_processes: "{{ ansible_processor_vcpus * 2 }}"
max_open_files: >-
  {{ '65536' if ansible_memtotal_mb > 8192 else '32768' }}  

# Dynamic lists based on the inventory
monitoring_endpoints: >-
  {{ groups['appservers']
     | map('extract', hostvars, 'ansible_host')
     | map('regex_replace', '^', 'http://')
     | map('regex_replace', '$', ':9090/metrics')
     | list }}  

The main advantage of using Jinja2 in variables is being able to defer evaluation to when the variable is used. Variables defined earlier in the same file can reference each other because all variables are re-evaluated every time Ansible loads the variable file.

ANTI-PATTERN vs CORRECT: Jinja2 in Variables vs Tasks #

# ANTI-PATTERN: create tasks for variables that can be written directly
tasks:
  - name: Calculate the log directory
    set_fact:
      app_log_dir: "/var/log/{{ app_name }}"
  - name: Set the DB URL
    set_fact:
      db_url: "postgresql://{{ db_user }}:{{ vault_db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}"

# CORRECT: write Jinja2 directly in variables, no extra tasks needed
vars:
  app_log_dir: "/var/log/{{ app_name }}"
  db_url: "postgresql://{{ db_user }}:{{ vault_db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}"

The ANTI-PATTERN version adds two tasks that must run just to calculate strings. This slows down the playbook, adds noise to the logs, and creates new facts that are actually just derivatives of existing variables. The CORRECT version writes variables directly as Jinja2 expressions — evaluated only when the variable is accessed, and adds no tasks.

Summary #

• Macros for repeated template blocks — define once, call many times with different parameters, exactly like functions. Default-value parameter support makes macros flexible for configurations with variations.
• namespace for loop accumulators — regular variables set inside a loop can’t be accessed after the loop ends because of Python scoping. set ns = namespace(...) then ns.attr = value modifies an object visible in all scopes.
• Filter chaining with selectattr, map, groupby, unique, sort — complex list transformations can be written in one expressive expression. Faster than manual loops with set_fact per iteration.
• Custom tests (is private_ip, is valid_hostname) for reusable boolean logic in templates and assert tasks. Tests differ from filters: tests for booleans, filters for transformations, and is vs |.
• Use - (dash) to remove whitespace in Jinja2: {{- expr }} or {%- block -%} — produces cleaner output especially for whitespace-sensitive configurations like nginx and systemd.
• Jinja2 can be used in variable values, not just template files — very useful for building dynamic strings from multiple variables, or conditional values like {{ '65536' if memtotal > 8192 else '32768' }}.
• is defined to check whether a key exists, | default(val) for default values, and | default(val, true) for defaults only when undefined (not for false) — these three patterns cover 90% of incomplete data cases.
• Filter pipelines are debuggable: insert | length in the middle to see how many items pass at each stage, so we can pinpoint at which filter the expected transformation didn’t happen.

← Previous: Collection Next: Strategy & Serial →