Logging #

A server that can’t be investigated when a problem occurs is a server that can’t be managed. Logs are the most fundamental source of information for understanding what’s happening in a system — but logs scattered across dozens of servers without centralized aggregation are almost as useless as having no logs at all. Automating logging infrastructure setup with Ansible ensures every server sends logs to the right place with a consistent format, from the moment the server is first created — not after the first incident happens and we realize there’s no trace at all to analyze.

This article will take us from the fundamental concept of the three log pillars — application logs, system logs, and access logs — to concrete setup of log shippers, storage backends, log rotation, and integration with the /en/observability/monitoring/ stack we’ve already built. After finishing reading, we’ll have Ansible playbooks and roles ready to use directly for deploying a logging pipeline across our entire server fleet.

Anatomy of a Logging Pipeline #

Before writing Ansible roles, we need to understand the complete flow of a log from the text line the application writes to being queryable on a dashboard. Each component has a specific role, and a failure at one point breaks the entire log delivery chain:

flowchart LR
    A["Application<br/>stdout / file"] --> B["Log Shipper<br/>Filebeat / Promtail"]
    C["OS & Services<br/>syslog / journald"] --> B
    B --> D["Buffer / Queue<br/>in-memory"]
    D --> E["Backend Storage<br/>Elasticsearch / Loki"]
    E --> F["Query Layer<br/>Kibana / Grafana"]
    F --> G["SRE / Developer<br/>Incident investigation"]

    style A stroke:#b45309,stroke-width:2px
    style C stroke:#b45309,stroke-width:2px
    style B stroke:#1d4ed8,stroke-width:2px
    style E stroke:#15803d,stroke-width:2px
    style F stroke:#be185d,stroke-width:2px
    style G stroke:#7e22ce,stroke-width:2px

The log shipper runs on every host. Its task is very specific: read log files, parse each line, and send them to the backend. The log shipper isn’t a place for long-term log storage — that’s the backend’s job. Understanding this responsibility separation is important so we don’t choose the wrong tool: if we need full-text queries with fast indexing, Elasticsearch is the choice. However, if we need a lightweight solution that’s disk-efficient and integrates smoothly with Prometheus, Loki is the best choice.

Logs that reach the backend are only as useful as their format. Plain text without structure forces us to write different regexes for every microservice. Structured JSON with consistent fields (level, timestamp, service, trace_id) makes backend queries a single line of PromQL or KQL. Investing two hours up front to set up structured logging saves hundreds of debugging hours in a year.

Three Log Formats You Must Know #

The log format choice determines how the backend can do querying, aggregation, and alerting. Each has its own characteristics in infrastructure:

Currently, JSON is the default we should choose for new applications. Almost all languages have structured logger libraries (like Python structlog, Go zap, Java Logback + logstash-encoder, and Node pino) that produce JSON without significant overhead. For legacy applications still producing plain text, we can do parsing at the log shipper with pipeline processors — but that’s a workaround, not a long-term solution.

Setting Up Filebeat as a Log Shipper #

Filebeat is Elastic’s lightweight agent that reads log files and sends them to Elasticsearch or Logstash. This choice makes sense if our monitoring stack is already Elastic-based — if not, please jump to the Setting Up Loki as a Backend section below.

# roles/filebeat/tasks/main.yml
---
- name: Add the Elastic GPG key
  apt_key:
    url: https://artifacts.elastic.co/GPG-KEY-elasticsearch
    state: present

- name: Add the Elastic repository
  apt_repository:
    repo: "deb https://artifacts.elastic.co/packages/8.x/apt stable main"
    state: present

- name: Install Filebeat
  apt:
    name: "filebeat={{ filebeat_version }}"
    state: present
    update_cache: true

- name: Deploy the Filebeat configuration
  template:
    src: filebeat.yml.j2
    dest: /etc/filebeat/filebeat.yml
    owner: root
    group: root
    mode: '0644'
  notify: Restart Filebeat

- name: Enable and run Filebeat
  systemd:
    name: filebeat
    state: started
    enabled: true

Pay attention to checksums — in the install task below we’ll see how to add them for security. The Filebeat configuration below adds mandatory fields to every log event: hostname, environment, and service name. These fields are the only way we can filter logs from 200 servers in our fleet — if they’re absent from the start, the service:api-gateway query in Kibana won’t find anything:

{# roles/filebeat/templates/filebeat.yml.j2 #}
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log
      - /var/log/syslog
    fields:
      host: {{ inventory_hostname }}
      environment: {{ env }}
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      service: {{ app_name }}
      host: {{ inventory_hostname }}
      environment: {{ env }}
    fields_under_root: true
    json.keys_under_root: true   # Parse JSON logs automatically
    json.overwrite_keys: true    # JSON fields rise to the root, not nested in 'message'
    json.add_error_key: true     # Track parse errors so failed parses aren't silent

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id"]
      ignore_missing: true

output.elasticsearch:
  hosts: {{ elasticsearch_hosts | to_json }}
  index: "logs-{{ '{{' }} fields.service {{ '}}' }}-%{+yyyy.MM.dd}"
  username: "{{ elasticsearch_username }}"
  password: "{{ vault_elasticsearch_password }}"
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
  ssl.verification_mode: full

setup.template.name: "logs"
setup.template.pattern: "logs-*"
setup.ilm.enabled: true   # Index Lifecycle Management — auto delete old logs

Don’t forget json.overwrite_keys: true when the log input is JSON. Without that flag, all our JSON fields are stored under the message key as a single object, and the level:error query in Kibana won’t find anything. This is the most common bug in Filebeat configuration with a debugging process that can take hours.

Anti-Pattern Pair: Empty Fields vs Mandatory Fields #

The most common case that appears during logging configuration reviews in teams that just set up logging:

# ANTI-PATTERN: only reading /var/log/*.log without adding metadata fields
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log
# Problem: when 50 servers send logs, we can't filter
# "show errors from service X in production" because there's no
# service/host/environment field. Every query must scan the entire dataset.

# CORRECT: add fields to every input
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      service: {{ app_name }}
      host: {{ inventory_hostname }}
      environment: {{ env }}
    fields_under_root: true
# Result: the Kibana query is just `service:api-gateway AND level:error AND
# environment:production` — done in 2 seconds across 50 servers.

Validating Binary Downloads with Checksums #

Filebeat binaries are downloaded directly from the internet. Without checksum verification, one compromised release server could inject a backdoor into our entire fleet. Add this verification to the role:

# roles/filebeat/tasks/main.yml — add after the apt install
- name: Verify the Filebeat binary checksum
  stat:
    path: /usr/share/filebeat/bin/filebeat
    checksum_algorithm: sha256
  register: filebeat_binary_stat

- name: Fail if the checksum doesn't match
  fail:
    msg: "Filebeat binary checksum mismatch — the file is likely corrupted or compromised"
  when: filebeat_binary_stat.stat.checksum != filebeat_expected_sha256
  # Get the value from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.x.x-linux-x86_64.tar.gz.sha512
# Checksum verification isn't paranoia — supply chain attacks against monitoring
# tools are a real scenario. Add the checksum in defaults/main.yml and validate
# before the binary is executed. Five minutes of setup saves a whole fleet compromise.

Log Rotation Management #

Logs that aren’t rotated fill the disk. Set up logrotate for all important logs — and important means: application logs, nginx logs, PostgreSQL logs, not just /var/log/syslog:

# roles/logrotate/tasks/main.yml
---
- name: Deploy the logrotate configuration for the application
  template:
    src: app-logrotate.j2
    dest: "/etc/logrotate.d/{{ app_name }}"
    owner: root
    group: root
    mode: '0644'

- name: Ensure the daily logrotate cron is active
  cron:
    name: "logrotate daily"
    minute: "0"
    hour: "1"
    user: root
    job: "/usr/sbin/logrotate /etc/logrotate.conf"
    cron_file: ansible_logrotate
  when: logrotate_manage_cron | default(false)

{# roles/logrotate/templates/app-logrotate.j2 #}
{{ app_log_dir }}/*.log {
    daily
    rotate {{ log_rotate_days | default(14) }}
    dateext              # date suffix, not .1.gz
    dateformat -%Y%m%d   # YYYYMMDD format
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    copytruncate         # for applications that can't be signalled to reopen
    postrotate
        systemctl reload {{ app_name }} > /dev/null 2>&1 || true
    endscript
}

copytruncate is an important option for applications that don’t handle SIGHUP correctly (like some Java applications): instead of renaming the log file (which makes the application keep writing to the file with the old inode), copytruncate copies the file contents then truncates the original file in place. The application doesn’t need to know anything.

Anti-Pattern Pair: Without dateext vs With dateext #

# ANTI-PATTERN: logrotate default rotation
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
}
# Problem: the resulting files are myapp.log.1.gz, myapp.log.2.gz, ...
# We can't know when those logs were rotated without opening their contents.
# When restoring from backup, the .1 .2 .3 file order isn't informative.

# CORRECT: use dateext + dateformat
/var/log/myapp/*.log {
    daily
    rotate 14
    dateext
    dateformat -%Y%m%d
    compress
}
# Result: myapp.log-20260607.gz, myapp.log-20260606.gz, ...
# Clearly visible: what date this file was rotated on. Searching in backup
# tools (or `ls`) is immediately clear. Plus, the rotation order is unambiguous.

Setting Up Loki as a Lightweight Logging Backend #

For teams that don’t want to manage Elasticsearch, Grafana Loki is a much lighter alternative. Loki stores logs in a Prometheus-like format (label-based, not full-text index), so it fits perfectly with the /en/observability/monitoring/ stack we’ve already deployed:

# roles/loki/tasks/main.yml
---
- name: Create the loki user
  user:
    name: loki
    system: true
    shell: /usr/sbin/nologin
    home: /opt/loki
    create_home: true

- name: Download the Loki binary
  get_url:
    url: "https://github.com/grafana/loki/releases/download/v{{ loki_version }}/loki-linux-amd64.zip"
    dest: /tmp/loki.zip
    mode: '0644'

- name: Extract Loki
  unarchive:
    src: /tmp/loki.zip
    dest: /usr/local/bin/
    remote_src: true
    creates: /usr/local/bin/loki-linux-amd64

- name: Rename the Loki binary
  command: mv /usr/local/bin/loki-linux-amd64 /usr/local/bin/loki
  args:
    creates: /usr/local/bin/loki

- name: Deploy the Loki configuration
  template:
    src: loki-config.yml.j2
    dest: /etc/loki/config.yml
    owner: loki
    mode: '0640'
  notify: Restart Loki

- name: Deploy the Loki systemd unit file
  template:
    src: loki.service.j2
    dest: /etc/systemd/system/loki.service
  notify:
    - Reload systemd
    - Restart Loki

{# roles/loki/templates/loki-config.yml.j2 #}
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: {{ loki_retention_days | default(30) }}d
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_entries_limit_per_query: 5000

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

ruler:
  alertmanager_url: http://alertmanager:9093
  storage:
    type: local
    local:
      directory: /var/lib/loki/rules

Note the retention_period: 30d in limits_config — this is the log storage retention period policy. Set it according to our business compliance needs. The fintech and healthcare industries usually require 1–7 years, while standard startups only need 30–90 days. Removing the retention policy means storing forever, which ends up inflating storage costs.

Promtail: The Log Shipper for Loki #

Promtail is Loki’s agent — lighter than Filebeat for use cases that don’t need Elastic features. Most importantly: Promtail produces Prometheus-style labels we can query using LogQL with syntax similar to PromQL:

# roles/promtail/tasks/main.yml
---
- name: Download and install Promtail
  get_url:
    url: "https://github.com/grafana/loki/releases/download/v{{ promtail_version }}/promtail-linux-amd64.zip"
    dest: /tmp/promtail.zip

- name: Extract Promtail
  unarchive:
    src: /tmp/promtail.zip
    dest: /usr/local/bin/
    remote_src: true
    creates: /usr/local/bin/promtail-linux-amd64

- name: Rename the Promtail binary
  command: mv /usr/local/bin/promtail-linux-amd64 /usr/local/bin/promtail
  args:
    creates: /usr/local/bin/promtail

- name: Deploy the Promtail configuration
  template:
    src: promtail-config.yml.j2
    dest: /etc/promtail/config.yml
  notify: Restart Promtail

- name: Deploy the Promtail systemd unit
  template:
    src: promtail.service.j2
    dest: /etc/systemd/system/promtail.service
  notify:
    - Reload systemd
    - Restart Promtail

{# templates/promtail-config.yml.j2 #}
server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: info

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://{{ loki_host }}:3100/loki/api/v1/push
    batchwait: 1s
    batchsize: 1048576
    backoff_config:
      min_period: 500ms
      max_period: 5m

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: {{ inventory_hostname }}
          env: {{ env }}
          __path__: /var/log/*log

  - job_name: {{ app_name }}
    static_configs:
      - targets:
          - localhost
        labels:
          job: {{ app_name }}
          host: {{ inventory_hostname }}
          env: {{ env }}
          __path__: {{ app_log_dir }}/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
            request_id: request_id
      - labels:
          level:
      - metrics:
          log_lines_total:
            type: Counter
            description: "Total log lines for {{ app_name }}"
            config:
              match_all: true
            action: inc
# Promtail produces the `log_lines_total` metric from the number of log lines
# scraped. This metric can be scraped by Prometheus via
# `http://promtail-host:9080/metrics` — meaning we can alert when the
# log rate drops drastically (an indication the application crashed without writing logs).
# This bridge is very powerful: missing application metrics could mean a missing application,
# or it could also mean a broken logging pipeline.

Structured Logging from Applications #

Log shippers can only group logs if their fields are consistent. Pushing structured logging from applications is far more effective than parsing plain text at the shipper — we have full context (trace ID, user ID, duration) that can never be reliably derived from regex. Example for a Python application:

# app/utils/logger.py
import logging
import sys
import json
from pythonjsonlogger import jsonlogger

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        log_record['timestamp'] = self.formatTime(record, self.datefmt)
        log_record['level'] = record.levelname
        log_record['logger'] = record.name
        log_record['service'] = '{{ app_name }}'  # injected by Ansible
        log_record['environment'] = '{{ env }}'

logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
formatter = CustomJsonFormatter('%(timestamp)s %(level)s %(name)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Usage in the application:
# logger.info("user_login", extra={"user_id": 42, "ip": "10.0.0.1"})
# Output: {"timestamp":"2026-06-07T08:01:23Z","level":"INFO","service":"api",
#          "environment":"production","message":"user_login",
#          "user_id":42,"ip":"10.0.0.1"}

Ansible can be integrated to inject the app_name and env values during deployment — use the template module to create the logger configuration file with values relevant to that environment.

Decision Tree: Filebeat or Promtail? #

The log shipper choice determines the backend and operational costs. Use this decision tree as an initial guide:

flowchart TD
    A["Is there already a<br/>monitoring stack?"] -->|"Elasticsearch"| B["Use Filebeat"]
    A -->|"Grafana + Prometheus"| C["Use Promtail"]
    A -->|"Not yet"| D{"Log volume<br/>per day?"}

    D -->|"< 10 GB"| E["Promtail + Loki<br/>lighter"]
    D -->|"> 50 GB"| F["Filebeat + Elasticsearch<br/>stronger index"]

    B --> G{"Need complex<br/>parsing?"}
    C --> H{"Need integration<br/>with alertmanager?"}
    G -->|"Yes"| I["Filebeat + Logstash<br/>richer pipeline"]
    G -->|"No"| J["Filebeat directly<br/>to Elasticsearch"]
    H -->|"Yes"| K["Promtail + Loki Ruler<br/>built-in alerting"]
    H -->|"No"| L["Standard Promtail"]

    style A stroke:#b45309,stroke-width:2px
    style E stroke:#15803d,stroke-width:2px
    style F stroke:#15803d,stroke-width:2px
    style I stroke:#1d4ed8,stroke-width:2px
    style J stroke:#1d4ed8,stroke-width:2px

Anti-Pattern Pair: Plaintext Passwords vs Ansible Vault #

Backend credentials should never exist in configuration files in plaintext. This pair shows the difference we must avoid during review:

{# ANTI-PATTERN: password hardcoded in a Jinja2 template #}
{# roles/filebeat/templates/filebeat.yml.j2 #}
output.elasticsearch:
  hosts: ["{{ elasticsearch_host }}:9200"]
  username: "elastic"
  password: "MyS3cretP@ssw0rd"   # ✗ DANGER — it's in Git, visible in `ps`, leaks during debug
  # Problem: the password is stored in Git history forever. When developers
  # run the playbook on laptops, the password appears in log output. When the
  # file is shared for debugging, the credential spreads with it.

{# CORRECT: credentials from Ansible Vault, injected when the template renders #}
{# roles/filebeat/templates/filebeat.yml.j2 #}
output.elasticsearch:
  hosts: {{ elasticsearch_hosts | to_json }}
  username: "{{ elasticsearch_username }}"
  password: "{{ vault_elasticsearch_password }}"   # ✓ from the vault
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

How to inject the vault into the template:

# Create the vault (done once, store the vault password in a password manager)
ansible-vault create group_vars/all/vault.yml
# Contents:
# vault_elasticsearch_password: "MyS3cretP@ssw0rd"

# Run the playbook with the vault password
ansible-playbook site.yml --ask-vault-pass
# Or automatically from CI:
ansible-playbook site.yml --vault-password-file ~/.vault_pass

Never let log output display rendered templates. Add no_log: true on tasks that render templates containing credentials:

- name: Deploy the Filebeat configuration
  template:
    src: filebeat.yml.j2
    dest: /etc/filebeat/filebeat.yml
  no_log: true   # ✓ hides output that could leak credentials

Without no_log: true, passwords can appear in Ansible logs if an error occurs. For CI/CD pipelines that store log artifacts, this is equivalent to writing passwords to public storage.

Integrating Logs with Metric Collection #

Logs and metrics complement each other. When Prometheus alerting tells us CPU usage rose on host-42, we want to jump straight to that host’s logs to see what’s happening. This integration requires consistent labels on both sides:

# roles/loki/tasks/integration.yml — labels must match the ones
# used by Prometheus (see the metric-collection article)
---
- name: Ensure Loki labels are consistent with Prometheus
  lineinfile:
    path: /etc/loki/config.yml
    regexp: '^  external_labels:'
    line: |
      external_labels:
        cluster: {{ cluster_name }}
        environment: {{ env }}      
    insertafter: 'common:'
  notify: Restart Loki

# In the Promtail scrape config, the host label must also be consistent
# with the `host` label in the Prometheus node_exporter scrape

Combined queries in Grafana — taking metrics from Prometheus, seeing the relevant logs from Loki in the same panel:

# LogQL query in Grafana Explore, filtering using the same labels as PromQL
{service="api-gateway", env="production", host="api-42"}
  | json
  | level="error"
  | line_format "{{.message}}"

Summary #

• Filebeat for the Elastic ecosystem (Elasticsearch/Logstash); Promtail for the Grafana ecosystem (Loki) — choose based on the monitoring stack that already exists, don’t force a new one.
• Always add fields in the log shipper configuration: host, environment, service — without these, queries across 50+ servers in the backend are nearly impossible.
• logrotate must be set up for all application logs — logs without rotation will eat the disk within weeks on production servers and cause cascading failures.
• For logs in JSON format, enable json.keys_under_root: true (Filebeat) or pipeline_stages: json (Promtail) so log fields can be searched and filtered individually.
• Use Ansible Vault to store Elasticsearch or Loki credentials — passwords must not be plaintext in configuration files or Git history.
• Pin versions of Filebeat/Promtail/Loki in defaults/main.yml — uncontrolled minor updates can change log formats, labels, or index structures.
• Enable json.overwrite_keys: true when the Filebeat input is JSON — without this flag, JSON fields are nested inside message and can’t be queried per-field.
• Add no_log: true on tasks that render templates containing credentials — prevents password leaks in Ansible logs or CI artifacts.

← Previous: Testing   Next: Monitoring →