Metric Collection #

Metrics are the foundation of all data-based monitoring systems. Node Exporter or standard vendor-built exporters are very helpful for tracking basic infrastructure health. However, when faced with special scenarios — like collecting physical server sensor temperatures, monitoring remaining third-party API quotas, or parsing application log files in real-time — we need a more flexible metric collection agent and a time-series database (TSDB) capable of handling high-speed data delivery. This article discusses applying the Telegraf and InfluxDB technology stack as a custom metric collection solution, how to design the push versus pull communication models, and how to write custom collectors using bash scripts and Grok-pattern log parsers — with the entire deployment process automated by Ansible.

Metric Collection Architecture: Push vs Pull #

In the observability world, there’s a classic debate about the best method for moving metrics from target servers to a centralized database: whether the storage server pulls data from targets (Pull), or agents on targets send data to the storage (Push). Both models have deep implications for network design, security, and system reliability.

Here’s a comparison diagram of the communication flows between the Pull model (Prometheus) and the Push model (Telegraf/InfluxDB):

flowchart TD
    subgraph PullModel["Pull Model (Pull - Prometheus)"]
        direction LR
        PROM["Prometheus Server"] -->|"HTTP GET /metrics"| T1["Target Agent (Exporter)"]
    end
    subgraph PushModel["Push Model (Push - Telegraf/InfluxDB)"]
        direction LR
        T2["Target Agent (Telegraf)"] -->|"HTTP POST (Line Protocol)"| INF["InfluxDB TSDB"]
    end

The Pull Model #

The Pull model is represented by systems like Prometheus. Here, every target runs a small web service (exporter) exposing metrics on a specific port. Prometheus periodically sends HTTP GET requests to the /metrics endpoint on each target.

• Advantages:
  • Target Death Detection: If Prometheus fails to contact a target, the system immediately knows that target is down.
  • Load Control: The monitoring server fully controls how often metrics are fetched, preventing the server from being flooded with data.
  • Simple on the Target Side: Target agents don’t need to know the monitoring server’s address or have write authentication tokens.
• Disadvantages:
  • Firewall Problems: The monitoring server must be able to reach target ports directly. This is hard to apply if targets are behind private networks or NAT.
  • Ephemeral Services: Less suitable for serverless functions (like AWS Lambda) that only light up for a few milliseconds then die.

The Push Model #

The Push model is represented by Telegraf sending data to InfluxDB. Telegraf agents are installed on target servers, collect data locally, then actively send HTTP POST payloads containing metric data to the InfluxDB endpoint.

• Advantages:
  • Network Interoperability: Target agents only need outbound access to the internet/TSDB. We don’t need to open inbound ports on target servers.
  • Supports Serverless and Batch Jobs: Very ideal for short-lived scenarios where instances dynamically start and immediately send their work status before being shut down.
  • Collection Scalability: Data collection happens independently on the client side, lightening the polling load on the TSDB server side.
• Disadvantages:
  • Agent Status Loss: If an agent stops sending data, the TSDB doesn’t automatically know whether the agent died or just has no new metrics to send.
  • Credential Management: Every agent must be equipped with a secure write token to write to InfluxDB.

In our infrastructure, we can combine both. We use Telegraf to collect metrics from edge servers behind private NAT, then configure that Telegraf to push data to the centralized InfluxDB cluster.

TSDB Data Stack Deployment: InfluxDB and Telegraf #

To build a push-based metric collection infrastructure, we’ll deploy InfluxDB v2 as the central TSDB and Telegraf as the collection agent on target servers using Ansible.

1. Deploying InfluxDB v2 via Ansible #

We’ll install InfluxDB v2 on the main monitoring server. Initial configuration requires defining the organization, default bucket, and administrative token.

# playbooks/deploy_influxdb.yml
---
- name: Deploy the InfluxDB Server
  hosts: monitoring_servers
  become: true
  vars:
    influxdb_version: "2.7.5"
    influxdb_org: "badritech"
    influxdb_bucket: "system_metrics"
  tasks:
    - name: Download the InfluxDB installation package
      get_url:
        url: "https://dl.influxdata.com/influxdb/releases/influxdb2-{{ influxdb_version }}-amd64.deb"
        dest: "/tmp/influxdb2.deb"
        mode: '0640'

    - name: Install the InfluxDB deb package
      apt:
        deb: "/tmp/influxdb2.deb"
        state: present

    - name: Ensure the InfluxDB service is running and enabled
      systemd:
        name: influxdb
        state: started
        enabled: true

    - name: Wait until the InfluxDB port is ready to accept connections
      wait_for:
        port: 8086
        delay: 3
        timeout: 30

    - name: Check whether InfluxDB has been initialized
      command: influx setup --check
      register: influx_check
      failed_when: false
      changed_when: false

    - name: Run the initial InfluxDB setup
      command: >
        influx setup
        --username "{{ influx_admin_user }}"
        --password "{{ vault_influx_admin_password }}"
        --org "{{ influxdb_org }}"
        --bucket "{{ influxdb_bucket }}"
        --token "{{ vault_influx_admin_token }}"
        --force        
      when: influx_check.rc == 0
      no_log: true

2. Deploying the Telegraf Agent on Target Servers #

After the InfluxDB server is ready, we install the Telegraf agent on all target servers. Telegraf is configured to collect built-in system metrics and send them to InfluxDB using the token we created.

# roles/telegraf_agent/tasks/main.yml
---
- name: Add the InfluxData repository key
  apt_key:
    url: https://repos.influxdata.com/influxdata-archive_compat.key
    state: present

- name: Add the official InfluxData repository
  apt_repository:
    repo: "deb https://repos.influxdata.com/debian stable main"
    state: present

- name: Install the Telegraf package
  apt:
    name: telegraf
    state: present
    update_cache: true

- name: Deploy the Telegraf configuration file
  template:
    src: telegraf.conf.j2
    dest: /etc/telegraf/telegraf.conf
    owner: root
    group: telegraf
    mode: '0640'
  notify: Restart Telegraf

- name: Ensure Telegraf is running in the background
  systemd:
    name: telegraf
    state: started
    enabled: true

Here’s the telegraf.conf.j2 configuration template managed by Ansible:

{# roles/telegraf_agent/templates/telegraf.conf.j2 #}
[global_tags]
  environment = "{{ env }}"
  hostname = "{{ inventory_hostname }}"

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "ns"
  hostname = ""
  omit_hostname = false

[[outputs.influxdb_v2]]
  urls = ["{{ influxdb_internal_url }}"]
  token = "{{ vault_telegraf_write_token }}"
  organization = "{{ influxdb_org }}"
  bucket = "{{ influxdb_bucket }}"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.mem]]

[[inputs.system]]

Custom Bash Metrics Collector #

When we need specific metrics not provided by Telegraf’s built-in plugins, we can create custom shell scripts and execute them using the inputs.exec plugin. This script must output data in the Influx Line Protocol format so Telegraf can understand it directly.

The basic Influx Line Protocol format is as follows:

<measurement>,<tag_key>=<tag_value> <field_key>=<field_value> <timestamp>

Let’s create a bash script to monitor backup status on target servers:

#!/usr/bin/env bash
# file: /usr/local/bin/backup_metric.sh
# Script to track the size and success status of the last backup.

BACKUP_DIR="/var/backups/postgres"
LAST_BACKUP_FILE=$(find "$BACKUP_DIR" -type f -name "*.sql.gz" -printf '%T@ %p\n' 2>/dev/null | sort -n | tail -1)

if [ -z "$LAST_BACKUP_FILE" ]; then
    # If there's no backup file
    echo "backup_status,status=failed size_bytes=0i,age_seconds=-1i"
    exit 0
fi

# Get the current timestamp and the file timestamp
CURRENT_TIME=$(date +%s)
FILE_EPOCH=$(echo "$LAST_BACKUP_FILE" | cut -d' ' -f1 | cut -d'.' -f1)
FILE_PATH=$(echo "$LAST_BACKUP_FILE" | cut -d' ' -f2-)

# Calculate the file age and size
FILE_AGE=$((CURRENT_TIME - FILE_EPOCH))
FILE_SIZE=$(stat -c%s "$FILE_PATH" 2>/dev/null || echo 0)

# Determine the status based on the file age (considered failed if older than 26 hours)
STATUS="success"
if [ "$FILE_AGE" -gt 93600 ]; then
    STATUS="failed"
fi

# Output data in Influx Line Protocol format
# Integers are identified with the 'i' suffix
echo "backup_status,status=${STATUS} size_bytes=${FILE_SIZE}i,age_seconds=${FILE_AGE}i"

Here’s the flow of how Telegraf interacts with our custom bash script:

flowchart TD
    CRON["Telegraf Internal Timer"] -->|"Triggers interval execution"| EXEC["inputs.exec Plugin"]
    EXEC -->|"Run the script"| BASH["/usr/local/bin/backup_metric.sh"]
    BASH -->|"Check the file directory"| BACKUP["Postgres Backup Directory"]
    BACKUP -- "Size & time metadata" --> BASH
    BASH -->|"Output Line Protocol data"| EXEC
    EXEC -->|"Send the data batch"| OUT["outputs.influxdb_v2 Plugin"]

We automate the deployment of this script and the plugin registration in Telegraf through Ansible tasks:

- name: Copy the custom backup monitoring script
  copy:
    src: backup_metric.sh
    dest: /usr/local/bin/backup_metric.sh
    owner: root
    group: root
    mode: '0755'

- name: Deploy the Telegraf exec input configuration
  copy:
    content: |
      [[inputs.exec]]
        commands = ["/usr/local/bin/backup_metric.sh"]
        timeout = "5s"
        data_format = "influx"      
    dest: /etc/telegraf/telegraf.d/exec_backup.conf
    owner: root
    group: root
    mode: '0640'
  notify: Restart Telegraf

Logging Metrics Parser #

Log files often contain high-value metric information not exposed by the application’s internal API. For example, from Nginx access logs, we can extract metrics for the number of requests per HTTP status code and response duration. We can use the Telegraf inputs.tail plugin with Grok patterns to parse log files in real-time into structured time-series data.

Let’s configure Telegraf to read the Nginx access log file:

# Example standard Nginx access log format:
127.0.0.1 - - [17/Jun/2026:10:25:32 +0700] "GET /api/v1/users HTTP/1.1" 200 452 0.125

The Grok pattern for the format above is:

%{IPORHOST:client_ip} - %{USER:ident} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status:int} %{NUMBER:bytes:int} %{NUMBER:response_time:float}

We deploy this log parser configuration using Ansible to web servers:

- name: Deploy the Telegraf log parser configuration for Nginx
  copy:
    content: |
      [[inputs.tail]]
        files = ["/var/log/nginx/access.log"]
        from_beginning = false
        pipe = false
        data_format = "grok"
        grok_patterns = ['%{IPORHOST:client_ip} - %{USER:auth} \\[%{HTTPDATE:timestamp}\\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:resp_code:int} (?:%{NUMBER:resp_bytes:int}|-) %{NUMBER:resp_time:float}']
        name_override = "nginx_access_log"      
    dest: /etc/telegraf/telegraf.d/nginx_logparser.conf
    owner: root
    group: root
    mode: '0640'
  notify: Restart Telegraf

Through this configuration, every new log line entering /var/log/nginx/access.log is read by Telegraf, parsed based on the Grok pattern, and sent to InfluxDB as metrics with tags like verb (HTTP Method) and fields like resp_time and resp_code. This allows us to monitor HTTP performance without ever touching the application code.

Metric Collection Anti-Patterns and Solutions #

During custom metric collection implementation, there are several fatal mistakes we must avoid to maintain system performance and data security.

1. Storing Authentication Tokens Openly #

• Anti-Pattern: Writing InfluxDB write tokens directly in playbook files or Git templates in plaintext. Anyone with access to the Git repository can steal the token and corrupt data in InfluxDB.
• Solution: Encrypt all sensitive variables using Ansible Vault. At runtime, Ansible securely decrypts those variables in memory before deploying them to target servers.

2. Using Exec Scripts Without Time Limits (Timeout) #

• Anti-Pattern: Running custom scripts via inputs.exec that do slow database connections or external API queries without timeouts. If the network is slow, the script process hangs, locks Telegraf’s execution thread, and stops other system metric collection.
• Solution: Always set a strict timeout parameter (for example timeout = "5s") in the inputs.exec configuration block. Additionally, make sure our internal scripts also implement internal timeouts on every network connection.

3. Ignoring Log File Rotation on the Tail Parser #

• Anti-Pattern: Configuring inputs.tail to read gigabyte-sized log files without a log rotation system. This makes Telegraf consume large CPU memory at startup because it must track file position changes.
• Solution: Make sure the logrotate service is properly configured on the operating system to split log files periodically (for example daily or based on size). Telegraf natively supports log rotation and automatically detects file switches without losing data.

Here’s a comparison of handling sensitive variables (Tokens) between the wrong method and the correct method using Ansible Vault:

# ANTI-PATTERN: Token written directly in the configuration file (Plain Text)
# group_vars/all.yml
telegraf_influx_token: "my-super-secret-admin-token-write-direct"

# CORRECT: Using an Ansible Vault encrypted variable
# group_vars/all.yml (File encrypted with ansible-vault encrypt)
# $ ansible-vault view group_vars/all.yml
vault_telegraf_write_token: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          36353934333939393739663836373839626330386230323337653634356461323331393666666465
          6430313136363462313639643532393361303831626233360a333333333333333333333333333333
          3333333333333333333333333333333333333333330a383236356265613337626365313962663962

By adopting variable encryption standards, we strengthen the security posture of the entire metric collection chain.

Summary #

• Telegraf and InfluxDB form the ideal technology stack for push-based metric collection architectures, which are very suitable for servers behind private NAT.
• The Push model lets agents independently send data to the central TSDB through outbound port 8086 access, simplifying target firewall configuration.
• Influx Line Protocol is a concise text-based metric delivery format, dividing data into measurements, tags (index), fields (non-index), and timestamps.
• Telegraf’s inputs.exec plugin makes custom system metric integration easy by periodically executing local bash or python scripts.
• Grok patterns inside the inputs.tail plugin allow us to parse raw logs into valuable performance metrics like HTTP error rates and response latency.
• Ansible Vault encryption must be used to protect InfluxDB write authorization tokens so they aren’t stored as plaintext in Git repositories.
• Execution timeout limits must be strictly defined on every external input module to prevent Telegraf worker thread deadlocks.
• Logrotate integration with the tail parser prevents system performance degradation from processing a single overly large log file.

← Previous: Dashboard Next: Tracing →