Dashboard #

Data visualization is the bridge connecting the rows of metric numbers in a database with human understanding when monitoring systems. Grafana dashboards created manually through the graphical interface (UI) often become undocumented technical debt — a Grafana server crash, migration to a new instance, or backup recovery failure can wipe out all the important visualizations we’ve assembled. The dashboard-as-code approach offers a solution by treating dashboard definitions as declarative code (either raw JSON format or Grafonnet templates) that is version-managed in a Git repository and automatically deployed using Ansible. With this method, we can ensure visualization consistency across all working environments, make change tracking easier, and speed up disaster recovery processes.

The Dashboard as Code Concept: JSON vs Grafonnet #

Before applying automation, we need to understand the two main formats for representing Grafana dashboards as code: raw JSON and Grafonnet.

Raw JSON #

By default, Grafana stores dashboard definitions in one giant JSON file containing thousands of lines. Inside this JSON file, every panel, query, grid layout coordinates, color thresholds, and graph axis configuration are defined explicitly.

The advantage of the raw JSON format is easy direct export from the Grafana web interface. However, its disadvantages stand out when working in teams:

  • High Duplication: Similar panel configurations must be written repeatedly for every metric.
  • Hard to Review: Changing one panel coordinate position in the UI triggers changes to hundreds of x, y, w, and h coordinate lines in the JSON file, making pull request review processes very hard to read.
  • Error Prone: Manually editing giant JSON files often causes bracket or comma syntax errors that break the entire dashboard.

Grafonnet (Jsonnet Template) #

Grafonnet is a Jsonnet-based library (Google’s data templating language) specifically designed to generate Grafana dashboard JSON. Compared to raw JSON, Grafonnet acts like a high-level programming language that lets us define dashboard components modularly and reusable.

Let’s compare the representation of creating a simple dashboard using Grafonnet:

// file: dashboard.jsonnet
local grafana = import 'github.com/grafana/grafonnet-lib/grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local row = grafana.row;
local prometheus = grafana.prometheus;
local graphPanel = grafana.graphPanel;

dashboard.new(
  title='Production API Overview',
  tags=['production', 'api'],
  editable=false,
)
.addRow(
  row.new(title='System Resources')
)
.addPanel(
  graphPanel.new(
    title='CPU Usage per Host',
    datasource='Prometheus',
    span=12,
  )
  .addTarget(
    prometheus.target(
      expr='sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)',
      legendFormat='{{instance}}',
    )
  ),
  gridPos={ x: 0, y: 0, w: 24, h: 8 }
)

To compile the file above into raw JSON understood by Grafana, we must use the jsonnet utility after installing its library dependencies:

# Install the jsonnet-bundler package to manage Grafonnet libraries
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

# Initialize jb in our project directory
jb init

# Install the grafonnet-lib library into the vendor/ directory
jb install github.com/grafana/grafonnet-lib/grafonnet@master

# Compile the .jsonnet file into a Grafana JSON file
jsonnet -I vendor dashboard.jsonnet > files/dashboards/production-api-overview.json

By switching from raw JSON editing to Grafonnet writing, we can cut code line duplication by more than 70 percent, because we can define a standard graph template once and reuse it across dozens of different dashboards.

Automating Datasource and Dashboard Provisioning with Ansible #

Grafana supports loading configuration declaratively through the provisioning folder when the service starts (startup). We can automate placing these configuration files using Ansible so Grafana directly connects to data sources (datasources) and loads dashboards without manual intervention.

The provisioning folder structure we target on the Grafana server is as follows:

/etc/grafana/provisioning/
├── datasources/
│   ├── datasources.yml         # Connection configuration to Prometheus and Loki
└── dashboards/
    ├── provider.yml            # Sets the JSON file reading directory

Here are the Ansible tasks to prepare the directories and dynamically copy provisioning configuration files to the Grafana server:

# roles/grafana_provisioning/tasks/main.yml
---
- name: Ensure the Grafana provisioning directories have the right permissions
  file:
    path: "{{ item }}"
    state: directory
    owner: grafana
    group: grafana
    mode: '0750'
  loop:
    - /etc/grafana/provisioning/datasources
    - /etc/grafana/provisioning/dashboards
    - /var/lib/grafana/dashboards

- name: Deploy the datasource configuration file
  template:
    src: datasources.yml.j2
    dest: /etc/grafana/provisioning/datasources/datasources.yml
    owner: grafana
    group: grafana
    mode: '0640'
  notify: Restart Grafana

- name: Deploy the dashboard provider configuration
  template:
    src: provider.yml.j2
    dest: /etc/grafana/provisioning/dashboards/provider.yml
    owner: grafana
    group: grafana
    mode: '0640'
  notify: Reload Grafana dashboards

- name: Synchronize the dashboard JSON files to the target folder
  copy:
    src: "{{ item }}"
    dest: "/var/lib/grafana/dashboards/{{ item | basename }}"
    owner: grafana
    group: grafana
    mode: '0640'
  with_fileglob:
    - "files/dashboards/*.json"
  notify: Reload Grafana dashboards

Here’s the template for the J2 datasource file:

{# roles/grafana_provisioning/templates/datasources.yml.j2 #}
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: "{{ prometheus_internal_url }}"
    isDefault: true
    jsonData:
      httpMethod: POST
      timeInterval: 15s
    editable: false

  - name: Loki
    type: loki
    access: proxy
    url: "{{ loki_internal_url }}"
    jsonData:
      maxLines: 1000
    editable: false

And the template for the dashboard provider configuration:

{# roles/grafana_provisioning/templates/provider.yml.j2 #}
apiVersion: 1

providers:
  - name: "Ansible Managed Dashboards"
    orgId: 1
    folder: "Infrastructure"
    type: file
    disableDeletion: true
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

We use a handler to trigger dashboard reload without having to do a full Grafana service restart. This keeps active user sessions from being disconnected:

# roles/grafana_provisioning/handlers/main.yml
---
- name: Restart Grafana
  systemd:
    name: grafana-server
    state: restarted

- name: Reload Grafana dashboards
  uri:
    url: "http://localhost:3000/api/admin/provisioning/dashboards/reload"
    method: POST
    user: "{{ grafana_admin_username }}"
    password: "{{ vault_grafana_admin_password }}"
    force_basic_auth: true
    status_code: 200

Implementing Monitoring Templates #

To avoid the need to create a new dashboard every time a new server or instance is added to the cluster, we must leverage Grafana’s variable (templating) feature. Variables let users dynamically select instances from a drop-down menu at the top of the dashboard.

Inside the dashboard JSON file, the variable structure is defined in the templating block. We use a Prometheus query to automatically populate the variable options based on incoming data labels:

{
  "templating": {
    "list": [
      {
        "current": {},
        "datasource": "Prometheus",
        "definition": "label_values(node_cpu_seconds_total, instance)",
        "hide": 0,
        "includeAll": true,
        "multi": true,
        "name": "instance",
        "options": [],
        "query": {
          "query": "label_values(node_cpu_seconds_total, instance)",
          "refId": "PrometheusTemplateQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "type": "query"
      },
      {
        "current": {},
        "datasource": "Prometheus",
        "definition": "label_values(node_cpu_seconds_total, env)",
        "hide": 0,
        "includeAll": false,
        "multi": false,
        "name": "env",
        "options": [],
        "query": {
          "query": "label_values(node_cpu_seconds_total, env)",
          "refId": "PrometheusEnvQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "type": "query"
      }
    ]
  }
}

Ansible manages deploying these JSON templates by ensuring the environment variable (env) is adjusted automatically during the file synchronization process. By leveraging template variables, one dashboard JSON file can monitor thousands of servers simultaneously in a structured way.

Audit and Access Control in Grafana #

In corporate environments with many teams, we don’t want all organization members to have the same access rights. Security policies require applying Role-Based Access Control (RBAC) and periodic audits of access activity.

Ansible provides integrated modules to automate creating organizations, teams, and folder access rights settings:

# playbooks/grafana_rbac.yml
---
- name: Configure Grafana Teams and Access Rights
  hosts: localhost
  connection: local
  vars:
    grafana_api_url: "http://grafana.internal.zone:3000"
    grafana_token: "{{ vault_grafana_admin_token }}"
  tasks:
    - name: Create a New Organization for the Finance Team
      community.grafana.grafana_organization:
        url: "{{ grafana_api_url }}"
        api_key: "{{ grafana_token }}"
        name: "Finance Department"
        state: present

    - name: Create the Core Developers Team in the Main Organization
      community.grafana.grafana_team:
        url: "{{ grafana_api_url }}"
        api_key: "{{ grafana_token }}"
        name: "Core Developers"
        email: "[email protected]"
        state: present
      register: core_dev_team

    - name: Restrict the Infrastructure Folder Access Rights
      uri:
        url: "{{ grafana_api_url }}/api/folders/infra_folder_uid/permissions"
        method: POST
        headers:
          Authorization: "Bearer {{ grafana_token }}"
          Content-Type: "application/json"
        body_format: json
        body:
          items:
            - role: Viewer
              permission: 1
            - role: Editor
              permission: 2
            - teamId: "{{ core_dev_team.team_id }}"
              permission: 2 # Give Edit access rights to the Developer Team
        status_code: 200

For audit needs, we configure Grafana to record every login activity, dashboard change, and data export to a centralized system log file. We manage this configuration through an Ansible task modifying /etc/grafana/grafana.ini:

- name: Enable audit log recording in Grafana
  ini_file:
    path: /etc/grafana/grafana.ini
    section: log
    option: level
    value: info
  notify: Restart Grafana

- name: Ensure audit logs are written to a separate file
  ini_file:
    path: /etc/grafana/grafana.ini
    section: log.audit
    option: enabled
    value: "true"
  notify: Restart Grafana

Active audit log recording helps security teams detect suspicious configuration changes or illegal access to sensitive business metrics.

Incident Exploration Flow: From Alert to Root Cause #

When an incident occurs in a production environment, the on-call team’s reaction speed depends heavily on how our dashboards are designed. The visualization flow must guide technicians from the high-level system overview to a directed root cause analysis.

Here’s a flow diagram illustrating on-call team navigation when responding to a system alert:

flowchart TD
    A["Start: Alert Notification Arrives in Slack"] --> B["Click the Runbook / Dashboard Link in the Alert Message"]
    B --> C["Open the Overview Dashboard (High-Level)"]
    C --> D{"Are the Host CPU / Memory Metrics Normal?"}
    D -- "No (Resource Exhausted)" --> E["Open the Node Detail Dashboard via the Instance Variable"]
    D -- "Yes (Normal)" --> F["Open the Application Performance Dashboard (Latency/Error Rate)"]
    E --> G["Identify the Disrupted Process via SSH / Node Exporter"]
    F --> H{"Is the Error Localized to One Service?"}
    H -- "Yes" --> I["Open the Service-Specific Dashboard (Drilldown)"]
    H -- "No" --> J["Open the Distributed Tracing Dashboard (Tempo)"]
    I --> K["Search Related Logs via Loki Integration"]
    J --> L["Track Slow Spans & Database Queries"]
    K --> M["Root Cause Found"]
    L --> M
    M --> N["Apply the Fix via an Ansible Playbook"]
    N --> O["Done: Verify the Dashboard Is Green Again"]

And here’s a sequence diagram showing how observability service interactions happen in the background when users explore visualizations:

sequenceDiagram
    participant U as "On-Call Technician"
    participant G as "Grafana Gateway"
    participant P as "Prometheus TSDB"
    participant L as "Loki Log Engine"
    participant T as "Tempo Trace Engine"

    U->>G: "Open the Main Dashboard"
    G->>P: "Fetch service availability metrics"
    P-->>G: "Availability metric drops to 94.2%"
    G-->>U: "Display the red graph (Incident Mode)"

    U->>G: "Click the Error Rate panel (Drilldown)"
    G->>P: "Fetch error contribution per endpoint"
    P-->>G: "The /payment endpoint produces 500 Internal Server Error"
    G-->>U: "Display the disrupted endpoint detail table"

    U->>G: "Click the 'Explore Logs' button"
    G->>L: "Search logs with the problematic trace ID filter"
    L-->>G: "Log: 'Connection timeout to payment gateway'"
    G-->>U: "Display the raw log in the right-side panel"

    U->>G: "Click the 'Inspect Trace' button"
    G->>T: "Fetch the span waterfall from the trace ID"
    T-->>G: "Trace Span: DB query = 50ms, external HTTP call = 5000ms"
    G-->>U: "Display the trace timeline visualization (Tempo)"

Dashboard Management Anti-Patterns and Solutions #

When adopting the dashboard-as-code concept, there are several bad practices often done by operations teams that just migrated.

1. Direct Modification in the Grafana UI #

  • Anti-Pattern: When an incident occurs, the on-call team edits panels or queries directly from the Grafana UI, then saves them. When the Ansible pipeline runs again, those emergency changes are automatically removed by the synchronization system.
  • Solution: Permanently disable UI editing features in production environments by setting allowUiUpdates: false in the provider file. Changes must be made in the staging environment, exported to JSON, added to the Git repository via a Pull Request, then automatically deployed by Ansible.

2. Hardcoding Datasource IDs (Datasource UID) #

  • Anti-Pattern: Using unique static UIDs for datasources inside JSON panels, like "uid": "prometheus-prod-xyz123". When the dashboard is deployed to the staging environment, visualizations break because staging uses a different UID.
  • Solution: Use global variable references across all JSON panels, like "datasource": "${DS_PROMETHEUS}". Grafana automatically maps this variable to the default datasource installed in each environment.

3. Uploading JSON Without Clear Formatting (Unformatted JSON) #

  • Anti-Pattern: Storing dashboard JSON files in a long one-line (minified) format or with inconsistent key order in Git. This makes the git diff command useless because the entire file is detected as one line of change.
  • Solution: Use an automatic cleaner and formatter script before committing JSON files to the repository. We can apply a pre-commit hook that consistently formats JSON files using the Python json.tool module or the jq program.

Here’s an example dashboard JSON cleaner script we should run before committing code to Git:

#!/usr/bin/env python3
# scripts/cleanup_dashboard.py
import json
import sys

def cleanup(file_path):
    with open(file_path, 'r') as f:
        data = json.load(f)
    
    # Remove source instance-specific data
    data.pop('id', None)
    data.pop('version', None)
    data.pop('iteration', None)
    
    # Normalize the default refresh interval
    data['refresh'] = '30s'
    
    # Save back with neat indentation and sorted keys
    with open(file_path, 'w') as f:
        json.dump(data, f, indent=2, sort_keys=True)
        f.write('\n')

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: cleanup_dashboard.py <path_to_json_file>")
        sys.exit(1)
    cleanup(sys.argv[1])

Dashboard Versioning and Rollback #

Because dashboard JSON files are stored in Git, we have full control over the change history. If a query change causes the dashboard to load data slowly or display misleading information, we can quickly roll back to the previous version.

Steps to roll back a dashboard using a Git and Ansible combination:

# 1. Find the last stable commit hash for the dashboard file
git log --oneline -- files/dashboards/api-performance.json

# 2. Restore the file to its state at that commit
git checkout a1b2c3d4 -- files/dashboards/api-performance.json

# 3. Run the Ansible playbook to deploy the changes
ansible-playbook -i inventories/production playbooks/sync_dashboards.yml

# 4. Create a new Git tag to mark the stable dashboard release
git tag -a dashboards-v1.4.2 -m "Roll back the api-performance dashboard to the stable version"

By integrating the Git repository into the dashboard management lifecycle, we treat the monitoring infrastructure with the same level of discipline as application code.

Summary #

  • Dashboard-as-Code is the best practice ensuring all visualizations are stored declaratively in Git, so they’re safe from data loss risks due to server crashes.
  • Grafonnet solves the duplication problem in raw JSON files by providing a Jsonnet-based library for generating dashboard files dynamically and modularly.
  • File-based Provisioning Configuration with the disableDeletion: true parameter prevents accidental dashboard deletion by users through the web interface.
  • Dynamic Variables in dashboard queries are essential to ensure one template file can monitor many servers without needing panel duplication.
  • Role-Based Access Control (RBAC) and audit logs must be applied at the production level to guarantee security compliance and limit dashboard editing rights.
  • Good observability data correlation allows technicians to move seamlessly from abnormal metrics in Prometheus to log lines in Loki and trace spans in Tempo.
  • Consistent JSON Formatting maintained by cleaner scripts or hooks before entering Git makes visualization change reviews easier during code review.
  • A fast rollback scheme based on Git checkout and re-invoking Ansible tasks minimizes recovery time if a dashboard configuration error occurs.

← Previous: Alerting Next: Metric Collection →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact