Best Practice #
All articles in this section discuss how to build observability components — logging, monitoring, alerting, tracing, health checks, SLO/SLA, up to incident response. This article discusses how to make them work well in the real world. Effective observability isn’t just a collection of installed tools — it’s a practice that lets teams understand systems, respond to incidents faster, and make data-based decisions. Here are eight principles that distinguish a mature observability setup from one that merely “runs”, plus a list of the most commonly found anti-patterns and a review checklist we can use right away.
stateDiagram-v2
[*] --> Reactive
Reactive: "Alerts from user complaints"
Proactive: "Alerts appear before users are affected"
Predictive: "Alerts based on trends and anomalies"
Reactive --> Proactive: "Add monitoring + SLO"
Proactive --> Predictive: "Add ML anomaly detection"
Predictive --> [*]: "Continuous improvement"The lifecycle above shows three maturity levels commonly encountered: reactive (only knowing there’s a problem when users complain), proactive (alerts appear before users are affected), and predictive (being able to anticipate problems from trends). The best practices discussed in this article help us move to at least the proactive level.
1. Observability Must Be Automatic, Not Optional #
New servers must be immediately monitored, their logs immediately sent to the centralized backend, and their health checks immediately functional — without extra manual steps. If observability is a “when we get around to it” step after provisioning, it will never be installed on all servers.
# ANTI-PATTERN: observability as an optional manual step
# "We'll install monitoring when we get around to it..."
# Result: new servers are never actually monitored
#
# Manual procedure after deploy:
# 1. SSH to the server
# 2. wget node_exporter.tar.gz
# 3. Extract and set up the systemd unit
# 4. Add the target to Prometheus
# 5. Restart Prometheus
# → 5 easy steps that will definitely be skipped on 30% of servers
# CORRECT: observability as part of server provisioning
# roles/common/tasks/main.yml — runs on EVERY new server
- import_tasks: install.yml
- import_tasks: configure.yml
- import_tasks: node-exporter.yml # System monitoring — always
- import_tasks: filebeat.yml # Centralized logging — always
- import_tasks: health-check.yml # Health endpoint — always
- import_tasks: promtail.yml # Log collector — always
Separate a common role containing the observability setup from application roles. Every application role (e.g. web, db, cache) imports the common role at the start. This ensures that no new server escapes observability — structurally, not by policy.
Define thecommonrole as a meta-role dependency inmeta/main.ymlso Ansible refuses to run application roles withoutcommonfirst. This turns “best practice” into a “hard requirement” that can’t be bypassed.
2. The Three Observability Pillars Must Be Connected #
Logs, metrics, and traces standing alone are only half useful. Real value appears when all three can be cross-linked. A Prometheus alert must be clickable to open the Grafana dashboard at the same time, from there click to the trace in Tempo, and from a slow trace click to the relevant log lines. This is only possible if the three pillars share the same identifier.
flowchart LR
subgraph Sources["Sources"]
App["Application<br/>(OTel instrumentation)"]
end
subgraph Three["Three Pillars"]
M["METRICS<br/>Prometheus<br/>counter, gauge, histogram"]
L["LOGS<br/>Loki / ELK<br/>structured events"]
T["TRACES<br/>Tempo / Jaeger<br/>request span tree"]
end
subgraph Correlation["Correlation Layer"]
Labels["Shared labels<br/>service, env, host, trace_id"]
end
subgraph Visual["Visualization"]
G["Grafana<br/>dashboard + derived fields"]
end
App --> M
App --> L
App --> T
M -. "use" .-> Labels
L -. "use" .-> Labels
T -. "use" .-> Labels
Labels --> G
G -- "click alert" --> M
G -- "click trace_id" --> T
G -- "click log line" --> LThe three pillars in the middle (Metrics, Logs, Traces) come from the same application, and all use consistent labels (service, environment, host, trace_id). The correlation layer below is the glue: the same identifier enables cross-navigation in Grafana through derived fields and Explore. Without label consistency, the three pillars remain three silos that don’t understand each other.
# group_vars/all.yml — define standard labels ONCE
observability_labels:
environment: "{{ env }}"
cluster: "{{ cluster_name }}"
service: "{{ app_name | default('unknown') }}"
host: "{{ inventory_hostname }}"
region: "{{ aws_region | default('local') }}"
# Use in ALL templates: Prometheus rules, Filebeat config, OTel resources
# ANTI-PATTERN: different hardcoded labels in every file
# file 1: env=production, hostname=app01
# file 2: environment=prod, host=app-01
# file 3: stage=prd, server=APP_01
# → cross-system queries are impossible, aggregation breaks
# CORRECT: one Ansible variable source, used in all templates
labels:
env: "{{ observability_labels.environment }}"
instance: "{{ observability_labels.host }}"
service: "{{ observability_labels.service }}"
Add trace_id to every log entry — usually through the OpenTelemetry SDK which automatically injects trace context into loggers. Configure Grafana derived fields to extract trace_id from logs and navigate to Tempo. This is a small investment that pays back many times over during incident response.
3. Alerts Must Be Actionable and Structured #
An alert that doesn’t make it clear what to do is useless — even dangerous because it causes alert fatigue. Every alert we create must pass five questions: who receives it, what should be done, does a runbook exist, is the severity right, and how many false alarms in the last month.
flowchart TD
Start["Alert firing"] --> T1{"Has an owner<br/>and runbook?"}
T1 -- "No" --> X1["Reject:<br/>complete it first"]
T1 -- "Yes" --> T2{"False alarms<br/>more than 1x/week?"}
T2 -- "Yes" --> X2["Investigate:<br/>threshold or inhibition"]
T2 -- "No" --> T3{"Severity<br/>matches the impact?"}
T3 -- "No" --> X3["Adjust:<br/>critical/warning/info"]
T3 -- "Yes" --> OK["Valid alert:<br/>send to the team"]The decision tree above is the minimum filter every new alert must pass. If any “No” branch triggers, the alert isn’t ready to be added to the active rules. This filter can also be validated through promtool test rules or a GitHub Action running assertions against every alert rule’s metadata.
# ANTI-PATTERN: an alert without context
- alert: HighMemory
expr: node_memory_MemAvailable_bytes < 500000000
annotations:
summary: "Low memory"
# CORRECT: an alert with actionable context
- alert: CriticallyLowMemory
expr: >
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
for: 5m
labels:
severity: critical
team: platform
slo: availability
annotations:
summary: "Critical memory on {{ '{{' }} $labels.instance {{ '}}' }}"
description: >
Only {{ '{{' }} $value | humanizePercentage {{ '}}' }} memory remaining
for the last 5 minutes. The process with the highest RSS is usually
the cause — see the runbook for diagnosis steps.
runbook_url: "https://wiki.company.com/runbooks/low-memory"
dashboard_url: "https://grafana.company.com/d/memory?var-host={{ '{{' }} $labels.instance {{ '}}' }}"
Notice the difference: the first alert doesn’t tell the receiver what to do. The second alert sets a ratio-based threshold (more accurate than an absolute value), adds a team label (who must be called) and a slo label (what’s the business context), includes a runbook URL, and adds a dashboard link for quick investigation. The receiver doesn’t need to guess — all needed links are already in the notification.
4. The Observability Infrastructure Must Also Be Monitored #
This is often forgotten: who monitors Prometheus? Who monitors Alertmanager? If the monitoring pipeline dies in the middle of the night, we won’t know there are other problems until users file complaints. This irony is real: unmonitored monitoring infrastructure is a distributed silent failure only discovered when it’s already too late.
# Prometheus monitoring itself
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
# Alert if Alertmanager is unreachable
- alert: AlertmanagerDown
expr: up{job="alertmanager"} == 0
for: 1m
labels:
severity: critical
team: platform
annotations:
summary: "Alertmanager is not running — no alerts will be sent!"
runbook_url: "https://wiki.company.com/runbooks/alertmanager-down"
# Alert if Prometheus fails to scrape a target
- alert: PrometheusTargetMissing
expr: up == 0
for: 5m
labels:
severity: warning
team: platform
annotations:
summary: "Scrape target unreachable: {{ '{{' }} $labels.job {{ '}}' }}/{{ '{{' }} $labels.instance {{ '}}' }}"
# Alert if Prometheus runs out of memory
- alert: PrometheusHighMemory
expr: process_resident_memory_bytes{job="prometheus"} > 4 * 1024 * 1024 * 1024
for: 10m
labels:
severity: warning
annotations:
summary: "Prometheus is using >4GB RAM — likely high cardinality"
The three alerts above are the minimum that must exist: monitor itself (self-monitoring), monitor the Alertmanager dependency, and monitor Prometheus’s own resource usage. The third alert in particular often becomes the early warning for cardinality problems (discussed in more detail in Principle 6).
Also add alerts for disk usage on the storage backends (Prometheus TSDB, Loki S3, etc.) — a full disk = permanent data loss. Set the threshold at 70% so there’s time for retention rotation or storage expansion before it’s truly full.
5. Economical Trace Sampling #
Storing 100% of traces is a luxury that must be rejected early. Trace storage is the most expensive of the three pillars — one span can be tens of kilobytes, one request can be hundreds of spans, and normal requests dominate the population. Proper sampling preserves insight from problematic traces while cutting storage costs by up to 90%.
# ANTI-PATTERN: 100% head-based sampling
# OTel Collector config — stores ALL traces
processors:
batch:
timeout: 5s
send_batch_size: 1000
# No filters, all traces pass through
# CORRECT: tail-based sampling prioritizing problematic traces
processors:
tail_sampling:
decision_wait: 10s
num_traces: 50000
expected_new_traces_per_sec: 200
policies:
# Policy 1: ALWAYS store error traces
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
# Policy 2: ALWAYS store slow traces
- name: slow-traces
type: latency
latency: { threshold_ms: 1000 }
# Policy 3: Randomly store 5% of normal traces
- name: baseline
type: probabilistic
probabilistic: { sampling_percentage: 5 }
The configuration above stores 100% of error traces, 100% of traces slower than 1 second, and only 5% of normal traces. The result: storage drops 90% but we never lose the traces we actually need. The rule “error traces and slow traces are always stored” is a non-negotiable constant.
Tail-based sampling needs a sufficiently largedecision_wait(5-10 seconds) to give all spans from one trace_id time to complete before the decision is made. Trade-off: the longerdecision_wait, the more accurate sampling but the higher memory usage in the collector.
For very high-traffic services, add initial head-based sampling (e.g. 1-2% in the application SDK) as a first filter, then let the tail-based sampler in the collector filter further. This prevents the collector from being overwhelmed.
6. Observability Cost Awareness #
Good observability isn’t the most complete observability — it’s the one most proportional to business value and cost. High cardinality, long retention, and full-fidelity traces are the three biggest cost sources we must be aware of from the start.
# ANTI-PATTERN: unbounded cardinality
# Every label whose value is unique per request will explode storage
metrics:
- name: http_requests_total
labels:
- method
- path # /users/12345, /users/12346, ... = thousands of series
- user_id # one series per user = MILLIONS of series
- request_id # every request differs = storage OOM within hours
# CORRECT: controlled cardinality
metrics:
- name: http_requests_total
labels:
- method
- path_template # /users/:id, NOT /users/12345
- status_class # 2xx, 4xx, 5xx — not 200, 201, 404, 500, ...
# Cardinality: 3 methods × 50 paths × 5 statuses = 750 series
# Compare with 3 × thousands × millions × millions = exploding cardinality
Rules of thumb for Prometheus cardinality:
SAFE label values:
✓ method (GET, POST, PUT) → ~5 values
✓ status_class (2xx, 3xx, 4xx, 5xx) → ~5 values
✓ environment (prod, staging, dev) → ~3 values
✓ service (app, db, cache) → ~10 values
✓ region (ap-southeast-1, us-east-1) → ~5 values
DANGEROUS label values:
✗ user_id → millions
✗ request_id → infinite
✗ email / name → millions
✗ path with IDs → thousands per application
✗ timestamp → infinite
For high-cardinality data (user_id, request_id, email), use logs or traces — not metrics. Logs and traces are indeed more expensive per event, but the number of high-cardinality events is usually far smaller than the number of metric series. A healthy trade-off.
Retention policy is also part of cost awareness. Prometheus defaults to storing 15 days of data, Loki 30 days, Tempo 14 days. Adjust according to compliance and incident review needs: data older than 90 days is usually only needed for annual audits — move it to much cheaper S3 cold storage.
7. Executable Runbooks and Alert Owners #
An alert without a runbook is a nuisance. A runbook without a clear owner is a dusty document. The combination of both — alerts pointing to executable runbooks and runbooks pointing to clear owners — is a reliable operational foundation during incidents.
<!-- ANTI-PATTERN: a runbook that's not executable and has no owner -->
# Runbook: Low Memory
If memory is full, restart the service.
<!-- Missing: owner, diagnosis steps, when to escalate, rollback plan -->
<!-- CORRECT: an executable runbook with clear ownership -->
# Runbook: Critical Memory on a Production Host
**Owner:** Platform Team (on-call: @oncall-platform)
**Severity:** Critical
**Escalation:** If not resolved within 30 minutes, page manager @platform-lead
**Dashboard:** https://grafana.company.com/d/memory
**Related SLO:** availability 99.9% (see the availability SLO runbook)
## Diagnosis (5 minutes)
1. Open the memory dashboard, identify the host from the alert.
2. SSH to the host: `ssh {{ '{{' }} hostname {{ '}}' }}`
3. Check the processes with the highest RSS:
```bash
ps aux --sort=-%mem | head -20
- Check whether the OOM killer is active:
dmesg | grep -i "killed process" | tail -20 - Check whether any process’s memory keeps rising (memory leak):
# Compare process X's RSS from the last 5 minutes
Quick Mitigation (10 minutes) #
- Option A: Restart the service (if the service is known)
systemctl restart <service-name> - Option B: Drain traffic (if unsure about the root cause)
# Remove from the load balancer ansible-playbook -i inv/prod playbooks/drain-host.yml -e "host={{ '{{' }} hostname {{ '}}' }}" - Option C: Kill the wasteful process (if one process is truly the cause)
# Confirm in Slack #platform first kill -15 <pid>
After the Incident #
- Write a post-mortem (template: wiki.company.com/postmortem)
- Add an alert for early detection (if not already present)
- Review whether this is a pattern that will repeat
Notice the correct runbook structure: there's an owner, an escalation path, a dashboard link, the affected SLI/SLO, and diagnosis steps *before* mitigation steps. Many teams jump to mitigation without proper diagnosis — which often makes the problem return within 1-2 hours.
Keep runbooks in the code repository (GitOps), not in a separate wiki. This ensures runbooks are reviewed when changes happen, have version history, and can be directly linked from alert annotations.
---
## 8. Observability as Code via Ansible
All observability configuration — Prometheus scrape configs, alert rules, Grafana dashboards, Filebeat pipelines, Alertmanager routing — must be defined in Ansible and version-controlled in Git. No observability configuration is changed manually through the UI, except for temporary experiments that will be promoted to code.
```yaml
# Observability role directory structure
roles/observability/
├── tasks/
│ ├── main.yml
│ ├── prometheus.yml
│ ├── grafana.yml
│ ├── alertmanager.yml
│ └── filebeat.yml
├── templates/
│ ├── prometheus.yml.j2
│ ├── alerts/
│ │ ├── infrastructure.yml.j2
│ │ ├── application.yml.j2
│ │ └── slo.yml.j2
│ ├── alertmanager.yml.j2
│ └── dashboards/
│ ├── api-overview.json.j2
│ └── slo-overview.json.j2
├── files/
│ └── dashboards/
│ ├── api-overview.json # Static JSON, copied as-is
│ └── slo-overview.json
└── defaults/
└── main.yml # retention, scrape interval, etc.
The structure above separates three file types:
Template files (.j2):
→ Configurations needing Ansible variables
→ Prometheus scrape configs, alert rules
→ Rendered when ansible-playbook runs
Static files (JSON/YAML):
→ Configurations that don't need variables
→ Dashboard JSON from Grafana exports
→ Copied as-is via the copy module
Defaults files:
→ Variables that can be overridden per environment
→ retention_days, scrape_interval, alertmanager_url
The recommended workflow:
1. A developer needs a new dashboard for service X
→ Export JSON from staging Grafana
→ Put it in roles/observability/files/dashboards/service-x.json
→ Add a task in roles/observability/tasks/grafana.yml
2. A developer needs a new alert rule
→ Edit templates/alerts/application.yml.j2
→ Add the receiver and route in templates/alertmanager.yml.j2
→ Submit a PR labeled "observability"
3. CI runs:
- promtool check rules templates/alerts/*.yml
- amtool check-config templates/alertmanager.yml.j2 --syntax-only
- Grafana dashboard lint (JSON schema validation)
- Test deployment to staging
4. After the merge, ansible-playbook deploys to production
→ Prometheus reloads automatically (notify handler)
→ The new dashboard appears in Grafana
→ The new alert rule starts evaluating
# ANTI-PATTERN: observability configuration changed via the Grafana/Prometheus UI
# "Try it in the UI first, export it once it's good"
# Problems:
# - No version history
# - No code review
# - Drift between staging and production
# - Re-deploying = configuration lost
# CORRECT: observability as code, everything through Ansible
# All changes go through PRs, are reviewed, tested in staging, then deployed
- name: Deploy the Prometheus alert rules
template:
src: "alerts/{{ item }}.yml.j2"
dest: "/etc/prometheus/rules/{{ item }}.yml"
owner: prometheus
mode: '0644'
validate: "promtool check rules %s"
loop:
- infrastructure
- application
- slo
notify: Reload Prometheus
This approach may look slower at the start compared to clicking in the UI, but at production scale (10+ services, 50+ alerts, 20+ dashboards), observability as code is the only sustainable way. Without it, drift and inconsistency will become the main source of incidents.
Anti-Patterns to Avoid #
Here are the most commonly found anti-patterns in production observability setups. Each anti-pattern comes with a concise solution we can adopt immediately.
# ✗ Anti-pattern 1: Unbounded cardinality
# Labels with unique values per request/user/path
metrics:
- name: api_requests_total
labels: [user_id, path, request_id]
# Consequence: Prometheus OOM within hours
# ✓ Solution: use path_template (low-cardinality) and put user_id
# in logs/traces, not in metrics
# ✗ Anti-pattern 2: Alerts without owners
- alert: HighErrorRate
expr: rate(http_errors[5m]) > 0.05
annotations:
summary: "High error rate"
# Consequence: nobody is responsible, the alert gets snoozed/ignored
# ✓ Solution: add a team label and runbook URL,
# integrate with per-team PagerDuty rotations
# ✗ Anti-pattern 3: Storing logs without a retention policy
# Logs from 2 years ago still exist, storage is full, queries are slow
# ✓ Solution: set explicit retention
# - Hot storage (Loki/ES): 30 days
# - Warm storage (S3 + Athena): 1 year
# - Cold storage (Glacier): 5 years
# Delete automatically after retention passes, and don't forget to monitor
# the disk usage of the storage backends themselves.
# ✗ Anti-pattern 4: Dashboards nobody reads
# Dozens of Grafana dashboards, nobody knows which one to open during incidents
# ✓ Solution: create a "runbook dashboard" per service, one dashboard
# showing everything needed during investigation
# (4 golden signals, dependency health, recent deploys, logs).
# ✗ Anti-pattern 5: Alert storms during big problems
# One problem triggers 50 alerts at once because all metrics are affected
# ✓ Solution: use inhibit_rules in Alertmanager to suppress derived alerts,
# and error budget alerts (see the SLO/SLA article) which alert once
# per SLO violation, not per metric
# Example inhibit_rules to reduce alert storms
# alertmanager.yml
inhibit_rules:
# If Alertmanager is down, don't send AlertmanagerDown alerts from other hosts
- source_match:
alertname: AlertmanagerDown
target_match:
alertname: AlertmanagerDown
equal: ['cluster']
# If there's a big outage, don't send warnings for every service
- source_match:
severity: critical
target_match:
severity: warning
equal: ['cluster', 'service']
# ✗ Anti-pattern 6: Logs without consistent structure
"Error: database connection failed"
"2024-03-15 ERROR db conn fail"
"[ERR] could not connect to postgres"
# Consequence: can't be queried, can't be filtered,
# can't be correlated with trace_id
# ✓ Solution: structured logs with consistent fields
{"timestamp":"2024-03-15T14:30:00Z","level":"error","service":"myapp",
"trace_id":"abc123","span_id":"def456","message":"database connection failed",
"error":"dial tcp: connection refused","host":"app-01","db_host":"db-01"}
# Every field can be filtered, aggregated, and correlated
# ✗ Anti-pattern 7: An observability stack running without configuration backups
# Prometheus crashes, restored from backup but all alert rules and
# scrape configs are gone because there's no version control
# ✓ Solution: git is the backup. Every change goes through a PR,
# Ansible deploys from Git, recovery = git clone + ansible-playbook
Observability Review Checklist #
Use this checklist every time you set up observability for a new service, or every time you review an existing observability setup. Check all items before declaring a service “production-ready”.
LOGGING
□ Logs are written in structured JSON format
□ Consistent fields: timestamp, level, service, trace_id, message
□ Log levels used correctly (debug/info/warn/error)
□ Logs sent to a centralized backend (Loki/Elasticsearch) via Filebeat/Promtail
□ Log retention set (hot 30 days, warm 1 year)
□ Log rotation on the application/local file side
□ No sensitive data (passwords, PII) in logs
□ Log sampling for debug level (if volume is high)
MONITORING
□ Node/system exporter running on hosts
□ Application metrics available at /metrics (Prometheus format)
□ RED metrics for services: Rate, Error, Duration
□ USE metrics for hosts: Utilization, Saturation, Errors
□ Service is a scrape target in Prometheus
□ Recording rules for complex queries
□ Service dashboard available in Grafana
□ Metric cardinality audited (no user_id/request_id)
ALERTING
□ Alert for availability (high error rate)
□ Alert for latency (high P99)
□ Alert for saturation (resources almost full)
□ Alert for SLO error budget burn rate
□ Every alert has a severity label (critical/warning/info)
□ Every alert has a team label (who gets called)
□ Every alert has a runbook_url annotation
□ Every alert has a dashboard_url annotation
□ Inhibit_rules configured to prevent alert storms
□ Routing to the right channels (PagerDuty critical, Slack warning)
TRACING
□ Application instrumented with the OpenTelemetry SDK
□ Trace IDs propagated between services (context propagation)
□ Tail-based sampling active (errors and slow traces always stored)
□ Traces available in Tempo/Jaeger
□ Sampling rate adjusted to traffic volume
□ Trace retention 7-14 days
□ Trace backend monitored (storage usage, ingestion rate)
HEALTH CHECK
□ /health/live (liveness) endpoint available and lightweight
□ /health/ready (readiness) endpoint available and checks dependencies
□ Load balancer configured for health checks
□ Kubernetes probes configured (if on K8s)
□ Health checks don't check external dependencies (for liveness)
□ Startup probe configured for slow-starting services
SLO
□ SLI defined (availability, latency, throughput)
□ SLO target set (e.g. 99.9% of requests < 200ms)
□ Error budget calculated per month
□ Recording rules for the SLI created
□ Multi-window burn rate alerts configured (1h+5m, 6h+30m)
□ SLO dashboard shows error budget remaining
□ Monthly SLA report automatically generated
COST & PERFORMANCE
□ Metric cardinality audited every quarter
□ Explicit retention policy for all storage backends
□ Disk usage monitoring for Prometheus/Loki/Tempo storage
□ Tracing sampling rate adjusted to traffic
□ Log sampling at debug/info level
□ Storage backends use tiers (hot/warm/cold) by data age
CULTURE & PROCESS
□ Runbooks up-to-date for every alert
□ Runbooks have owners and escalation paths
□ On-call rotation is clear (per team, per severity)
□ Blameless post-mortem for every critical incident
□ Observability review in every PR adding a new service
□ Observability configuration via Ansible/Git, not UI clicks
□ CI runs lint for alert rules and dashboard JSON
□ Alerts that haven't fired in 3 months are reviewed (needed or not?)
Summary #
- Observability must be automatic — integrate it into the Ansible
commonrole so every new server is immediately monitored, its logs centralized, and health checks functional without manual steps.- The three pillars must be connected — use consistent labels (service, environment, host, trace_id) across logs, metrics, and traces; cross-navigation in Grafana via derived fields provides far more value than each pillar separately.
- Alerts must be actionable — every alert must answer “what to do” with a runbook link, and “who gets called” with a team label. Context-free alerts are noise that destroys team trust.
- Monitor the monitoring infrastructure — a down Alertmanager, an OOM Prometheus, and a full storage backend are silent failures only discovered when it’s too late. Self-monitoring alerts are the minimum investment.
- Economical trace sampling — tail-based sampling: store 100% of error and slow traces, sample 5-10% of normal traces. Storage saves 90% while debugging insight stays intact.
- Cardinality and retention are the main costs — high-cardinality labels (user_id, request_id) must be avoided in metrics; such data fits better in logs or traces. Set explicit retention for all storage backends.
- Executable runbooks with clear owners — runbooks must contain diagnosis steps before mitigation, mention owners and escalation paths, and be stored in Git (not a separate wiki) so they get reviewed when changes happen.
- Observability as code via Ansible — all configuration (scrape configs, alert rules, dashboards, alertmanager routing) goes through Ansible roles and Git, linted in CI, deployed via
promtool/amtoolvalidation. No UI-click changes.