Health Check #

Monitoring tells us when something is already broken. Health checks prevent traffic from being sent to components that aren’t ready or are already broken. Both are needed — but health checks work at a more real-time and more operational layer: load balancers rely on them for routing, Kubernetes relies on them for automatic restarts, and deployment pipelines rely on them to determine whether a deployment succeeded or needs rollback. This article discusses how to implement and automate proper health checks using Ansible.

Two Different Types of Health Checks #

Before implementing, understand the difference between the two health check types that are often misunderstood:

Liveness Check — "Is the application still alive?"
  Purpose: Detect deadlocks or conditions that can't recover on their own
  Response on failure: Container is restarted
  Example endpoint: GET /health/live → 200 if the process is running
  What it must NOT check: database connections, external dependencies
  Reason: If the database is down, we don't want all containers restarting too
          and worsening the situation with cascading failure

Readiness Check — "Is the application ready to accept traffic?"
  Purpose: Prevent traffic to instances that aren't ready or are overwhelmed
  Response on failure: Instance is removed from the load balancer rotation
  Example endpoint: GET /health/ready → 200 if all dependencies are ready
  What it MAY check: database connections, cache, critical dependencies
  Reason: If the database is unreachable, the instance indeed isn't ready to serve
          and shouldn't receive traffic until it recovers

Mixing both in one endpoint is a common source of bugs — a health check that calls the database makes Kubernetes restart all containers when the database dies, which actually adds pressure to the newly recovering database. This is why livenessProbe and readinessProbe in Kubernetes are two separate fields.

State Diagram: Application Health Cycle #

Applications move between states depending on their internal conditions and dependencies. Understanding these transitions is important for designing correct health checks:

stateDiagram-v2
    [*] --> Starting: "container start"
    Starting --> Healthy: "startup probe OK"
    Starting --> Unhealthy: "startup timeout"
    Healthy --> Degraded: "dependency down"
    Degraded --> Unhealthy: "timeout exceeded"
    Degraded --> Healthy: "dependency recovered"
    Healthy --> Overloaded: "resource exhaustion"
    Overloaded --> Degraded: "load dropped"
    Unhealthy --> Recovering: "restart / intervention"
    Recovering --> Healthy: "liveness OK"
    Healthy --> [*]: "graceful shutdown"
    Unhealthy --> [*]: "kill / OOM"

Notice that Degraded and Overloaded are different states from Unhealthy. A degraded application can still answer some types of requests — maybe reads but not writes, or requests that don’t need the cache. A readiness check that forces the instance out of rotation in the degraded state ensures traffic is routed to still-healthy instances, without restarting an instance that’s actually still functional for part of the load.

Probe Types: HTTPGet, TCPSocket, and exec #

Kubernetes supports three probe types, each for different cases. Choosing the wrong type is a common source of health check bugs:

flowchart TD
    A["Need a health check?"] --> B{"Can it do HTTP?"}
    B -- "Yes" --> C["httpGet probe"]
    B -- "No" --> D{"Is TCP open<br/>enough?"}
    D -- "Yes" --> E["tcpSocket probe"]
    D -- "No" --> F["exec probe"]
    C --> G{"What's the check goal?"}
    E --> G
    F --> G
    G -- "Application<br/>responsive" --> H["httpGet"]
    G -- "Port open" --> I["tcpSocket"]
    G -- "Custom logic" --> J["exec / custom HTTP"]

Probe Type	What It Checks	When to Use	Example
`httpGet`	HTTP endpoint returns a 2xx code	Applications serving HTTP/HTTPS	GET /health/live → 200
`tcpSocket`	TCP connection to the port can open	Non-HTTP applications (databases, message queues)	nc -z db-host 5432
`exec`	Command inside the container returns 0	Validations needing complex logic (check files, run scripts)	`cat /tmp/ready && exit 0`

Decision Tree: Choose the Right Probe Type #

flowchart TD
    A["Start: need a probe"] --> B{"Does the app<br/>serve HTTP?"}
    B -- "Yes" --> C["httpGet probe<br/>to /health/live or /health/ready"]
    B -- "No" --> D{"Is the protocol<br/>port enough info?"}
    D -- "Yes" --> E["tcpSocket probe<br/>to the service port"]
    D -- "No" --> F["exec probe<br/>run a validation script"]
    C --> G{"Need to check<br/>business logic?"}
    G -- "Yes" --> H["httpGet custom endpoint<br/>returning JSON with details"]
    G -- "No" --> I["httpGet simple endpoint<br/>returning 200 only"]

Anti-Pattern: tcpSocket for Web Servers vs httpGet #

# ANTI-PATTERN: tcpSocket probe for a web server
livenessProbe:
  tcpSocket:
    port: 8080
# ✗ Problem: tcpSocket only checks whether the port is open and can handshake.
# Our web server can crash at the application layer (e.g. thread pool deadlock,
# panic in a handler, or memory corruption) BUT the port is still open
# because listen() is still running on the socket. tcpSocket returns SUCCESS
# even though the application actually can't serve requests anymore.

# CORRECT: httpGet probe to an endpoint guaranteeing application logic runs
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3
# ✓ httpGet forces the application to process an HTTP request — if a handler panics,
# if the thread pool is exhausted, if there's a deadlock, the response never
# returns within timeoutSeconds, and the probe is counted as failed.
# Kubernetes knows the application is truly unhealthy and may restart it.

Implementing Health Endpoints in Applications #

For Python/Flask applications, add health check endpoints that clearly separate liveness and readiness:

# roles/app-health/tasks/main.yml
---
- name: Deploy the health check module to the application
  template:
    src: health.py.j2
    dest: "{{ app_dir }}/health.py"
    owner: "{{ app_user }}"
    mode: '0644'
  notify: Restart the application

# templates/health.py.j2
# Health check endpoints for {{ app_name }}
from flask import Blueprint, jsonify
import psycopg2
import redis
import time

health_bp = Blueprint('health', __name__)

# Timeout for the dependency check — readiness must be fast
HEALTH_CHECK_TIMEOUT = 2  # seconds


@health_bp.route('/health/live')
def liveness():
    """Liveness: check the process is still running — doesn't check dependencies"""
    # ✓ This endpoint MUST be fast and not touch I/O.
    # Its only purpose is proving the process can still handle requests.
    return jsonify({
        "status": "ok",
        "service": "{{ app_name }}",
        "version": "{{ app_version }}",
        "timestamp": time.time()
    }), 200


@health_bp.route('/health/ready')
def readiness():
    """Readiness: check all dependencies are ready — return 503 if any fails"""
    checks = {}
    overall_status = "ok"
    failed = []

    # Check the database
    try:
        conn = psycopg2.connect(
            "{{ db_url }}",
            connect_timeout=HEALTH_CHECK_TIMEOUT
        )
        conn.close()
        checks["database"] = "ok"
    except Exception as e:
        checks["database"] = f"failed: {str(e)}"
        overall_status = "degraded"
        failed.append("database")

    # Check Redis
    try:
        r = redis.from_url("{{ redis_url }}")
        r.ping()
        checks["redis"] = "ok"
    except Exception as e:
        checks["redis"] = f"failed: {str(e)}"
        overall_status = "degraded"
        failed.append("redis")

    # Status code 503 for degraded — the load balancer removes it from rotation
    status_code = 200 if overall_status == "ok" else 503
    return jsonify({
        "status": overall_status,
        "checks": checks,
        "failed": failed,
        "timestamp": time.time()
    }), status_code

Anti-Pattern: One Endpoint for Everything vs Separate Endpoints #

# ANTI-PATTERN: One endpoint that calls all dependencies
@app.route('/health')
def health():
    # ✗ This endpoint is used for BOTH liveness and readiness
    # The Kubernetes liveness probe calls this endpoint
    db_ok = check_database()
    redis_ok = check_redis()
    if not db_ok or not redis_ok:
        return "unhealthy", 500
    return "ok", 200
# Problem: when the database dies, the endpoint returns 500.
# The Kubernetes liveness probe fails → Kubernetes RESTARTS the container.
# All containers restart at the same time when the database dies,
# then try connecting to the newly recovering database, all fail,
# a restart loop happens, and the application never recovers.

# CORRECT: Two endpoints with different responsibilities
@app.route('/health/live')
def liveness():
    # ✓ Doesn't check dependencies — always 200 while the process runs
    return jsonify({"status": "ok"}), 200

@app.route('/health/ready')
def readiness():
    # ✓ Checks dependencies, returns 503 if any fails
    db_ok = check_database()
    redis_ok = check_redis()
    if not db_ok or not redis_ok:
        return jsonify({"status": "degraded"}), 503
    return jsonify({"status": "ok"}), 200
# Advantage: database dies → readiness returns 503 → the instance leaves
# the load balancer rotation, BUT Kubernetes does NOT restart it.
# The instance stays alive, saves resources, and immediately re-enters rotation
# once the database recovers without a restart loop.

Ansible-Based Health Checks for Post-Deployment #

After deployment, Ansible needs to verify all components are healthy before the deployment is considered successful. Without this verification, a “successful” deployment could leave an application unable to accept traffic:

# playbooks/verify-deployment.yml
---
- name: Verify health after deployment
  hosts: appservers
  gather_facts: false

  tasks:
    - name: Wait for the application port to open
      wait_for:
        port: "{{ app_port }}"
        host: "{{ inventory_hostname }}"
        timeout: 60
        state: started

    - name: Verify the liveness endpoint
      uri:
        url: "http://{{ inventory_hostname }}:{{ app_port }}/health/live"
        method: GET
        status_code: 200
        timeout: 10
      register: liveness_result
      until: liveness_result.status == 200
      retries: 6
      delay: 10

    - name: Verify the readiness endpoint
      uri:
        url: "http://{{ inventory_hostname }}:{{ app_port }}/health/ready"
        method: GET
        status_code: 200
        timeout: 10
      register: readiness_result
      until: readiness_result.status == 200
      retries: 12
      delay: 10
      failed_when: readiness_result.status not in [200]

    - name: Verify the running version matches the target
      assert:
        that:
          - readiness_result.json.version is defined
          - readiness_result.json.version == app_version
        fail_msg: >
          Version mismatch!
          Expected: {{ app_version }}
          Actual: {{ readiness_result.json.version | default('not detected') }}          

    - name: Display the health check summary
      debug:
        msg:
          - "✓ Liveness: {{ liveness_result.json.status }}"
          - "✓ Readiness: {{ readiness_result.json.status }}"
          - "✓ Version: {{ readiness_result.json.version }}"
          - "✓ Database: {{ readiness_result.json.checks.database }}"
          - "✓ Redis: {{ readiness_result.json.checks.redis }}"

This post-deployment verification is usually combined with the Ansible rolling update (serial: 1 or serial: 25%) so only a small part of the fleet is updated and verified at one time. If verification fails, Ansible stops and the deployment is considered failed before the fleet is half-way on the new version.

Health Check Configuration at the Load Balancer #

Deploy health check configuration to HAProxy using Ansible. HAProxy supports HTTP, TCP, and custom script-based health checks:

# roles/haproxy/tasks/health-check.yml
---
- name: Deploy the HAProxy configuration with health checks
  template:
    src: haproxy.cfg.j2
    dest: /etc/haproxy/haproxy.cfg
    validate: haproxy -c -f %s
  notify: Reload HAProxy

{# templates/haproxy.cfg.j2 #}
global
    log /dev/log local0
    maxconn 50000

defaults
    log global
    mode http
    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend http_front
    bind *:80
    default_backend app_servers

backend app_servers
    balance roundrobin
    # ✓ Health check to the readiness endpoint, not liveness
    option httpchk GET /health/ready HTTP/1.1\r\nHost:\ localhost
    http-check expect status 200

    # Important fall/rise parameters:
    #   inter 10s: check every 10 seconds
    #   fall 3: 3 consecutive failures = out of rotation
    #   rise 2: 2 consecutive successes = back in rotation
    # This prevents flapping when there's a single momentary failed response.
{% for host in groups['appservers'] %}
    server {{ host }} {{ hostvars[host]['ansible_default_ipv4']['address'] }}:{{ app_port }} \
        check inter 10s fall 3 rise 2
{% endfor %}

The fall 3 rise 2 configuration means: remove the server from rotation after 3 consecutive failed health checks, return it after 2 successful health checks. These parameters are very important for stability — without thresholds, a single momentary failed response (like a network blip) could remove an instance from rotation and unnecessarily reduce capacity.

Never set fall 1 on health checks. Single failure = single eviction = one packet loss = instance leaves the load balancer. Always need a minimum of 2-3 consecutive failures to consider unhealthy, and a minimum of 1-2 successes to consider recovered.

Health Checks for Kubernetes Probes #

Configure liveness, readiness, and startup probes in Kubernetes Deployments using Ansible. All three have different purposes and different parameters:

- name: Deploy the Deployment with correctly configured probes
  kubernetes.core.k8s:
    kubeconfig: "{{ k8s_kubeconfig }}"
    state: present
    definition:
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: "{{ app_name }}"
        namespace: "{{ app_namespace }}"
      spec:
        template:
          spec:
            containers:
              - name: "{{ app_name }}"
                image: "{{ app_image }}:{{ app_version }}"
                # ✓ Liveness: only checks the app is responsive, restarts if hung
                livenessProbe:
                  httpGet:
                    path: /health/live
                    port: "{{ app_port }}"
                  initialDelaySeconds: 15    # Wait 15 seconds before starting checks
                  periodSeconds: 10          # Check every 10 seconds
                  failureThreshold: 3        # Restart after 3 failures
                  timeoutSeconds: 5
                # ✓ Readiness: checks dependencies, leaves traffic if not ready
                readinessProbe:
                  httpGet:
                    path: /health/ready
                    port: "{{ app_port }}"
                  initialDelaySeconds: 5     # Faster than liveness
                  periodSeconds: 5
                  failureThreshold: 3        # Remove from traffic after 3 failures
                  successThreshold: 1        # Return after 1 success
                  timeoutSeconds: 3
                # ✓ Startup: tolerates long startup time, prevents liveness kill during init
                startupProbe:
                  httpGet:
                    path: /health/live
                    port: "{{ app_port }}"
                  failureThreshold: 30       # Give 300 seconds (30 x 10s) for startup
                  periodSeconds: 10

Anti-Pattern: Health Checks That Call Dependencies in Liveness #

# ANTI-PATTERN: a livenessProbe that calls external dependencies
livenessProbe:
  httpGet:
    path: /health/check-all   # endpoint that checks DB, Redis, all dependencies
    port: 8080
  failureThreshold: 3
# Problem: when the database dies, the livenessProbe fails, Kubernetes
# kills the container, the container restarts, tries connecting to the DB again, fails,
# kills again, restart loop. A database that should recover in
# 5 minutes instead makes the application unavailable for 30 minutes
# because of the thundering herd restart.

# CORRECT: a livenessProbe that only checks the process, a readinessProbe that checks dependencies
livenessProbe:
  httpGet:
    path: /health/live       # endpoint without dependency checks
    port: 8080
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready      # endpoint that checks the DB and dependencies
    port: 8080
  failureThreshold: 3
# When the database dies: readinessProbe fails → the pod leaves the Service endpoint
# (receives no traffic), BUT the pod stays alive. Once the database recovers,
# the readinessProbe returns 200 → the pod receives traffic again. No restart.

Kubernetes Probe Flow #

Here’s a sequence diagram showing how Kubernetes coordinates the three probes when a pod starts and when problems occur:

sequenceDiagram
    participant K as Kubelet
    participant SP as Startup Probe
    participant LP as Liveness Probe
    participant RP as Readiness Probe
    participant SVC as Service
    participant Pod as Application

    Note over K,Pod: "Startup Phase"
    K->>SP: "Check pod (every 10s)"
    SP->>Pod: "GET /health/live"
    Pod-->>SP: 200
    SP-->>K: Success
    K->>LP: "Enable liveness probe"
    K->>RP: "Enable readiness probe"

    Note over K,Pod: "Normal Phase"
    RP->>Pod: "GET /health/ready"
    Pod-->>RP: 200
    RP-->>K: Ready
    K->>SVC: "Add pod to the endpoint"
    SVC->>Pod: "Route traffic"

    Note over K,Pod: "Dependency Down"
    RP->>Pod: "GET /health/ready"
    Pod-->>RP: "503 (DB down)"
    RP-->>K: "Not Ready"
    K->>SVC: "Remove pod from the endpoint"
    Note over LP: "Liveness STILL 200 (the process still runs)"

    Note over K,Pod: "Application Hung"
    LP->>Pod: "GET /health/live"
    Pod-->>LP: "timeout (5s)"
    LP->>LP: "failure count++"
    LP-->>K: "3x failure = restart"
    K->>Pod: "Kill & restart"

Notice how the startup probe is only active until the pod first succeeds, after which liveness takes over. This prevents the classic scenario: an application needs 60 seconds to start (e.g. loading a large cache, connecting to many services), but the liveness probe is set to 30 seconds with a failure threshold of 3 — the pod gets killed before it ever starts.

Health Check Dashboard and Alerts #

Create a Grafana dashboard displaying the health status of all services. This dashboard usually shows aggregated health from several services at once, with green/yellow/red visual indicators:

- name: Deploy the health check dashboard
  copy:
    src: files/dashboards/health-overview.json
    dest: /var/lib/grafana/dashboards/health-overview.json
    owner: grafana
    mode: '0640'
  notify: Reload Grafana dashboards

Alert rules for failed health checks:

# roles/prometheus/templates/rules/health-checks.yml.j2
groups:
  - name: health_checks
    rules:
      - alert: ServiceHealthCheckFailing
        expr: >
          probe_success{job="blackbox"} == 0          
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Health check failed: {{ '{{' }} $labels.instance {{ '}}' }}"
          description: "Endpoint {{ '{{' }} $labels.instance {{ '}}' }} unresponsive for 2 minutes"
          runbook_url: "https://wiki.company.com/runbooks/service-down"

      - alert: ServiceHealthDegraded
        expr: >
          probe_success{job="blackbox"} == 0          
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "Service degraded: {{ '{{' }} $labels.instance {{ '}}' }}"
          description: "Intermittent health check failure detected"

Notice the difference between for: 2m vs for: 30s — critical alerts need longer confirmation time (reducing false positives from momentary blips), while warnings are more sensitive for early detection.

Health Checks for Cron and Background Jobs #

Health checks aren’t only for HTTP services. For cron jobs and background workers, the common pattern is a heartbeat file updated every time the job runs, then monitored by a health check service:

# roles/cron-health/tasks/main.yml
---
- name: Set up the cron job with a heartbeat
  cron:
    name: "data-sync-heartbeat"
    minute: "*/5"
    job: >
      /opt/app/run-sync.sh &&
      /usr/bin/date +%s > /var/run/app-sync.heartbeat      
    user: "{{ app_user }}"

- name: Set up the health check service that monitors the heartbeat
  template:
    src: heartbeat-check.sh.j2
    dest: /opt/monitoring/heartbeat-check.sh
    mode: '0755'

- name: Cron job for the health check service
  cron:
    name: "heartbeat-monitor"
    minute: "*/1"
    job: "/opt/monitoring/heartbeat-check.sh {{ app_name }} 600"
    user: monitoring

{# templates/heartbeat-check.sh.j2 #}
#!/bin/bash
# Monitor the heartbeat file and update the exporter metric
SERVICE=$1
MAX_AGE_SECONDS=$2
HEARTBEAT_FILE="/var/run/${SERVICE}.heartbeat"

if [ ! -f "$HEARTBEAT_FILE" ]; then
    echo "heartbeat_missing{service=\"$SERVICE\"} 1" | \
        curl --data-binary @- http://localhost:9115/metrics/job/heartbeat
    exit 0
fi

HEARTBEAT=$(cat "$HEARTBEAT_FILE")
NOW=$(date +%s)
AGE=$((NOW - HEARTBEAT))

if [ $AGE -gt $MAX_AGE_SECONDS ]; then
    echo "heartbeat_stale{service=\"$SERVICE\"} 1" | \
        curl --data-binary @- http://localhost:9115/metrics/job/heartbeat
else
    echo "heartbeat_ok{service=\"$SERVICE\"} 1" | \
        curl --data-binary @- http://localhost:9115/metrics/job/heartbeat
fi

This heartbeat pattern can also be used to monitor ETL jobs, scheduled tasks, or data synchronization processes that don’t expose an HTTP endpoint.

When Health Checks Aren’t Enough #

Health checks are real-time operational controls. For deeper visibility (latency, error rate, throughput), we need metrics and tracing (see Metric Collection and Tracing). For more proactive alerts before users file complaints, we need alerting (see Alerting). Health checks are the foundation that must exist first — the three observability pillars complement each other:

flowchart TD
    Obs["Observability"] --> Logs["Logs"]
    Obs --> Metrics["Metrics"]
    Obs --> Tracing["Tracing"]
    Logs --> LogsDesc["What happened (event narrative)"]
    Metrics --> MetricsDesc["How often / how slow (aggregated numbers)"]
    Tracing --> TracingDesc["Where a request is slow along its journey"]

Health checks work at yet another level — they’re not part of the three pillars because they’re not about “what happened” but “is it OK to receive traffic again”. They’re decision support for orchestrators (Kubernetes, load balancers) that need real-time yes/no answers.

Summary #

Liveness and readiness are two different endpoints with different purposes — don’t combine them in one endpoint.
Liveness probes must not check dependencies (database, Redis) — if a dependency is down, we don’t want all containers restarting too and worsening the situation.
Readiness probes may and must check dependencies — an instance whose dependencies are unreachable indeed isn’t ready to serve traffic.
Choose the right probe type: httpGet for web services, tcpSocket for simple port checks, exec for complex custom validations.
A tcpSocket probe isn’t enough for web servers — it only checks the port, not whether handlers can respond. Use httpGet to the /health/live endpoint.
The fall and rise parameters in HAProxy prevent flapping — fall 3 rise 2 is more stable than fall 1.
startupProbe in Kubernetes for applications with long startup times — prevents liveness probes from killing applications still in the startup process.
Heartbeat files for health checking cron jobs and background workers that don’t expose HTTP endpoints.
In Ansible deployment pipelines, always verify deployments with health checks after rolling updates before marking the deployment as successful.
Health checks are the foundation — for deeper visibility you need logs, metrics, and tracing (see Logging, Metric Collection, Tracing).

← Previous: Tracing Next: Incident Response →