Production Readiness #
There’s a very wide gap between systems that merely “run in production” and systems that are truly “production ready”. An Ansible playbook might successfully execute once on our production server, but that doesn’t prove our system is ready for traffic spikes, disk failures, or human configuration errors. Production readiness isn’t just the status when our code runs successfully without red errors in the terminal. Production readiness is about resilience, recoverability, observability, and the guarantee that our configuration can be safely operated by the whole team even in crisis situations.
When we manage infrastructure with Ansible, we hold control over hundreds or thousands of servers at once. This scale of power brings an equal risk: one small error in a playbook variable can cripple an entire cluster in seconds. Therefore, we must apply production readiness discipline strictly. We need to build validation mechanisms for conditions before changes, post-change smoke testing tactics, and disaster recovery playbooks tested periodically.
The Safe Infrastructure Change Lifecycle #
To maintain production environment stability, every configuration modification must go through a standardized lifecycle. We must not blindly apply changes without preparing a safety net first.
Here’s the safe infrastructure change lifecycle workflow:
flowchart TD
A["Start the Change"] --> B["Capture the Baseline State"]
B --> C["Apply the Change (Playbook)"]
C --> D{"Did Execution Succeed?"}
D -- "No" --> E["Start Disaster Recovery / Rollback"]
D -- "Yes" --> F["Verify Post-Change (Smoke Test)"]
F --> G{"Did Verification Pass?"}
G -- "No" --> E
G -- "Yes" --> H["Send a Success Notification"]
H --> I["Done (Production Ready)"]By adopting this cycle, we ensure every change always starts with recording the current system state (baseline) and ends with thorough verification guaranteeing no service degradation.
Production Readiness Checklist #
Before we raise our Ansible repository’s status to “production ready”, we must audit all infrastructure and playbook components using the following review criteria:
1. Deployment and Testing Category #
- Representative Staging Testing: We must ensure playbooks are tested in a staging environment with system specifications, network topology, and data volumes resembling the production environment.
- Standalone Dry Run Review: Before real execution, run
ansible-playbook -i inventory/production site.yml --check --diffand do a line-by-line review of the resulting diff output. - Tested Rollback Procedures: Every Pull Request changing sensitive configuration must be accompanied by concrete rollback step documentation that has been simulated successfully.
2. Backup and Recovery Category #
- Backup Before Maintenance: Our maintenance playbooks must have automatic tasks to back up the database or important configuration files before performing any modification actions.
- Backup Integrity Verification: We must schedule periodic automatic restore tests. Remember, backup files never tested for restore aren’t valid backups.
- RTO and RPO Definitions: Our team must have written agreements on Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for each infrastructure service tier.
3. Observability Category #
- Health Check Endpoints: Every application deployed by Ansible must have a
/healthor/readyendpoint verifying internal application status (like database connections and remaining disk storage). - Configuration Alerting: Make sure Ansible also configures alert rules on Prometheus or Datadog so we immediately get notified if newly installed services encounter problems after deployment.
4. Security Category #
- Comprehensive Vault Encryption: There must be no database passwords, API tokens, or encryption keys stored in plaintext format in Git. All sensitive variables must be in Ansible Vault.
- Least Privilege Principle: Use
become: trueandbecome_userparameters selectively only on tasks needing high administrative (root) access; avoid applying them at the global playbook level if not needed.
System State Validation Before Major Changes #
One of the most fatal mistakes during maintenance is not knowing the system’s initial state before making changes. As a result, when an error occurs after the playbook finishes, we struggle to determine whether the error was caused by our new configuration or existed before the maintenance began.
To solve this problem, we must create a dedicated playbook named capture-baseline.yml. This playbook collects operational server status data, installed package versions, and active configuration files, then saves them as a local archive on our computer before maintenance starts.
Here’s the standard production capture-baseline.yml playbook we use:
# playbooks/capture-baseline.yml
---
- name: Capture the baseline state before major changes
hosts: "{{ target_hosts | default('all') }}"
gather_facts: true
become: true
vars:
baseline_dir: "/tmp/ansible-baselines/{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"
pre_tasks:
- name: Ensure the local baseline directory is available
file:
path: "{{ baseline_dir }}/{{ inventory_hostname }}"
state: directory
mode: '0755'
delegate_to: localhost
become: false
tasks:
- name: 1. Snapshot the list and versions of installed packages
package_facts:
manager: auto
- name: Save the package list to a local file
copy:
content: >
{{ ansible_facts.packages
| dict2items
| selectattr('value.0.version', 'defined')
| map(attribute='key')
| sort
| join('\n') }}
dest: "{{ baseline_dir }}/{{ inventory_hostname }}/packages-installed.txt"
delegate_to: localhost
become: false
- name: 2. Snapshot the service status
service_facts:
- name: Save the running service status to a local file
copy:
content: >
{{ ansible_facts.services
| dict2items
| selectattr('value.state', 'equalto', 'running')
| map(attribute='key')
| sort
| join('\n') }}
dest: "{{ baseline_dir }}/{{ inventory_hostname }}/services-running.txt"
delegate_to: localhost
become: false
- name: 3. Snapshot the TCP ports currently listening
shell: ss -tlnp
register: listen_ports
changed_when: false
- name: Save the TCP port list to a local file
copy:
content: "{{ listen_ports.stdout }}"
dest: "{{ baseline_dir }}/{{ inventory_hostname }}/tcp-ports-listen.txt"
delegate_to: localhost
become: false
- name: 4. Fetch copies of current critical configuration files
fetch:
src: "{{ item }}"
dest: "{{ baseline_dir }}/{{ inventory_hostname }}/configs/"
flat: yes
loop:
- /etc/nginx/nginx.conf
- /etc/sysctl.conf
- /etc/hosts
ignore_errors: true
- name: 5. Record the remaining disk storage space
shell: df -h /
register: disk_usage
changed_when: false
- name: Save the disk info to a local file
copy:
content: "{{ disk_usage.stdout }}"
dest: "{{ baseline_dir }}/{{ inventory_hostname }}/disk-usage.txt"
delegate_to: localhost
become: false
The playbook above leverages built-in modules like package_facts and service_facts to abstract data collection without relying on OS-specific commands. Running this playbook produces the following local archive folder structure on our laptop:
/tmp/ansible-baselines/2026-06-18_21-30-00/
├── web-server-01/
│ ├── packages-installed.txt
│ ├── services-running.txt
│ ├── tcp-ports-listen.txt
│ ├── disk-usage.txt
│ └── configs/
│ ├── nginx.conf
│ └── sysctl.conf
└── db-server-01/
├── packages-installed.txt
└── ...
With this baseline archive, we have a very clear recovery reference point if our new configuration breaks service status on the servers.
Post-Change Verification #
After the deployment process finishes running through Ansible, we must not assume everything is fine just because the execution log shows no red. We must do automatic smoke testing to verify the real functionality of our applications and servers.
Here’s the verify-post-change.yml playbook designed to run a series of business logic and system stability tests after changes are applied:
# playbooks/verify-post-change.yml
---
- name: Verify system readiness post-change
hosts: "{{ target_hosts | default('all') }}"
gather_facts: true
become: true
vars:
critical_services:
- nginx
- postgresql
health_check_endpoints:
- name: "Main API Application"
port: 8080
path: "/health"
- name: "Frontend Web Portal"
port: 80
path: "/"
tasks:
- name: 1. Get the post-change service status
service_facts:
- name: Ensure all critical services are in the running state
assert:
that:
- ansible_facts.services[item + '.service'] is defined
- ansible_facts.services[item + '.service'].state == 'running'
fail_msg: "CRITICAL ERROR: The {{ item }} service is detected down after deployment!"
success_msg: "Success: The {{ item }} service is running safely."
loop: "{{ critical_services }}"
- name: 2. Test HTTP health check endpoints locally
uri:
url: "http://localhost:{{ item.port }}{{ item.path }}"
status_code: 200
timeout: 10
loop: "{{ health_check_endpoints }}"
loop_control:
label: "{{ item.name }}"
register: health_results
- name: Ensure health check responses contain an OK status
assert:
that:
- "'ok' in item.json.status or item.status == 200"
fail_msg: "CRITICAL ERROR: The {{ item.item.name }} health check endpoint returns an unhealthy status!"
loop: "{{ health_results.results }}"
loop_control:
label: "{{ item.item.name }}"
- name: 3. Check for new errors appearing in system logs (last 5 minutes)
shell: |
journalctl --since "5 minutes ago" -p err --no-pager | wc -l
register: system_errors
changed_when: false
- name: Give a warning if new log errors are detected
debug:
msg: "⚠️ WARNING: Detected {{ system_errors.stdout }} new errors in journald in the last 5 minutes!"
when: system_errors.stdout | int > 0
By running this verification playbook after the main playbook execution, we can immediately detect performance degradation or dead services without waiting for user complaints.
Tested Disaster Recovery Playbooks #
Advanced production readiness demands we have written procedures to recover systems when worst-case scenarios occur (e.g. database data corruption or total disk failure). Writing recovery instructions while our servers are down and management is panicking is the best recipe for fatal operational mistakes.
We must design defensive disaster recovery playbooks: forcing explicit operator confirmation, doing preventive backups before wiping old data, and validating backup file availability before execution starts.
Below is an example of a safe PostgreSQL database recovery playbook from S3 backups for production environments:
# playbooks/disaster-recovery/restore-database.yml
---
# EMERGENCY WARNING: This playbook will overwrite existing production database data!
# Execute only after coordinating with the Incident Commander.
- name: Execute the database recovery from the S3 backup
hosts: db_servers
become: true
serial: 1 # Do it one-by-one to avoid total cluster outage if mis-targeted
vars:
s3_bucket: "company-production-backups-secure"
restore_target_date: "{{ target_date | mandatory }}" # Format: YYYY-MM-DD
confirm_action: "{{ confirm | default('no') }}"
pre_tasks:
- name: 1. Protect with explicit operator confirmation
assert:
that:
- confirm_action == "I-AM-AWARE-THIS-WILL-OVERWRITE-PRODUCTION-DATA"
fail_msg: >
Failed: We must include an explicit confirmation argument to run this recovery!
Example run:
ansible-playbook -i inventory/production restore-database.yml -e "target_date=2026-06-18 confirm=I-AM-AWARE-THIS-WILL-OVERWRITE-PRODUCTION-DATA"
- name: 2. Record the initial disaster recovery activity log
lineinfile:
path: /var/log/disaster-recovery-audit.log
line: >
[{{ ansible_date_time.iso8601 }}] RESTORE DB STARTED by user: {{ ansible_user }}
using backup date: {{ restore_target_date }}
create: yes
owner: root
group: root
mode: '0600'
tasks:
- name: 3. Stop the application services connected to the database
systemd:
name: "{{ item }}"
state: stopped
loop:
- node-api-service
- background-worker-service
delegate_to: "{{ item }}"
loop_control:
label: "Stopping the {{ item }} service"
# We assume the application service hosts are registered in the inventory
- name: 4. Create an emergency backup (Snapshot) of the current database before overwriting
shell: |
pg_dump -U postgres -d main_production -F c -b -v -f "/tmp/emergency-pre-restore-{{ ansible_date_time.epoch }}.dump"
become_user: postgres
ignore_errors: yes # Continue the restore even if the emergency backup fails (e.g. disk full)
- name: 5. Download the backup file from AWS S3
amazon.aws.s3_object:
bucket: "{{ s3_bucket }}"
object: "backups/postgres/{{ restore_target_date }}/main_production.dump"
dest: "/tmp/restore-target.dump"
mode: get
register: s3_download
until: s3_download is success
retries: 3
delay: 10
- name: 6. Clean up and recreate the empty database
postgresql_db:
name: main_production
state: absent
become_user: postgres
- name: Recreate the empty database with the standard schema
postgresql_db:
name: main_production
state: present
encoding: UTF-8
become_user: postgres
- name: 7. Apply the data recovery from the dump file
shell: |
pg_restore -U postgres -d main_production -v "/tmp/restore-target.dump"
become_user: postgres
register: restore_execution
failed_when: restore_execution.rc not in [0, 1] # pg_restore sometimes returns rc=1 for non-critical warnings
- name: 8. Bring the database and application services back up
systemd:
name: "{{ item }}"
state: started
loop:
- node-api-service
- background-worker-service
delegate_to: "{{ item }}"
- name: 9. Delete the temporary recovery files to clean the disk
file:
path: "{{ item }}"
state: absent
loop:
- "/tmp/restore-target.dump"
post_tasks:
- name: Record the final recovery activity log
lineinfile:
path: /var/log/disaster-recovery-audit.log
line: >
[{{ ansible_date_time.iso8601 }}] RESTORE DB FINISHED status: {{ 'SUCCESS' if not ansible_failed_result is defined else 'FAILED' }}
delegate_to: localhost
become: false
Why is this disaster recovery playbook safe for production levels?
- Explicit Safeguards (
assert): The playbook won’t execute anything if the operator mistypes the command or accidentally presses enter. Operators are forced to type a long, specific confirmation string. - Controlled Serialization (
serial: 1): If we apply it to a database cluster, the playbook runs one-by-one to ensure we don’t break all servers at once if a parameter is wrong. - Emergency Backup Snapshot: Before destroying the old production database, the playbook tries to create a final local dump. If it turns out our restore file is corrupt, we still have a database copy from right before the recovery process started.
- Audit Trail: All recovery activity is recorded to a local audit log file with precise timestamps, helping post-incident analysis (post-mortem).
Visibility and Monitoring Integration (Observability) #
Production-ready systems must have their health status monitorable at all times. As part of our Ansible deployment lifecycle, make sure every server role we install automatically registers itself to the central monitoring system.
We must automate the following steps on every deployment target:
- Node Exporter Installation: Install the Prometheus Node Exporter on every VM to export basic system metrics (CPU, RAM, Disk, Network).
- Log Shipping Configuration: Use Ansible to configure Promtail or Filebeat to send system logs in real-time to Loki or Elasticsearch.
- Monitoring Target Registration: Use the Ansible HTTP module (
uri) to call the Prometheus server API and dynamically register the new node’s IP into the monitoring configuration.
By integrating observability aspects directly into our infrastructure code, we eliminate the bad habit of teams forgetting to install monitoring on new servers.
Summary #
- Production Readiness Is a Culture — Not just the absence of error messages in playbooks, but the guarantee of system resilience, fast recovery capability, and complete operational visibility.
- The Importance of Baselines — Always run initial system state recording before major maintenance using snapshot playbooks (
capture-baseline.yml) to accurately track configuration regressions.- Automatic Smoke Testing — Integrate post-change readiness test playbooks (
verify-post-change.yml) doing real-time service status assertions and HTTP health checks before closing maintenance sessions.- Defensive Recovery Playbooks — Design disaster handling playbooks with strict safety nets: force explicit string confirmations, do emergency backups before overwriting data, and record execution history in audit logs.
- Automate Monitoring — Every VM or service installed via Ansible must be configured to export basic metrics and send system logs to a centralized log repository from day one.