Team Workflow #

Ansible managed by one person is fundamentally different from Ansible managed collaboratively by the whole team. When we work alone, all assumptions, configuration histories, and playbook logic are neatly stored in our heads. However, as our team grows, every undocumented configuration decision and every playbook without a standard structure becomes a time bomb. Without an agreed structured workflow, our Ansible repository turns into a scary collection of code — a place where no engineer dares to make modifications for fear of breaking production servers.

To avoid that scenario, we must treat infrastructure configuration the same way we treat application code (Infrastructure as Code or IaC). We need a disciplined Git branching strategy, a strict yet productive code review process, a Pull Request template carrying deep context, a smooth onboarding process for new team members, and a collection of tactical runbooks handling daily operational scenarios. By adopting these collaboration standards, our team can move fast without sacrificing system stability and security.

Git Branching Strategy for Infrastructure #

In traditional software development, we often see complex branching strategies like GitFlow. However, for Ansible-based infrastructure management, overly complex strategies often create obstacles. Infrastructure has unique characteristics: a change in one part of the code can directly impact the status of a system running in real-time. Therefore, a simple, transparent, testing-focused branching strategy (like GitHub Flow) proves safer and more effective for DevOps teams.

Here’s an overview of the branching workflow we use:

flowchart TD
    A["Developer (Local)"] -->|"1. Create a Feature Branch"| B["Feature Branch"]
    B -->|"2. Push & Create a PR"| C["GitHub / GitLab PR"]
    C -->|"3. CI Pipeline (Lint & Test)"| D{"CI Valid?"}
    D -->|"No: Fix Code"| B
    D -->|"Yes: 4. Code Review & Approval"| E{"Reviewer Agrees?"}
    E -->|"No: Revise Code"| B
    E -->|"Yes: 5. Squash Merge"| F["Main Branch"]
    F -->|"6. CD Deploy (Staging)"| G["Staging Environment"]
    G -->|"7. Promotion (Production)"| H["Production Environment"]

Below are the rules we must consistently apply across the whole team:

1. Main Branch #

The main branch is the single source of truth for our entire infrastructure state.

• Always Deployable State: Code on the main branch must always be in a stable state, ready to deploy to staging or production at any time.
• Branch Protection: We must enable branch protection on main in the Git platform (like GitHub or GitLab). Team members are strictly forbidden from pushing directly to the main branch.
• Merge Requirements: For a change to be merged into main, the code must pass automated checks (CI pipeline) and get at least one approval from another team member through the code review process.

2. Feature & Fix Branches #

Every time we want to make a change — whether adding a new role, changing configuration parameters, or fixing a bug — we must create a new branch from main.

• Naming Convention: Use descriptive branch names reflecting the change’s purpose.
  • feature/add-redis-role for adding new components or features.
  • fix/nginx-ssl-config for bug or wrong configuration fixes.
  • hotfix/critical-security-patch for emergency fixes needing immediate handling.
• Short Lifetime: Try to keep feature branches short-lived. We should frequently merge small changes (frequent commits) rather than hoarding giant changes in one branch for weeks, which only triggers painful merge conflicts.

3. Merge Policy #

To keep the commit history on main clean and readable, we apply the following merge rules:

• Squash and Merge: Recommended for small or medium Pull Requests (PRs). This method combines all experimental commits from our feature branch into one clean single commit on main.
• Merge Commit: Used for large architectural changes involving several sub-systems, so detailed change history is preserved for future audit needs.

Pull Request Template as Context Communication #

The main problem in infrastructure collaboration is the lack of information about the impact of proposed changes. A reviewer shouldn’t have to guess what a playbook contains or what effects will occur on the system after the playbook runs. To bridge this communication, we must use a standardized Pull Request template. This template guides the PR author to explain the change description, local test results, impact analysis, and recovery plans if failures occur.

We need to place this template file in the repository at .github/pull_request_template.md (or .gitlab/merge_request_templates/default.md if using GitLab).

Here’s the standard Pull Request template we should use:

<!-- .github/pull_request_template.md -->

## 📝 Change Description
<!-- Write a concise explanation of what changed, why this change is needed, and what problem we're solving. -->
Example: "Upgrading the TLS configuration on Nginx to support TLS 1.3 and disabling TLS 1.0/1.1 to raise PCI-DSS compliance security standards."

## 🛠️ Change Type
- [ ] New role (adding a new infrastructure component)
- [ ] Changes to an existing Role
- [ ] General Playbook modification
- [ ] Configuration variable or template file updates
- [ ] Bug fix / Hotfix
- [ ] Refactoring (improving code structure without changing function)

## 🧪 Testing Results
<!-- Prove that this change has been tested locally or in a non-production environment. -->
- [ ] `ansible-lint` has run locally with no errors/warnings found.
- [ ] `ansible-playbook --syntax-check` executed successfully without errors.
- [ ] Molecule-based testing has been run and passed (for modified roles).
- [ ] Manual testing succeeded in the environment: [name the environment, e.g. Dev/Staging]
- [ ] Idempotency verification succeeded (playbook run twice in a row, second run shows `changed=0`).

## ⚡ Impact & Risk (Impact Analysis)
- **Affected Environments:** [ ] Staging  [ ] Production  [ ] All Environments
- **Affected Services:** (Name the touched services, e.g. Nginx, PostgreSQL, application daemons)
- **Downtime Required:** [ ] Yes  [ ] No
  *If Yes, estimate the downtime duration and mitigation steps:*
- **Rollback Plan (In Case of Problems):**
  <!-- Describe step-by-step instructions to restore configuration to the state before this PR was applied. -->
  1. Run the rollback playbook with the recovery tag: `ansible-playbook -i inventory/production site.yml --tags rollback-nginx`
  2. Or revert this commit in Git, then re-run the main playbook on the main branch.

## 📋 Security & Code Quality Checklist
- [ ] No sensitive information (passwords, API keys, private keys) written in plaintext. All secrets are encrypted with Ansible Vault.
- [ ] New variables are defined with safe default values in the `defaults/main.yml` folder.
- [ ] All Ansible tasks have descriptive, human-readable `name` parameters.
- [ ] `shell` or `command` code blocks include `changed_when` to maintain idempotency status accuracy.

By forcing this information to be filled in on every Pull Request, we minimize the risk of operational errors from one-sided assumptions. Reviewers can quickly assess the change’s risk level and give relevant feedback.

A Productive Code Review Process for Ansible #

Code review isn’t just a formality to press the “Approve” button. In the Infrastructure as Code context, code review is our first line of defense in preventing production outages. When we review a teammate’s Ansible code, we’re not looking for who writes the best code — we’re collectively ensuring that the code to be executed on hundreds of servers is safe, efficient, and easy to understand in the future.

For productive reviews, we must divide review focus into four main pillars:

1. Technical Correctness and Idempotency #

• Validate Shell/Command Tasks: Every time we see a shell or command module usage, we should question its appropriateness. Can a built-in Ansible module (like template, copy, apt, or user) be used instead? Built-in modules guarantee system state stability, while custom shell scripts often break the idempotency principle.
• Double Run Testing: Make sure the PR author has verified their playbook is safe to run repeatedly. If we see a task modifying a configuration file by adding new text lines (e.g. using shell: echo "..." >> /file), ask whether there’s a prevention for duplicating those configuration lines.
• Error Handling: Check how tasks handle failure conditions. Do they use block, rescue, and always blocks to ensure cleanup processes still run when errors occur?

2. Security and Secrets Compliance #

• Scan for Leaked Secrets: This is the highest priority. We must carefully review new code lines to ensure no API tokens, database passwords, or SSL private keys are accidentally written directly in YAML files.
• Use no_log: On tasks dealing with sensitive data — like creating new users with default passwords or interacting with secret APIs — make sure the no_log: true property is added so Ansible doesn’t spill sensitive variable contents into terminal or CI server log output.

3. Maintainability #

• Variable Readability: Are the variable names used easy to understand? A variable name like db_port is far better than a generic variable like port which could collide with the web server port.
• Default Variable Documentation: All variables used inside roles must have well-defined default values in defaults/main.yml. We must also ensure inline documentation in the form of comments explains the function and data type of each variable.

4. Code Style Consistency #

• Linting Automation: We must not spend human energy debating spacing, indentation, or quote issues. Cosmetic matters like these should already be resolved by ansible-lint and yamllint at the CI Pipeline level. Human code review should focus on the architecture and logic of changes.

Let’s look at the difference between unproductive review comments (anti-pattern) and constructive, solution-oriented review comments (correct):

// ANTI-PATTERN: Review comments without concrete or subjective solutions

"Don't use the shell module here, find another way."

"This code is a mess. Please clean it up."

// CORRECT: Provide clear technical reasons and alternative solution examples that can be used

"This task uses the 'shell' module to install a package. For security and platform consistency, we should use the built-in 'apt' module. Here's an example implementation:
- name: Ensure nginx is installed
  apt:
    name: nginx
    state: present
    update_cache: yes"

With a collaborative communication style, the code review process feels like a shared learning opportunity to improve team skills, not a tense interrogation session.

Systematically Onboarding New Engineers #

One measure of mature infrastructure management success is how quickly a new engineer can contribute to our project. Often, teams let new engineers struggle to figure out their local computer setup, manually request credential access from various parties, and finally accidentally break configurations due to a lack of standardized setup guides.

We must provide a systematic onboarding flow divided into clear weekly targets. This way, the transition process runs safely and measurably.

Here’s the Ansible infrastructure onboarding worksheet we should give to every newly joined team member:

Local Work Environment Setup Guide (Day 1) #

New team members must follow these local environment setup steps to ensure configuration runs uniformly with other team members:

1. Isolate the Python Environment with a Virtual Environment #

We don’t want global Python modules colliding with the Ansible dependencies our project needs. Therefore, we always recommend using a Python virtual environment.

# Move to the main working directory
cd /path/to/our/ansible-infrastructure

# Create a new Python virtual environment
python3 -m venv .venv

# Activate the virtual environment
source .venv/bin/activate

# Make sure pip is the latest version
pip install --upgrade pip

2. Install Standardized Dependencies #

We install ansible-core, ansible-lint, and other testing tools using the requirements.txt file we’ve committed to Git to maintain version consistency.

# Create a requirements.txt file if it doesn't exist, then install dependencies
pip install ansible-core==2.16.4 ansible-lint==24.2.0 pre-commit==3.6.1 molecule[docker]==24.2.0

3. Configure Pre-commit Hooks #

So our code is always clean before being pushed to remote servers, we must enable local pre-commit hooks.

# Initialize the pre-commit hooks configuration
pre-commit install

4. Test Configuration Access in Dry Run Mode #

After successfully setting up the SSH key to the dev/staging Managed Node, run the following test command to ensure all local access rights are correctly configured without changing anything on the server:

# Run a dry-run for the whole staging configuration
ansible-playbook -i inventory/staging/ site.yml --check --diff

Onboarding Activity Plan (First 30 Days) #

Runbooks for Common Operational Scenarios #

In crisis situations or when daily operations run at a fast pace, teams must not rely on individual memory for critical tasks. We must document frequently performed operational scenarios in ready-to-use tactical runbooks. These runbooks contain flow guides, CLI commands to execute, and supporting playbooks.

Below are three main operational runbooks we often face in the field:

Runbook 1: Adding a New Server Node to Production #

When scaling out our infrastructure, follow this workflow to safely bring a new server into Ansible management:

New Node Addition Flow:

flowchart LR
    A["Provision the New VM"] --> B["Record the IP/DNS"]
    B --> C["Register in the Inventory"]
    C --> D["Test Ping"]
    D --> E["Dry-Run the Playbook"]
    E --> F["Run the Playbook (Limit Target)"]

Step 1: Register the New Host in the Inventory #

Open our production inventory file (e.g. inventory/production/hosts.yml) and register the new server under the appropriate group. Don’t forget to include unique variables if any:

# inventory/production/hosts.yml
all:
  hosts:
    web-production-05.company.internal:
      ansible_host: 10.0.1.25
      ansible_user: deploy-user
  children:
    webservers:
      hosts:
        web-production-05.company.internal:

Step 2: Test the Initial SSH Connection #

Before running complex playbooks, make sure Ansible can communicate with the new node using the built-in ping module:

# Verify connectivity to the newly added host
ansible -i inventory/production/ hosts.yml web-production-05.company.internal -m ping

Step 3: Execute the Playbook with a Limit (Dry Run) #

We must not run the whole playbook for all servers if we only want to add one new server. Use the --limit and --check options to verify changes specifically on the new server:

# Run a limited dry-run on the new server
ansible-playbook -i inventory/production/ site.yml \
  --limit web-production-05.company.internal \
  --check --diff

Step 4: Apply the Configuration (Real Run) #

If the dry-run result shows the change status matching our expectations, remove the --check option to apply the real configuration:

# Apply the configuration to the new server
ansible-playbook -i inventory/production/ site.yml \
  --limit web-production-05.company.internal

Runbook 2: Rotating Ansible Vault Encryption Credentials #

Infrastructure security requires us to periodically rotate Vault Password files. Follow this guide to rotate without breaking the availability of secret data in our repository:

Step 1: Prepare a New Vault Password #

Create a new password file outside our Git repository working directory. For example, store it in a secure folder on our local computer:

# Generate a new random password and store it in a temporary secure location
openssl rand -base64 32 > ~/.ansible_vault_key_new

Step 2: Reconstruct the Encryption with the New Password #

Ansible provides the built-in ansible-vault rekey utility to change a file’s encryption password without first decrypting it to plaintext. Run this command for our secret variable files:

# Rekey the secret file by providing the old password file and pointing to the new password file
ansible-vault rekey \
  --vault-password-file ~/.ansible_vault_key_old \
  --new-vault-password-file ~/.ansible_vault_key_new \
  inventory/production/group_vars/all/vault.yml

Step 3: Update the Vault Password Configuration in CI/CD Systems #

Don’t forget to update the ANSIBLE_VAULT_PASSWORD environment variable value in Jenkins, GitHub Actions, or our GitLab CI/CD with the new password we just created. After success, delete the temporary password file on our local laptop:

# Delete the temporary password file for security
rm ~/.ansible_vault_key_new

Runbook 3: Emergency Rollback Using Special Tags #

When a production configuration change causes unexpected obstacles, we need a fast response to restore the system to a stable state. Instead of debugging on the production server while the system is down, it’s safer to roll back the configuration first.

We can design our playbook to support emergency recovery tags (rollback) on crucial configuration tasks:

# playbooks/roles/nginx/tasks/main.yml
---
- name: Copy the main Nginx configuration file
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    backup: yes  # Saves the old file with a timestamp (e.g. nginx.conf.XXXX~)
  notify: Reload nginx
  tags: nginx_config

- name: Restore the emergency backup configuration (Rollback)
  shell: |
    latest_backup=$(ls -t /etc/nginx/nginx.conf.*~ 2>/dev/null | head -n 1)
    if [ -n "$latest_backup" ]; then
      cp "$latest_backup" /etc/nginx/nginx.conf
      nginx -t && systemctl reload nginx
    else
      echo "No backup file found!" && exit 1
    fi    
  when: trigger_rollback | default(false) | bool
  tags: rollback_nginx

To execute this emergency rollback on the problematic server, run the following command:

# Force the Nginx rollback task execution on the production web server group
ansible-playbook -i inventory/production/ site.yml \
  --limit webservers \
  --tags rollback_nginx \
  -e "trigger_rollback=true"

By providing instant recovery strategies like this, we drastically reduce service downtime during unexpected incidents.

Building a Sustainable Infrastructure as Code Culture #

Tools and technology are just supporting instruments; the real success of infrastructure management lies in the culture of the team operating it. To keep our Ansible codebase healthy and relevant in the long term, we must cultivate the following good habits in our team:

1. The Boy Scout Rule for Infrastructure #

Get used to always leaving code in a better state than when we first found it. If we’re reading a playbook to add a new task and find an old task without a descriptive name or using outdated syntax, take 5 minutes to fix it in our Pull Request. These consistent small changes prevent technical debt accumulation later.

2. Write ADRs (Architecture Decision Records) #

Every time the team makes a major architectural decision — like switching from one database module to another, changing the main directory structure, or setting new variable naming standards — document that decision in a simple markdown file in the docs/adr/ folder. This documentation becomes a valuable history guide for future team members to understand why our infrastructure was built in a certain way.

3. Peer-Programming Sessions and Task Rotation #

Don’t let one person monopolize knowledge about a particular role or playbook (e.g. only A knows how to manage the database cluster). Do periodic peer-programming sessions and rotate maintenance ticket handling responsibilities. The more evenly team members understand the infrastructure codebase, the more resilient our team is in emergency situations.

Summary #

• Simple Git Branching — Use a GitHub Flow-based branching strategy (short feature branches merged directly to main via Pull Requests) to avoid complicated merge conflicts.
• Mandatory PR Template — Apply a comprehensive Pull Request template in the repository to force risk analysis disclosure, testing methods, and clear rollback plans before code is reviewed.
• Objective Code Review — Focus the code review process on the pillars of technical correctness, idempotency, security compliance, and maintainability. Avoid manually debating cosmetic writing styles in PRs; leave that to automatic linters.
• Guided Onboarding — Provide isolated local setup guides using Python virtual environments, pre-commit hooks, and standardized requirements.txt so new team members can contribute safely from the first week.
• Ready-to-Use Runbooks — Document crucial operational scenarios like adding new servers, rotating Vault files, and handling emergency recovery in CLI step forms executable by whoever is on duty.
• Foster a Sustainable Culture — Apply the Boy Scout Rule to infrastructure code, write Architecture Decision Records (ADR) files for every major decision, and periodically rotate knowledge among team members.

← Previous: Testing Strategy Next: Production Readiness →