Scheduled Task #

Automating infrastructure isn’t just about reactively triggering changes when there’s new code or an emergency problem. Most daily operations tasks are routine and periodic: backing up databases every night, applying security patches every weekend, cleaning temporary storage directories, and regularly auditing configuration compliance. All these are scheduled tasks that run automatically on a schedule without manual intervention. However, running automated scripts without human supervision demands a much higher level of resilience. We must ensure tasks don’t run overlapping, failures are detected instantly, and execution logs are neatly recorded for compliance audit purposes.

Scheduling Systems: Cron vs Systemd Timer #

When we want to schedule Ansible executions on the control node, we have two main mechanism choices on Linux operating systems: the traditional cron scheduler or the modern systemd timer scheduler. Although cron has been the industry standard for decades because of its simplicity, systemd timer offers far superior control in large-scale production environments.

Here’s the logical flow diagram of safe scheduled task execution with overlap detection and failure notification mechanisms:

flowchart TD
      A["Schedule Triggered (Cron/Timer)"] --> B["Check Lock File Availability (/var/run/job.lock)"]
      B --> C{"Does the Lock File Exist?"}
      C -- "Yes (Overlapping)" --> D["Stop Execution & Send Duplication Alert"]
      C -- "No" --> E["Create Lock File (Touch)"]
      E --> F["Run Pre-flight Assertions (e.g. Disk Space)"]
      F --> G{"Do Assertions Pass?"}
      G -- "No" --> H["Send Failure Alert & Remove Lock File"]
      G -- "Yes" --> I["Execute the Main Task (Backup/Patch)"]
      I --> J{"Was It Successful?"}
      J -- "No" --> H
      J -- "Yes" --> K["Remove Lock File (Cleanup)"]
      K --> L["Send Success Report & Done"]

Scheduler Characteristic Comparison #

Let’s compare both options in depth to help us design the right automation architecture:

Configuring Scheduling with Ansible #

Here’s an example of how we use Ansible to configure both scheduler types on target machines.

Implementation with the Cron Module #

- name: Setup Daily Cron Job for Database Backup
  hosts: control_node
  become: true
  tasks:
    - name: Schedule the database backup execution every day at 02:00 AM
      ansible.builtin.cron:
        name: "Production Database Backup"
        minute: "0"
        hour: "2"
        job: "/usr/bin/ansible-playbook -i /opt/inventory/ /opt/playbooks/backup.yml >> /var/log/ansible-backup.log 2>&1"
        user: "ansible"
        state: present

Implementation with Systemd Timer #

To schedule with systemd timer, first we create the service unit file (/etc/systemd/system/ansible-backup.service):

[Unit]
Description=Run the Ansible Database Backup Playbook
After=network.target

[Service]
Type=oneshot
User=ansible
ExecStart=/usr/bin/ansible-playbook -i /opt/inventory/ /opt/playbooks/backup.yml
StandardOutput=journal
StandardError=journal

Second, we create the timer unit file (/etc/systemd/system/ansible-backup.timer):

[Unit]
Description=Weekly Schedule for Ansible Backup

[Timer]
OnCalendar=Sun *-*-* 03:00:00
Persistent=true
Unit=ansible-backup.service

[Install]
WantedBy=timers.target

We can automate the deployment of both files and enable the timer using the ansible.builtin.systemd module in our controller configuration playbook.

Concurrency Handling and Lock File Protection #

The most frequent problem with scheduled automation tasks is overlapping concurrency (job overlap). Imagine scheduling a data synchronization or disk backup task every 30 minutes. One day, the network slows down drastically so the first task running at 08:00 hasn’t finished when the next schedule triggers at 08:30.

If we don’t protect our workflow, the second task starts running in parallel on the same server. Two processes trying to manipulate data or write to the same backup file simultaneously cause race conditions, CPU/RAM overload, and end in fatal data corruption.

Using the tempfile and file Modules for Lock File Protection #

We must apply a locking mechanism at the start of our playbook. Before executing the main task, the playbook checks for the existence of a lock file. If that file exists, the playbook stops safely immediately. If not, the playbook creates that file first, runs the main task, and removes it again after the entire process completes.

Here’s a defensive playbook implementation applying lock file protection:

# playbooks/backup-safe.yml
---
- name: Scheduled Backup with Lock File Protection
  hosts: db_servers
  become: true
  vars:
    lock_file_path: "/var/run/ansible-backup.lock"
    backup_dest: "/mnt/backup-nas"

  tasks:
    - name: Evaluate Lock File Existence
      block:
        - name: Check whether there's a lock file from the previous execution
          ansible.builtin.stat:
            path: "{{ lock_file_path }}"
          register: current_lock_status

        - name: Stop execution if the previous process is still active
          ansible.builtin.fail:
            msg: "Failure: Lock file {{ lock_file_path }} found! The previous process is still running."
          when: current_lock_status.stat.exists

        - name: Create a new lock file (Touch Lock File)
          ansible.builtin.file:
            path: "{{ lock_file_path }}"
            state: touch
            mode: '0600'
            owner: root
            group: root

        - name: Run Pre-flight Assertions (Verify Disk Space)
          ansible.builtin.setup:
            filter: "ansible_mounts"

        - name: Calculate the storage space availability
          set_fact:
            mount_point: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | first }}"

        - name: Ensure at least 10GB of free space
          ansible.builtin.assert:
            that:
              - "mount_point.size_available > 10737418240"
            fail_msg: "Insufficient storage space to hold the new backup!"

        - name: Execute the Main Backup Task
          ansible.builtin.command: "/opt/scripts/backup-db.sh"
          register: backup_execution_result
          changed_when: true

      always:
        - name: Clean Up the Lock File After Completion (Cleanup)
          ansible.builtin.file:
            path: "{{ lock_file_path }}"
            state: absent
          when: current_lock_status is defined and not current_lock_status.stat.exists
          # We only remove the lock file if we created it at the start.
          # If the playbook stops because it detected someone else's lock file, we must not remove it.

In the implementation above, the always block inside Ansible’s error handling structure guarantees the lock file is cleanly removed even if the backup process crashes midway. This prevents our cluster from staying in a permanently locked state (deadlock).

Enterprise-Level Orchestration with AWX and Ansible Tower #

Although Linux CLI-based scheduling (cron and systemd) is very reliable for small-scale infrastructure, we encounter limitations when managing thousands of servers with large operations teams. Some of the limitations are:

• No graphical interface (UI) to quickly see task success status.
• Credentials (like SSH keys and database passwords) must be physically stored on the control node.
• Hard to divide access roles (who can edit schedules vs who can only view).

To answer enterprise-class needs, we must use AWX (the open-source version) or Red Hat Ansible Automation Platform (AAP / formerly Ansible Tower).

AWX acts as a centralized control plane providing:

1. Centralized Logging: All playbook outputs are recorded in a centralized database and can be sent to external systems like Elasticsearch or Splunk.
2. Credential Vault: AWX stores SSH keys, cloud tokens, and Vault passwords encrypted. AWX runners inject these credentials into container memory during execution and remove them immediately after completion.
3. Enterprise Scheduler: A user-friendly web interface for arranging execution schedules (cron-like expressions) complete with national holiday calendars to delay automatic executions.
4. Drift Detection: Periodically runs playbooks with the --check (dry-run) option. If there are manual configuration changes on target servers, AWX detects those differences (drift) and sends alerts or automatically reconciles back to the original state.

Designing Defensive Playbooks for Routine Tasks #

Routine tasks like security updates (patch management) demand extra defensive playbook design. We must not update all servers randomly and simultaneously because if the update package has compatibility issues, all our services die totally at the same time.

Rolling Patching Orchestration #

We must leverage Ansible’s serial parameter to arrange gradual updates. For example, by setting serial: "25%", Ansible only updates 25% of the total inventory servers at one time. If the update on the first group fails, Ansible stops the entire playbook, limiting the damage radius to only 25% of our infrastructure, while the remaining 75% stays safe running and serving users.

Here’s the weekly defensive patch management playbook:

# playbooks/weekly-patching.yml
---
- name: Weekly Security Update Application in Rolling Fashion
  hosts: web_servers
  become: true
  serial: "25%" # Process 25% of servers per group (Rolling update)
  vars:
    reboot_timeout_seconds: 300

  pre_tasks:
    - name: Record the current kernel version before patching
      ansible.builtin.command: uname -r
      register: kernel_before
      changed_when: false

  tasks:
    - name: Apply updates only for security packages (Debian/Ubuntu)
      ansible.builtin.apt:
        upgrade: dist
        update_cache: true
        only_upgrade: true
      when: ansible_os_family == "Debian"
      register: apt_update_result

    - name: Check whether the operating system requires a reboot
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_status

    - name: Reboot the Server If Necessary
      ansible.builtin.reboot:
        msg: "Automatic reboot by Ansible to complete the kernel patch installation."
        reboot_timeout: "{{ reboot_timeout_seconds }}"
        post_reboot_delay: 15
      when: reboot_status.stat.exists

    - name: Verify the application port availability post-reboot
      ansible.builtin.wait_for:
        port: 80
        delay: 5
        timeout: 60
        state: started

  post_tasks:
    - name: Get the kernel version after patching
      ansible.builtin.command: uname -r
      register: kernel_after
      changed_when: false

    - name: Report the kernel version transition
      ansible.builtin.debug:
        msg: "Server {{ inventory_hostname }} successfully updated. Kernel: {{ kernel_before.stdout }} -> {{ kernel_after.stdout }}"
      when: kernel_before.stdout != kernel_after.stdout

Notifications and Monitoring of Scheduled Task Failures #

One of the biggest dangers of unattended automatic scheduling tasks is silent failure. We feel safe because we’ve created an automatic backup script every night. However, after six months of running, when our server experiences disk damage and we need that backup data, we just realize the backup script has been failing since the first month because it ran out of disk space.

We must configure automatic reporting mechanisms to team communication platforms like Slack or Microsoft Teams every time a scheduled task execution fails.

Here’s an example Ansible playbook sending instant notifications to a Slack webhook if it detects a failure in the backup process:

# playbooks/backup-with-alert.yml
---
- name: Scheduled Database Backup with Slack Notifications
  hosts: db_servers
  become: true
  vars:
    slack_webhook_url: "https://hooks.slack.com/services/T00/B00/X00" # Store in Ansible Vault for security
    backup_file_path: "/var/backups/db-latest.sql"

  tasks:
    - name: Main Backup Execution Flow
      block:
        - name: Execute the database dump
          ansible.builtin.command: "/usr/bin/mysqldump --all-databases > {{ backup_file_path }}"
          register: mysqldump_result
          changed_when: true

        - name: Validate the backup file size (Must be > 1MB)
          ansible.builtin.stat:
            path: "{{ backup_file_path }}"
          register: file_status

        - name: Ensure the backup file is valid and not empty
          ansible.builtin.assert:
            that:
              - "file_status.stat.size > 1048576"
            fail_msg: "The backup file is too small, dump data corruption detected!"

      rescue:
        - name: Send the failure notification to Slack
          delegate_to: localhost
          become: false
          ansible.builtin.uri:
            url: "{{ slack_webhook_url }}"
            method: POST
            body_format: json
            body:
              text: "🚨 *SCHEDULED BACKUP FAILURE* 🚨\n*Server:* {{ inventory_hostname }}\n*Time:* {{ ansible_date_time.iso8601 }}\n*Error Details:* {{ ansible_failed_result.msg | default('The database dump process experienced a runtime failure.') }}"
            status_code: 200
          ignore_errors: true

        - name: Propagate the failure so the pipeline records the failed status
          ansible.builtin.fail:
            msg: "The backup playbook was stopped due to an internal failure."

Through the error handling method above, we can ensure every problem that happens at night is immediately reported to the operations team’s on-call channel by morning, so handling can be done quickly before it has a bad impact.

Summary #

• Use Systemd Timer for Flexibility — Systemd timers offer integrated log recording via journald, automatic post-boot recovery, and far superior resource allocation management compared to traditional cron.
• Apply Lock File Protection — Always use a lock file at the start of your playbook to prevent overlapping double executions that trigger race conditions.
• Implement Block-Always for Cleanup — Wrap the execution flow inside a block-always structure so the lock file removal process still runs even if the main task fails.
• Limit Impact with Serial Patching — When scheduling system updates, use the serial: "25%" parameter to update servers gradually and limit the damage radius if update packages have problems.
• Avoid Silent Failure with Alerting — Install a rescue block that triggers Slack/Teams API calls to send instant notifications when detecting failures in scheduled scripts.
• Evaluate AWX/Tower Features — Consider migrating scheduling to centralized platforms like AWX for centralized logging needs, RBAC-based access management, and configuration compliance audits for large-scale clusters.

← Previous: CI/CD Integration Next: Error Handling →