Service #

Installing software packages on a server is only half of the infrastructure setup process. After the application is installed, you must make sure the service is running, configured correctly, and can recover after a server restart. On modern Linux operating systems, systemd has become the de facto standard for the init system and service management. Ansible simplifies this management through the ansible.builtin.systemd and ansible.builtin.service modules. With a deep understanding of the service lifecycle in systemd, you can keep your application uptime maintained without performing unnecessary restart actions.

Managing the Service Lifecycle: The Difference Between the service and systemd Modules #

In Ansible, you’ll find two main modules that appear to have the same function: ansible.builtin.service and ansible.builtin.systemd. Understanding the difference between them is very important for determining which module best fits a given scenario.

The service module is a high-level built-in module that’s generic. It’s designed for cross-operating-system compatibility. Behind the scenes, it detects which init system is running on the target server (whether old SysVinit, Upstart from old Ubuntu versions, systemd, or OpenRC on Alpine) and translates your instructions to the appropriate tool. This module is ideal if you’re creating generic playbooks that must work on very old systems or alternative systems like Alpine Linux.

Meanwhile, the systemd module is a native module designed exclusively to communicate directly with the systemd system manager through the systemctl interface. Because it focuses on one init system, the systemd module provides access to advanced parameters that the generic service module doesn’t have. Some of those important parameters include daemon_reload, masked, scope, user, and no_block.

Because nearly all modern Linux server distributions today (like Ubuntu 16.04+, Debian 8+, RHEL 7+, Rocky Linux, and AlmaLinux) use systemd as their default standard, it’s highly recommended to use the systemd module directly for richer, more precise control features.

flowchart TD
    A["New Service Installed"] --> B{"Desired Action?"}
    B -- "Run the Service" --> C["state: started"]
    B -- "Enable at Boot" --> D["enabled: true"]
    B -- "Stop the Service" --> E["state: stopped"]
    B -- "Disable at Boot" --> F["enabled: false"]
    B -- "Totally Disable (Block)" --> G["masked: true"]
    C --> H["Service Active (Running)"]
    D --> I["Service Active After Reboot"]
    E --> J["Service Inactive"]
    F --> K["Service Won't Start After Reboot"]
    G --> L["Service Locked (Masked) - Cannot Be Started"]

Essential systemd Module Parameters #

To control the service state, the systemd module provides several important options on the state and enabled parameters. Here’s an in-depth explanation of those options:

• state: started: Ensures the service is running. If the service is down, Ansible starts it. If the service is already running, Ansible takes no action (idempotent).
• state: stopped: Ensures the service is down. If the service is running, Ansible stops it.
• state: restarted: Forces the service to stop first, then run again. This action is not idempotent because Ansible always restarts every time the playbook runs.
• state: reloaded: Sends a signal (usually SIGHUP) to the service process to re-read the configuration file in memory without killing the main process. Like restarted, this option is also not idempotent and runs every time it’s called.
• enabled: true: Ensures the service is registered in systemd to run automatically when the server boots.
• enabled: false: Ensures the service won’t start automatically when the server boots.
• masked: true: This is a special status that totally locks the service by creating a symbolic link (symlink) from the service unit file to /dev/null. A masked service cannot be started, either manually via systemctl start or automatically when triggered by other service dependencies. This is very useful if you want to make sure risky built-in services (like telnet or conflicting built-in firewall services) are permanently disabled.

Let’s look at the implementation of these parameters in a playbook:

# Ensure the postgresql database service is running and enabled at boot
- name: Run and enable PostgreSQL
  systemd:
    name: postgresql
    state: started
    enabled: true

# Stop and disable a service that risks conflicting with Nginx
- name: Stop and disable Apache2
  systemd:
    name: apache2
    state: stopped
    enabled: false

# Totally block a service so other processes can't start it
- name: Mask the snapd daemon service
  systemd:
    name: snapd
    masked: true
    state: stopped

The Significant Difference: restarted vs reloaded #

One common mistake in system administration with Ansible is overusing state: restarted to apply configuration changes. You must understand the consequences of both actions on service availability in production environments.

The restarted action works by completely stopping the application process (SIGTERM/SIGKILL), releasing network sockets, clearing memory, then starting the process from scratch. During this stop-and-start process (which can take from a few milliseconds to several seconds), your server can’t accept new connections and active connections get cut off. This creates a downtime gap for your application users.

The reloaded action works much more elegantly. Instead of killing the process, systemd sends a configuration reload signal to the running process. The main application process re-reads the configuration file from disk, validates it, loads the new configuration into memory, and gradually stops old workers (graceful shutdown) after new workers are ready to accept traffic. Active connections keep being served without interruption, so you get a zero-downtime configuration reload.

Let’s look at the comparison scenario of when to use restarted and reloaded:

# CORRECT: Using reloaded for web servers serving active user traffic
- name: Reload the Nginx configuration without downtime
  systemd:
    name: nginx
    state: reloaded

# CORRECT: Using restarted for language runtimes that don't support hot reload
- name: Restart the Python Gunicorn application (requires a full restart)
  systemd:
    name: myapp-gunicorn
    state: restarted

As a rule of thumb, if the application supports hot configuration reload (like Nginx, HAProxy, PostgreSQL, Apache, and SSH), use reloaded. If the application doesn’t support that feature (like Node.js, Go binaries, or Java Spring Boot), only then are you forced to use restarted.

Automating Daemon Reload after Unit File Changes #

In modern environments, you don’t only manage built-in operating system services — you also often deploy your own custom applications. To run a custom application as a service, you need to write a service description file called a systemd unit file (usually stored with a .service extension in the /etc/systemd/system/ directory).

Every time you create a new unit file or modify an existing one (for example changing memory limits, changing the executable path, or adding new environment variables), systemd doesn’t automatically know about the change. If you directly try to run the service, systemd triggers a warning that the configuration file on disk differs from the configuration loaded in the systemd daemon’s memory.

To tell systemd that a unit file changed, you must run the daemon reload operation (equivalent to systemctl daemon-reload). In Ansible, this operation is safely integrated using the daemon_reload: true parameter inside the systemd module.

Let’s look at the difference between the inefficient anti-pattern and the correct declarative solution:

# ANTI-PATTERN: Running daemon-reload using a shell command on every Ansible run
- name: Reload the systemd daemon manually
  shell: systemctl daemon-reload
  changed_when: true

# CORRECT: Using daemon_reload inside a handler triggered by unit file changes
- name: Deploy the custom service unit file
  template:
    src: templates/myapp.service.j2
    dest: /etc/systemd/system/myapp.service
    owner: root
    group: root
    mode: '0644'
  notify: Reload daemon and restart service

# Define the handler at the bottom of our playbook
handlers:
  - name: Reload daemon and restart service
    systemd:
      name: myapp
      state: restarted
      daemon_reload: true
      enabled: true

In the correct approach, the daemon_reload: true operation only runs if the myapp.service file actually changes. If the template file on the control node matches the file on the managed node, the template task reports an ok status (not changed), the handler isn’t triggered, and you avoid wasting server resources reloading the systemd daemon.

Coordinating Configuration Changes with Handlers #

Handlers are one of the most important features in Ansible for keeping target servers stable. Handlers are basically regular tasks, but they don’t execute unless explicitly triggered by another task using the notify instruction.

Here are the important rules about Handler behavior you must understand:

1. Execution at the End of the Play: By default, all triggered (notified) handlers during the play are collected and run together at the end of the play, right after all main tasks finish. This prevents services from restarting multiple times mid-way if many configuration file changes trigger them.
2. Only Runs on Changed Status: Handlers only run if the task calling them returns a changed status. If the task returns ok (because the server configuration already matches the desired state), the handler doesn’t run. This is very important for guaranteeing idempotency.
3. Single Execution: If five different tasks modify the Nginx configuration and all call the same Restart Nginx handler, Ansible only runs that handler once at the end of the play. You avoid repeated downtime from unnecessary restarts.
4. Cancellation on Error: If the playbook fails mid-way before reaching the end of the play, all already-registered handlers won’t execute. This is a safety measure preventing services from starting back up with half-finished or broken configurations.

Let’s look at a complex coordination implementation example:

- name: Configure the Nginx web server
  hosts: webservers
  become: true
  
  tasks:
    - name: Deploy the main nginx.conf configuration file
      template:
        src: templates/nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify: Reload Nginx

    - name: Deploy the application vhost configuration
      template:
        src: templates/vhost.conf.j2
        dest: /etc/nginx/sites-available/myapp.conf
      notify: Reload Nginx

    - name: Create a symlink to enable the vhost
      file:
        src: /etc/nginx/sites-available/myapp.conf
        dest: /etc/nginx/sites-enabled/myapp.conf
        state: link
      notify: Reload Nginx

  handlers:
    - name: Reload Nginx
      systemd:
        name: nginx
        state: reloaded

In the scenario above, even though three different tasks can trigger Reload Nginx, Nginx only receives the reload signal once at the end of the play after all main configuration, vhost, and symlink files are perfectly installed.

Waiting for Service Readiness (Service Health Verification) #

When you instruct Ansible to start a service using state: started, Ansible only detects whether the service’s main process was successfully triggered by systemd. Ansible doesn’t know whether the application inside the service has finished memory initialization, loaded internal configuration files, or connected itself to the database.

For example, when a Java Spring Boot or PostgreSQL service starts, the systemd process immediately reports active (running) status. However, the application might need 10 to 30 seconds to truly be ready to listen on network ports and accept traffic. If your playbook immediately continues to the next task (like running database migrations or API endpoint tests) without waiting for the application to be truly ready, those next tasks are guaranteed to fail due to connection refusal.

To bridge this initialization gap, you must use health verification modules like ansible.builtin.wait_for or ansible.builtin.uri.

# Scenario 1: Waiting for the database port to open before running migrations
- name: Restart PostgreSQL
  systemd:
    name: postgresql
    state: restarted

- name: Wait until PostgreSQL is ready to accept connections on port 5432
  wait_for:
    port: 5432
    host: 127.0.0.1
    delay: 3        # Wait 3 seconds first before starting the first check
    timeout: 45     # Limit the maximum wait to 45 seconds, abort if exceeded
    state: started  # Make sure the port is "open"

- name: Run our application database migration
  command: /opt/myapp/venv/bin/python manage.py migrate

# Scenario 2: Polling the application HTTP Health Endpoint
- name: Restart the API backend service
  systemd:
    name: myapp-backend
    state: restarted

- name: Poll the healthcheck endpoint until it returns HTTP 200
  uri:
    url: "http://127.0.0.1:8080/api/health"
    status_code: 200
  register: api_health
  until: api_health.status == 200
  retries: 15      # Try a maximum of 15 times
  delay: 2         # The pause between attempts is 2 seconds

By applying these verification steps, your playbook becomes much more robust because it ensures the target server state is truly functionally ready before continuing to the next configuration step.

Checking Service Status Idempotently #

In some cases, you only want to check whether a service is running or not, without changing its state (for example for monitoring or reporting needs). To do this safely without triggering a task status change in Ansible, you must register the systemd check output and include the changed_when: false parameter.

- name: Check the Docker service active status
  systemd:
    name: docker
  register: docker_service_info
  changed_when: false  # This task changes nothing, make sure it's always green (ok)

- name: Display Docker's internal status
  debug:
    msg: >
      Docker ActiveState: {{ docker_service_info.status.ActiveState }}
      Docker LoadState: {{ docker_service_info.status.LoadState }}
      Docker SubState: {{ docker_service_info.status.SubState }}      

- name: Take conditional action if Docker is down
  debug:
    msg: "WARNING: The Docker service is inactive on this host!"
  when: docker_service_info.status.ActiveState != "active"

The status property returned by the systemd module contains a complete representation of the systemd D-Bus object for that service, giving you very deep information without having to parse raw text output from shell commands.

Case Study: Deploying a Custom Node.js Service with systemd #

Let’s combine all your understanding into a real scenario: deploying a custom Node.js application as a systemd service, ensuring it runs with a safe restart policy, configuring boot auto-start, triggering a daemon reload if the unit file changes, and verifying port health before ending the deployment process.

Here’s the dynamic service file template (templates/nodejs-app.service.j2):

{# templates/nodejs-app.service.j2 #}
[Unit]
Description=Node.js Backend Service {{ app_name }}
After=network.target mongodb.service
Requires=mongodb.service

[Service]
Type=simple
User={{ app_user }}
Group={{ app_group }}
WorkingDirectory={{ app_dir }}
Environment=NODE_ENV={{ app_env }}
Environment=PORT={{ app_port }}
ExecStart=/usr/bin/node {{ app_dir }}/server.js
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodejs-{{ app_name }}

[Install]
WantedBy=multi-user.target

And here’s the complete playbook automating the service deployment:

# playbooks/deploy-nodejs-service.yml
---
- name: Deploy the Node.js Service Independently
  hosts: appservers
  become: true
  
  vars:
    app_name: "payment-api"
    app_user: "nodeapp"
    app_group: "nodeapp"
    app_dir: "/var/www/payment-api"
    app_env: "production"
    app_port: 3000

  tasks:
    - name: Ensure the Node.js application user and group exist
      block:
        - name: Create the application group
          group:
            name: "{{ app_group }}"
            state: present

        - name: Create the non-interactive application user
          user:
            name: "{{ app_user }}"
            group: "{{ app_group }}"
            shell: /usr/sbin/nologin
            create_home: false
            state: present

    - name: Deploy the systemd unit file for the Node.js application
      template:
        src: templates/nodejs-app.service.j2
        dest: "/etc/systemd/system/{{ app_name }}.service"
        owner: root
        group: root
        mode: '0644'
      notify: Reload the systemd daemon and restart the nodejs app

    - name: Ensure the service is enabled at boot and started
      systemd:
        name: "{{ app_name }}"
        state: started
        enabled: true

    # Force Ansible to run pending handlers right now
    # so we can immediately verify the port after the restart happens
    - name: Apply pending configuration changes (flush handlers)
      meta: flush_handlers

    - name: Verify the Node.js port readiness before ending the playbook
      wait_for:
        port: "{{ app_port }}"
        host: 127.0.0.1
        delay: 2
        timeout: 20
        state: started

  handlers:
    - name: Reload the systemd daemon and restart the nodejs app
      systemd:
        name: "{{ app_name }}"
        state: restarted
        daemon_reload: true
        enabled: true

In this case study, we use the meta: flush_handlers feature. This feature is very important because by default handlers only execute at the end of the playbook. By calling flush_handlers, we force Ansible to execute the daemon reload and service restart process right away. After that, the wait_for task below can safely and accurately verify the port of the freshly restarted application.

Summary #

• Native systemd Module — Use the systemd module instead of the generic service module on modern distros for full control over the target service lifecycle.
• Reload Advantage — Always prioritize the state: reloaded option over restarted for services that support it to get configuration updates without downtime.
• Repeated Restart Danger — Remember that the restarted and reloaded statuses aren’t idempotent; use handlers so these actions only trigger once when there’s a real change.
• Service Masking — Use the masked: true option to permanently disable services so other processes can’t start them for security reasons.
• Daemon Reload Automation — Make sure the daemon_reload: true parameter is included in systemd handler tasks every time you deploy or change systemd unit files (.service).
• Port Readiness Verification — Always use the wait_for module after starting database or API services to ensure the network port is truly open before other tasks run.
• HTTP Health Endpoint Polling — Combine the uri module with the until parameter to check web server health before ending the deployment playbook.
• Safe Status Inspection — Use changed_when: false and register the check output status (register) to read service conditions without triggering playbook status changes.

← Previous: Package Next: Copy Module →