Keeping Linux up to date with Aria Automation Config — part 2

In a recent post (available here), we created a simple Aria Automation Config (formerly SaltStack Config) state file which reported on and applied available Linux OS updates. In this post we’ll revisit a minor change to this state file.

After creating the previous state which applies available updates every Saturday morning, I noticed sometimes logging into Linux VMs would return a message of *** System restart required ***. I found that this text was coming from the file /var/run/reboot-required which was created when a package required a system restart.

I’ve modified the state file applied by my scheduled job to accommodate this reboot as shown below:

update_pkg:
{% if grains['os'] == 'VMware Photon OS' %}
  pkg.uptodate:
    - refresh: True
{% else %}
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True
{% endif %}

{# Check if the system requires a reboot, and if so schedule it to happen in the next 15 minutes, randomize to prevent boot storm #}
{%- if salt['file.file_exists']('/var/run/reboot-required') %}
Reboot-if-needed:
  module.run:
    - name: system.reboot
    - tgt: {{ grains.id }}
    - at_time: {{ range(1,15) | random }}
{%- endif %}

In this version, we continue to use pkg.uptodate to apply updates, but after doing so we check for the presence of /var/run/reboot-required. If found, we schedule a system reboot to happen at least one minute in the future (to give the salt-minion time to report back). In this case we are randomizing the time of these reboots to minimize a boot storm, with a maximum future time of 15 minutes.