Keeping Linux up to date with Aria Automation Config

I have several persistent Windows & Linux VMs in my lab. The Windows VMs get their OS updates via Windows Server Update Service (WSUS) managed by group policy. This works pretty consistently and keeps everything current. The Linux VMs are a mix of Photon OS 4 and 5 as well as Ubuntu 20.04, and every time I ssh in I see a notification that updates are available. If I have a few minutes I’ll usually take the opportunity to get current… but these VMs could run for weeks without an interactive ssh login, leaving some security risk on the table.

In my lab I have an Aria Automation Config (formerly known as Salt Stack Config) appliance that I’ll use occasionally. It has functionality to run scheduled jobs on managed minions, but I’ve not taken time to set that up — until today.

Inventory

The first step I wanted to tackle was an inventory of patches that are required for the various endpoints. Looking around at available out-of-box functions in Aria Automation Config, I found a command pkg.list_upgrades that looked promising. I ran it against all minions and then looked at the resulting out. The raw return is available as JSON, so I imported it into Powershell (Get-Content timestamp-return.json | ConvertFrom-Json) and started looking at the details. There is an item for each minion and that item contains a return note property which looks similar to this:

$json[0].return |fl


libpq5          : 11.22-0+deb10u1
python-urllib3  : 1.24.1-1+deb10u2
python3-urllib3 : 1.24.1-1+deb10u2

For my purposes I’m mostly interested in the count of the return items… the number of packages that need to be updated. To find these counts, I came up with the following oneliner:

$json | Select-Object minion_id, has_errors, @{N='NumUpdates';E={($_.return | Get-Member -Type NoteProperty | Measure-Object).Count}} | ?{$_.NumUpdates -ne 0} |Sort-Object minion_id

minion_id                              has_errors NumUpdates
---------                              ---------- ----------
dr-extdns-21.lab.enterpriseadmins.org       False          8
h135-linux-01.lab.enterpriseadmins.org      False          7
h135-linux-02.lab.enterpriseadmins.org      False        105
net-cont-21.lab.enterpriseadmins.org        False         75
net-wanrtr-01.lab.enterpriseadmins.org      False         46
raspberrypi                                 False          3
saltmaster                                  False         34
svcs-cont-21.lab.enterpriseadmins.org       False         61

Looking at this list I can see that my updates are all over the place, some are fairly current, others are way off.

Apply the Updates

To apply updates to Linux, I first created a small state file called updates/os.sls and entered the following contents:

update_pkg:
  pkg.uptodate:
    - refresh: True

I ran applied this state to an Ubuntu box, and once complete logged in to check the system to make sure it was fully patched. Running an apt list upgradable I found some missing updates. I re-applied the state file, hoping that another round of updates would get me current, but the same updates were still missing. After doing some searching online, I found that I needed to include an extra line in my state file to include dist_upgrade so version 2 looked like this:

update_pkg:
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True

Applying this revised state to the Ubuntu VM got it all matched up with the apt list upgradable results. I then expanded this task to a few more VMs, but ran into a problem when applying the revised version 2 of the state to a Photon OS VM. The results of the Photon VMs showed an error message of: No such option: --dist_upgrade , which is the additional line I had just added to get better patching coverage.

I could have created two different state files, jobs, and associated schedules — one to target Ubuntu and another for Photon, using both of the above versions of the state file, but I wanted to have a more generic / shared update process for all Linux systems. Instead of creating duplicate states/jobs, I decided to add some if logic to my state file. The out of the box sse/apache/init.sls showed a perfect example of how to accomplish this. For the final version of my state file, I look to see if the OS is Photon, and if so apply the pkg.uptodate without the dist_upgrade flag, otherwise I do include it. The syntax looks like:

update_pkg:
{% if grains['os'] == 'VMware Photon OS' %}
  pkg.uptodate:
    - refresh: True
{% else %}
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True
{% endif %}

Now when I apply this state to a Linux VM, it updates both system types without error. The sample apache state also has some else if examples if we need to get more specific in the future… for example, pulling in Windows patching with the same state. For now, the single if/else works just fine.

Report on the Updates

The output from the above job which applies the updates/os.sls state returns a JSON body similar to our first reporting task. I tried importing this JSON into Powershell to see what sort of interesting reporting was available. One set of data that we can see is the list of packages which were updated, including the old & new version numbers. Here is an example from one minion:

$json[0].return.'pkg_|-update_pkg_|-update_pkg_|-uptodate'.changes


apt                          : @{new=2.0.10; old=2.0.6}
ufw                          : @{new=0.36-6ubuntu1.1; old=0.36-6}
bolt                         : @{new=0.9.1-2~ubuntu20.04.2; old=0.8-4ubuntu1}
<list truncated for readability>

For comparison to the initial report of which updates were needed, I also wanted to summarize what was updated to see if my counts were similar. The following output is from an initial test run on a subset of minions:

$json | Select-Object Minion_id, has_errors, @{N='NumPkgs';E={ ($_.full_ret.return.'pkg_|-update_pkg_|-update_pkg_|-uptodate'.changes| Get-Member -Type NoteProperty | Measure-object).Count }} | Sort-Object minion_id

minion_id                              has_errors NumPkgs
---------                              ---------- -------
dr-extdns-21.lab.enterpriseadmins.org       False       8
h135-linux-01.lab.enterpriseadmins.org      False       7
h135-linux-02.lab.enterpriseadmins.org      False     105
net-wanrtr-01.lab.enterpriseadmins.org      False      46

Comparing the above output to the initial report we can see that all available updates were applied.

I’ve since created a schedule to run each of these jobs on a regular basis. For now, the inventory job will run daily so that I can see/track progress, and the update job will run every weekend. Hopefully the next time I ssh into a Linux VM I won’t be presented with a laundry list of required updates.