Enterprise Admins.org

I have several persistent Windows & Linux VMs in my lab. The Windows VMs get their OS updates via Windows Server Update Service (WSUS) managed by group policy. This works pretty consistently and keeps everything current. The Linux VMs are a mix of Photon OS 4 and 5 as well as Ubuntu 20.04, and every time I ssh in I see a notification that updates are available. If I have a few minutes I’ll usually take the opportunity to get current… but these VMs could run for weeks without an interactive ssh login, leaving some security risk on the table.

In my lab I have an Aria Automation Config (formerly known as Salt Stack Config) appliance that I’ll use occasionally. It has functionality to run scheduled jobs on managed minions, but I’ve not taken time to set that up — until today.

Inventory

The first step I wanted to tackle was an inventory of patches that are required for the various endpoints. Looking around at available out-of-box functions in Aria Automation Config, I found a command pkg.list_upgrades that looked promising. I ran it against all minions and then looked at the resulting out. The raw return is available as JSON, so I imported it into Powershell (Get-Content timestamp-return.json | ConvertFrom-Json) and started looking at the details. There is an item for each minion and that item contains a return note property which looks similar to this:

$json[0].return |fl


libpq5          : 11.22-0+deb10u1
python-urllib3  : 1.24.1-1+deb10u2
python3-urllib3 : 1.24.1-1+deb10u2

For my purposes I’m mostly interested in the count of the return items… the number of packages that need to be updated. To find these counts, I came up with the following oneliner:

$json | Select-Object minion_id, has_errors, @{N='NumUpdates';E={($_.return | Get-Member -Type NoteProperty | Measure-Object).Count}} | ?{$_.NumUpdates -ne 0} |Sort-Object minion_id

minion_id                              has_errors NumUpdates
---------                              ---------- ----------
dr-extdns-21.lab.enterpriseadmins.org       False          8
h135-linux-01.lab.enterpriseadmins.org      False          7
h135-linux-02.lab.enterpriseadmins.org      False        105
net-cont-21.lab.enterpriseadmins.org        False         75
net-wanrtr-01.lab.enterpriseadmins.org      False         46
raspberrypi                                 False          3
saltmaster                                  False         34
svcs-cont-21.lab.enterpriseadmins.org       False         61

Looking at this list I can see that my updates are all over the place, some are fairly current, others are way off.

Apply the Updates

To apply updates to Linux, I first created a small state file called updates/os.sls and entered the following contents:

update_pkg:
  pkg.uptodate:
    - refresh: True

I ran applied this state to an Ubuntu box, and once complete logged in to check the system to make sure it was fully patched. Running an apt list upgradable I found some missing updates. I re-applied the state file, hoping that another round of updates would get me current, but the same updates were still missing. After doing some searching online, I found that I needed to include an extra line in my state file to include dist_upgrade so version 2 looked like this:

update_pkg:
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True

Applying this revised state to the Ubuntu VM got it all matched up with the apt list upgradable results. I then expanded this task to a few more VMs, but ran into a problem when applying the revised version 2 of the state to a Photon OS VM. The results of the Photon VMs showed an error message of: No such option: --dist_upgrade , which is the additional line I had just added to get better patching coverage.

I could have created two different state files, jobs, and associated schedules — one to target Ubuntu and another for Photon, using both of the above versions of the state file, but I wanted to have a more generic / shared update process for all Linux systems. Instead of creating duplicate states/jobs, I decided to add some if logic to my state file. The out of the box sse/apache/init.sls showed a perfect example of how to accomplish this. For the final version of my state file, I look to see if the OS is Photon, and if so apply the pkg.uptodate without the dist_upgrade flag, otherwise I do include it. The syntax looks like:

update_pkg:
{% if grains['os'] == 'VMware Photon OS' %}
  pkg.uptodate:
    - refresh: True
{% else %}
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True
{% endif %}

Now when I apply this state to a Linux VM, it updates both system types without error. The sample apache state also has some else if examples if we need to get more specific in the future… for example, pulling in Windows patching with the same state. For now, the single if/else works just fine.

Report on the Updates

The output from the above job which applies the updates/os.sls state returns a JSON body similar to our first reporting task. I tried importing this JSON into Powershell to see what sort of interesting reporting was available. One set of data that we can see is the list of packages which were updated, including the old & new version numbers. Here is an example from one minion:

$json[0].return.'pkg_|-update_pkg_|-update_pkg_|-uptodate'.changes


apt                          : @{new=2.0.10; old=2.0.6}
ufw                          : @{new=0.36-6ubuntu1.1; old=0.36-6}
bolt                         : @{new=0.9.1-2~ubuntu20.04.2; old=0.8-4ubuntu1}
<list truncated for readability>

For comparison to the initial report of which updates were needed, I also wanted to summarize what was updated to see if my counts were similar. The following output is from an initial test run on a subset of minions:

$json | Select-Object Minion_id, has_errors, @{N='NumPkgs';E={ ($_.full_ret.return.'pkg_|-update_pkg_|-update_pkg_|-uptodate'.changes| Get-Member -Type NoteProperty | Measure-object).Count }} | Sort-Object minion_id

minion_id                              has_errors NumPkgs
---------                              ---------- -------
dr-extdns-21.lab.enterpriseadmins.org       False       8
h135-linux-01.lab.enterpriseadmins.org      False       7
h135-linux-02.lab.enterpriseadmins.org      False     105
net-wanrtr-01.lab.enterpriseadmins.org      False      46

Comparing the above output to the initial report we can see that all available updates were applied.

I’ve since created a schedule to run each of these jobs on a regular basis. For now, the inventory job will run daily so that I can see/track progress, and the update job will run every weekend. Hopefully the next time I ssh into a Linux VM I won’t be presented with a laundry list of required updates.

In my home lab I have a container running Nginx Proxy Manager (discussed in this previous post). This proxy allows for friendlier host names and SSL for various services in the lab. Using a wildcard DNS record and wildcard SSL certificates makes for a super easy way to onboard new services.

To get started, I first needed to pick a parent domain name to use for services. I already have a DNS zone for example.com so I decided to put these services under apps.example.com. To make this super easy to manage, I created a new domain under example.com with the name apps. It has a single CNAME record of asterisk (*) and the FQDN points to the host name of my container host. Screenshot of the DNS record from my Windows DNS server below:

I then created a wildcard certificate for *.apps.example.com from my internal CA. There are many ways to create a certificate signing request (CSR), but since I have a lot of Aria Suite products in the lab, I like using the Aria Suite Lifecycle > Locker > Certificates > Generate CSR button. This gives me a UI to populate the fields and kicks out a single file with both the CSR & private key. I use the CSR to generate a web server certificate from my internal CA, then download the base64 certificate. I edit the resulting .cer file and append my CAs public key to create a proper chain. Now that I have a certificate and private key, I can move into the Nginx Proxy Manager UI.

From the Nginx Proxy Manager UI, I select the SSL Certificates tab. I add a new SSL certificate and populate the required fields, screenshot below:

When I go to the Hosts > Proxy hosts tab I can now very easily add hosts with SSL capabilities. I no longer need to make a certificate for each service or even manually create DNS records. For example, lets say my internal IPAM solution needs a certificate. Instead of creating a ‘friendly’ DNS record and dedicated certificate, I can use this Nginx Proxy and wildcard certificate. We can simply add a new proxy host, enter a domain name such as ipam.apps.example.com, enter the correct host/port details, and select the correct certificate.

On the SSL tab of the new host we can pick our wildcard.apps.example.com certificate and select force SSL.

Now when I browse to http://ipam.apps.example.com/, I’m automatically redirected to the secure version of the site:

This does inject a new dependency — the Nginx Proxy Manager container needs to be running for me to reach these secure services — but in this case the container is running on a host that is typically online/working.