Which Linux Template?

I was recently showing someone my homelab and they noticed that I had templates for multiple Linux distributions including Ubuntu Server, Photon OS, and Tiny Core, as well as different versions of each. They asked why I didn’t standardize on just one Linux distribution or version. The version issue is easy — I can be lazy when it comes to cleaning up old templates. The reason for multiple distributions required a bit of extra explanation, so I figured I would type it up here.

Tiny Core is a very small Linux distribution that I can clone in seconds. If I want to test something like network connectivity, a new DHCP scope, scripting something against a handful of VMs, or something that deals with the disk of a VM like vSphere Replication or a VM image backup/restore Tiny Core is perfect… just enough of a VM to get the job done. Instructions on how to create such a template can be found in this previous post: https://enterpriseadmins.org/blog/lab-infrastructure/lightweight-vm-for-testing-tinycore-linux/.

Photon OS is a minimal Linux distribution optimized to run on VMware platforms. Its already available as an OVA here: https://github.com/vmware/photon/wiki/Downloading-Photon-OS, so I just imported it once and saved it as a template. It only takes a couple minutes to deploy one of these and configure it to run Docker containers. For an example of something you can use it for, check out this previous post https://enterpriseadmins.org/blog/scripting/vrealize-operations-alerts-using-rest-notification-plugin/.

Ubuntu Server is the largest Linux template I have on disk, but is still relatively small compared to a Windows template (my configured Ubuntu Server 18.04 template is ~5GB). Ubuntu is a consumer friendly distribution with current support for a lot of different packages. I have one of these VMs running PiHole and another used as a 2FA/RADIUS server that I use for Two Factor Authentication with Horizon UAG. Here is a previous post for building an Ubuntu 18.04 template: https://enterpriseadmins.org/blog/virtualization/lab-updates-ubuntu-18-04-template/

What Linux distribution do you like that I am missing?

Scripts to max out CPU and Memory

Most of the time we want our virtual machines to run as optimally as possible. We don’t want to see CPU contention or high memory conditions. However, on occasion we may want to have some stress to see what that looks like in monitoring tools like vRealize Operations. I created two small scripts that will run in TinyCore Linux, one consuming CPU and the other memory. For info on creating a TinyCore template, you may want to check out this post: https://enterpriseadmins.org/blog/lab-infrastructure/lightweight-vm-for-testing-tinycore-linux/. Here are the scripts for reference:


cpus=$(grep -ci processor /proc/cpuinfo)
echo "System has $cpus CPUs, starting thread for each."

for i in $( seq 1 $cpus )
  echo " ..starting background process #$i to consume CPU."
  sha1sum /dev/zero &

echo "You can check processes with 'top' sorting by CPU with 'P'."
echo "To end all processes run 'killall sha1sum'"

Note: this file is available at https://raw.githubusercontent.com/bwuch/code-snips/master/cpubusy.sh


echo "This script will fill memory to 90% with zeros."
# disable swap, only use RAM
sudo swapoff -a

# find current system memory
mem=$(sudo grep -i memtotal /proc/meminfo | awk '{print $2}')

# calculate 90% of current memory, using 100% will cause instability
fillmem=$(expr $mem / 100 \* 90)

# tmpfs is mounted as 50% by default, remount with our 90% number
echo " ..remounting /dev/shm to use 90% of MemTotal"
sudo mount -o remount,size=$fillmem"k" /dev/shm

# show the current size of tmpfs
df -h | grep tmpfs

# fill that space with 1k block zeros
echo " ..starting memory fill process."
dd if=/dev/zero of=/dev/shm/fill bs=1k

Note: this file is available at: https://raw.githubusercontent.com/bwuch/code-snips/master/memfill.sh

I placed these files in the tc user home directory (/home/tc) and set them to executable with chmod +x filename.sh.

If you’d like, you can add entries to have these scripts start automatically at boot — if you want an appliance that maxes out resources all the time. To do this, use sudo vi /opt/bootsync.sh and add entries at the end of the file for /home/tc/cpubusy.sh & and/or /home/tc/memfill.sh &. Note: the ending ampersand causes the script to run in the background and not wait for completion.

Typing backup will allow you to make these files & changes persistent.

Note: its much easier getting this text copied over if ssh is installed/configured on your tinycore VM. There is a very good write up on how to do this here: https://iotbytes.wordpress.com/configure-ssh-server-on-microcore-tiny-linux/. This post also covers how to include credentials (etc/shadow) in the file list backed up by TinyCore, which is also very useful.

Lightweight VM for testing – TinyCore Linux

I’ve often found the need to have a very small VM for testing. Some use cases would be testing if a DHCP scope was working, to verify if a VM port group was correctly configured, to confirm if a site-to-site VPN tunnel was up/passing traffic, to deploy/modify with PowerCLI, and many, many more others. Typically all I’m looking for is a command prompt, a network stack, and preferably VMware Tools running. Many years ago I stumbled on TinyCore Linux — a very small distribution that with a GUI and open-vm-tools installed only took 53mb of disk space as an OVA. Such a small distribution was easy and fast to deploy and used nearly no resources, so it was ideal for testing. This OVA was super handy, I had it on various file shares, a web URL, and even as a VM Template so that I could easily get to it for testing. The OVA I created was based on TinyCore 3.6 (Linux kernel 2.6) — like I said, I’ve had this for many years. I recently went looking to see if TinyCore was still being developed and if a newer version was available. What I found was TinyCore 11 (Linux kernel 5.4) that was released earlier this month. The steps below show how you can create your very own lightweight VM based on this latest distribution.

To begin we should download the ISO. I used the CorePlus-11.0.iso from http://tinycorelinux.net/11.x/x86/release/. Once I had the ISO, I uploaded it to a datastore. (Note: These instructions were tested on an ESXi 6.7u3 host, but should be similar on different versions and/or hypervisors.) . Here are the settings I used for the new VM

  • Compatible with ESXi 6.0 and later (hardware version 11). Note: as of this writing ESXi 6.0 is still supported, so I made this choice to ensure a the newest features would be available while maintaining maximum backwards compatibility (to all supported platforms).
  • Guest OS = Linux \ Other 3.x Linux (32-bit). Note: if using a newer version of virtual hardware, additional / newer Other Linux options may exist. TinyCore is based on Linux kernel 5.4, so we want to pick the nearest option available.
  • Memory = 256mb. Note: I tested with lower amounts of RAM, but this was the lowest that worked consistently. Your mileage may vary.
  • New Hard Disk – Remove
  • New CD/DVD Drive > Datastore ISO File, browse to CorePlus-11.0.iso uploaded earlier and select Connect…
  • Expand CD/DVD Drive and change Virtual device Node to IDE1:0
  • Add a new device > hard disk, size = 64mb, expand details and change virtual device node to IDE 0:0
  • Remove the new SCSI controller as it is not used
  • Under VM Options tab \ Boot Options \ confirm that Firmware = BIOS (recommended). Note: if you use a newer version of virtual hardware, a different firmware option may be selected by default.
  • Power On VM
  • Open with Remote Console (not web console). Note: this requires VMRC to be installed. By default in TinyCore the mouse pointer does not work with the web console.

Once the VM boots, you should see a menu of different boot options. Select the option “Boot Core with X/GUI (TinyCore) + Installation Extension.” Here are additional settings I selected:

  • Once the GUI has loaded, run tc-install (the last icon on the bar at the bottom of the screen)
  • Select Whole Disk and sda (Frugal and install boot loader should be checked by default)
  • Next > ext4 > next > next > Core Only (Text Based Interface) > next > Proceed
  • The wizard should say ‘Installation has completed’
  • Click Exit on the tool bar at the bottom of the screen, then Reboot

The VM will now boot into the shell prompt of the locally installed copy of TinyCore. We will now install some additional packages from the command line. This assumes the network in use has DHCP available and we already have an IP / internet connectivity.

tce-load -wi pcre.tcz
tce-load -wi open-vm-tools.tcz
tce-load -wi curl.tcz

Note: for me, the open-vm-tools install ended with text about fuse.conf not being a file or directory.  This did not cause any issues in my testing.

Next we check to ensure the new packages (pcre.tcz, open-vm-tools.tcz, and curl.tcz) are listed (one per row) in the list of packages loaded at boot (using command cat /etc/sysconfig/tcedir/onboot.lst).

In my testing, I noticed that sometimes open-vm-tools is not started correctly at boot. I’m not sure what causes this, but found a simple workaround by using a script that runs automatically at boot to restart the service. I did this by typing sudo vi /opt/bootlocal.sh

At the end of this file, I pressed i (to get into insert mode) and added the text /usr/local/etc/init.d/open-vm-tools restart.  We then save & exit from file by pressing the escape key (to exit insert mode) and typing :wq (to write changes and quit the vi editor).

By default the hostname of the VM is ‘box.’ However, I wanted to set a custom value. To do this we edit the bootsync.sh script by typing sudo vi /opt/bootsync.sh. We then replace the text ‘box’ on the line for sethostname with our FQDN. Note: this isn’t how hostnames are typically set in Linux, and I would assume this would not be updated by a vCenter customization spec. I’m only adding a value for reference when I see the hostname in vCenter that is presented by VMware Tools.

When the VM boots up, there is some text that’s automatically displayed from the message of the day (MOTD) file. We want to edit this to include some simple instructions. However, the changes to the MOTD file are not persistent by default. To include this file in the list of persistent files, we need to add an entry to a file. We can do this by typing vi /opt/.filetool.lst and adding an entry for “etc/motd” (without the quotes). Now that these changes persist, we can make changes to the MOTD with the command sudo vi /etc/motd. In this example I’ve added instructions for how to set a static IP address and to use curl to test a specific TCP port:

As a cleanup step, I like to remove the command history from templates. To do this for TinyCore we can type rm .ash_history to remove the file.

To trigger a backup of the files we’d like to persist, we simply type backup and press y for yes.

With all our changes complete, we can test that the VM starts up correctly by issuing the command sudo reboot.

Confirm that VMware Tools is working and that your message of the day is displayed.  On ESXi or vCenter you can confirm that VMware Tools are working by reviewing the status tab of the running VM. In this screenshot you can see that tools version 11269 is running. Clicking more info confirms that this is version 11.0.5 (the latest available as of this post).

You may notice that the Guest OS is showing Other 4.x or later Linux, even though we selected Other 3.x when creating the VM. This is because VMware Tools is presenting the property from within the guest, instead of defaulting to the property from the VM creation wizard.

You can now shutdown the VM and change the CD ROM back to Client Device (to detach the ISO image). From here we can export the VM to an OVF or convert to a virtual machine template. As an OVF this VM is roughly 19mb on disk. It will deploy very quick and is just enough of an appliance for many test cases.

Note: there are a few other recommended changes you may want to consider if you are building a template for use in VMware Fusion or Workstation that relate to display resizing and copy/paste. Those are documented here: http://wiki.tinycorelinux.net/wiki:vmware_installation.

vRealize Operations Alerts using Rest Notification Plugin

I have created several vRealize Operations (vROps) alerts in the past, mainly using the Log File Plugin and Standard Email Plugin. However, I recently had someone ask for more information on using the Rest Notification Plugin. I hadn’t used this, so I started looking for more detail on how to get started.

I found a couple really good blog posts on this, specifically https://blogs.vmware.com/management/2017/01/vrealize-webhooks-infinite-integrations.html and https://blogs.vmware.com/management/2018/02/extending-vrealize-operations-collaboration-tools-restful-apis-webhook-shims-david-davis-vrealize-operations-post-50.html . Both of these posts describe using an intermediary to accept what vROps is sending and convert it into a format that another endpoint expects. There are a handful of integrations provided, so I started looking at one that I could test with. The following post will describe the steps to get this working.

The Test Service
For testing, I’m going to use vROps to send an alert to Slack. This was pretty straight forward, I created a new channel where I wanted the alerts to appear, and then created a new incoming webhook by visiting https://api.slack.com/apps/new. I created a new app called vrealize-bot using my workspace. Once the app was created, I toggled on the ‘incoming webhooks’ feature, and mapped to my channel. This resulted in a webhook URL that looked like this:


To confirm this was working, I used a quick PowerShell script to try and post to that webhook URL. This isn’t needed, but did prove that my web hook was correctly created.

Invoke-WebRequest `
   -Uri $webhookURL `
   -Method POST `
   -Headers @{"Content-type"="application/json"} `
   -Body (@{"text"="This is a test"}|ConvertTo-Json)

The ‘Shim’
We need a piece of code to covert from the vROps Rest output into a format Slack will accept. The first blog post mentioned above calls out a prebuilt tool to do this — called the loginsightwebhookdemo. There are instructions available on getting this running, but the easiest route for me was to use the docker image. I started by downloading the Photon OS 3.0 OVA, deployed it to an ESXi host, and then enabled docker using these instructions. I ran three commands… only the middle one is required, the other two will just show some supplemental info.

systemctl status docker
systemctl start docker
docker in

Once docker is running, you can start the webhook-shims container using these instructions. As described in the instructions, you launch the bash shell, which gives you the ability to edit files in the container file system to add things like our Slack API URL. If you choose this route, once the files are edited in the loginsightwebhookdemo directory, you’ll need to run ./runserver.py from the webhook-shims directory. However, since we are only using the Slack shim in this example, there is an easier way. All we need to do is pull & run the container using these commands:

docker pull vmware/webhook-shims
docker run -it -p 5001:5001 vmware/webhook-shims

With the container running we can access its info page at http://dockerhostnameorip:5001/. This will show everything is up and running and that you can connect to the website.

The vROps Alerts
From the Alerts > Alert Settings > Notification Settings area, we can add a new rule. We will select the Rest Notification Plugin method and add a new instance. We’ll name our instance SlackWebhook_vrealize-bot. We could enter anything we want here, but want to be descriptive as possible. This shows the service we are using, how we are connecting, and the application that will be doing the posting, which seems sufficient. The URL is where the magic happens. We’ll enter a URL like this:


This is the name of the host our container is on, the port that the service is exposed on, the endpoint/slack is to specifies which shim we want to use, and the TTTTTTTTT/BBBBBBBBB/alphaNumericStr0ngOfText comes from our Slack webhook at the beginning of the article. We will leave the username and password blank (all that authentication is done in our custom webhook URL). For content type we’ll select application/json. Pressing TEST should result in two new posts to our Slack channel.

Slack Channel posting from vRealize Operations Rest Notification Plugin / webhook shim

Now all we need to finish up is to define the filtering criteria for which alerts we want to be sent to Slack. For testing, I just set criticality to Immediate or Critical, but we will likely want to narrow that down over time as it is a bit too chatty.

Using Log Insight Agent to find insecure LDAP binds

I recently read an interesting article on the vSphere Blog: https://blogs.vmware.com/vsphere/2020/01/microsoft-ldap-vsphere-channel-binding-signing-adv190023.html, which states Microsoft is making a change to the default behavior of LDAP servers that are part of Active Directory. This change will require secure LDAP channel binding by default, and is scheduled to be implemented in March 2020. The article goes on to say that vCenter using Integrated Windows Authentication (IWA) is not affected, so my lab should be fine…right? I have systems other than vCenter connecting over LDAP, so I have a need to double check. I found a really good article from 2016 ( https://docs.microsoft.com/en-us/archive/blogs/russellt/identifying-clear-text-ldap-binds-to-your-dcs) that shows how to increase logging levels and also includes an event viewer view and PowerShell script to find these events. However, I wanted to make this same data visible as a dashboard in Log Insight. The following post will recap the required steps to build a similar dashboard for your environment.

The first step was to enable the additional logging. I did this by adding the following registry key to all domain controllers in the environment. I did this in a lab with very few domain controllers, so I just ran this command on each, one at a time:

# Enable Simple LDAP Bind Logging
reg add HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics /v "16 LDAP Interface Events" /t REG_DWORD /d 2 

My domain controllers were already running the Log Insight agent and the Active Directory content pack was already installed and configured. You can check this in your environment by going to Settings > Content Packs. If you do not see Microsoft-Active Directory listed under the Installed Content Packs header, you can get it from the Marketplace. When you install the content pack from Marketplace, it will display a set of instructions you need to follow to enable appropriate collection. If you have already installed the content pack, you can find these setup instructions by browsing to the content pack, clicking the ‘gear’ icon and selecting ‘setup instructions’ (screenshot below):

For example, you may need to enable additional audit logs in your Default Domain Controllers policy – the details are included in the popup. You will also need to create a copy of the agent configuration from Settings > Administration > Agents (among other settings, this will ensure the Directory Services log is captured — and that is where event 2889 is logged). Here is a sample of the agent configuration, showing that our newly created configuration applies to both of my lab domain controllers.

Now that Windows is logging eventid 2889, and Log Insight agent is picking that event up, we can focus on extracted fields and dashboards in Log Insight.

Extracted Fields
An extracted field is a regular expression definition of text we want to find in our event. If we look at a sample of one such event (screenshot below) we can see that there three fields that we likely want to capture:
– Client IP Address
– Identity the client attempted to authenticate as
– Binding Type

The easiest way to extract this event is to highlight the text we want to extract. A popup will appear and give us filtering choices as well as an option to extract a field. If we select that option, a new field definition will appear to the right of the page.

We can see that the default criteria is looking for an IPv4 address that appears after the text address: and before the text :38130. This is close to the text we want, but the pre-text is not as specific as we could be and the post text is too specific (the source port can be any random high numbered port). We can edit this default criteria
– Pre text: Client IP address:\s* [adding the Client IP prefix as shown in the example event]
– Post text: : [removing the ending port number, leaving only the single colon ]
We will also give this field a name (Ldap2889_ClientIP) and make it visible to all users. The resulting field definition should look like this:

Now that we know how to extract the data we want, we will repeat the process to create other extracted fields for Ldap2889_Identity and Ldap2889_BindingType.

Once we have extracted interesting fields, we can build visuals to consume that information. The easiest way to do this is from Interactive Analytics. We start by applying filters — in this example we add a filter for eventid = 2889. We can then add interesting groups of data. In the top section of interactive analytics we can see a timeline of findings. In the bottom left area we see a couple of drop downs. The first one starts ‘Count of events,’ which is typically the most logical grouping. There are other choices available so you should review those options. The second drop down list defaults to ‘over time,’ and this one is the one that is typically adjusted. For example, we can select ‘Non-time series’ and then Group By one of our custom extracted fields — for example, Ldap2889_ClientIP.

Screenshot of count of events grouped by Ldap2889_ClientIP

Using the options to the right, we could easily turn this visual from a column to a pie graph. Once we have it the way we like, we can use the ‘Add to Dashboard’ button in the top right to save this view for later. The popup will ask us for a visual name, and then to select the dashboard where we’d like to see this data. We can make a new dashboard from this same screen if needed, and even share with all other users. We can create other visuals, for example, groupings by Ldap2889_Identity or BindingType, or even include some time series metrics to see when most of the logins are occurring. Looking at my dashboard, I can see that I may have a bit more work ahead of that March 2020 deadline.