VMware Skyline Insights API PowerShell Module

VMware Skyline is a proactive support tool to help customers avoid problems before they occur. More information on the service can be found here: https://www.vmware.com/support/services/skyline.html. One feature of this service is a GraphQL based API known as the Skyline Insights API. Using this API, you can query for active findings (the problems known/covered in the Skyline catalog) and for affected objects (the inventory items impacted by these findings). This was my first attempt at using a GraphQL based interface and it had a few learning curves.

The first learning curve with GraphQL was dealing with iterating through the results set. By default, the Insights API only returns 200 results per query. Once you have the first 200 records, if the query has more results you need to ask for the next batch of 200, and so on until you have all the results. This is easy enough to do, however the API will eventually start rate limiting queries against a Skyline Organization. Due to this, we also need some logic to account for these HTTP 429 rate limiting responses. As I attempted to solve these issues, I ended up creating a PowerShell module that would account for these lessons learned. The remainder of this post will cover how to use this new VMware.Skyline.InsightsApi PowerShell module.

To get started, you’ll need a API token. The process to create one is well documented here: https://blogs.vmware.com/kb/2021/12/skyline-insights-api-getting-started.html.

Second, you’ll need the module. All of the code for this is available in the VMware PowerCLI-Example-Scripts repo at https://github.com/vmware/PowerCLI-Example-Scripts/tree/master/Modules/VMware.Skyline.InsightsApi and also available in the PowerShell Gallery. The easiest way to install this module is with Install-Module VMware.Skyline.InsightsApi. This module contains 7 functions:

Get-Command -Module VMware.Skyline.InsightsApi

CommandType Name                             Version Source
----------- ----                             ------- ------
Function    Connect-SkylineInsights          1.0.0   VMware.Skyline.InsightsApi
Function    Disconnect-SkylineInsights       1.0.0   VMware.Skyline.InsightsApi
Function    Format-SkylineResult             1.0.0   VMware.Skyline.InsightsApi
Function    Get-SkylineAffectedObject        1.0.0   VMware.Skyline.InsightsApi
Function    Get-SkylineFinding               1.0.0   VMware.Skyline.InsightsApi
Function    Invoke-SkylineInsightsApi        1.0.0   VMware.Skyline.InsightsApi
Function    Start-SkylineInsightsApiExplorer 1.0.0   VMware.Skyline.InsightsApi

The first two functions listed (Connect-SkylineInsights and Disconnect-SkylineInsights) are used to connect to the API. The first requires an -apiKey parameter, which we obtained earlier. With this apiKey a global variable is created ($Global:DefaultSkylineConnection) containing the bearer token used to query the API. The second function simply clears out this global variable. As a safety mechanism, logic exists in the helper function to prevent the other functions from executing if this global variable is not present.

The Format-SkylineResult function is an optional function that helps with converting some of the objects returned by the API into strings. This is useful if you want to export the output into something like a CSV file. By default, if you attempt to pass the output from one of the other functions to a CSV, like Get-SkylineFinding | Export-Csv D:tmp\mySkylineFindings.csv many of the columns will end up with System.Object[] and the date values will be stored as long integers. If we also use this function, such as Get-SkylineFinding | Format-SkylineResult | Export-Csv D:tmp\mySkylineFindings.csv the objects are converted to strings (that can be separated by the value you pass to -separator) and the dates are converted to PowerShell dates.

The next function listed, Get-SkylineAffectedObject will return the list of affected inventory findings. It requires a -findingId and a -products input parameter to be passed in, either by property name or pipeline. A ‘product’ in this context is the case sensitive name of an endpoint in the Skyline Inventory, such as a vCenter Server name, Horizon Connection Server, vROps instance or the like. The ‘findingId’ is the case sensitive ID of the finding / issue that Skyline is aware of. Both of these properties, in the expected case, can be uncovered with the next function.

Next up, Get-SkylineFinding is a function that requires no input parameters, but does support three — the same -findingId and -products described above, as well as -severity. You can specify any number of these parameters, either by name or pipeline. The severity parameter is implemented client side, so all records are returned to the function, but the function will only return those matching one of the Skyline finding severities (Critical, Moderate, or Trivial). The output of this function can be piped to the above AffectedObject function, such as Get-SkylineFinding -severity:CRITICAL | Get-SkylineAffectedObject .

The Invoke-SkylineInsightsApi function is a proxy function that is consumed by both Get-SkylineAffectedObject and Get-SkylineFinding and is usually not directly consumed. It is exposed in the function for testing and any sort of future use. This is where much of the logic is implemented so that it can be shared by the two get functions.

Last, but not least, is the Start-SkylineInsightsApiExplorer function. This function will take the bearer token from the Connect-SkylineInsights function, put it in the clipboard, then launch the Skyline Insights API Explorer website in a web browser. From here, you can paste the bearer token into the ‘Request Headers’ area and interactively explore the GraphQL query for Skyline Findings.

I hope you find this module useful. If you have any feedback please leave a comment below or open an issue in the PowerCLI-Example-Scripts repo here: https://github.com/vmware/PowerCLI-Example-Scripts/issues.

Posted in Scripting, Virtualization | 1 Comment

Uncovering missing Active Directory subnets with vRealize Log Insight

In a recent post (https://enterpriseadmins.org/blog/virtualization/domain-controllers-and-micro-segmentation/) I described an issue where authentication may not work as desired when Active Directory sites and Services Subnets are not properly defined. There is often a disconnect in large enterprises where network/subnet creation and active directory aren’t managed by the same folks, so occurrences like the one I described are all too common. I remember many years ago writing a VBScript that parsed a log file to try and find new networks so that we could create subnet definitions. I decided to see what new options existed in this space and was surprised to see that things were mostly unchanged.

Active Directory authentications from clients without subnets defined are still logged to C:\WINDOWS\Debug\netlogon.log all these years later. This file contains entries such as:

05/12 21:28:11 [6772] LAB: NO_CLIENT_SITE: EUC-VIEWCS-21

This suggests that the subnet I use for VDI Management VMs is not mapped to a site in AD Sites and Services through a properly defined subnet. In this case I know that the network should map to my US-East-IN site in Active Directory. This is an easy fix, but in dynamic environments something similar is going to happen again.

The old VBScript would still work to parse this file, and I could run that as a scheduled task, and occasionally look for these types of events. However, thanks to vRealize Log Insight, I have better ways to deal with log files in my lab these days. All of the systems deployed in my lab run the Log Insight agent, which can be used to pickup this file. I already have a custom Agent Group for my domain controllers, so I can just edit its configuration so that it also picks up the file. To do this, I browse to Management > Agents > select the group “Domain Controllers” > File Logs > New and create an entry for the path in question:

As you can see, we are looking in the C:\Windows\Debug\ directory, specifically for one file named netlogon.log. After adding this entry I selected ‘Save Agent Group’. After a couple of minutes I searched Interactive Logs for no_client_site and had a few hits. This works well, but what I really want to see is which clients are showing up without needing to parse through all of these individual rows. To help with this I can make a custom dashboard based off an extracted field.

Extracted Field

I can see that the data I want is right there at the end of the string, so I can highlight the text and click ‘Extract field’. This brings up a ‘Manage Fields’ screen in the right navigation. By default, the wizard knows that I want to extract an IP Address, but it thinks I only want the one that comes after a specific hostname:

I can simply change this from EUC\-VIEWCS\-21 to (a single space) and it automatically highlights all the entries, not just this one. I can name the field and select save.

Custom Dashboard

From the explore logs view, I queried for no_client_site I changed the dashboard selections at the top to ‘Count of events’ and grouped by ‘WinDebug_NoClientSite_IP’ which is the name of the extracted field from above. This resulted in a bar graph by the authenticating client where I could easily see the handful of interesting clients.

In the top right of this visual is an ‘Add to Dashboard’ button. I used that button to add this newly created chart to a ‘Active Directory – Custom’ dashboard that I have started.

I now have a visual that will show me the clients that are in subnets not mapped to an Active Directory site. Once I research these subnets and get them properly defined this query should return no results — until the next time a network is created.

Posted in Lab Infrastructure, Virtualization | 1 Comment

Domain Controllers and Micro-segmentation

I was recently reminded of the importance of Active Directory Sites and Services as it relates to micro-segmentation. I was working on an engagement where an organization wanted to implement a zero-trust / micro-segmentation policy by default. As part of this effort, they created a new network with default deny/any firewall rules. The first system to be deployed to this network was a vCenter Server 6.7 system using Integrated Windows Authentication (IWA). Note: IWA is deprecated in vCenter Server 7.0 and will be removed in a future release per https://kb.vmware.com/s/article/78506.

When using IWA, a vCenter Server is joined to the domain, similar to a Windows client system. To support this configuration, a firewall rule was added to allow the client (vCenter Server) to access Active Directory servers in the local site (and remote domain controller hosting the PDC Emulator role, to support password changes). All ports documented at http://ports.vmware.com were included in the rule, but for ‘Active Directory Domain Controllers’ only a subset of the environment was listed.

Attempts to join the domain were failing with a generic error message. We attempt to join from the command line instead, with syntax similar to:

/opt/likewise/bin/domainjoin-cli join domain.com Domain_Administrator Password

Which returned an error that indicated the domain was not reachable. As part of troubleshooting, all domain controllers from the necessary domains were added to the domain controller rule on the firewall. This attempt was successful — indicating that a non-local domain controller was being contacted for our domain join. We checked the status of our vCenter Server Likewise configuration with this command:

/opt/likewise/bin/lw-lsa get-status

Which confirmed that the domain controller in use was not part of the local site. That’s when we checked Active Directory Sites and Services. Remember how I said this was a new network? The subnet had not been defined in AD Sites and Services, so the client didn’t know which site to use. A new subnet was created in AD Sites and Services and properly mapped to the correct/local site. The temporary firewall rule was reverted (so we again only listed local DCs and the PDC emulator role) and a domain join was retried — SUCCESS!

A few other relevant settings came up while investigating this issue, but were not required for this specific engagement. I’m including them below as I believe they could be relevant depending on the micro-segmentation project.

Posted in Virtualization | 1 Comment

Photon OS 4.0 Grow Filesystem using Logical Volume Management (LVM)

I recently needed to grow the file system on a Linux VM. I remembered writing a post on this before, so I headed here:

Unfortunately this previous post assumed a non-LVM disk setup.

I needed to increase the size of the /data mountpoint using the filesystem /dev/mapper/data_vg-data and the earlier instructions did not work. My VM had three disks. To figure out which one I needed to grow, I used the Linux lsblk command. This showed three disks, sda, sdb, and sdc. The /data mountpoint went to sdb, which was 60GB in size. Looking at the VM, Hard Disk 2 was the only one that was 60 GB. I increased it to the necessary size in the vCenter UI.

Back in the VM I needed to resize the filesystem to match the new disk size. To do this I first ran pvresize /dev/sdb which returned the message that 1 physical volume(s) resized or updated.

I then ran lvdisplay looking for the ‘data’ volume group, specifically the LV Path property, which in this case was /dev/data_vg/data.

From here I ran lvextend -l +100%FREE /dev/data_vg/data which returned a message that the size of logical volume had changed to the new size of the disk.

Finally I ran resize2fs /dev/data_vg/data which confirmed that the filesystem was resized on-line.

The command df -h now shows the expected size for the mount.

Posted in Lab Infrastructure, Virtualization | Leave a comment

Horizon Virtual Desktop – Non Persistent Ubuntu 20.04

I recently built a new Ubuntu Desktop 20.04 machine to be used as an instant clone golden image for a Horizon 7.13 environment. I kept some notes on the steps I followed on my desktop, but after seeing a funny tweet this week (https://twitter.com/DennisCode/status/1475343774695886849), I decided I should share this document. This blog post will outline those steps to create a Ubuntu 20.04 Desktop for use as a Horizon 7.13 non persistent pool.

The first step was creating a virtual machine.

  • VM Name: vdi_g01_ubuntu-2004 — this is my naming convention for VDI golden images. The VDI is pretty obvious, the g01 is for golden image #1 and then the last part is the OS name.
  • Compatible with: ESXi 7.0 U2 and later (vmx-19) — my test hosts for VDI are running the latest available 7.0 release. I typically only visit compatibility levels when VMs are initially created. Since I’m able to run this version, and knowing 6.5/6.7 reach end of support in just a few months, I decided to go with the latest available. You’ll want to specify something that is capable of running on the versions of ESXi available in your environment.
  • Guest OS Family: Linux
  • Guest OS Version: Ubuntu Linux (64-bit)
  • Virtual Hardware Configuration
    • vCPU: 1
    • Memory: 2GB
    • New Hard disk: 24GB, thin provisioned
    • SCSI Controller: LSI Logic Parallel
    • New Network: — this is the port group I use for VDI desktops
    • Video Card > Number of displays 2
    • Video Card > Total video memory: 32mb
  • VM Options

I then selected ‘launch remote console’ to open the VMware Remote Console. I find this is the easiest way to install operating systems from ISO image. During the install I skipped the file check (just to save time) and selected ‘Install Ubuntu’ and accepted defaults. For my name I entered template-admin and for computer name I entered vdig01ubu2004.lab.enterpriseadmins.org. I entered a good password, selected require password to log in and didn’t enable Active Directory (we’ll do that later).

After the final reboot, we’ll login as our template-admin user, launch the terminal and install sshd, so that we can use it for the rest of the configuration. We’ll do this by entering sudo apt install openssh-server -y. With that complete we can find our IP address — either from the vCenter Server > VM > Summary tab, or by typing ip addr at that terminal session. We can then ssh into the system for the majority of the next configuration items.

The first few changes I made were generic system wide changes.

# apply any system updates & remove any obsolete packages
sudo apt update && sudo apt upgrade -y
sudo apt clean && sudo apt -y autoremove --purge

# Prevent ctrl-alt-del from causing a reboot
sudo systemctl mask ctrl-alt-del.target

# Disable auto-suspend
sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target

# refresh systemctl
sudo systemctl daemon-reload

# Disable auto-updates
sudo sed -i 's/APT::Periodic::Update-Package-Lists "1"/APT::Periodic::Update-Package-Lists "0"/' /etc/apt/apt.conf.d/20auto-upgrades

# Disable LTS Upgrade MOTD
sudo sed -i '16 s/.*Prompt.*/Prompt=never/' /etc/update-manager/release-upgrades

# Remove some of the initial setup packages
sudo apt remove --purge gnome-initial-setup gnome-online-accounts update-manager-core -y

# setup script to generate new ssh keys at boot
cat << 'EOL' | sudo tee /etc/rc.local
#!/bin/sh -e
# rc.local
test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server
exit 0

# make sure the script is executable
sudo chmod +x /etc/rc.local

I then made a few changes related to authentication and Active Directory membership. This is likely a questionable security practice, as we are letting all members of Domain Users to be able to not only login but run commands with sudo. For purposes of this lab desktop it works like I like, but you may want to make a few changes for your environment. In the past, I’ve used pbis-open for Active Directory authentication, but it appears that project is no longer maintained as of 2019. For this example, I will switch to sssd, as it is called out as supported by Horizon documentation and is recommended for ease of deployment (https://docs.vmware.com/en/VMware-Horizon/2111/linux-desktops-setup/GUID-D8E3A4AA-83E9-46A4-8BBA-824027146E93.html).

# install required components
sudo apt install sssd sssd-tools realmd libnss-sss adcli samba-common-bin  -y

# join the domain
echo 'VMware1!' | sudo realm join lab.enterpriseadmins.org --user svc-windowsjoin --computer-ou='OU=EUC VDI Linux Parents,OU=LAB Computers,DC=lab,DC=enterpriseadmins,DC=org' --os-name='other' --verbose

# Tell pam.d to create the home directory at login
echo "session required pam_mkhomedir.so skel=/etc/skel umask=0022" | sudo tee /etc/pam.d/common-session

# Since we only have the one domain, lets use short names for most things
sudo sed -i 's/use_fully_qualified_names = True/use_fully_qualified_names = False/g' /etc/sssd/sssd.conf

echo '%domain\ users  ALL=(ALL) ALL,!ROOTONLY' | sudo tee -a /etc/sudoers
# Note that '\ ' is to escape the space between domain & users.  In PBIS-Open spaces were denoted as ^ and those had to be changed for group names to show the escaped space (example: '\ ' without the quotes).

# Note: I did need to delegate extra permissions to this service account compared to PBIS Open.  I followed this guide:
# https://www.computertechblog.com/active-directory-permissions-required-to-join-linux-and-windows-computers-to-a-domain/

I also like to suppress some of the initial popup and configuration steps. I use these desktop for testing purposes and they are disposed of at logoff — there isn’t a lot of value in getting comfy and setting up everything perfectly as a user as those settings are lost at logoff. There were a couple different options listed here: https://askubuntu.com/questions/1028822/disable-the-new-ubuntu-18-04-welcome-screen and I applied them all as they seemed somewhat hit or miss in my testing.

sudo mkdir /etc/skel/.config
sudo touch /etc/skel/.config/gnome-initial-setup-done
sudo sed -i '/\[daemon\]/a InitialSetupEnable=false' /etc/gdm3/custom.conf

I like to install a few packages to do administrative type activities. These include:

  • firefox — a web browser (installed by default)
  • net-tools — the legacy network tools like ifconfig
  • powershell / powercli — a scripting language & module I use often
  • remmina — a remote desktop client
  • zenmap — a GUI for the nmap scanner
# install net-tools
sudo apt install net-tools -y

# install Powershell & PowerCLI
# Download the Microsoft repository GPG keys
wget -q https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb

# Register the Microsoft repository GPG keys
sudo dpkg -i packages-microsoft-prod.deb

# Update the list of products
sudo apt-get update

# Install PowerShell
sudo apt-get install -y powershell

# Start PowerShell, but as sudo se we can install PowerCLI for all users
sudo pwsh
Install-Module vmware.powercli -scope:allusers -Confirm:$false
Set-PowerCLIConfiguration -ParticipateInCeip $true -Scope:AllUsers -Confirm:$false

# Confirm that VMware* modules are available in the /usr/local/share/powershell/Modules path:
ls /usr/local/share/powershell/Modules
exit # this is to exit pwsh

# Install remmina
sudo apt install remmina -y

# Install zenmap
sudo apt install curl ndiff -y

wget http://archive.ubuntu.com/ubuntu/pool/universe/p/pygtk/python-gtk2_2.24.0-5.1ubuntu2_amd64.deb
sudo apt install ./python-gtk2_2.24.0-5.1ubuntu2_amd64.deb -y
wget http://archive.ubuntu.com/ubuntu/pool/universe/n/nmap/zenmap_7.60-1ubuntu5_all.deb
sudo apt install ./zenmap_7.60-1ubuntu5_all.deb -y

# cleanup & remove installers
rm python-gtk2_2.24.0-5.1ubuntu2_amd64.deb packages-microsoft-prod.deb zenmap_7.60-1ubuntu5_all.deb

There are a few edits I made to keep these apps pinned to the menu on the desktop. The following lines should make that happen.

echo "user-db:user" | sudo tee /etc/dconf/profile/user
echo "system-db:local" | sudo tee -a /etc/dconf/profile/user
sudo mkdir /etc/dconf/db/local.d
echo "[org/gnome/shell]" | sudo tee /etc/dconf/db/local.d/00-favorite-apps
echo "favorite-apps = ['firefox.desktop', 'org.remmina.Remmina.desktop', 'nautilus.desktop', 'gedit.desktop', 'gnome-terminal.desktop']" | sudo tee -a /etc/dconf/db/local.d/00-favorite-apps
sudo dconf update

I also like to leave this desktop open for long periods of time and like to come back to it without needing to reauthenticate. Again, for production purposes this is a questionable practice, but this is a lab. You can leave this out if you’d like, but I wanted to leave the example in case. The gsettings commands are per user only, so to make this setting available to other users that may use the pool, we’ll write them to a profile script that is executed at login.

echo 'gsettings set org.gnome.desktop.lockdown disable-lock-screen true' | sudo tee /etc/profile.d/02-weak-security.sh
echo 'gsettings set org.gnome.desktop.screensaver lock-delay 3600' | sudo tee -a /etc/profile.d/02-weak-security.sh
echo 'gsettings set org.gnome.desktop.screensaver lock-enabled false' | sudo tee -a /etc/profile.d/02-weak-security.sh
echo 'gsettings set org.gnome.desktop.screensaver idle-activation-enabled false' | sudo tee -a /etc/profile.d/02-weak-security.sh

Now to install the Horizon Agent. I could download the installer from inside the desktop, but I try to launch the fewest apps possible in my golden image & don’t like logging into websites from that template. Instead, I download the agent installer and post it to an internal web server, then download that copy with wget as shown below.

cd /tmp
wget http://www.example.com/VMware-horizonagent-linux-x86_64-7.13.1-19066964.tar.gz
tar -xzf VMware-horizonagent-linux-x86_64-7.13.1-19066964.tar.gz
cd VMware-horizonagent-linux-x86_64-7.13.1-19066964/
./install_viewagent.sh -A yes -G yes

# Lets set clipboard redirection system wide, both directions.  Default is 2. 0 means disable; 1 means both direction; 2 means from client to agent only, 3 means from agent to client only.
sudo sed -i 's/#Clipboard.Direction=1/Clipboard.Direction=1/g' /etc/vmware/config

# Update the config to use sssd instead of pbis-open (https://communities.vmware.com/t5/Horizon-for-Linux/True-SSO-for-CentOs-7-Instant-Clone-agent-initialization-state/td-p/509274)
echo "OfflineJoinDomain=sssd" | sudo tee -a /etc/vmware/viewagent-custom.conf

I also like to have my Certificate Authority root cert installed as a trusted authority for both the system and Firefox. I previously wrote a post on that here: https://enterpriseadmins.org/blog/lab-infrastructure/installing-windows-ca-root-certificate-on-linux-and-firefox/. I’m going to follow those steps in this golden image as well.

To shut things down (now, and anytime we make changes to our template) I like to run the following:

# clean up the command history of the VM template-admin user
sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*
sudo rm -f /etc/ssh/ssh_host_*
history -c
sudo shutdown -h now

I then like to edit the ‘Notes’ property of the template with the date & a note about what was changed. This field is copied each time the VM is cloned, so I have a bit of an idea of what was included. For example, I would enter something like 2021-12-29: Initial creation of Ubuntu 20.04 Non Persistent VDI Desktop. Once the note is added, I would create a snapshot with the same message. From here, I was able to create a Horizon Instant Clone Pool, selecting our new VM and snapshot.

I hope you find this post helpful, feel free to drop any suggestions or comments below.

Posted in Lab Infrastructure, Virtualization | Leave a comment