Enterprise Admins.org

I was recently speaking with a customer who mentioned an issue that occurred when using the PowerCLI Get-TagAssignment cmdlet. They had a tag category which applied to clusters only. If a host were offline or in maintenance mode, they were getting an error that a local datastore was not accessible. I was able to recreate this issue and observed the following error text.

Get-TagAssignment -Category h225-clusteronly-category

Tag                                      Entity
---                                      ------
h225-clusteronly-category/h225-cluste... h206-cluster-storagepath
Get-TagAssignment : 7/14/2024 10:30:04 AM       Get-TagAssignment               Datastore 'Local-h206-vesx-02' is not accessible. No
connected and accessible host is attached to this datastore.
At line:1 char:1
+ Get-TagAssignment -Category h225-clusteronly-category
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-TagAssignment], InaccessibleDatastore
    + FullyQualifiedErrorId : Client20_TaggingServiceCisImpl_GetVDisks_Error,VMware.VimAutomation.ViCore.Cmdlets.Comma
   nds.Tagging.GetTagAssignment

This is a peculiar error as the tag category only specifies ClusterComputeResource as an Associable Entity, so datastores should not be in scope and it is unexpected to see them called out in this context.

While looking into this error message, I stumbled across this blog post from a couple of years ago: https://virtuallyjason.blogspot.com/2022/02/powercli-and-get-tagassignment.html. The article doesn’t touch specifically on this error message, but discuses performance related impact on specifying the -Entity parameter of the Get-TagAssignment cmdlet. I did a bit of testing and confirmed that not only does specifying the entity improve performance, it also suppresses the error message that occurs for offline datastores.

To double check performance, I wanted run each command several times. The loop below runs each command 100 times, then outputs the min, avg, and max run times for each command. In my case, specifying -Entity (Get-Cluster) caused the execution to complete on average 5x quicker.

$myCommand = @()
1..100 | %{
  $myCommand += [pscustomobject][ordered]@{
    Iteration = $_
    RunTime1 = (measure-command {Get-TagAssignment -Category h225-clusteronly-category}).TotalSeconds
    RunTime2 = (measure-command {Get-TagAssignment -Category h225-clusteronly-category -Entity (Get-Cluster)}).TotalSeconds
  }
}

$myCommand | Measure-Object -Property RunTime1 -Minimum -Average -Maximum 
$myCommand | Measure-Object -Property RunTime2 -Minimum -Average -Maximum 

When using Get-TagAssignment there are a couple of good reasons to limit the scope of the cmdlet to only those entities required.

A customer recently asked me if it was possible to find a virtual machine BIOS serial number from a virtual machine object or vmx file. While looking into this, I found an old post that provided a very useful hint: https://peppercrew.nl/2011/04/get-virtual-machine-bios-serial-number/. This post contains a PowerShell function to convert a VM UUID to the BIOS serial number. Looking at the code, and reviewing some test VMs, I realized that this is only a string replacement exercise. I’ve seen these serial numbers and UUIDs for years and never made the connection that they were so closely related.

The image below has one example value. The top line is the virtual machine serial number obtained from inside the guest OS (using wmic bios get serialnumber on Windows or dmidecode -s system-serial-number on Linux). The bottom line is the virtual machine UUID (from PowerCLI we can find this with: $vm.ExtensionData.Config.Uuid). The color coding has been added to make the pattern easier to see.

This actually helped explain something I’ve seen before with virtual machines created by storage array-based clones having duplicate UUIDs. The top value (minus the ‘VMware-‘ prefix) is stored in the vmx file as uuid.bios. When a direct clone of the VM file is registered into inventory, the VMs UUID (and by extension, BIOS serial number) would be a duplicate of the original/source VM.

We can see in the following command/output that two VMs have different names / IDs, but have duplicate UUIDs. I’ve confirmed with dmidecode that these two guests also have the same system-serial-number.

Get-VM h207-vm-* | Select-Object Name, ID, @{N='UUID';E={$_.extensiondata.config.uuid}}

Name       Id                        UUID
----       --                        ----
h207-vm-01 VirtualMachine-vm-3147125 42023081-331b-58e1-2730-ca560789e551
h207-vm-02 VirtualMachine-vm-3147127 42023081-331b-58e1-2730-ca560789e551

If we want to change our VM UUID, to ensure all our VMs have unique serial numbers, we can do that with PowerCLI as well. For illustration purposes, I’m going to update the UUID value for the second VM to end with the number 2. However, we could also have PowerShell generate a completely new UUID with [guid]::NewGuid().

$vm = Get-VM h207-vm-02
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.uuid = '42023081-331b-58e1-2730-ca560789e552'
$vm.extensiondata.ReconfigVM_Task($spec)

The above code will update our second VM to have a UUID which ends in 552 (the string we provided). In the below code block we’ll get the same two VMs as above and note that our UUIDs have changed.

Get-VM h207-vm-* | Select-Object Name, ID, @{N='UUID';E={$_.extensiondata.config.uuid}}

Name       Id                        UUID
----       --                        ----
h207-vm-01 VirtualMachine-vm-3147125 42023081-331b-58e1-2730-ca560789e551
h207-vm-02 VirtualMachine-vm-3147127 42023081-331b-58e1-2730-ca560789e552

Knowing the BIOS serial number and UUID relationship can be understood with string manipulation, I created a quick function to reverse the original example we found. In this function we can provide a VM system-serial-number and have it select the pieces of the string required to build our UUID.

function Get-VMUuidFromSerial {
  param($vmSerial)
  if ($vmSerial.length -ne 54) { write-warning "The provided serial number $vmSerial does not appear to be the correct length." }

  $myResult = $vmSerial -Replace 'VMware-', '' -Replace ' ', ''
  $myResult.substring(0, 8) + '-' + $myResult.substring(8,4) + '-' + $myResult.SubString(12,4) + '-' + $myResult.Substring(17,4) + '-' + $myResult.Substring(21, $myResult.length - 21)
}

The original code we found expected a parameter of type [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl], but our VM objects were being passed in as [VMware.VimAutomation.ViCore.Impl.V1.VM.UniversalVirtualMachineImpl], so I updated the type definition in the function. To be able to test this using a list of UUIDs from a CSV file, I ended up removing that type definition and checked the length of the string, similar to the above function. I also adjusted the function name to match the format we used above. I’m including it below for reference/safe keeping.

function Get-VMSerialFromUUID {
  param($vmUUID)
  if ($vmUUID.length -ne 36) { write-warning "The provided UUID $vmUUID does not appear to be the correct length." }

  $myResult = 'VMware-'
  $serialnumtmp = $vmUUID.Replace('-','')
  for ($i = 0; $i -lt $serialnumtmp.length; $i += 2) { $myResult += $serialnumtmp.substring($i, 2); if ($myResult.Length -eq 30) {$myResult += '-' } else { $myResult += ' ' } }
  $myResult
}

Hopefully this post will help if you ever need to compare VM UUIDs with BIOS serial numbers and/or resolve duplicate serial number issues.

I was recently helping a customer who had inherited an existing vSphere deployment that used NFS storage. They were tasked with migrating the old VMs to newer infrastructure, but they first wanted to find the NFS storage array backing the datastores.

Looking at the datastores in vCenter, they couldn’t find the hostname/IP address of the storage target. Instead, they saw somewhat random values in the device backing > server field, sort of like this demo red datastore I mocked up that shows a server of network.nfs.1 where we’d normally expect to see the hostname or IP. The value observed, network.nfs.1 in this case, wasn’t a name that was resolvable using the customers DNS.

Looking at host networking, one VMkernel adapter was clearly the one used for storage access, similar to this mocked up screenshot:

It seemed logical that network.nfs.1 and the other seemingly random names were devices on this 10.3.3.0/25 network. We wanted to try and issue a ping from this ESXi host, but the root password was unknown and we were not able to login to the console to do so. However, since we had access to vCenter, I went looking for a way to send a ping from esxcli, hoping we could then use the Get-EsxCli PowerCLI cmdlet to issue our pings. I found esxcli network diag ping and testing in a lab worked as expected, so we tried it in this environment:

$esxcli = Get-EsxCli -VMHost $thisVmHost -v2
$networkDiagPing = $esxcli.network.diag.ping.CreateArgs()
$networkDiagPing.host = 'network.nfs.1'
$networkDiagPing.interface = 'vmk1'
$pingResults = $esxcli.network.diag.ping.Invoke($networkDiagPing)

Unfortunately, this resulted in the error sendto() failed (Network is unreachable). Surprising, as the NFS datastore was online and we had specified that we wanted to use the VMkernel interface on the storage network. In this case, the host had 4 VMkernel interfaces, so we stepped through each, trying to find out if the storage traffic was using a different interface. The last interface we tried, vmk0, received a response.

As best we could tell, the vmk1 interface was unused. The portgroup named ‘storage’ had a VLAN backing that didn’t actually exist in the environment & the VMkernel IP address wasn’t a network that existed either. Once we knew which network adapter was actually in use, the ping response returned an IP address of a known NAS. We did a bit more digging and found host entries that were obfuscating the actual IP addresses of known storage targets. For reference, here is how we found the host file entries, again using esxcli.

$esxcli.network.ip.hosts.list.invoke() | Select-Object HostName, IPaddress

HostName      IPaddress
--------      ---------
network.nfs.2 192.168.10.26
network.nfs.1 192.168.10.26
network.nfs.9 192.168.67.21

After the fact I put together a quick script to help in the odd event I ever see something like this again. It finds all the unique hostnames/IPs used by NFS datastores and then for each VMkernel interface attempts to ping the NFS host, only showing the successful ping responses.

$thisVmHost = 'h197-vesx-04.lab.enterpriseadmins.org'
foreach ($thisDatastoreBacking in Get-vmhost $thisVmHost | get-datastore |?{$_.ExtensionData.info.nas.type -eq 'NFS'} | select-object @{N='RemoteHostNames';E={$_.ExtensionData.info.nas.RemoteHostNames}} -Unique) {
  foreach ($thisVmk in Get-VMHostNetworkAdapter -VMHost $thisVmHost -VMKernel) {
    $esxcli = Get-EsxCli -VMHost $thisVmHost -v2
    $networkDiagPing = $esxcli.network.diag.ping.CreateArgs()
    $networkDiagPing.host = $thisDatastoreBacking.RemoteHostNames
    $networkDiagPing.interface = $thisVmk.name
    try {$pingResults = $esxcli.network.diag.ping.Invoke($networkDiagPing); $uniqueHosts = [string]::Join(', ', ($pingResults.Trace.host | select-object -Unique))} catch { $pingResults=$null }
    
    if ($pingResults) { "Pinging $($thisDatastoreBacking.RemoteHostNames) from $($thisVmk.name) [IP $($thisVmk.IP)] took path $uniqueHosts" }
  } # end vmkernel loop
} # end Datastore backing loop

In a lab with a similar configuration, the script above produces output similar to:

Pinging network.nfs.1 from vmk0 [IP 192.168.10.19] took path 192.168.10.26
Pinging network.nfs.2 from vmk0 [IP 192.168.10.19] took path 192.168.10.26
Pinging network.nfs.9 from vmk0 [IP 192.168.10.19] took path 192.168.57.21, 192.168.10.1, 192.168.127.252

The final row in that output shows an NFS target that was not on the local network and took a few hops to get to the final destination, which might be helpful to see.