Finding VMs that will boot into the BIOS screen

A few months ago, after a round of Windows OS patching on development servers, a server admin notified me that their server did not come back online after the patch initiated reboot. Looking at the VM console he noted that the server was at the BIOS setup screen. After exiting the BIOS Windows booted as expected.

The only explanation I could think of was someone checked the box ‘The next time the virtual machine boots, force entry into the BIOS setup screen.’ in the virtual machine properties. I looked into the issue and came up with a simple one-liner looking for any virtual machines with this option set. Here is that script:

Get-View -ViewType VirtualMachine -Property Name,Config.BootOptions | where {$_.Config.BootOptions.EnterBIOSSetup -eq $true} | select Name

I did locate a couple of production VMs that had this option selected. I was able to remove the ‘EnterBIOSSetup’ flag from these servers before patching. I was unable to determine how/why/who this setting ended up getting changed, but thought I’d share the one liner in case anyone else encounters a similar issue.

VMware Certified Professional 5

I recently completed the requirements for the VMware Certified Professional 5 certification. Achieving certifications isn’t a priority for my current employer (a VMware customer) and as such it had been over 10 years since I last attempted an IT certification. I passed the exam on the first attempt but the questions were a lot harder than I had anticipated. I wanted to point out a couple of very solid resources I used to study for the exam.

First, I watched the 7 part brownbag series on the VCP 5 exam. You can find these episodes here: http://professionalvmware.com/brownbags/. Each one is between 1 and 2 hours long, so put them on your iPad and kick back when you have a chance. This was my first experience watching the brownbags — but it won’t be my last. These are very solid technical presentations and I highly recommend subscribing to in iTunes.

Next, I read through the VCP blueprint from VMware MyLearn. It lists all of the content that could appear on the exam. It is a very short paper, but worth a look. I printed a copy and was going to use it as a check list to figure out what I needed to study…until I found something better.

Finally, I found and used the study guide created by Josh Coen and Jason Langer. It is a 130 page PDF document that is organized by objective just like the VCP blueprint. You can find it here: http://www.virtuallanger.com/vcp5/. This is better than the blueprint as it includes content about each objective and references the associated supporting document you should review.

Here is a bit of free advice. When you study for the exam, don’t discount anything you see in the blueprint. I remember thinking…Oh, nobody would ever use this feature so I don’t need to know it for the exam. That is simply not true. If the feature exists — and more specifically exists on the blueprint — you’ll want to know the material for the exam. Additionally, and this will sound fairly straight forward, be sure to study those things you do NOT know. If you only use vSphere Distributed Switches be sure and study the limitations and use cases of standard switches. If you only use NAS storage be sure to study block storage. Knowing these things will greatly help in passing the exam. Good luck!

Power On virtual machine – Unable to access file since it is locked

Several times in the past few weeks I have ran into virtual machines that do not power on because of locked files. Steps to troubleshoot this issue are described in KB10051: Virtual machine does not power on because of locked files. While looking into this topic, I found the following article on vi-tips.com Could not power on VM – lock was not free, which describes the same issue. My issue was encountered on several ESXi 5.0 Update 1 hosts, and according to this vi-tips.com article the issue also exists on ESXi 4.1.

The vi-tips.com post includes several screenshots of how to isolate/troubleshoot this issue. After a few steps, you can use vmkfstools -D /vmfs/volumes/sanvolname/filename-flat.vmdk to determine the MAC address of the VMKernel interface maintaining the lock. A little bit of PowerCLI goes a long way into finding which host uses this interface:

1
2
3
4
5
6
7
Function Get-VMHostNameByMac ([string]$macAddress) {
#The MAC address passed to this function does not need separators, we will remove them if provided
$macAddress = $macAddress.Replace(":","").Replace("-","")

#The MAC property returned by the Get-VMHostNetworkAdapter cmdlet will have a colon separator, so we will remove it for comparison purposes
Get-VMHost | Get-VMHostNetworkAdapter | where {$_.Mac.Replace(":","") -eq $macAddress} | select-Object VMHost, Name, MAC
}

Here is example usage of this command:

Get-VMHostNameByMac '001ec9123456'

Once the locking VMKernel interface is located, we can continue troubleshooting. The vi-tips.com article suggests cold migration of the VM to the locking host and powering on the virtual machine on that host — but I was unable to get that option to work. However, I found that restarting the management agents on the locking host would resolve my issues. Once the management agents were restarted I could power on the VM using any host.

Verify VAAI settings with PowerCLI

I recently had a need to validate that all hosts in a specific vCenter had appropriate vStorage API for Array Integration (VAAI) settings. I didn’t want to change the values, I was just looking for the existing values. This short code will do just that:

1
2
3
4
5
6
7
8
9
10
11
12
13
$myreport = @()
Get-VMHost | %{
    $DataMoverHardwareAcceleratedMove = $_ | Get-VMHostAdvancedConfiguration -Name DataMover.HardwareAcceleratedMove
    $VMFS3HardwareAcceleratedLocking = $_ | Get-VMHostAdvancedConfiguration -Name VMFS3.HardwareAcceleratedLocking
    $DataMoverHardwareAcceleratedInit = $_ | Get-VMHostAdvancedConfiguration -Name DataMover.HardwareAcceleratedInit
    $myreport += new-object psobject -property @{
        Host = $_.Name
        DataMoverHardwareAcceleratedMove = [string]$DataMoverHardwareAcceleratedMove.Values
        VMFS3HardwareAcceleratedLocking = [string]$VMFS3HardwareAcceleratedLocking.Values
        DataMoverHardwareAcceleratedInit = [string]$DataMoverHardwareAcceleratedInit.Values
    }
}
$myreport

The resulting output will list the setting for each option value for each host. Note: 0 = disabled and 1 = enabled.

If you need to make changes to these values on a large number of hosts, check out this article: http://nickapedia.com/2011/03/01/powercli-and-vaai-quick-and-dirty-script/.

Creating and Reviewing Windows crash dumps in VMware virtual machines

The other day I was assisting a co-worker who needed a memory dump of a Windows virtual machine. He was working with Microsoft support and the support engineer mentioned that the process for gathering this data was different since we were using a virtual machine. I Googled vmware windows memory dump and found VMware KB 1001624, which links to a Microsoft KB article that documents how to make CTRL+ScrollLock+ScrollLock cause a blue screen (which creates a memory dump). This process works the same on both physical and virtual servers.

While that is a fun setting, and has a possible April Fools use case, it doesn’t really help troubleshoot issues. Using this option, the memory dump is caused by the user and requires an outage is required (a reboot to enable the registry key and a blue screen to create the memory dump).

The tool the Microsoft engineer was talking about is vmss2core.exe and it ships with VMware Workstation. I haven’t had a chance to use it in the past, but it is very simple. Here are the steps I used to test this process:

  1. Create a snapshot of a virtual machine with the ‘snapshot the virtual machines memory’ option selected.
  2. Browse to the virtual machine’s folder on the VMFS or NFS volume and download the ‘-snapshot1.vmsn’ file to a temporary directory.
  3. Copy the ‘vmss2core.exe’ file from a VMware Workstation installation to this temporary directory.
  4. Run the following command (where ‘w2k3test01-snapshot1.vmsn’ is the name of your snapshot file):
vmss2core.exe w2k3test01-snapshot1.vmsn -W

This process creates a memory.dmp file in the current directory. This file can be analyzed to determine what was running at the time of the snapshot creation. Note: the vmsn file is slightly larger than the amount of vRAM allocated to the virtual machine and the memory dump is equal to the amount of RAM installed. For a VM with 8GB of RAM you’ll need at least 16GB of disk space to complete this process.

How do I open these memory.dmp files to verify they are working? I’m glad you asked. I found a Network World post from 2005 that outlines the process: http://www.networkworld.com/news/2005/041105-windows-crash.html. You only need to download and install two applications (note: a few dependencies exist for these apps, such as the .NET Framework).

Download and Install Debugging Tools for Windows
Download Windows Symbol Packages

These downloads are free and give you the tools to open/review a crash dump. I don’t have the detailed understanding on how to work with these files, but I wanted to make sure the memory.dmp export worked correctly before sending an 8GB file to Microsoft for analysis.

Disable SSL for VMware Converter 5 using powershell

Starting with VMware Converter Standalone 5.0, the default action is to secure the P2V data with SSL. While this is nice if you are converting machines across an untrusted network or need the additional level of security, it can really slow down the P2V process. There is a blog post at Jon Kohler’s Blog and a thread on the VMware Communities about this subject.

I’ve been working on a handful of P2Vs, and I always need to find my notes about the NFC SSL setting to remember exactly what I need to change. While I was waiting on one of these P2Vs to complete, I opened up powershell looking for a way to modify the XML file. I now have a simple script on my management server that:

1.) Finds the right file (based off operating system)
2.) Makes a backup copy of the XML
3.) Adds the useSSL element if it does not exist
4.) Modifies the value of config.nfs.useSSL
5.) Restarts the vmware-converter-agent service

If you are interested in the script, you can download it here: VMwareConverterDisableSSL.ps1

I did notice two possible concerns that aren’t important in my environment but wanted to share them just in case. The backup XML file will be overwritten if this script is ran more than one time. This could be fixed by appending a date stamp or random number to the file name (see line 38). Second, the powershell modified version of the XML file does not preserve white space in the comments, so the resulting output is less lines than the source file. The change works as expected and the service does start, so I wasn’t worried about this difference.

Why does the ‘Update Tools’ button not work from within VMware Tools?

I recently upgraded an ESXi 4.1 Update 1 cluster to ESXi 5.0 Update 1 — but have not upgraded the tools yet. One of the Windows admins I work with mentioned that he noticed a development guest needed a tools upgrade, and since the server needed a reboot anyway, he decided to click the ‘Upgrade Tools’ button from within the guest OS — but nothing appeared to happen. After a few minutes he tried again, and when nothing happened he rebooted the virtual machine manually. When the same issue appeared on another virtual machine a few days later he asked me if something was going on.

After some testing of my own, I was able to recreate and confirm this issue.

If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

Here are the details:

  • Windows guest (confirmed with XP and 2003)
  • VMware Tools 8.3.7-381511 (version number 8295)
  • ESXi 5.0 Update 1 (build 623860)
  • Symptom: Clicking ‘Upgrade Tools’ from within the guest OS does not update VMware Tools. No error message or warning is displayed.
  • Selecting ‘VM > Guest > Install/Upgrade Tools…’ from within vSphere Client works as expected.
  •  
    I tried upgrading the tools to the ESXi 5.0 non-Update 1 version [VMware Tools 8.6.0-515842 (version number 8384)]. The tools were still outdated, but on a more recent version. After those tools were installed, I then attempted to upgrade the tools from within the guest and received the error message: “Update Tools failed. Edit the virtual machine’s vmx file, add the line below and try again. Please read KB article 1714 on tips for editing a vmx file. isolation.tools.guestInitiatedUpgrade.disable=FALSE.” Here is a screenshot of that error message

    Somewhere along the line, the default option was changed to prevent tools upgrades initiated from within guest operating systems. The error handling code for this error must not exist in older tools versions. If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

    Instead of updating the vmx file (as suggested by the error message — and this KB article), I would recommend breaking out some PowerCLI. Here is code that will change the guest initiated upgrade behavior back to the previous 4.x style for a virtual machine named 2k3-test01:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    $gIU = New-Object VMware.Vim.optionvalue
    $gIU.Key="isolation.tools.guestInitiatedUpgrade.disable"
    $gIU.Value="FALSE"

    #Create a Machine Config Spec using the three option values specified above
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $vmConfigSpec.extraconfig += $gIU

    #Get a VM View collection of all the VMs that need to have these options
    $vms = get-view -viewtype virtualmachine |where {$_.name -eq "2k3-test01"} # remove 'where' statement to update all virtual machines
    $vms | %{ $_.ReconfigVM($vmConfigSpec) }

    Use PowerCLI to mark disk as as SSD

    I’ve recently been working with a few new ESXi hosts. These systems have a pair of solid state disks that I planned to use for host cache. However, ESXi 5.0 did not detect these disks as an SSD type. I found a couple articles describing how to trick ESXi into thinking a data store was SSD:

    Yellow-Bricks: Swap to host cache aka swap to SSD
    VirtuallyGhetto: How to Trick ESXi 5 in seeing an SSD Datastore

    After confirming with my third party vendors that this was a supported configuration, I tested and confirmed that using the enable_ssd option would resolve my issues. Since my host build script is written in PowerCLI, I wanted to re-write the vMA commands using Get-EsxCli.

    1
    2
    3
    4
    5
    6
    $localDisk = Get-ScsiLun | where {$_.ExtensionData.DisplayName -match "Local LSI Disk"}
    $canName = $localDisk.CanonicalName
    $esxcli = Get-EsxCli
    $satp = ($esxcli.storage.nmp.device.list() | where {$_.Device -eq $canName }).StorageArrayType
    $esxcli.storage.nmp.satp.rule.add($null,$null,$null,$canname,$null,$null,$null,"enable_ssd",$null,$null,$satp,$null,$null,$null)
    $esxcli.storage.core.claiming.reclaim($canName)

    The local VMFS data store needs to be created first. I encountered some errors when running the commands against an unformatted/RAW device.

    PowerCLI, say hello to UCS PowerTool

    I’ve recently been digging into Cisco’s Unified Computing System (UCS). I wanted a report showing which ESXi host was in each chassis. I don’t have a lot of blades (yet) so making this list by hand wouldn’t have been a lot of work. However, I had heard about the UCS PowerTool (http://developer.cisco.com/web/unifiedcomputing/pshell-download) and thought this would be a perfect chance to kick the tires. What I had in mind was something that would look like this:

    VMHostName              UCSProfileName     UCSBladeSlot
    ----------              --------------     ------------
    esxbl51.bwuch.local     esxbl5-1           sys/chassis-1/blade-8
    esxbl52.bwuch.local     esxbl5-2           sys/chassis-2/blade-8
    esxbl53.bwuch.local     esxbl5-3           sys/chassis-1/blade-7
    esxbl54.bwuch.local     esxbl5-4           sys/chassis-2/blade-7
    esxbl55.bwuch.local     esxbl5-5           sys/chassis-1/blade-6
    esxbl56.bwuch.local     esxbl5-6           sys/chassis-2/blade-6
    

    Unfortunately the ESXi host name I wanted could not be found in the UCS interface. I was able to find the UUID of the service profile, which maps directly to the UUID in the system.hardware section of Get-View cmdlet for PowerCLI. Since both of the tools are powershell based it isn’t very tough to use them together. (The following code may look long, but that’s because of a lot of comments.)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    # Author: Brian Wuchner
    # Date: 2012/03/03
    # Description: This is a sample inventory report combining information from Cisco UCS Manager with information from VMware vCenter
    # The report uses Windows PowerShell and the following modules available from each vendor:
    #   Cisco UCS PowerTool: http://developer.cisco.com/web/unifiedcomputing/pshell-download
    #   VMware PowerCLI: http://vmware.com/go/powercli

    # Define UCS connection details
    $ucsSysName = "ucs.bwuch.local"
    $ucsUserName = "admin"
    $ucsPassword = "ucsadminpassword"

    # Define vCenter connection info
    $vcSysName = "vcenter.bwuch.local"

    # Import the UCS PowerTool module and the VMware PowerCLI snapin
    Import-Module "C:\Program Files (x86)\Cisco\Cisco UCS PowerTool\CiscoUcsPS.psd1"
    Add-PSSnapin vmware.VimAutomation.core

    # The UCSM connection requires a PSCredential to login, so we must convert our plain text password to make an object
    $ucsPassword = ConvertTo-SecureString -String $ucsPassword -AsPlainText -Force
    $cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $ucsUserName, $ucsPassword

    # Create connection to UCS system
    $ucsConnection = Connect-Ucs $ucsSysName -Credential $cred

    # Collect the deployed service profiles and convert them to a hash table for easy reference
    $serviceProfiles = Get-UcsServiceProfile | where {$_.AssignState -eq "assigned"} | select name, PnDn, SrcTemplName, Uuid
    $spHT = $serviceProfiles | Group-Object uuid -AsHashTable -AsString

    # Thats all we need from UCS, lets go ahead and logout
    $ucsConnection = Disconnect-Ucs

    # Connect to vCenter
    $vcConnection = Connect-VIServer $vcSysName # if needed the -user and -password parameters can be passed here, otherwise SSPI is used (current windows credentials)

    # Collect vCenter information on Cisco hardware
    $hs = Get-View -ViewType HostSystem -Property Name, Summary.Hardware, Config.Product -Filter @{"Summary.Hardware.Vendor"="Cisco Systems Inc"}

    # We are done with vCenter, lets go ahead and logout
    $vcConnection = Disconnect-VIServer * -Confirm:$false

    # Put the to lists of information together
    $myReport = @() # collection to store all the results
    $hs | %{ # loop through the host system vCenter information
        # create a variable to store the specific Cisco UCS record by UUID
        $thisCiscoDevice = $spHT[$_.Summary.Hardware.Uuid][0]
       
        # populate an object using all the applicable fields needed
        $myReport += New-Object -Type PSObject -Property @{
            VMHostName = $_.Name
            VMHostVersion = $_.Config.Product.FullName
            CpuModel = $_.Summary.Hardware.CpuModel
            CpuMhz = $_.Summary.Hardware.CpuMhz
            NumCpuPkgs = $_.Summary.Hardware.NumCpuPkgs
            NumCpuCores = $_.Summary.Hardware.NumCpuCores
            UCSBladeSlot = $thisCiscoDevice.PnDn
            UCSProfileName = $thisCiscoDevice.Name
            UCSProfileTemplate = $thisCiscoDevice.SrcTemplName
        } # end new-object
    } # end foreach-object

    # Select a few of the columns in the order we want to see them...display to screen
    $myReport | Select VMHostName, UCSProfileName, UCSBladeSlot

    For my report needs, I only needed three columns. I added a handful of additional properties to the $myReport variable for demonstration purposes, but there are many more properties available.

    CpuModel           : Intel(R) Xeon(R) CPU E7- 2850  @ 2.00GHz
    NumCpuCores        : 20
    UCSProfileTemplate : VMware_Blade_5x
    UCSBladeSlot       : sys/chassis-1/blade-8
    CpuMhz             : 1997
    UCSProfileName     : esxbl5-1
    NumCpuPkgs         : 2
    VMHostVersion      : VMware ESXi 5.0.0 build-515841
    VMHostName         : esxbl51.bwuch.local
    

    How vCenter assigns MAC addresses

    I’ve recently posted several entries on MAC address conflicts within vCenter:

    Multiple vCenters and MAC address conflicts
    Virtual Machines with Duplicate MAC addresses

    If you look at the first post (and VMware KB 1024025) you’ll notice that vCenter uses a random instance ID to come up with the automatically generated MAC addresses. If you look at the second post, you’ll notice that it is possible for virtual machines to retain their automatic MAC assignment when moving between vCenters — and for the source vCenter to reissue that MAC. This can lead to problems with multiple MAC addresses on the same network.

    What we need to know now is which virtual machines have automatic MAC addresses generated by another vCenter.

    I looked at a couple of vCenter instance IDs and compared them to the automatically generated MAC addresses. Here is what I found:
    vCenter with instance ID 50 will generate a MAC prefix of 00:50:56:b2
    vCenter with instance ID 60 will generate a MAC prefix of 00:50:56:bc

    The important things to note are that:
    1.) only the fourth octet of the MAC changed
    2.) the fourth octet changed by exactly 10 in hexadecimal

    Converting B2 in hexadecimal returned 178 in decimal. BC in hexadecimal is 188 in decimal. Since the fourth octet of the MAC address grows linearly with the instance ID, I took 178 and subtracted the instance ID of 50, which returned 128. To prove this idea, I then changed the instance ID on a test vCenter to 0, rebooted and then added a network adapter to a VM. As expected, the fourth octet of the MAC address was 80 (or 128 in decimal).

    The automatically generated MAC address has a fourth octet of 128 + the vCenter instance ID converted to hexadecimal.

    With this understanding of how the MAC address is generated, I turned back to PowerCLI to help figure out which virtual machines had automatically assigned MAC addresses from other vCenters.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    # http://communities.vmware.com/thread/339358
    # Modified to find MAC addresses generated from other vCenter servers
    $myreport = $null
    $global:DefaultVIServers | %{
        Write-Host "Starting vCenter $($_.name)"
        $si = Get-View ServiceInstance -Server $_.name
        $set = Get-View $si.Content.Setting -Server $_.name
        $vCenterInstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq "instance.id"}).value
        $vCenterMac4 = [Convert]::ToString((128 + $vCenterInstanceID), 16)
        $vCenterMacPrefix = "00:50:56:$vCenterMac4"
        Write-Host "Expected MAC prefix of $vCenterMacPrefix"
        $vCenterName = $_.Name
       
        $myreport += Get-View -ViewType VirtualMachine -Server $_.name -Property Name, Config.Hardware.Device -Filter @{"Config.Template"="False"} | %{
            $viewThisVM = $_
            $viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
                New-Object -Type PSObject -Property @{
                    VMname = $viewThisVM.Name
                    NICtype = $_.GetType().Name
                    MacAddress = $_.MacAddress
                    AddressType = $_.AddressType
                    vCenterName = $vCenterName
                    vCenterMacPrefix = $vCenterMacPrefix
                } ## end new-object
            } ## end foreach-object
        }
    }

    $myreport | where {$_.MacAddress -notmatch "^$($_.vCenterMacPrefix)" -AND $_.AddressType -eq "assigned"}
    #The caret in the notmatch comparison signifies the start of string in regex

    The returned list includes the virtual machines that have automatically assigned MAC addresses that do not match their current vCenter. I know exactly which virtual machines need new/modified MAC addresses to resolve my duplicate MAC issues — without them coming back in the future.

    I have tested the following code to get vCenter to generate a new automatic MAC address for a virtual machine:

    1
    2
    3
    4
    $thisAdapter = Get-NetworkAdapter VMNAME
    $thisAdapter.ExtensionData.AddressType = "Generated"
    $thisAdapter.ExtensionData.MacAddress = ""
    Set-NetworkAdapter $thisAdapter -confirm:$false

    Virtual Machines will need to be powered off/on for the guest operating system to be made aware of this change. Testing shows that Windows virtual machines retain static IP configuration after this change, but Linux guests detect a new network adapter and required re-setting IP information. Your mileage may vary, as always, test before making changes in your production environment.