Enterprise Admins.org

I recently upgraded an ESXi 4.1 Update 1 cluster to ESXi 5.0 Update 1 — but have not upgraded the tools yet. One of the Windows admins I work with mentioned that he noticed a development guest needed a tools upgrade, and since the server needed a reboot anyway, he decided to click the ‘Upgrade Tools’ button from within the guest OS — but nothing appeared to happen. After a few minutes he tried again, and when nothing happened he rebooted the virtual machine manually. When the same issue appeared on another virtual machine a few days later he asked me if something was going on.

After some testing of my own, I was able to recreate and confirm this issue.

If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

Here are the details:

 
I tried upgrading the tools to the ESXi 5.0 non-Update 1 version [VMware Tools 8.6.0-515842 (version number 8384)]. The tools were still outdated, but on a more recent version. After those tools were installed, I then attempted to upgrade the tools from within the guest and received the error message: “Update Tools failed. Edit the virtual machine’s vmx file, add the line below and try again. Please read KB article 1714 on tips for editing a vmx file. isolation.tools.guestInitiatedUpgrade.disable=FALSE.” Here is a screenshot of that error message

Somewhere along the line, the default option was changed to prevent tools upgrades initiated from within guest operating systems. The error handling code for this error must not exist in older tools versions. If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

Instead of updating the vmx file (as suggested by the error message — and this KB article), I would recommend breaking out some PowerCLI. Here is code that will change the guest initiated upgrade behavior back to the previous 4.x style for a virtual machine named 2k3-test01:
[cc lang=”powershell”]
$gIU = New-Object VMware.Vim.optionvalue
$gIU.Key=”isolation.tools.guestInitiatedUpgrade.disable”
$gIU.Value=”FALSE”

#Create a Machine Config Spec using the three option values specified above
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraconfig += $gIU

#Get a VM View collection of all the VMs that need to have these options
$vms = get-view -viewtype virtualmachine |where {$_.name -eq “2k3-test01”} # remove ‘where’ statement to update all virtual machines
$vms | %{ $_.ReconfigVM($vmConfigSpec) }
[/cc]

I’ve recently been working with a few new ESXi hosts. These systems have a pair of solid state disks that I planned to use for host cache. However, ESXi 5.0 did not detect these disks as an SSD type. I found a couple articles describing how to trick ESXi into thinking a data store was SSD:

Yellow-Bricks: Swap to host cache aka swap to SSD
VirtuallyGhetto: How to Trick ESXi 5 in seeing an SSD Datastore

After confirming with my third party vendors that this was a supported configuration, I tested and confirmed that using the enable_ssd option would resolve my issues. Since my host build script is written in PowerCLI, I wanted to re-write the vMA commands using Get-EsxCli.

[cc lang=”powershell”]
$localDisk = Get-ScsiLun | where {$_.ExtensionData.DisplayName -match “Local LSI Disk”}
$canName = $localDisk.CanonicalName
$esxcli = Get-EsxCli
$satp = ($esxcli.storage.nmp.device.list() | where {$_.Device -eq $canName }).StorageArrayType
$esxcli.storage.nmp.satp.rule.add($null,$null,$null,$canname,$null,$null,$null,”enable_ssd”,$null,$null,$satp,$null,$null,$null)
$esxcli.storage.core.claiming.reclaim($canName)
[/cc]

The local VMFS data store needs to be created first. I encountered some errors when running the commands against an unformatted/RAW device.

I’ve recently posted several entries on MAC address conflicts within vCenter:

Multiple vCenters and MAC address conflicts
Virtual Machines with Duplicate MAC addresses

If you look at the first post (and VMware KB 1024025) you’ll notice that vCenter uses a random instance ID to come up with the automatically generated MAC addresses. If you look at the second post, you’ll notice that it is possible for virtual machines to retain their automatic MAC assignment when moving between vCenters — and for the source vCenter to reissue that MAC. This can lead to problems with multiple MAC addresses on the same network.

What we need to know now is which virtual machines have automatic MAC addresses generated by another vCenter.

I looked at a couple of vCenter instance IDs and compared them to the automatically generated MAC addresses. Here is what I found:
vCenter with instance ID 50 will generate a MAC prefix of 00:50:56:b2
vCenter with instance ID 60 will generate a MAC prefix of 00:50:56:bc

The important things to note are that:
1.) only the fourth octet of the MAC changed
2.) the fourth octet changed by exactly 10 in hexadecimal

Converting B2 in hexadecimal returned 178 in decimal. BC in hexadecimal is 188 in decimal. Since the fourth octet of the MAC address grows linearly with the instance ID, I took 178 and subtracted the instance ID of 50, which returned 128. To prove this idea, I then changed the instance ID on a test vCenter to 0, rebooted and then added a network adapter to a VM. As expected, the fourth octet of the MAC address was 80 (or 128 in decimal).

The automatically generated MAC address has a fourth octet of 128 + the vCenter instance ID converted to hexadecimal.

With this understanding of how the MAC address is generated, I turned back to PowerCLI to help figure out which virtual machines had automatically assigned MAC addresses from other vCenters.

[cc lang=”powershell”]
# http://communities.vmware.com/thread/339358
# Modified to find MAC addresses generated from other vCenter servers
$myreport = $null
$global:DefaultVIServers | %{
Write-Host “Starting vCenter $($_.name)”
$si = Get-View ServiceInstance -Server $_.name
$set = Get-View $si.Content.Setting -Server $_.name
$vCenterInstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq “instance.id”}).value
$vCenterMac4 = [Convert]::ToString((128 + $vCenterInstanceID), 16)
$vCenterMacPrefix = “00:50:56:$vCenterMac4”
Write-Host “Expected MAC prefix of $vCenterMacPrefix”
$vCenterName = $_.Name

$myreport += Get-View -ViewType VirtualMachine -Server $_.name -Property Name, Config.Hardware.Device -Filter @{“Config.Template”=”False”} | %{
$viewThisVM = $_
$viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
New-Object -Type PSObject -Property @{
VMname = $viewThisVM.Name
NICtype = $_.GetType().Name
MacAddress = $_.MacAddress
AddressType = $_.AddressType
vCenterName = $vCenterName
vCenterMacPrefix = $vCenterMacPrefix
} ## end new-object
} ## end foreach-object
}
}

$myreport | where {$_.MacAddress -notmatch “^$($_.vCenterMacPrefix)” -AND $_.AddressType -eq “assigned”}
#The caret in the notmatch comparison signifies the start of string in regex
[/cc]

The returned list includes the virtual machines that have automatically assigned MAC addresses that do not match their current vCenter. I know exactly which virtual machines need new/modified MAC addresses to resolve my duplicate MAC issues — without them coming back in the future.

I have tested the following code to get vCenter to generate a new automatic MAC address for a virtual machine:
[cc lang=”powershell”]
$thisAdapter = Get-NetworkAdapter VMNAME
$thisAdapter.ExtensionData.AddressType = “Generated”
$thisAdapter.ExtensionData.MacAddress = “”
Set-NetworkAdapter $thisAdapter -confirm:$false
[/cc]
Virtual Machines will need to be powered off/on for the guest operating system to be made aware of this change. Testing shows that Windows virtual machines retain static IP configuration after this change, but Linux guests detect a new network adapter and required re-setting IP information. Your mileage may vary, as always, test before making changes in your production environment.

I was recently asked if it would be possible for two virtual machines to be automatically assigned the same MAC addresses. Knowing that vCenter handles these assignments and ensures uniqueness, I figured it wouldn’t be possible…but it got me to thinking. Since I have multiple vCenters, what would prevent each vCenter from reusing MAC addresses? My search lead to this KB article: http://kb.vmware.com/kb/1024025 which in turn resulted in this blog post: http://enterpriseadmins.org/blog/scripting/multiple-vcenters-and-mac-address-conflicts/.

Question answered — this would not automatically happen.

A few days later, this same guy shows up with two guest names and one MAC address asking me who actually has that MAC? I check — then double check — only to confirm that both VMs have that same MAC address automatically assigned! One interesting thing to note was that each VM was in fact in a different vCenter.

Thats when I turned to some solid code I remembered seeing from mattboren on the VMware PowerCLI communities page (http://communities.vmware.com/thread/339358). A few slight modifications and I had the super fast code shown below to kick out a list of any duplicate MAC addresses:

[cc lang=”powershell”]
# http://communities.vmware.com/thread/339358
# Modified to find duplicate MAC addresses from virtual machines across vCenter environments.
$myreport = Get-View -ViewType VirtualMachine -Property Name, Config.Hardware.Device -Filter @{“Config.Template”=”False”} | %{
$viewThisVM = $_
$viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
New-Object -Type PSObject -Property @{
VMname = $viewThisVM.Name
NICtype = $_.GetType().Name
MacAddress = $_.MacAddress
AddressType = $_.AddressType
vCenterAPI = $viewThisVM.Client.serviceUrl -replace(“https://”,””) -replace(“:443/sdk”,””)
} ## end new-object
} ## end foreach-object
}
$myreport | group MacAddress |where {$_.Count -gt 1} |select -ExpandProperty Group | Select VMname, vCenterAPI, AddressType, MacAddress, NICtype
[/cc]
Side note, the Client.serviceUrl property was something I found pretty quick…I wouldn’t call that a ‘best practice’ on figuring out which vCenter an object is part of. Also, the code that is removing the https and 443/sdk is a very bad approach and isn’t going to work in all scenarios. I just wanted something quick and dirty to get the info I needed.

The above code showed that the problem was much larger than just 2 virtual machines — I had nearly 50 virtual machines sharing 25 MAC addresses!

After looking into several of the virtual machines, I started seeing a pattern — a pattern I created a few months back. I had moved a bunch of virtual machines from my production vCenter a disaster recovery site vCenter (separated for SRM) on the same network. To limit the amount of downtime, the move was completed by presenting one temporary LUN to hosts in each vCenter, using storage VMotion to move the data, then during a change window the VMs were removed from inventory in production and added to the disaster recovery inventory. Virtual machines were then storage VMotioned to the proper LUNs and the temporary LUN removed. During this move, virtual machines kept their original MAC addresses, but as they had been removed from production inventory, the production vCenter was able to reissue those unused addresses.

There were also a couple examples where a similar approach was used to ‘fail back’ select virtual machines after a disaster recovery test, leaving the production vCenter having MAC addresses generated by the disaster recovery site vCenter.

I have a little more information on this issue — including details on how vCenter generates MAC addressess — that I will share in another post. Stay tuned!