Why does the ‘Update Tools’ button not work from within VMware Tools?

I recently upgraded an ESXi 4.1 Update 1 cluster to ESXi 5.0 Update 1 — but have not upgraded the tools yet. One of the Windows admins I work with mentioned that he noticed a development guest needed a tools upgrade, and since the server needed a reboot anyway, he decided to click the ‘Upgrade Tools’ button from within the guest OS — but nothing appeared to happen. After a few minutes he tried again, and when nothing happened he rebooted the virtual machine manually. When the same issue appeared on another virtual machine a few days later he asked me if something was going on.

After some testing of my own, I was able to recreate and confirm this issue.

If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

Here are the details:

 
I tried upgrading the tools to the ESXi 5.0 non-Update 1 version [VMware Tools 8.6.0-515842 (version number 8384)]. The tools were still outdated, but on a more recent version. After those tools were installed, I then attempted to upgrade the tools from within the guest and received the error message: “Update Tools failed. Edit the virtual machine’s vmx file, add the line below and try again. Please read KB article 1714 on tips for editing a vmx file. isolation.tools.guestInitiatedUpgrade.disable=FALSE.” Here is a screenshot of that error message

Somewhere along the line, the default option was changed to prevent tools upgrades initiated from within guest operating systems. The error handling code for this error must not exist in older tools versions. If you are using tools from ESXi 4.1 on virtual machine guest running on an ESXi 5.0 Update 1 host, selecting ‘Update Tools’ from within the guest operating system will not work. This is because the running tools version does not understand the setting ‘isolation.tools.guestInitiatedUpgrade.disable’ and does not display an appropriate error message.

Instead of updating the vmx file (as suggested by the error message — and this KB article), I would recommend breaking out some PowerCLI. Here is code that will change the guest initiated upgrade behavior back to the previous 4.x style for a virtual machine named 2k3-test01:
[cc lang=”powershell”]
$gIU = New-Object VMware.Vim.optionvalue
$gIU.Key=”isolation.tools.guestInitiatedUpgrade.disable”
$gIU.Value=”FALSE”

#Create a Machine Config Spec using the three option values specified above
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraconfig += $gIU

#Get a VM View collection of all the VMs that need to have these options
$vms = get-view -viewtype virtualmachine |where {$_.name -eq “2k3-test01”} # remove ‘where’ statement to update all virtual machines
$vms | %{ $_.ReconfigVM($vmConfigSpec) }
[/cc]