Enterprise Admins.org

I’ve recently posted several entries on MAC address conflicts within vCenter:

Multiple vCenters and MAC address conflicts
Virtual Machines with Duplicate MAC addresses

If you look at the first post (and VMware KB 1024025) you’ll notice that vCenter uses a random instance ID to come up with the automatically generated MAC addresses. If you look at the second post, you’ll notice that it is possible for virtual machines to retain their automatic MAC assignment when moving between vCenters — and for the source vCenter to reissue that MAC. This can lead to problems with multiple MAC addresses on the same network.

What we need to know now is which virtual machines have automatic MAC addresses generated by another vCenter.

I looked at a couple of vCenter instance IDs and compared them to the automatically generated MAC addresses. Here is what I found:
vCenter with instance ID 50 will generate a MAC prefix of 00:50:56:b2
vCenter with instance ID 60 will generate a MAC prefix of 00:50:56:bc

The important things to note are that:
1.) only the fourth octet of the MAC changed
2.) the fourth octet changed by exactly 10 in hexadecimal

Converting B2 in hexadecimal returned 178 in decimal. BC in hexadecimal is 188 in decimal. Since the fourth octet of the MAC address grows linearly with the instance ID, I took 178 and subtracted the instance ID of 50, which returned 128. To prove this idea, I then changed the instance ID on a test vCenter to 0, rebooted and then added a network adapter to a VM. As expected, the fourth octet of the MAC address was 80 (or 128 in decimal).

The automatically generated MAC address has a fourth octet of 128 + the vCenter instance ID converted to hexadecimal.

With this understanding of how the MAC address is generated, I turned back to PowerCLI to help figure out which virtual machines had automatically assigned MAC addresses from other vCenters.

# http://communities.vmware.com/thread/339358
# Modified to find MAC addresses generated from other vCenter servers
$myreport = $null
$global:DefaultVIServers | %{
	Write-Host "Starting vCenter $($_.name)"
	$si = Get-View ServiceInstance -Server $_.name
	$set = Get-View $si.Content.Setting -Server $_.name
	$vCenterInstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq "instance.id"}).value
	$vCenterMac4 = [Convert]::ToString((128 + $vCenterInstanceID), 16)
	$vCenterMacPrefix = "00:50:56:$vCenterMac4"
	Write-Host "Expected MAC prefix of $vCenterMacPrefix"
	$vCenterName = $_.Name
	
	$myreport += Get-View -ViewType VirtualMachine -Server $_.name -Property Name, Config.Hardware.Device -Filter @{"Config.Template"="False"} | %{
	    $viewThisVM = $_
	    $viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
	        New-Object -Type PSObject -Property @{
	            VMname = $viewThisVM.Name
	            NICtype = $_.GetType().Name
	            MacAddress = $_.MacAddress
	            AddressType = $_.AddressType
				vCenterName = $vCenterName
				vCenterMacPrefix = $vCenterMacPrefix
	        } ## end new-object
	    } ## end foreach-object
	}
}

$myreport | where {$_.MacAddress -notmatch "^$($_.vCenterMacPrefix)" -AND $_.AddressType -eq "assigned"}
#The caret in the notmatch comparison signifies the start of string in regex

The returned list includes the virtual machines that have automatically assigned MAC addresses that do not match their current vCenter. I know exactly which virtual machines need new/modified MAC addresses to resolve my duplicate MAC issues — without them coming back in the future.

I have tested the following code to get vCenter to generate a new automatic MAC address for a virtual machine:

$thisAdapter = Get-NetworkAdapter VMNAME
$thisAdapter.ExtensionData.AddressType = "Generated"
$thisAdapter.ExtensionData.MacAddress = ""
Set-NetworkAdapter $thisAdapter -confirm:$false

Virtual Machines will need to be powered off/on for the guest operating system to be made aware of this change. Testing shows that Windows virtual machines retain static IP configuration after this change, but Linux guests detect a new network adapter and required re-setting IP information. Your mileage may vary, as always, test before making changes in your production environment.

I was recently asked if it would be possible for two virtual machines to be automatically assigned the same MAC addresses. Knowing that vCenter handles these assignments and ensures uniqueness, I figured it wouldn’t be possible…but it got me to thinking. Since I have multiple vCenters, what would prevent each vCenter from reusing MAC addresses? My search lead to this KB article: http://kb.vmware.com/kb/1024025 which in turn resulted in this blog post: http://enterpriseadmins.org/blog/scripting/multiple-vcenters-and-mac-address-conflicts/.

Question answered — this would not automatically happen.

A few days later, this same guy shows up with two guest names and one MAC address asking me who actually has that MAC? I check — then double check — only to confirm that both VMs have that same MAC address automatically assigned! One interesting thing to note was that each VM was in fact in a different vCenter.

Thats when I turned to some solid code I remembered seeing from mattboren on the VMware PowerCLI communities page (http://communities.vmware.com/thread/339358). A few slight modifications and I had the super fast code shown below to kick out a list of any duplicate MAC addresses:

# http://communities.vmware.com/thread/339358
# Modified to find duplicate MAC addresses from virtual machines across vCenter environments.
$myreport = Get-View -ViewType VirtualMachine -Property Name, Config.Hardware.Device -Filter @{"Config.Template"="False"} | %{
    $viewThisVM = $_
    $viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
        New-Object -Type PSObject -Property @{
            VMname = $viewThisVM.Name
            NICtype = $_.GetType().Name
            MacAddress = $_.MacAddress
            AddressType = $_.AddressType
            vCenterAPI = $viewThisVM.Client.serviceUrl -replace("https://","") -replace(":443/sdk","")
        } ## end new-object
    } ## end foreach-object
} 
$myreport | group MacAddress |where {$_.Count -gt 1} |select -ExpandProperty Group | Select VMname, vCenterAPI, AddressType, MacAddress, NICtype

Side note, the Client.serviceUrl property was something I found pretty quick…I wouldn’t call that a ‘best practice’ on figuring out which vCenter an object is part of. Also, the code that is removing the https and 443/sdk is a very bad approach and isn’t going to work in all scenarios. I just wanted something quick and dirty to get the info I needed.

The above code showed that the problem was much larger than just 2 virtual machines — I had nearly 50 virtual machines sharing 25 MAC addresses!

After looking into several of the virtual machines, I started seeing a pattern — a pattern I created a few months back. I had moved a bunch of virtual machines from my production vCenter a disaster recovery site vCenter (separated for SRM) on the same network. To limit the amount of downtime, the move was completed by presenting one temporary LUN to hosts in each vCenter, using storage VMotion to move the data, then during a change window the VMs were removed from inventory in production and added to the disaster recovery inventory. Virtual machines were then storage VMotioned to the proper LUNs and the temporary LUN removed. During this move, virtual machines kept their original MAC addresses, but as they had been removed from production inventory, the production vCenter was able to reissue those unused addresses.

There were also a couple examples where a similar approach was used to ‘fail back’ select virtual machines after a disaster recovery test, leaving the production vCenter having MAC addresses generated by the disaster recovery site vCenter.

I have a little more information on this issue — including details on how vCenter generates MAC addressess — that I will share in another post. Stay tuned!