Enterprise Admins.org

I was recently asked if it would be possible for two virtual machines to be automatically assigned the same MAC addresses. Knowing that vCenter handles these assignments and ensures uniqueness, I figured it wouldn’t be possible…but it got me to thinking. Since I have multiple vCenters, what would prevent each vCenter from reusing MAC addresses? My search lead to this KB article: http://kb.vmware.com/kb/1024025 which in turn resulted in this blog post: http://enterpriseadmins.org/blog/scripting/multiple-vcenters-and-mac-address-conflicts/.

Question answered — this would not automatically happen.

A few days later, this same guy shows up with two guest names and one MAC address asking me who actually has that MAC? I check — then double check — only to confirm that both VMs have that same MAC address automatically assigned! One interesting thing to note was that each VM was in fact in a different vCenter.

Thats when I turned to some solid code I remembered seeing from mattboren on the VMware PowerCLI communities page (http://communities.vmware.com/thread/339358). A few slight modifications and I had the super fast code shown below to kick out a list of any duplicate MAC addresses:

[cc lang=”powershell”]
# http://communities.vmware.com/thread/339358
# Modified to find duplicate MAC addresses from virtual machines across vCenter environments.
$myreport = Get-View -ViewType VirtualMachine -Property Name, Config.Hardware.Device -Filter @{“Config.Template”=”False”} | %{
$viewThisVM = $_
$viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
New-Object -Type PSObject -Property @{
VMname = $viewThisVM.Name
NICtype = $_.GetType().Name
MacAddress = $_.MacAddress
AddressType = $_.AddressType
vCenterAPI = $viewThisVM.Client.serviceUrl -replace(“https://”,””) -replace(“:443/sdk”,””)
} ## end new-object
} ## end foreach-object
}
$myreport | group MacAddress |where {$_.Count -gt 1} |select -ExpandProperty Group | Select VMname, vCenterAPI, AddressType, MacAddress, NICtype
[/cc]
Side note, the Client.serviceUrl property was something I found pretty quick…I wouldn’t call that a ‘best practice’ on figuring out which vCenter an object is part of. Also, the code that is removing the https and 443/sdk is a very bad approach and isn’t going to work in all scenarios. I just wanted something quick and dirty to get the info I needed.

The above code showed that the problem was much larger than just 2 virtual machines — I had nearly 50 virtual machines sharing 25 MAC addresses!

After looking into several of the virtual machines, I started seeing a pattern — a pattern I created a few months back. I had moved a bunch of virtual machines from my production vCenter a disaster recovery site vCenter (separated for SRM) on the same network. To limit the amount of downtime, the move was completed by presenting one temporary LUN to hosts in each vCenter, using storage VMotion to move the data, then during a change window the VMs were removed from inventory in production and added to the disaster recovery inventory. Virtual machines were then storage VMotioned to the proper LUNs and the temporary LUN removed. During this move, virtual machines kept their original MAC addresses, but as they had been removed from production inventory, the production vCenter was able to reissue those unused addresses.

There were also a couple examples where a similar approach was used to ‘fail back’ select virtual machines after a disaster recovery test, leaving the production vCenter having MAC addresses generated by the disaster recovery site vCenter.

I have a little more information on this issue — including details on how vCenter generates MAC addressess — that I will share in another post. Stay tuned!

VMware KB article 1024025 explains an issue with MAC address conflicts due to multiple vCenter configurations with the same instance ID. Here is a technical description straight from the KB article:

Each vCenter Server system has a vCenter Server instance ID. This ID is a number between 0 and 63 that is randomly generated at installation time, but can be reconfigured after installation. 

vCenter Server uses the vCenter instance ID to generate MAC addresses and UUIDs for virtual machines. If two vCenter Server systems have the same vCenter instance ID, they might generate identical MAC addresses for virtual machines. This can cause conflicts if the virtual machines are on the same network, leading to packet loss and other problems. 

The solution is to verify that all of your vCenter configurations use different instance IDs for MAC address generation. This wouldn’t be difficult if you only had two or three vCenters, but as you scale out it becomes more of a challenge. Here is some PowerCLI code that will help you check for duplicate instance IDs.

[cc lang=”Powershell”]
# Virtual machine MAC address conflicts – http://kb.vmware.com/kb/1024025
$vCentersToCheck = “vcenter1″,”vcenter2″,”vcenter3”

$config = Get-PowerCLIConfiguration
if($config.DefaultVIServerMode -eq “Single”){Set-PowerCLIConfiguration -DefaultVIServerMode Multiple}
foreach ($vCenter in $vCentersToCheck){connect-viserver $vCenter | Out-Null}

$myReport = @()
$global:DefaultVIServers | %{
$si = Get-View ServiceInstance -Server $_.Name
$set = Get-View $si.Content.Setting -Server $_.Name
$myReport += new-object -type PSObject -Property @{
InstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq “instance.id”}).value
vCenter = $_.Name
}
}

$myResults = $myReport | Group-Object InstanceID | where {$_.Count -gt 1} | select -ExpandProperty Group | select vCenter, InstanceID
if ($myResults.count -gt 1) { $myResults } else { “No duplicate InstanceIDs were identified in the vCenters checked” }
[/cc]

By default, the script will let you know if you have any duplicates. However, if you want to know what all instance IDs are in use, they are stored in the $myReport variable. You can type “$myReport” (without the quotes) in the console window after this script completes.