How vCenter assigns MAC addresses

I’ve recently posted several entries on MAC address conflicts within vCenter:

Multiple vCenters and MAC address conflicts
Virtual Machines with Duplicate MAC addresses

If you look at the first post (and VMware KB 1024025) you’ll notice that vCenter uses a random instance ID to come up with the automatically generated MAC addresses. If you look at the second post, you’ll notice that it is possible for virtual machines to retain their automatic MAC assignment when moving between vCenters — and for the source vCenter to reissue that MAC. This can lead to problems with multiple MAC addresses on the same network.

What we need to know now is which virtual machines have automatic MAC addresses generated by another vCenter.

I looked at a couple of vCenter instance IDs and compared them to the automatically generated MAC addresses. Here is what I found:
vCenter with instance ID 50 will generate a MAC prefix of 00:50:56:b2
vCenter with instance ID 60 will generate a MAC prefix of 00:50:56:bc

The important things to note are that:
1.) only the fourth octet of the MAC changed
2.) the fourth octet changed by exactly 10 in hexadecimal

Converting B2 in hexadecimal returned 178 in decimal. BC in hexadecimal is 188 in decimal. Since the fourth octet of the MAC address grows linearly with the instance ID, I took 178 and subtracted the instance ID of 50, which returned 128. To prove this idea, I then changed the instance ID on a test vCenter to 0, rebooted and then added a network adapter to a VM. As expected, the fourth octet of the MAC address was 80 (or 128 in decimal).

The automatically generated MAC address has a fourth octet of 128 + the vCenter instance ID converted to hexadecimal.

With this understanding of how the MAC address is generated, I turned back to PowerCLI to help figure out which virtual machines had automatically assigned MAC addresses from other vCenters.


# http://communities.vmware.com/thread/339358
# Modified to find MAC addresses generated from other vCenter servers
$myreport = $null
$global:DefaultVIServers | %{
	Write-Host "Starting vCenter $($_.name)"
	$si = Get-View ServiceInstance -Server $_.name
	$set = Get-View $si.Content.Setting -Server $_.name
	$vCenterInstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq "instance.id"}).value
	$vCenterMac4 = [Convert]::ToString((128 + $vCenterInstanceID), 16)
	$vCenterMacPrefix = "00:50:56:$vCenterMac4"
	Write-Host "Expected MAC prefix of $vCenterMacPrefix"
	$vCenterName = $_.Name
	
	$myreport += Get-View -ViewType VirtualMachine -Server $_.name -Property Name, Config.Hardware.Device -Filter @{"Config.Template"="False"} | %{
	    $viewThisVM = $_
	    $viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
	        New-Object -Type PSObject -Property @{
	            VMname = $viewThisVM.Name
	            NICtype = $_.GetType().Name
	            MacAddress = $_.MacAddress
	            AddressType = $_.AddressType
				vCenterName = $vCenterName
				vCenterMacPrefix = $vCenterMacPrefix
	        } ## end new-object
	    } ## end foreach-object
	}
}

$myreport | where {$_.MacAddress -notmatch "^$($_.vCenterMacPrefix)" -AND $_.AddressType -eq "assigned"}
#The caret in the notmatch comparison signifies the start of string in regex

The returned list includes the virtual machines that have automatically assigned MAC addresses that do not match their current vCenter. I know exactly which virtual machines need new/modified MAC addresses to resolve my duplicate MAC issues — without them coming back in the future.

I have tested the following code to get vCenter to generate a new automatic MAC address for a virtual machine:


$thisAdapter = Get-NetworkAdapter VMNAME
$thisAdapter.ExtensionData.AddressType = "Generated"
$thisAdapter.ExtensionData.MacAddress = ""
Set-NetworkAdapter $thisAdapter -confirm:$false

Virtual Machines will need to be powered off/on for the guest operating system to be made aware of this change. Testing shows that Windows virtual machines retain static IP configuration after this change, but Linux guests detect a new network adapter and required re-setting IP information. Your mileage may vary, as always, test before making changes in your production environment.

Posted in Scripting, Virtualization | Leave a comment

Virtual Machines with duplicate MAC addresses

I was recently asked if it would be possible for two virtual machines to be automatically assigned the same MAC addresses. Knowing that vCenter handles these assignments and ensures uniqueness, I figured it wouldn’t be possible…but it got me to thinking. Since I have multiple vCenters, what would prevent each vCenter from reusing MAC addresses? My search lead to this KB article: http://kb.vmware.com/kb/1024025 which in turn resulted in this blog post: http://enterpriseadmins.org/blog/scripting/multiple-vcenters-and-mac-address-conflicts/.

Question answered — this would not automatically happen.

A few days later, this same guy shows up with two guest names and one MAC address asking me who actually has that MAC? I check — then double check — only to confirm that both VMs have that same MAC address automatically assigned! One interesting thing to note was that each VM was in fact in a different vCenter.

Thats when I turned to some solid code I remembered seeing from mattboren on the VMware PowerCLI communities page (http://communities.vmware.com/thread/339358). A few slight modifications and I had the super fast code shown below to kick out a list of any duplicate MAC addresses:


# http://communities.vmware.com/thread/339358
# Modified to find duplicate MAC addresses from virtual machines across vCenter environments.
$myreport = Get-View -ViewType VirtualMachine -Property Name, Config.Hardware.Device -Filter @{"Config.Template"="False"} | %{
    $viewThisVM = $_
    $viewThisVM.Config.Hardware.Device | ?{$_ -is [VMware.Vim.VirtualEthernetCard]} | %{
        New-Object -Type PSObject -Property @{
            VMname = $viewThisVM.Name
            NICtype = $_.GetType().Name
            MacAddress = $_.MacAddress
            AddressType = $_.AddressType
            vCenterAPI = $viewThisVM.Client.serviceUrl -replace("https://","") -replace(":443/sdk","")
        } ## end new-object
    } ## end foreach-object
} 
$myreport | group MacAddress |where {$_.Count -gt 1} |select -ExpandProperty Group | Select VMname, vCenterAPI, AddressType, MacAddress, NICtype

Side note, the Client.serviceUrl property was something I found pretty quick…I wouldn’t call that a ‘best practice’ on figuring out which vCenter an object is part of. Also, the code that is removing the https and 443/sdk is a very bad approach and isn’t going to work in all scenarios. I just wanted something quick and dirty to get the info I needed.

The above code showed that the problem was much larger than just 2 virtual machines — I had nearly 50 virtual machines sharing 25 MAC addresses!

After looking into several of the virtual machines, I started seeing a pattern — a pattern I created a few months back. I had moved a bunch of virtual machines from my production vCenter a disaster recovery site vCenter (separated for SRM) on the same network. To limit the amount of downtime, the move was completed by presenting one temporary LUN to hosts in each vCenter, using storage VMotion to move the data, then during a change window the VMs were removed from inventory in production and added to the disaster recovery inventory. Virtual machines were then storage VMotioned to the proper LUNs and the temporary LUN removed. During this move, virtual machines kept their original MAC addresses, but as they had been removed from production inventory, the production vCenter was able to reissue those unused addresses.

There were also a couple examples where a similar approach was used to ‘fail back’ select virtual machines after a disaster recovery test, leaving the production vCenter having MAC addresses generated by the disaster recovery site vCenter.

I have a little more information on this issue — including details on how vCenter generates MAC addressess — that I will share in another post. Stay tuned!

Posted in Scripting, Virtualization | 5 Comments

Get-Beer Powershell function

Earlier in the week I had the privilege of presenting an introduction to Powershell/PowerCLI webex to a group of VMware customers. To help explain the Where-Object, Select-Object, and Sort-Object cmdlets, I thought it would be helpful to create a Get-Beer function. This function was a success, so I thought I would share it here. The function is very basic — it uses Import-CSV to read information from a file and returns the results ordered by name. The results can then be piped to other functions for demo purposes.

You can download a sample beer.csv file here.

Here is the function:


<#
.SYNOPSIS
List Miller Coors beer
.DESCRIPTION
This function returns a list of beer bottled by MillerCoors.
The object also includes nutrition and alcohol information.
.EXAMPLE
PS C:\> Get-Beer
.NOTES
This function was created for demo purposes only.  
VMware PowerCLI rocks!
#>
Function Get-Beer {
	$myResults = @()
	Import-Csv C:\tmp\beer.csv | Sort-Object Name | %{
		[decimal]$thisAlcoholPercent = $_."Alcohol%"
		[decimal]$thisCaloriesPer12oz = $_.CaloriesPer12oz
		$myResults += new-object -type PSObject -Property @{
			Name = $_.Name
			AlcoholPercent = $thisAlcoholPercent
			CaloriesPer12oz = $thisCaloriesPer12oz
			Category = $_.Category
		} #End New-Object
	} #End for each object
	
	# Return a custom PS object containing each beer
	return ($myResults | Select-Object Name, AlcoholPercent, CaloriesPer12oz, Category)
} #End Function

The function type casts the numeric values as decimal so that they properly sort. To run this, place a beer.csv file in the C:\tmp path (or adjust the filename/path in the function), paste the function into a powershell window and you are ready to go.

Here are a few examples of what you can do with this function:

Get-Beer

(Get-Beer).count

Get-Beer | Sort-Object AlcoholPercent -Descending | Select-Object Name, AlcoholPercent -First 5

Get-Beer | Where-Object {$_.CaloriesPer12oz -lt 200 -AND $_.Category -eq "Craft"} | Sort-Object AlcoholPercent -Descending | Select-Object -First 5
Posted in Scripting | Leave a comment

vSphere vCenter 5.0 SSL certificates

It seems every release of the VMware vSphere vCenter server service has more dependency on SSL.  I always replace the default self signed SSL certificates with ones created from an internal/trusted certificate authority.  There is a good four part guide to creating the properly formatted certificates available here: http://www.virtualvcp.com/vsphere-4-technical-guides/136-replace-ssl-certificates-prepare-openssl-and-microsoft-cs

As of last count, you need to place the custom SSL certificate in 4 places to make sure you don’t see any pesky SSL warnings. Here is a listing of those paths and instructions required to make the certificate work:

vCenter Server Service (VPXD)
SSL location: C:\ProgramData\VMware\VMware VirtualCenter\SSL
Run the command: “D:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe –p” to reset the vpxd password.
*Note: This will cause all of the hosts to become disconnected and require each to be reconnected to vCenter

vSphere Web Client Server
SSL location: D:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl
Restart the “vSphere Web Client” Service.

VMware vCenter Update Manager
SSL location: D:\Program Files\VMware\Infrastructure\Update Manager\SSL
Find/Replace the existing server name in the D:\Program Files (x86)\VMware\Infrastructure\Update Manager\extension.xml file and replace it with your servers alias/SSL certificates common name.
Run the command: “D:\Program Files (x86)\VMware\Infrastructure\Update Manager\vciInstallUtils.exe –vc myvcenter.mydomain.com –port 80 -U myusername -P mypassword -S extension.xml -C . -L . -O extupdate”
Where myvcenter.mydomain.com is the common name/friendly name of your SSL certificate and 80 is the http port of your vCenter.

vCenter Inventory Service:
SSL location: D:\Program Files\VMware\Infrastructure\Inventory Service\ssl
Stop thevCenter Inventory Service (Note: This will also stop the dependent service VMware vSphere Profile-Driven Storage Service)
Run the command: D:\Program Files\VMware\Infrastructure\Inventory Service\scripts\register.bat myvcenter.mydomain.com 443
Where myvcenter.mydomain.com is the common name/friendly name of your SSL certificate and 443 is the https port of your vCenter.
Start the vCenter Inventory Service
Start the VMware vSphere Profile-Driven Storage Service

Posted in Virtualization | 2 Comments

Multiple vCenters and MAC address conflicts

VMware KB article 1024025 explains an issue with MAC address conflicts due to multiple vCenter configurations with the same instance ID. Here is a technical description straight from the KB article:

Each vCenter Server system has a vCenter Server instance ID. This ID is a number between 0 and 63 that is randomly generated at installation time, but can be reconfigured after installation. 

vCenter Server uses the vCenter instance ID to generate MAC addresses and UUIDs for virtual machines. If two vCenter Server systems have the same vCenter instance ID, they might generate identical MAC addresses for virtual machines. This can cause conflicts if the virtual machines are on the same network, leading to packet loss and other problems. 

The solution is to verify that all of your vCenter configurations use different instance IDs for MAC address generation. This wouldn’t be difficult if you only had two or three vCenters, but as you scale out it becomes more of a challenge. Here is some PowerCLI code that will help you check for duplicate instance IDs.


# Virtual machine MAC address conflicts - http://kb.vmware.com/kb/1024025
$vCentersToCheck = "vcenter1","vcenter2","vcenter3"

$config = Get-PowerCLIConfiguration 
if($config.DefaultVIServerMode -eq "Single"){Set-PowerCLIConfiguration -DefaultVIServerMode Multiple}
foreach ($vCenter in $vCentersToCheck){connect-viserver $vCenter | Out-Null}

$myReport = @()
$global:DefaultVIServers | %{
	$si = Get-View ServiceInstance -Server $_.Name
	$set = Get-View $si.Content.Setting -Server $_.Name
	$myReport += new-object -type PSObject -Property @{
		InstanceID = ($set.Setting.GetEnumerator() | where {$_.key -eq "instance.id"}).value
		vCenter = $_.Name
	}
}

$myResults = $myReport | Group-Object InstanceID | where {$_.Count -gt 1} | select -ExpandProperty Group | select vCenter, InstanceID 
if ($myResults.count -gt 1) { $myResults } else { "No duplicate InstanceIDs were identified in the vCenters checked" }

By default, the script will let you know if you have any duplicates. However, if you want to know what all instance IDs are in use, they are stored in the $myReport variable. You can type “$myReport” (without the quotes) in the console window after this script completes.

Posted in Scripting, Virtualization | 2 Comments