EnterpriseAdmins.org

I was recently asked if it would be possible for two virtual machines to be automatically assigned the same MAC addresses. Knowing that vCenter handles these assignments and ensures uniqueness, I figured it wouldn’t be possible…but it got me to thinking. Since I have multiple vCenters, what would prevent each vCenter from reusing MAC addresses? My search lead to this KB article: http://kb.vmware.com/kb/1024025 which in turn resulted in this blog post: http://enterpriseadmins.org/blog/scripting/multiple-vcenters-and-mac-address-conflicts/.

Question answered — this would not automatically happen.

A few days later, this same guy shows up with two guest names and one MAC address asking me who actually has that MAC? I check — then double check — only to confirm that both VMs have that same MAC address automatically assigned! One interesting thing to note was that each VM was in fact in a different vCenter.

Thats when I turned to some solid code I remembered seeing from mattboren on the VMware PowerCLI communities page (http://communities.vmware.com/thread/339358). A few slight modifications and I had the super fast code shown below to kick out a list of any duplicate MAC addresses:

Side note, the Client.serviceUrl property was something I found pretty quick…I wouldn’t call that a ‘best practice’ on figuring out which vCenter an object is part of. Also, the code that is removing the https and 443/sdk is a very bad approach and isn’t going to work in all scenarios. I just wanted something quick and dirty to get the info I needed.

The above code showed that the problem was much larger than just 2 virtual machines — I had nearly 50 virtual machines sharing 25 MAC addresses!

After looking into several of the virtual machines, I started seeing a pattern — a pattern I created a few months back. I had moved a bunch of virtual machines from my production vCenter a disaster recovery site vCenter (separated for SRM) on the same network. To limit the amount of downtime, the move was completed by presenting one temporary LUN to hosts in each vCenter, using storage VMotion to move the data, then during a change window the VMs were removed from inventory in production and added to the disaster recovery inventory. Virtual machines were then storage VMotioned to the proper LUNs and the temporary LUN removed. During this move, virtual machines kept their original MAC addresses, but as they had been removed from production inventory, the production vCenter was able to reissue those unused addresses.

There were also a couple examples where a similar approach was used to ‘fail back’ select virtual machines after a disaster recovery test, leaving the production vCenter having MAC addresses generated by the disaster recovery site vCenter.

I have a little more information on this issue — including details on how vCenter generates MAC addressess — that I will share in another post. Stay tuned!

Earlier in the week I had the privilege of presenting an introduction to Powershell/PowerCLI webex to a group of VMware customers. To help explain the Where-Object, Select-Object, and Sort-Object cmdlets, I thought it would be helpful to create a Get-Beer function. This function was a success, so I thought I would share it here. The function is very basic — it uses Import-CSV to read information from a file and returns the results ordered by name. The results can then be piped to other functions for demo purposes.

You can download a sample beer.csv file here.

Here is the function:

The function type casts the numeric values as decimal so that they properly sort. To run this, place a beer.csv file in the C:\tmp path (or adjust the filename/path in the function), paste the function into a powershell window and you are ready to go.

Here are a few examples of what you can do with this function:

Get-Beer

(Get-Beer).count

Get-Beer | Sort-Object AlcoholPercent -Descending | Select-Object Name, AlcoholPercent -First 5

Get-Beer | Where-Object {$_.CaloriesPer12oz -lt 200 -AND $_.Category -eq "Craft"} | Sort-Object AlcoholPercent -Descending | Select-Object -First 5

It seems every release of the VMware vSphere vCenter server service has more dependency on SSL.  I always replace the default self signed SSL certificates with ones created from an internal/trusted certificate authority.  There is a good four part guide to creating the properly formatted certificates available here: http://www.virtualvcp.com/vsphere-4-technical-guides/136-replace-ssl-certificates-prepare-openssl-and-microsoft-cs

As of last count, you need to place the custom SSL certificate in 4 places to make sure you don’t see any pesky SSL warnings. Here is a listing of those paths and instructions required to make the certificate work:

vCenter Server Service (VPXD)
SSL location: C:\ProgramData\VMware\VMware VirtualCenter\SSL
Run the command: “D:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe –p” to reset the vpxd password.
*Note: This will cause all of the hosts to become disconnected and require each to be reconnected to vCenter

vSphere Web Client Server
SSL location: D:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl
Restart the “vSphere Web Client” Service.

VMware vCenter Update Manager
SSL location: D:\Program Files\VMware\Infrastructure\Update Manager\SSL
Find/Replace the existing server name in the D:\Program Files (x86)\VMware\Infrastructure\Update Manager\extension.xml file and replace it with your servers alias/SSL certificates common name.
Run the command: “D:\Program Files (x86)\VMware\Infrastructure\Update Manager\vciInstallUtils.exe –vc myvcenter.mydomain.com –port 80 -U myusername -P mypassword -S extension.xml -C . -L . -O extupdate”
Where myvcenter.mydomain.com is the common name/friendly name of your SSL certificate and 80 is the http port of your vCenter.

vCenter Inventory Service:
SSL location: D:\Program Files\VMware\Infrastructure\Inventory Service\ssl
Stop thevCenter Inventory Service (Note: This will also stop the dependent service VMware vSphere Profile-Driven Storage Service)
Run the command: D:\Program Files\VMware\Infrastructure\Inventory Service\scripts\register.bat myvcenter.mydomain.com 443
Where myvcenter.mydomain.com is the common name/friendly name of your SSL certificate and 443 is the https port of your vCenter.
Start the vCenter Inventory Service
Start the VMware vSphere Profile-Driven Storage Service

VMware KB article 1024025 explains an issue with MAC address conflicts due to multiple vCenter configurations with the same instance ID. Here is a technical description straight from the KB article:

Each vCenter Server system has a vCenter Server instance ID. This ID is a number between 0 and 63 that is randomly generated at installation time, but can be reconfigured after installation. 

vCenter Server uses the vCenter instance ID to generate MAC addresses and UUIDs for virtual machines. If two vCenter Server systems have the same vCenter instance ID, they might generate identical MAC addresses for virtual machines. This can cause conflicts if the virtual machines are on the same network, leading to packet loss and other problems.

The solution is to verify that all of your vCenter configurations use different instance IDs for MAC address generation. This wouldn’t be difficult if you only had two or three vCenters, but as you scale out it becomes more of a challenge. Here is some PowerCLI code that will help you check for duplicate instance IDs.

By default, the script will let you know if you have any duplicates. However, if you want to know what all instance IDs are in use, they are stored in the $myReport variable. You can type “$myReport” (without the quotes) in the console window after this script completes.

This is a very simple post, but it shows how to make the Powershell title bar contain useful information. By default the Powershell title contains the path to the powershell.exe file — here is an example:

To make this more useful, I’ve added a line to my powershell profile that looks like this:

$host.ui.rawui.WindowTitle="PowerShell $($psversiontable.psversion) as $($env:username)"

*Note: all this text should appear on the same line in your profile.

This change adds the Powershell version number and running user name to the title bar. This is useful in my environment because I often have the need to run Powershell as a different user. Here is an example of the update:

I placed this in the all users profile located here:

%windir%\system32\WindowsPowerShell\v1.0\profile.ps1 

This profile is machine specific and executes anytime you open a powershell console for any user. While you have the profile open, I have one more time saving suggestion. If you use the Send-MailMessage cmdlet often, add this line:

$PSEmailServer = "smtp-relay.mydomain.local"

With this variable specified, you no longer need the “-SmtpServer” attribute to send a message — this global variable will automatically be used.

Hope this helps!

In most of my Powershell scripts I make use of the PSCustomObject to store results. The example I first learned (and use all the time) looks just like this:

In the above example, you create a blank PSCustomObject on line 1 which has Name, RAM and CPU placehoders. Then on lines 2 through 4 you assign values to each attribute. Finally on line 5 you do something with the variable $item. In the above example, the contents of the $item variable are displayed onscreen. Normally you’d see something like $myreport += $item which stores the new item in a collection called $myreport.

The problem is adding an additional attribute to your object requires two steps — first, you must declare the attribute on line 1 and then assign something to the attribute after it is declared. For example, if I wanted to add PowerState to my report, I would have to add “, PowerState” on the first line. If you forget, which happens to me all the time, you get this error:

$item.PowerState = "PoweredOff"
Property 'PowerState' cannot be found on this object; make sure it exists and is settable.
At line:1 char:7
+ $item. <<<< PowerState = "PoweredOff"
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : PropertyAssignmentException

I recently found a new example that I like a lot better. Here is an example:

In the above example, attributes are created as needed, removing the need to pre-define them on the first line of the script. This method removes the need to specify "$item." for each attribute, making the code easier to read/maintain/support.

This is a very simple example, but I hope others find it useful.

I have been spending a lot of time working on Powershell scripts. Specifically, I have been re-writing many VB scripts and processes into more efficient powershell versions. In my last post I mentioned that I planed to share how I made a task that took 8 hours to execute run in only 28 minutes. There are two parts to this efficiency — Start-Job and Group-Object. In a future post I’ll write more about the Start-Job functionality, but the greatest benefit was from Group-Object — and thats what I’ll focus on in this post.

The task I re-wrote looked at about 50,000 user accounts and collected many details about the accounts for auditing purposes. Due to some constraints, I needed to use the lastLogon attribute instead of lastLogonTimeStamp. You can read more about the constraints here: http://www.rlmueller.net/Last%20Logon.htm, but this line pretty much sums it up: Because the lastLogon attribute is not replicated in Active Directory, a different value can be stored in the copy of Active Directory on each Domain Controller. In VB script, I was using an LDAP bind to each domain controller for each user account and then evaluated the lastLogon attribute, which was very inefficient.

Here is the powershell version of this code, which is much more efficient and flexible (as you can get the last login time from each/all domain controllers very easy).

After creating this code, I attempted to select user information from the $lastLogons variable. This was very slow — even slower than the previous VB script that did excessive LDAP binds. The efficiency in this script was gained by grouping the objects into a hash table ($groupedLastLogin) by distinguished name, and then accessing the specific key value as needed.

I know several of my posts have strayed from the normal VMware/vSphere/PowerCLI topics recently, but I have some plans to get back on track early next year. Please stay tuned!

I have been working on various scripts that populate information into a SQL table. The other day I was attempting to see how long some of these tables took to populate. I could have updated the PowerShell script to include some Measure-Command information, but then I’d have to re-execute my tasks and see how long they took to run. Since I already had all of the information populated into a table, and each row had a time stamp of when the data was entered, I could just use a T-SQL select statement to get the run time data I wanted.

SELECT CAST( (MAX(dataCollected) - MIN(dataCollected)) as time) as TotalRunTime, COUNT(*) as RecCount FROM tblTestInserts WHERE collectionID = MAX(collectionID)

The above example looks at two columns in my table. The first column (dataCollected) is of the datetime type and has a default binding of GetDate() — so it is automatically populated with the SQL Servers date and time every time a record is inserted. The second column (collectionID) is an integer that is calculated each time I execute the script — inside of powershell I select the maximum value of this column and then add 1. This provides a way know how many times the task has been executed and which data is the most current.

This gave me exactly the information I was looking for without needing to modify my PowerShell script and re-run the collection. It also allows me to compare current runs to previous runs (by changing the value of the collectionID in the WHERE clause). I can also be more specific…I can add an ‘AND’ to the where clause and see how long a specific part of the collection took to run.

In a future post I plan to share how I was able to take a vbscript that took 8 hours to complete condense it into a 28 minute execution time powershell script. (Here is a preview of the magic sauce — Group-Object -Property DN -AsHashTable -AsString)

I mentioned in a previous post (VMware View inside vCloud Director) that we had to make a last minute change to the lab content on the fly. This change was to Exercise 11 that covers the Get-ESXCli cmdlet. I should mention that this is a change to ESXi — if you are using PowerCLI 5.0 to access a 4.1 host, you will need to use the 4.1 methods.

The PowerCLI 4.x lab can be found here: Demo Days 4: PowerCLI Lab
The PowerCLI 5.x lab can be found here: GCVMUG PowerCLI 5.0 Lab

After connecting to the ESXi hosts directly with connect-viserver, we get the ESXCli and assign it to a variable:

$esxcli = Get-ESXCli

The changes apply to how you access virtual machine information.

# ESXi 4.1:
$esxcli.vms.vm.list()

# ESXi 5.0:
$esxcli.vm.process.list()

As you can see the root element for the VM namespace has changed as well as the child namespace. This has been updated in the lab and should be corrected for the next show.

Earlier this month I had the privilege of helping out with a PowerCLI hands on lab at the Greater Cincinnati VMUG regional event. The content of this GCVMUG PowerCLI 5.0 lab is now available online: GCVMUG PowerCLI 5.0 Lab

A special thanks goes to the GCVMUG, Jake Robinson, Ryan Birk, Bluelock and 10zig for making this lab possible.