Archive for Virtualization

Find ESXi Installation Date with PowerCLI

I recently noticed a link to a very interesting article on Twitter: Find ESXi Installation Date — http://vcdx56.com/2016/01/05/find-esxi-installation-date/. These steps are straight forward and show how to calculate the original ESXi install date, looking at the first section of the UUID. I didn’t realize so much useful information was coded into the UUID, but this could be useful information for certain reports. To understand the process, please check out the original article on vcdx56.com.

The following few lines of code will return this information for all hosts in a vCenter using PowerCLI.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Find ESXi install date: http://vcdx56.com/2016/01/05/find-esxi-installation-date/
# Convert HEX to DEC: http://michaelflanakin.com/Weblog/tabid/142/articleType/ArticleView/articleId/1073/Converting-ToFrom-Hex-with-PowerShell.aspx
# Convert epoch to date: http://stackoverflow.com/questions/10781697/convert-unix-time-with-powershell

Get-VMHost | Sort Name | %{
  $thisUUID = (Get-EsxCli -VMHost $_.name).system.uuid.get()
  $decDate = [Convert]::ToInt32($thisUUID.Split("-")[0], 16)
  $installDate = [timezone]::CurrentTimeZone.ToLocalTime(([datetime]'1/1/1970').AddSeconds($decDate))
  [pscustomobject][ordered]@{
    Name="$($_.name)"
    InstallDate=$installDate
  } # end custom object
} # end host loop

Setup VSAN without vCenter using PowerCLI

Recently @RuddyVCP (Blog | Twitter) and I worked on building a home lab to give away at the 2015 Indianapolis VMUG UserCon. The idea was we’d build a three node VSAN cluster using Intel NUC miniature PCs each with a 1TB 7200 RPM SATA disk and a 256GB M.2 SSD. Between Kyle and I we have plenty of content for a series of blog posts, as this was a much more challenging project than either of us initially assumed.

Today I want to write about the VSAN install, as it was part of a chicken vs. egg type discussion. Normally one would setup VSAN on a cluster from within vCenter. However, in our day-zero deployment we did not have a working vCenter yet, or a place to deploy the vCenter server. All of the local disks inside the NUCs needed to be claimed by the VSAN (three nodes with one SSD and one spinning disk are the bare minimum for VSAN), so we couldn’t use one of those disks as a temporary home for vCenter either. The catch was without vCenter we also couldn’t deploy VSAN… or could we? How can you setup VSAN without vCenter?

Our Technical Account Manager @eck79 (Blog | Twitter) pointed us at this gem of a document for just such an occasion. It covers using the shell to do just that — setup VSAN without vCenter for day-zero deployments: https://www.vmware.com/files/pdf/products/vsan/VMware-TechNote-Bootstrapping-VSAN-without-vCenter.pdf

This document is great… it covers the basics of VSAN and includes bootstrapping instructions using esxcli over SSH. I wanted to do a bit of testing to see if I could make this work from within PowerCLI. Here is some sample code that I tested on a handful of NUCs to do just that — enable VSAN without vCenter using PowerCLI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Set-PowerCLIConfiguration -DefaultVIServerMode:Multiple -InvalidCertificateAction:Ignore -Confirm:$false

Connect-VIServer 192.168.1.31,192.168.1.32,192.168.1.33 -User root -Password My#1Password
Get-VMHostNetworkAdapter -Name vmk0 | Set-VMHostNetworkAdapter -VsanTrafficEnabled:$true -Confirm:$false

Get-VMHost | Sort Name | %{
$vmh = $_
$luns = Get-ScsiLun -VMHost $vmh -LunType disk | ?{$_.IsLocal -eq $true -AND $_.CapacityGB -gt 32} | select CanonicalName, @{N="SSD";E={$_.ExtensionData.Ssd}}, CapacityGB
$dataLun = ($luns | ?{$_.ssd -eq $false} ).CanonicalName
$ssdLun = ($luns | ?{$_.ssd -eq $true} ).CanonicalName

$thisEsxCli = $vmh | Get-EsxCli
$thisEsxCli.vsan.storage.add($dataLun,$ssdLun)

# On the first pass we need to get a new UUID, future passes will reuse the existing value
if (!$newVsanUUID) {$newVsanUUID = $thisEsxCli.system.uuid.get()}
$thisEsxCli.vsan.cluster.join($newVsanUUID)
}

$thisEsxCli.vsan.cluster.get()

Lets break this code down a bit. In the first couple of lines we are connecting to all three of the NUCs (which are already running ESXi 6.0). We then enable VSAN traffic on the existing management network/vmk0 interface. We could easily have created a new portgroup/vmkernel interface, perhaps on its own VLAN, but to keep this simple I used the existing interface. When using unsupported hardware, best practices take the backseat.
We then loop through all the hosts (the three we have connected to earlier) and for each host we select the pair of local disks. The where-object statement that picks out devices larger than 32GB is just excluding the USB drive that contains the boot image. Once we have the disks, we use Get-EsxCli to claim them for VSAN. Finally we use the first host in the loop to generate a new UUID for the VSAN cluster (subsequent hosts will reuse this value) and we add the hosts to the VSAN cluster. Pretty snazzy. Here are some screenshots showing some of the various output.

Here is where we enabled the VSAN traffic on each host:
Enable VSAN traffic

Here you’ll see the code block that is selecting the disks, claiming them for VSAN (one true return per host) and then adding the host to the VSAN cluster (another true return for each host).
Host EsxCli loop code block with 6 true returns

Here you can see the status of the cluster is healthy.
esxi vsan cluster get at the very end

And finally, a screenshot showing the datastore is present — it appears three times, once per host.
Get-Datastore

Moving to XtremIO (or any other array)

I was recently asked to help move VMs from one datastore backed by traditional spinning disk to an EMC XtremIO datastore.  I was really excited as I wanted to see the power of an all flash array.

As a best practice, most all flash vendors recommend using their own multipathing software (PowerPath/VE) or round robin. In this environment, the customer choose to go with RoundRobin. This is pretty easy to do in the GUI, but you need to do it for each device and on each host in a cluster.  Not to worry, PowerCLI is willing to help.  The following code will find all XtremIO LUNs not currently set to RoundRobin and change them.

1
2
3
4
Get-VMHost "esx-cl3-*" | Get-VMHostHba -Type "FibreChannel" |
Get-ScsiLun -LunType "disk" |
where {$_.MultipathPolicy -ne "RoundRobin" -and $_.Vendor -eq 'XtremIO'} |
Set-ScsiLun -MultipathPolicy RoundRobin

Once we have the new datastores all prepared and ready to go, we can start moving virtual machines. In the customers environment, two different clusters of ESXi hosts shared a common set of datastores. As part of this migration, the customer wanted to dedicated datastores to each cluster (removing their cross cluster datastores). Again, PowerCLI was willing to help. In the following code, we will move all of the virtual machines from MyOldDataStore that are on a host whose name matches esx-cl3. We will sort the virtual machines by name, just for tracking purposes and kick out an email when the process is complete (just so we don’t have to babysit the process):

1
2
3
4
5
6
Get-Datastore MyOldDataStore | Get-VM | ?{$_.VMHost.Name -match 'esx-cl3'} | Sort Name | %{
   write-host "$($_.Name) will be moved to XtremIO..." -NoNewline
   [void]( $_ | Move-VM -Datastore xtrem-cl3-vol1 -DiskStorageFormat:EagerZeroedThick )
   Write-Host " done!"
}
Send-MailMessage -To 'notify@mydomainname.com' -Subject "Done with CL3 group" -From 'notify@mydomainname.com' -SmtpServer 'smtp.mydomainname.com'

That was pretty easy, and now the VMs will benefit from the consistent low-latency of an all flash array. Now you probably need some help managing and monitoring your array. Fortunately the team over at vNugglets have you covered. Check out their latest post at http://www.vnugglets.com/2014/04/xtremio-powershell-module-report-on.html. It is a fantastic module and really worth checking out.

PowerCLI: Getting LUN paths when using EMC PowerPath/VE

A few weeks back I wanted to verify some path counts per LUN. This is typically pretty easy and something that can be written as a one liner using standard PowerCLI cmdlets as such:

Get-VMHost | Get-ScsiLun | Get-ScsiLunPath

However, the above command wouldn’t return results in the customer environment. After doing some testing, I realized that the issue was likely related to the presence of EMC PowerPath/VE for multipathing on the hosts. When using the GUI to view storage/LUN properties other details like Path Selection Plugin (PSP) is also missing… but the path information I wanted was still available. It took a little bit of poking around in the Get-View output, but I was able to come up with something to get me the data I was looking for. Its not real pretty, but it is fast and helped me answer a couple questions. I figured I would share the code here here in case anyone else runs into this issue. If you have any comments/suggestions on how to make this code better/more complete please post them in the comments section.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$dsHt = Get-View -ViewType Datastore -Property "Info","Summary.Type" -Filter @{"Summary.Type"="VMFS"} |
Select @{N="DSName";E={$_.Info.Vmfs.Name}}, @{N="Capacity";E={[math]::round( ($_.Info.Vmfs.Capacity / 1024 / 1024), 0)}},
@{N="Extent";E={$_.Info.Vmfs.Extent[0].DiskName}}, @{N="VMFS Version";E={$_.Info.Vmfs.Version}} |
Group-Object "Extent" -AsHashTable -AsString

$results = @()
Get-View -ViewType HostSystem -SearchRoot (Get-Cluster ClusterPod8).id -Property Name, Config.StorageDevice | %{
  $thisHostName = $_.Name
  $_.Config.StorageDevice.PlugStoreTopology.path | ?{$_.Name -match 'naa'} | Group-Object LunNumber | Sort-object Name | %{
    try {
      $thisNaaId = ($_.Group[0].Name -split "naa.")[1]
      $results += New-Object psobject -Property @{
        HostName = $thisHostName
        LunNumber = $_.Name
        PathCount = $_.Count
        DSName = $dsHt["naa.$thisNaaId"][0].DSName
        Capacity = $dsHt["naa.$thisNaaId"][0].Capacity
        "VMFS Version" = $dsHt["naa.$thisNaaId"][0]."VMFS Version"
      }
    } catch {
      Write-Warning "Found something with $thisNaaId"
    }
  } # end this Lun
} # end this host

$results |Group-Object pathcount

Getting data out of vCOps

I’ve been troubleshooting a specific problem where storage latency jumps very high during very short periods of time, usually in the late evening/very early morning hours. The latency is very bad, sometimes in the 2,000ms+ neighborhood. My storage guys see an extreme increase in IOPS coming from my ESXi hosts just before the latency comes into play. The working thought was several VMs were kicking off some type of disk intensive batch job around the same time. This would be a perfect use of the vCOps troubleshooting Top N charts, but the issue doesn’t appear every day and is typically resolved before anyone noticed. Since the Top N charts are realtime they are not super useful in this situation.

What I needed was a way to export which VMs were contributing high IO around the time of the poor latency. Clicking around in vCOps I couldn’t find a way to get this data. (Side note: if anyone knows a good way to do this, please leave a comment.) However, a co-worker pointed me at an unofficial vCOps powershell module available here: http://velemental.com/2012/09/04/unofficial-vmware-vcenter-operations-powershell-module/. Using this module, I was able to get all the data points for disk commands by virtual machine during the time period in questions. With a little where-object goodness we can find only those VMs with over 300 IOPS. Looking at the data before applying this filter, I noticed this value would be around 3x the average IO normally seen during this period of time. This isn’t really a good visualization for the amount of data, but it can give me what I need to be able to continue to troubleshoot:

1
2
3
4
5
6
$startDate = Get-Date "9/27/2013 12:01 AM"
$endDate = Get-Date "9/27/2013 5:00 AM"
Get-Datacenter NestedLab | Get-VM |
Get-vCOpsResourceMetric -metricKey "virtualDisk:Aggregate of all instances|commandsAveraged_average" -startDate $startDate -endDate $endDate |
Select-Object Name, @{N="Value";E={[math]::round($_.value,0)}}, Date |
Where-Object {$_.Value -gt 300}

In my case, this method didn’t give me an obvious answer to my problems. However, it did give me a smaller list of virtual machines to focus on.