Enterprise Admins.org

A few weeks back I wanted to verify some path counts per LUN. This is typically pretty easy and something that can be written as a one liner using standard PowerCLI cmdlets as such:

Get-VMHost | Get-ScsiLun | Get-ScsiLunPath

However, the above command wouldn’t return results in the customer environment. After doing some testing, I realized that the issue was likely related to the presence of EMC PowerPath/VE for multipathing on the hosts. When using the GUI to view storage/LUN properties other details like Path Selection Plugin (PSP) is also missing… but the path information I wanted was still available. It took a little bit of poking around in the Get-View output, but I was able to come up with something to get me the data I was looking for. Its not real pretty, but it is fast and helped me answer a couple questions. I figured I would share the code here here in case anyone else runs into this issue. If you have any comments/suggestions on how to make this code better/more complete please post them in the comments section.

$dsHt = Get-View -ViewType Datastore -Property "Info","Summary.Type" -Filter @{"Summary.Type"="VMFS"} | 
Select @{N="DSName";E={$_.Info.Vmfs.Name}}, @{N="Capacity";E={[math]::round( ($_.Info.Vmfs.Capacity / 1024 / 1024), 0)}}, 
@{N="Extent";E={$_.Info.Vmfs.Extent[0].DiskName}}, @{N="VMFS Version";E={$_.Info.Vmfs.Version}} | 
Group-Object "Extent" -AsHashTable -AsString

$results = @()
Get-View -ViewType HostSystem -SearchRoot (Get-Cluster ClusterPod8).id -Property Name, Config.StorageDevice | %{
  $thisHostName = $_.Name
  $_.Config.StorageDevice.PlugStoreTopology.path | ?{$_.Name -match 'naa'} | Group-Object LunNumber | Sort-object Name | %{
    try {
      $thisNaaId = ($_.Group[0].Name -split "naa.")[1]
      $results += New-Object psobject -Property @{
        HostName = $thisHostName
        LunNumber = $_.Name
        PathCount = $_.Count
        DSName = $dsHt["naa.$thisNaaId"][0].DSName
        Capacity = $dsHt["naa.$thisNaaId"][0].Capacity
        "VMFS Version" = $dsHt["naa.$thisNaaId"][0]."VMFS Version"
      }
    } catch {
      Write-Warning "Found something with $thisNaaId"
    }
  } # end this Lun
} # end this host

$results |Group-Object pathcount

I’ve been troubleshooting a specific problem where storage latency jumps very high during very short periods of time, usually in the late evening/very early morning hours. The latency is very bad, sometimes in the 2,000ms+ neighborhood. My storage guys see an extreme increase in IOPS coming from my ESXi hosts just before the latency comes into play. The working thought was several VMs were kicking off some type of disk intensive batch job around the same time. This would be a perfect use of the vCOps troubleshooting Top N charts, but the issue doesn’t appear every day and is typically resolved before anyone noticed. Since the Top N charts are realtime they are not super useful in this situation.

What I needed was a way to export which VMs were contributing high IO around the time of the poor latency. Clicking around in vCOps I couldn’t find a way to get this data. (Side note: if anyone knows a good way to do this, please leave a comment.) However, a co-worker pointed me at an unofficial vCOps powershell module available here: http://velemental.com/2012/09/04/unofficial-vmware-vcenter-operations-powershell-module/. Using this module, I was able to get all the data points for disk commands by virtual machine during the time period in questions. With a little where-object goodness we can find only those VMs with over 300 IOPS. Looking at the data before applying this filter, I noticed this value would be around 3x the average IO normally seen during this period of time. This isn’t really a good visualization for the amount of data, but it can give me what I need to be able to continue to troubleshoot:

$startDate = Get-Date "9/27/2013 12:01 AM"
$endDate = Get-Date "9/27/2013 5:00 AM"
Get-Datacenter NestedLab | Get-VM | 
Get-vCOpsResourceMetric -metricKey "virtualDisk:Aggregate of all instances|commandsAveraged_average" -startDate $startDate -endDate $endDate | 
Select-Object Name, @{N="Value";E={[math]::round($_.value,0)}}, Date | 
Where-Object {$_.Value -gt 300}

In my case, this method didn’t give me an obvious answer to my problems. However, it did give me a smaller list of virtual machines to focus on.

I recently needed to resize system partitions on several Windows 2008R2 virtual machines. To do this with Set-HardDisk the virtual machines must be powered off and you need a helper VM. I was looking for a way to do this without downtime, as that can be arranged when executing the steps manually (grow the disk, log into the guest, rescan disks and then extend the partition). I came up with the following workaround and thought it would be worth sharing. The idea is to set the hard disk to the new size with Set-HardDisk and then use Invoke-VMScript to run diskpart from within the VM. I included the -ResizeGuestPartition switch on Set-HardDisk as that appears to complete the task of re-scanning for disks within disk manager.

$guestName="newDiskTest"
$guestUser="Administrator"
$guestPass="Aw3s0m3pwd"
$newSizeGB=40
 
Get-HardDisk -vm "newDiskTest" | 
?{$_.name -eq "hard disk 1"} | 
Set-HardDisk -CapacityKB ($newSizeGB*1MB) -ResizeGuestPartition -GuestUser $guestUser -GuestPassword $guestPass -confirm:$false -ErrorAction:SilentlyContinue

Invoke-VMScript -vm $guestName -ScriptText "echo select vol c > c:\diskpart.txt && echo extend >> c:\diskpart.txt && diskpart.exe /s c:\diskpart.txt" -GuestUser $guestUser -GuestPassword $guestPass -ScriptType BAT