A missing folder, a missing day

I recently moved some scheduled tasks from a Windows 2008R2 server to a Windows 2012R2 server. This should be the easy type of work: copy some files, enter some credentials, move on to another project. But as luck would have it, An exception occurred. Those three words.

One of the scheduled tasks uses a COM object to access Visio.Application. The script enumerates a listing of Visio diagrams, sees if they are newer than a static version on a web server and if so runs SaveAsWeb for each new document. After moving to a new server the ‘An exception occurred.’ error kept showing up for each document being published when the script ran as a scheduled task. Running the script interactively worked fine. A bunch of searching kept pointing at dcomcnfg, specifically for an entry named Visio Document and/or Microsoft Office Visio Document. This option did not show up for me. I even re-installed Visio and rebooted hoping to re-register the COM object. When I was just about ready to give up, I found this simple article, with a comment all the way at the end:

You need to ensure that the folder C:\Windows\System32\config\systemprofile\Desktop exists. It’s the same when you try to automate Word in a service.

Could it really be that simple? Would creating one folder solve the day long project? The answer — yes. I browsed to C:\Windows\System32\config\systemprofile and created a folder named Desktop. The scheduled task now works fine on the new server. One missing folder, one missing day in troubleshooting a simple task.

Moving to XtremIO (or any other array)

I was recently asked to help move VMs from one datastore backed by traditional spinning disk to an EMC XtremIO datastore.  I was really excited as I wanted to see the power of an all flash array.

As a best practice, most all flash vendors recommend using their own multipathing software (PowerPath/VE) or round robin. In this environment, the customer choose to go with RoundRobin. This is pretty easy to do in the GUI, but you need to do it for each device and on each host in a cluster.  Not to worry, PowerCLI is willing to help.  The following code will find all XtremIO LUNs not currently set to RoundRobin and change them.

Get-VMHost "esx-cl3-*" | Get-VMHostHba -Type "FibreChannel" |
Get-ScsiLun -LunType "disk" |
where {$_.MultipathPolicy -ne "RoundRobin" -and $_.Vendor -eq 'XtremIO'} |
Set-ScsiLun -MultipathPolicy RoundRobin

Once we have the new datastores all prepared and ready to go, we can start moving virtual machines. In the customers environment, two different clusters of ESXi hosts shared a common set of datastores. As part of this migration, the customer wanted to dedicated datastores to each cluster (removing their cross cluster datastores). Again, PowerCLI was willing to help. In the following code, we will move all of the virtual machines from MyOldDataStore that are on a host whose name matches esx-cl3. We will sort the virtual machines by name, just for tracking purposes and kick out an email when the process is complete (just so we don’t have to babysit the process):

Get-Datastore MyOldDataStore | Get-VM | ?{$_.VMHost.Name -match 'esx-cl3'} | Sort Name | %{
   write-host "$($_.Name) will be moved to XtremIO..." -NoNewline
   [void]( $_ | Move-VM -Datastore xtrem-cl3-vol1 -DiskStorageFormat:EagerZeroedThick )
   Write-Host " done!"
Send-MailMessage -To 'notify@mydomainname.com' -Subject "Done with CL3 group" -From 'notify@mydomainname.com' -SmtpServer 'smtp.mydomainname.com'

That was pretty easy, and now the VMs will benefit from the consistent low-latency of an all flash array. Now you probably need some help managing and monitoring your array. Fortunately the team over at vNugglets have you covered. Check out their latest post at http://www.vnugglets.com/2014/04/xtremio-powershell-module-report-on.html. It is a fantastic module and really worth checking out.

Lack of posts

I realized today that I hadn’t posted anything new since December. That’s four full months without a post… normally I would have averaged a dozen posts in this period of time. A few things have contributed to to this lack of posting.

  • Working on projects that really lack code worth sharing.  There were a few tips/tricks I picked up specifically around making graphical user interfaces with SAPIEN PowerShell Studio (that I plan to share soon) but overall the projects focused on solving very specific problems that don’t share well.
  • Book reviews.  I had the opportunity to review several vSphere related books for Packt Publishing.  This was an interesting project, but consumed some of the free time I used for blogging. I plan on posting reviews of these books soon.
  • A career change.  I accepted a promotion from a technical to managerial role.  This has been more of a change that I had anticipated and adjusting to the new demands has consumed some time.

Anyway, enough with excuses.  I have a few post ideas lined up and hope to have them out the door in the next few weeks.  Check back soon!

Migration to Office 365/Exchange Online

I recently had a chance to help a small business move their 70 mailboxes to the Exchange Online service. This company was running Windows 2008 with Exchange 2007 on premises installation and we helped them migrate to Office 365/Exchange Online over a couple weekends. During the migration we ran into several issues. I couldn’t find many online resources where people documented these errors, so I wanted to write them down on this site.

Issue #1
Outlook Anywhere (RPC over HTTPS) service was configured, running and available to end users. However we received various errors when trying to establish a migration endpoint. Even though the service was working, the Remote Connectivity Analyzer (https://testconnectivity.microsoft.com/) was failing on Outlook Anywhere tests.
Looking around online, we thought the issue was related to the Exchange 2007 service pack level, so we upgraded to SP3. I’m not sure if this actually helped, but I figured it was worth mentioning as it may have fixed other issues we could have seen during the migration. The actual fix for this issue (and unfortunately I can’t find the Office 365 communities post where we found the suggestion) was to create a hosts file entry on the exchange server containing the IP, server name and FQDN of the internal Exchange server name. DNS was working perfectly, so I’m not sure why this was needed. However, after the entry was added the Remote Connectivity Analyzer tests started working and we were able to move forward with the migration.

Issue #2
Batch loaded into Office 365 environment to begin migration, but after several hours the task fails
Error log may mention “MigrationPermanentException: Error: MapiExceptionLogonFailed: Unable to make connection to the server”
Verify permissions, specifically Receive-As rights to the database (http://community.office365.com/en-us/forums/158/t/18911.aspx).

Add-ADPermission -Identity "Mailbox Store" -User "Trusted User" -ExtendedRights Receive-As

Issue #3
When opening Outlook as a user when the PC is joined to the domain, the on premises mailbox is opened instead of the Office 365 mailbox. When using a non-domain joined test PC, the Office 365 mailbox was opened.
This was caused by the way Exchange handles autodiscover. You can read more about the process here: http://msdn.microsoft.com/en-us/library/office/jj900169(v=exchg.150).aspx. We found a pair of scripts (ExportO365UserInfo.ps1 and Exchange2007MBtoMEU.ps1) available here: http://community.office365.com/en-us/wikis/exchange/845.aspx which allow you to convert the users active directory account into a mail enabled user that references Office 365 instead of a mailbox user in the on premises install.

This was my first experience with Office 365/Exchange Online. I was surprised at how complicated some of the migration steps were. With the whole ‘cloud-based’ self service model, I assumed that the migration path would be just a few clicks. With the handful of lessons learned from going through this process once, I would feel more comfortable doing another migration (but its not something I would volunteer for). I hope someone finds this post helpful.

PowerCLI: Getting LUN paths when using EMC PowerPath/VE

A few weeks back I wanted to verify some path counts per LUN. This is typically pretty easy and something that can be written as a one liner using standard PowerCLI cmdlets as such:

Get-VMHost | Get-ScsiLun | Get-ScsiLunPath

However, the above command wouldn’t return results in the customer environment. After doing some testing, I realized that the issue was likely related to the presence of EMC PowerPath/VE for multipathing on the hosts. When using the GUI to view storage/LUN properties other details like Path Selection Plugin (PSP) is also missing… but the path information I wanted was still available. It took a little bit of poking around in the Get-View output, but I was able to come up with something to get me the data I was looking for. Its not real pretty, but it is fast and helped me answer a couple questions. I figured I would share the code here here in case anyone else runs into this issue. If you have any comments/suggestions on how to make this code better/more complete please post them in the comments section.

$dsHt = Get-View -ViewType Datastore -Property "Info","Summary.Type" -Filter @{"Summary.Type"="VMFS"} |
Select @{N="DSName";E={$_.Info.Vmfs.Name}}, @{N="Capacity";E={[math]::round( ($_.Info.Vmfs.Capacity / 1024 / 1024), 0)}},
@{N="Extent";E={$_.Info.Vmfs.Extent[0].DiskName}}, @{N="VMFS Version";E={$_.Info.Vmfs.Version}} |
Group-Object "Extent" -AsHashTable -AsString

$results = @()
Get-View -ViewType HostSystem -SearchRoot (Get-Cluster ClusterPod8).id -Property Name, Config.StorageDevice | %{
  $thisHostName = $_.Name
  $_.Config.StorageDevice.PlugStoreTopology.path | ?{$_.Name -match 'naa'} | Group-Object LunNumber | Sort-object Name | %{
    try {
      $thisNaaId = ($_.Group[0].Name -split "naa.")[1]
      $results += New-Object psobject -Property @{
        HostName = $thisHostName
        LunNumber = $_.Name
        PathCount = $_.Count
        DSName = $dsHt["naa.$thisNaaId"][0].DSName
        Capacity = $dsHt["naa.$thisNaaId"][0].Capacity
        "VMFS Version" = $dsHt["naa.$thisNaaId"][0]."VMFS Version"
    } catch {
      Write-Warning "Found something with $thisNaaId"
  } # end this Lun
} # end this host

$results |Group-Object pathcount