Enterprise Admins.org

I recently got my hands on a new Synology DS723+ and a couple of 4TB HDD disks for my home lab. My plan is to use this device as a backup target, either as a storage destination for my existing backup product, or perhaps switching to the Synology Active Backup application that can be enabled on the Synology device. I’ll be doing a bit more testing with this in the future, and do plan to write up those results in a future post.

However, since I had this new device, I wanted to revisit a decision I made several years ago — whether I should use iSCSI or NFS for my ESXi Datastores. I wanted to do a side-by-side comparison of the protocols to make sure I was using the optimial one. I also wanted to test NFS with & without the NFS VAAI plugin, to make sure it was worth installing as part of host builds. Several years ago, I purchased a Synology DS1817+ to use as the primary storage in my lab and decided to use NFS with the VAAI plugin. This has worked very well for me, but it never hurts to re-think a decision.

For this test I decided to perform a couple of virtual machine copy and move operations. I wanted to use the same virtual machine, but decided to use one smaller template and one that was a bit larger. Here are some storage details on the two VMs that I used for comparisons:

• photon5 – 16GB VMDK, thin provisioned & using about 860MB on disk
• win22 – 50GB VMDK, thin provisioned & using about 16GB on disk

Both test VMs were placed on a local NVME disk on a test host which also had a pair of NFS datastores and a pair of iSCSI datastores backed by the Synology DS723+, connected to the network with a single 1GbE port. The DS723+ has two gigabit connections available and can be upgraded to include a 10GbE connection. However, for this test I wanted to focus specifically on storage protocols and having this bottleneck in the network and using HDD instead of SSD seemed acceptable. Changing these factors would improve performance, but my focus of this test was the protocol & VAAI impact instead of pure performance.

To execute the tests & capture results, I decided to use PowerCLI commands inside of a Measure-Command code block and focus on total seconds. I had originally started this test using ESXi 7.0 only, but after seeing another thread mention that the Synology NFS VAAI VIB would work with vSphere 8.0 (although it is not officially supported), I decided to double up all the tests to see if drastic differences existed between the two versions. From the testing below, we can see that the ESXi version had no noticable impact on the testing. I listed the total seconds for the first test only, but did see some variance in multiple runs. I suspect if we averaged the results across 10 runs they would be even closer to the same value.

Test1: Full VM Copy across the network – completed & measured using syntax similar to: Measure-Command {New-VM -VM 'localdisk_photon5' -Name 'nfs_photon5' -VMHost 'testhost' -Datastore 'nfs1'} | Select-Object TotalSeconds

Based on the table above, we can see that the performance of all the New-VM operations are roughly the same for the same per source VM size.

Test2: Full VM Copy between datastores of the same type – completed & measured using syntax similar to: Measure-Command { Move-VM -VM 'nfs_photon5' -Datastore 'nfs2' } | Select-Object TotalSeconds

I did not include specific results, but the tests observed above for Move-VM had similar results to New-VM copies that used the same source/destination datastores. For example, a Windows 2022 New-VM from iSCSI to iSCSI (similar to the last 2 rows of the above table) only took 3.83 seconds. This is most likely due to the fact that both clone and migrate requests are using the XCOPY VAAI primitive, described here: https://core.vmware.com/resource/vmware-vsphere-apis-array-integration-vaai#sec9410-sub4.

Based on the table above, we observe some differences for NFS vs. iSCSI. Installing the NFS VAAI plugin offered a significant improvement vs. the default NFS configuration. However, the VAAI iSCSI performance for Move-VM operations was noticeably better than the best case NFS configuartion.

Summary: These are not the results I had initially expected. I had assumed NFS performance with VAAI enabled would have been much closer to the iSCSI performance numbers. I was very impressed by the iSCSI performance for clones/moves when using VAAI. Based on Test 1, I’m likely to leave the majority of VMs already deployed in their current NFS configuration. I don’t move the VMs around and the VMs aren’t observing any latency, so I can’t think of a compelling reason to go back & move them. However, based on Test 2, I do plan to make some changes. I currently have a couple Aria Automation deployments that use linked clones to improve provisioning times. I’m likely to revisit that configuration, moving the template VMs & target deployments to iSCSI datastores backed by the same Synology array to unlock the super-fast cloning.

Have you ever had a new site come online and want to validate that the items in a subscriber content library are in sync? Or experience some storage issues/corruption and want to make sure your sync’d library is still intact? This post will use a PowerCLI script to quickly validate items in a content library.

$contentLibraryName = 'h114-sub'
$testHost = 'lab-esxi-02.example.org'
$testDatastore = 'local-hdd'

foreach ($clitem in ( Get-ContentLibraryItem -ContentLibrary $contentLibraryName -ItemType ovf | Sort-Object Name ) ) {
  "Starting deploy of item $($clitem.name)..."
  $thisNewVM = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Datastore $testDatastore
  if ($thisNewVM) {
    "  New VM created. Will attempt power on."
    [void]($thisNewVM | Get-NetworkAdapter | Set-NetworkAdapter -StartConnected:$false -confirm:$false)
    $thisVMstarted = $thisNewVM | Start-VM | Wait-Tools
    if ($thisVMstarted) {
      "  Startup success, doing cleanup."
      [void]($thisVMstarted | Stop-VM -Confirm:$false)
      [void]($thisVMstarted | Remove-VM -DeletePermanently -Confirm:$false)
    } else {
      "  There was an issue with a test VM for content library item $($clitem.Name).  Cleanup not completed."
    }
  } else {
    write-warning "  Failure occured creating new VM based on $($clitem.Name)!"
  }
}

This script has three variables that must be defined — the name of the subscriber content library, a test host in the site/environment you wish to test, and a datastore from that test host to use for provisioning.

The script will retrieve a list of OVF templates, deploy a new VM from each using the creative name Test + the name of the content library item. Once the item is deployed, the VM will be powered on and wait for tools to start. If Tools start successfully, we can likely assume that the VM is in working order, so the script proceeds to power off/delete the test VM, otherwise it will throw a warning to the screen and continue.

During testing I encountered/created a few issues which I’ll describe below.

The first was caused by an ’empty’ VM which didn’t have an OS installed. It failed with an error that VMware Tools was not installed. The script didn’t delete this VM, but after reviewing an confirming this was expected behavior for the VM, I deleted it anyway.

Wait-Tools : 8/21/2023 6:53:45 AM       Wait-Tools              The specified VM 'Test_emptyVM-notools' does not have VMware Tools
installed.
At line:7 char:46
+     $thisVMstarted = $thisNewVM | Start-VM | Wait-Tools
+                                              ~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Test_emptyVM-notools:VirtualMachineImpl) [Wait-Tools], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_WaitTools_ToolsNotInstalledOrVersionIncorrect,VMware.VimAuto
   mation.ViCore.Cmdlets.Commands.WaitTools

To contrive the next error, I ssh’d to the test host, found the content library item on the filesystem, and intentionally corrupted the VMDK by running echo blah > emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf-flat.vmdk. This resulted in the following error (read below the next error for the resolution to this):

New-VM : 8/21/2023 6:15:29 AM   New-VM          Transfer failed: Error during transfer of
ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//Test_emptyVM-notools/Test_emptyVM-notools.vmdk: The virtual
disk is either corrupted or not a supported format..
At line:3 char:26
+ ... = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Da ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VM], ViError
    + FullyQualifiedErrorId : ViCore_VMServiceImpl_DeployFromLibraryItem_ViNetException,VMware.VimAutomation.ViCore.Cm
   dlets.Commands.NewVM

After capturing this error, I deleted the files associated with the content library item. I had assumed that these missing files would trigger the content library to re-sync the content. Instead, I received the following error:

New-VM : 8/21/2023 6:22:32 AM   New-VM          Transfer failed: Error during transfer of ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//contentlib-21c270a1-11
7e-4982-a014-ac122fdb74eb/088a872b-1098-441b-807a-251f76402991/emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf.vmdk?serverId=bad2b868-9a1e-4d68-ba7e-26
0f397cc0ca: Failed to resolve source datastore file URI(ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//contentlib-21c270a1-117e-4982-a014-ac122fdb74e
b/088a872b-1098-441b-807a-251f76402991/emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf.vmdk?serverId=bad2b868-9a1e-4d68-ba7e-260f397cc0ca).
At line:3 char:26
+ ... = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Da ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VM], ViError
    + FullyQualifiedErrorId : ViCore_VMServiceImpl_DeployFromLibraryItem_ViNetException,VMware.VimAutomation.ViCore.Cmdlets.Commands.NewVM

To address the previous two items, I found the ‘bad’ content library item in the web interface, right clicked and selected delete item content. I had expected this operation to fail, as I assumed that content could only be deleted from the publisher repo and not the subscriber repo. However, this command was successful, removing the ‘size’ property of the item from the content library as if it had never sync’d. Rerunning the script to deploy this content library item caused the item to replicate and the deployment was successful.

Hopefully this script can help you validate content library items if needed. If you have any suggestions on how to improve this script, please leave a comment.