Content Library item validation with Power CLI

Have you ever had a new site come online and want to validate that the items in a subscriber content library are in sync? Or experience some storage issues/corruption and want to make sure your sync’d library is still intact? This post will use a PowerCLI script to quickly validate items in a content library.

$contentLibraryName = 'h114-sub'
$testHost = 'lab-esxi-02.example.org'
$testDatastore = 'local-hdd'

foreach ($clitem in ( Get-ContentLibraryItem -ContentLibrary $contentLibraryName -ItemType ovf | Sort-Object Name ) ) {
  "Starting deploy of item $($clitem.name)..."
  $thisNewVM = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Datastore $testDatastore
  if ($thisNewVM) {
    "  New VM created. Will attempt power on."
    [void]($thisNewVM | Get-NetworkAdapter | Set-NetworkAdapter -StartConnected:$false -confirm:$false)
    $thisVMstarted = $thisNewVM | Start-VM | Wait-Tools
    if ($thisVMstarted) {
      "  Startup success, doing cleanup."
      [void]($thisVMstarted | Stop-VM -Confirm:$false)
      [void]($thisVMstarted | Remove-VM -DeletePermanently -Confirm:$false)
    } else {
      "  There was an issue with a test VM for content library item $($clitem.Name).  Cleanup not completed."
    }
  } else {
    write-warning "  Failure occured creating new VM based on $($clitem.Name)!"
  }
}

This script has three variables that must be defined — the name of the subscriber content library, a test host in the site/environment you wish to test, and a datastore from that test host to use for provisioning.

The script will retrieve a list of OVF templates, deploy a new VM from each using the creative name Test + the name of the content library item. Once the item is deployed, the VM will be powered on and wait for tools to start. If Tools start successfully, we can likely assume that the VM is in working order, so the script proceeds to power off/delete the test VM, otherwise it will throw a warning to the screen and continue.

During testing I encountered/created a few issues which I’ll describe below.

The first was caused by an ’empty’ VM which didn’t have an OS installed. It failed with an error that VMware Tools was not installed. The script didn’t delete this VM, but after reviewing an confirming this was expected behavior for the VM, I deleted it anyway.

Wait-Tools : 8/21/2023 6:53:45 AM       Wait-Tools              The specified VM 'Test_emptyVM-notools' does not have VMware Tools
installed.
At line:7 char:46
+     $thisVMstarted = $thisNewVM | Start-VM | Wait-Tools
+                                              ~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Test_emptyVM-notools:VirtualMachineImpl) [Wait-Tools], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_WaitTools_ToolsNotInstalledOrVersionIncorrect,VMware.VimAuto
   mation.ViCore.Cmdlets.Commands.WaitTools

To contrive the next error, I ssh’d to the test host, found the content library item on the filesystem, and intentionally corrupted the VMDK by running echo blah > emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf-flat.vmdk. This resulted in the following error (read below the next error for the resolution to this):

New-VM : 8/21/2023 6:15:29 AM   New-VM          Transfer failed: Error during transfer of
ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//Test_emptyVM-notools/Test_emptyVM-notools.vmdk: The virtual
disk is either corrupted or not a supported format..
At line:3 char:26
+ ... = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Da ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VM], ViError
    + FullyQualifiedErrorId : ViCore_VMServiceImpl_DeployFromLibraryItem_ViNetException,VMware.VimAutomation.ViCore.Cm
   dlets.Commands.NewVM

After capturing this error, I deleted the files associated with the content library item. I had assumed that these missing files would trigger the content library to re-sync the content. Instead, I received the following error:

New-VM : 8/21/2023 6:22:32 AM   New-VM          Transfer failed: Error during transfer of ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//contentlib-21c270a1-11
7e-4982-a014-ac122fdb74eb/088a872b-1098-441b-807a-251f76402991/emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf.vmdk?serverId=bad2b868-9a1e-4d68-ba7e-26
0f397cc0ca: Failed to resolve source datastore file URI(ds:///vmfs/volumes/639245e5-399bba0d-e1e8-000c2927a169//contentlib-21c270a1-117e-4982-a014-ac122fdb74e
b/088a872b-1098-441b-807a-251f76402991/emptyVM-notools-1_248c256e-cab1-4512-b7a0-48322ad4a7bf.vmdk?serverId=bad2b868-9a1e-4d68-ba7e-260f397cc0ca).
At line:3 char:26
+ ... = $clitem | New-VM -Name "Test_$($clitem.Name)" -VMHost $testHost -Da ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VM], ViError
    + FullyQualifiedErrorId : ViCore_VMServiceImpl_DeployFromLibraryItem_ViNetException,VMware.VimAutomation.ViCore.Cmdlets.Commands.NewVM

To address the previous two items, I found the ‘bad’ content library item in the web interface, right clicked and selected delete item content. I had expected this operation to fail, as I assumed that content could only be deleted from the publisher repo and not the subscriber repo. However, this command was successful, removing the ‘size’ property of the item from the content library as if it had never sync’d. Rerunning the script to deploy this content library item caused the item to replicate and the deployment was successful.

Hopefully this script can help you validate content library items if needed. If you have any suggestions on how to improve this script, please leave a comment.

Posted in Lab Infrastructure, Virtualization | Leave a comment

Aria Automation Internal IPAM reserve / release API

In the May 2023/8.12.1 release notes for Aria Automation, one of the what’s new items was the ability to reserve/unreserve an IP address to make it unavailable/available for deployment. In this release the functionality is supported via API only. This post will cover how to connect to and consume this API

The first thing we’ll look at is the network elements from the web interface to show which object we’ll be interacting with from the API. Under Assembler > Infrastructure > Network Profiles we see various networks. In this case we’ll use VLAN40, which in my lab contains physical network devices.

We will open this tile, then select the Networks tab. From here we can see the 192.168.40.0/24 network.

We will select the Network and then click Manage IP Ranges.

This ‘VLAN40 – Network Devices’ name (which I intentionally set different that the network profile name above) is the one we see when calling the API for a list of network-ranges. I defined the range as being 192.168.40.10 through 192.168.40.199, for a total of 190 usable addresses. Now that we’ve seen the elements in the web interface, we will interact with them using this new API.

Our first task will be logging into the API. We will provide a username/password to obtain a refresh token, then we’ll use that token to obtain a bearer token. This is based on the postman example from this KB article: https://kb.vmware.com/s/article/89129.

$vraHost = 'https://cm-vra-01.lab.enterpriseadmins.org'

$restBody = @{
  username='configadmin'
  password='VMware1!'
  domain='System Domain'
} | Convertto-Json

$refreshToken = Invoke-RestMethod -Uri "$($vraHost)/csp/gateway/am/api/login?access_token" -Method 'POST' -Body $restBody -ContentType 'application/json'
$bearerToken = Invoke-RestMethod -Uri "$($vraHost)/iaas/api/login" -Method 'POST' -Body $(@{refreshToken=$($refreshToken.'refresh_token')}|ConvertTo-Json) -ContentType 'application/json'

Once we have our bearer token, we can use it as a header in our future posts. For example, to obtain a list of network-ip-ranages, we’ll use:

$networkIpRanges = Invoke-RestMethod -Uri "$($vraHost)/iaas/api/network-ip-ranges" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}

We know from the web interface that the name of our network range is ‘VLAN40 – Network Devices’. We will use a where-object (using the question mark alias) to find the ID of this network.

$thisNetId=($networkIpRanges.content | ?{$_.Name -match 'vlan40'}).id

$networkIpRequest = Invoke-RestMethod -Uri "$($vraHost)/iaas/api/network-ip-ranges/$thisNetId/ip-addresses/allocate" -Method 'POST' -body $(@{description='I need some IPs for the demo app'; numberOfIps=2}|ConvertTo-Json) -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}

# wait a few seconds for the request to complete, then ask for results
$reqId = $networkIpRequest.id
foreach ($thisResource in (Invoke-RestMethod -Uri "$($vraHost)/iaas/api/request-tracker/$reqId" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}).Resources) {
  Invoke-RestMethod -Uri "$($vraHost)$($thisResource)?apiVersion=2021-07-15" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}
}

The networkIpRequest object that results will have an ID property to identify that specific network. In the codeblock below, we will use that ID to get the status of that request. Since we specified that we needed 2 addresses, the Resources property will be a collection of resource IDs. We will loop through the addresses and get more details on each item

$reqId = $networkIpRequest.id
foreach ($thisResource in (Invoke-RestMethod -Uri "$($vraHost)/iaas/api/request-tracker/$reqId" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}).Resources) {
  Invoke-RestMethod -Uri "$($vraHost)$($thisResource)?apiVersion=2021-07-15" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}
}

The above code will return the details of the two IP addresses that were allocated, results below.

ipAddress             : 192.168.40.10
ipAddressDecimalValue : 3232245770
ipVersion             : IPv4
ipAddressStatus       : ALLOCATED
ipAllocationType      : USER
description           : I need some IPs for the demo app
id                    : 3f025212-e50a-430e-a8f2-baf0e53deb8f
createdAt             : 2023-08-28
updatedAt             : 2023-08-28
orgId                 : e142c6b9-379b-4754-9790-00b4f4373ca4
_links                : @{network-ip-range=; self=}

ipAddress             : 192.168.40.11
ipAddressDecimalValue : 3232245771
ipVersion             : IPv4
ipAddressStatus       : ALLOCATED
ipAllocationType      : USER
description           : I need some IPs for the demo app
id                    : 32416121-39b5-4462-8d51-2bf6743368fa
createdAt             : 2023-08-28
updatedAt             : 2023-08-28
orgId                 : e142c6b9-379b-4754-9790-00b4f4373ca4
_links                : @{network-ip-range=; self=}

As you can see, 192.168.40.10 and 192.168.40.11 are the first two addresses in the pool and those are the ones given to us for our demo app.

Lets assume that we ended up only needing one of those addresses and now want to return / release one to the pool. We can do this with the API as well. The following code needs the Network ID from above, as well as the specific address we want to release. In this example we will return 192.168.40.11.

Invoke-RestMethod -Uri "$($vraHost)/iaas/api/network-ip-ranges/$thisNetId/ip-addresses/release"  -Method 'POST' -body $(@{ipAddresses=@('192.168.40.11')}|ConvertTo-Json) -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}

This will again return a task ID which looks like this:

progress : 1
status   : INPROGRESS
name     : IP Address Release Task
id       : ad5dc112-3dfc-4f7c-836a-6808084bcd56
selfLink : /iaas/api/request-tracker/ad5dc112-3dfc-4f7c-836a-6808084bcd56

To confirm this completed, we could post to the 'selfLink' to get a status, but instead we will look at our network-ip-ranage and see which addresses are in use. In this case we would only expect 1 address, the 192.168.40.10 IP.

(Invoke-RestMethod -Uri "$($vraHost)/iaas/api/network-ip-ranges/$thisNetId/ip-addresses?apiVersion=2021-07-15" -Method 'GET' -ContentType 'application/json' -Headers @{Authorization="Bearer $($bearerToken.token)"}).content


ipAddress             : 192.168.40.10
ipAddressDecimalValue : 3232245770
ipVersion             : IPv4
ipAddressStatus       : ALLOCATED
ipAllocationType      : USER
description           : I need some IPs for the demo app
id                    : 3f025212-e50a-430e-a8f2-baf0e53deb8f
createdAt             : 2023-08-28
updatedAt             : 2023-08-28
orgId                 : e142c6b9-379b-4754-9790-00b4f4373ca4
_links                : @{network-ip-range=; self=}

ipAddress             : 192.168.40.11
ipAddressDecimalValue : 3232245771
ipVersion             : IPv4
ipAddressStatus       : RELEASED
ipAllocationType      : NONE
description           :
id                    : 32416121-39b5-4462-8d51-2bf6743368fa
createdAt             : 2023-08-28
updatedAt             : 2023-08-28
orgId                 : e142c6b9-379b-4754-9790-00b4f4373ca4
_links                : @{network-ip-range=; self=}

As we can see above, the 192.168.40.10 address is allocated, while the 192.168.40.11 address has been released. In this case we can see that only 1 address is in use. If we are in the web interface (Assembler > Infrastructure > Network Profiles > specific tile > Networks > Manage IP Ranges) we can see the same — only 1 address is currently used.

I hope this walkthrough shows how to consume this new API to manage Aria Automation (formerly known as vRealize Automation) internal IPAM.

Posted in Lab Infrastructure, Scripting, Virtualization | Leave a comment

No repositories found for given version.

I was recently working with someone who was using vRealize Suite Lifecycle Manager 8.10 (now known as Aria Suite Lifecycle) to update vRealize Network Insight from version 6.7 to 6.9 (now known as Aria Operations for Networks). The Product Support Pack was updated to the correct version, the binary mapping task had been completed, and the environment trigger inventory sync task completed without issue. However the very next screen where one would normally select the desired version repository had an error which stated “no repositories found for given version.” Even more interestingly, the choices for product version only provided 5.2 and 5.3 as choices. We had expected to see versions 6.8 and 6.9 instead.

After validating we had the correct PSPak and binary again we happened to look at the Lifecycle Operations > Requests tab and sorted by Request Status. We noticed a VRNI upgrade task from this environment in pending action state from early 2021. This task had more options, including one option to delete the request.

On a hunch, we decided to delete this old task. After doing so, the upgrade wizard continued without showing the “no repositories” error, and the drop down list showed the expected versions 6.8 and 6.9.

In the requests UI, there were similar pending tasks for a few other products, like vRealize Automation and vRealize Operations. We didn’t validate, but expect that this issue could be present for other products/environments being updated from within vRealize Lifecycle Manager. To prevent future issues, we decided to delete those pending requests as well. We didn’t see any communities articles or blog posts on this issue, but since it was a rather easy fix I wanted to document it here. I hope this saves you a few minutes of trubleshooting!

Posted in Virtualization | Leave a comment

Using VI/JSON with Powershell

I recently saw a post on a new feature that is enabled by default in vSphere 8.0u1: https://blogs.vmware.com/code/2023/06/10/introducing-vi-json-a-modern-wire-protocol-for-vsphere-management. The article describes a new JSON-based, REST-like protocol named VI/JSON that allows access to all the familiar SOAP XML properties. It seemed interesting, so I decided to try it out with PowerShell.

$vc = 'vc3.example.com'

# Get SessionManager Value
$sessionManagerMoid = (Invoke-RestMethod "https://$VC/sdk/vim25/8.0.1.0/ServiceInstance/ServiceInstance/content").sessionManager.value

# Login/get sessionID
$sessionId = (Invoke-WebRequest -Uri "https://$VC/sdk/vim25/8.0.1.0/SessionManager/$sessionManagerMoid/Login" -Body ( @{'userName'='Administrator@vsphere.local'; 'password'='VMware1!'} | ConvertTo-Json ) -Method:POST -ContentType:'application/json').Headers.'vmware-api-session-id'

# Get output from a specific VM
(Invoke-WebRequest -Headers @{'vmware-api-session-id'=$sessionId} -Uri "https://$VC/sdk/vim25/8.0.1.0/VirtualMachine/vm-48/config" -ContentType:'application/json').Content | ConvertFrom-JSON

The above few lines logs in to the API and returns all the details of a specific VM (specifically moref vm-48 which is passed in via the last URI). I’ve included the first few lines of the output below for reference.

_typeName                    : VirtualMachineConfigInfo
changeVersion                : 2023-06-08T18:12:09.377525Z
modified                     : 1970-01-01T00:00:00Z
name                         : h006-vm-02
guestFullName                : Microsoft Windows Server 2016 or later (64-bit)
version                      : vmx-15
uuid                         : 423f0e56-bfe2-39c5-2096-52b59fdea9b8
createDate                   : 2022-12-05T13:42:45.640154Z
instanceUuid                 : 503fd66a-4f92-950d-18fd-1ab0a4f55ff2

This is an interesting way to get JSON output of objects and can allow for integrations with newer tooling. I hope this PowerShell example is helpful.

Posted in Scripting, Virtualization | Leave a comment

Emulating a Wide Area Network with Latency and Packet Loss

Have you ever found yourself needing to simulate network latency or packet loss? I recently wanted to test a replication scenario but needed to have at least 15ms or so of latency between the source and destination.

Years ago I used a WANem virtual appliance as a router to do something similar. This project can be found online (https://wanem.sourceforge.net), however it appears that development has stopped, with the latest releases being nearly 10 years old. WANem provides a PHP web interface allowing the user to configure latency and packet loss, among other things. When you apply settings, the web interface issues commands to Linux Traffic Control (tc) Network Emulator (netem) to enforce those settings.

Instead of using the WANem appliance, which ships as a very old and unpatched Knoppix Linux distribution, I decided to take the command it ran and try it on an Ubuntu template available in my lab.

I created an Ubuntu VM with 2 network interfaces. One interface connected to my routable lab network using a static IP (ens192: 192.168.40.11) and the other connected to non-routable VLAN 19 that was available and was assigned another static IP (ens224: 192.168.19.1) — which would become the default gateway for this new network.

I then added the following two lines to the /etc/rc.local file on this system:

echo "1" > /proc/sys/net/ipv4/ip_forward
tc qdisc add dev ens192 root handle 1: netem delay 19ms loss 2%

The first line makes the system act as a router, the second adds 19ms of latency and 2% packet loss to traffic which passes the network interface. More configuration options for this second command can be found here: https://man7.org/linux/man-pages/man8/tc-netem.8.html, including how to send corrupt/duplicate/reordered packets — really make your WAN experience terrible. Since it is in /etc/rc.local these commands will run automatically when the system starts.

I also wanted this system to act as a DHCP server for the clients at the remote site. I did this by installing a DHCP server (apt install isc-dhcp-server), making environment specific changes to /etc/dhcp/dhcpd.conf such as DNS servers and lease-time, and also adding my new subnet with:

subnet 192.168.19.0 netmask 255.255.255.0 {
   range 192.168.19.50 192.168.19.99;
   option subnet-mask 255.255.255.0;
   option routers 192.168.19.1;
}

To activate these DHCP changes, I ran /etc/init.d/isc-dhcp-server restart.

Finally I added a static route to the primary router in my lab, such that any requests for 192.168.19.0/24 go to the gateway at 192.168.40.11. Now any request in or out of my WAN site has latency injected at some occasional lost packets:

I’m now able to deploy virtual machines at a “remote location” about 19-20ms away, but actually run on the same vSphere cluster.

Posted in Lab Infrastructure, Virtualization | Leave a comment