Fine-Tuning Updates: Targeted Host Remediation with vSphere Lifecycle Manager

In vSphere 7.0 functionality was introduced to be able to manage a cluster of ESXi hosts with a single image. In addition to being a bit easier to configure than the prior Update Manager Baselines, this feature also integrates with Hardware Support Managers to be able to update host firmware.

PowerCLI 12.1 introduced the ability to remediate against these single images, using syntax like:

Get-Cluster img-test | Set-Cluster -Remediate -AcceptEULA

This and other vLCM cmdlets are discussed in this blog post: https://blogs.vmware.com/PowerCLI/2020/10/new-release-powercli-12-1-vlcm-enhancements.html.

This is a very simple command and works very well to programmatically interact with the clusters if you have many to remediate. However, sometimes we need to remediate hosts in a more controlled order, or as part of a larger workflow. For example, perhaps we need to notify an operations management product or team before a host goes into maintenance to prevent after hour pages or service desk issues being logged. This can be done, but requires a few extra cmdlets. The code block below provides sample commands to remediate a specific host in a cluster.

$clusterid = (Get-Cluster 'Img-Test').ExtensionData.MoRef.Value
$vmhostid = (Get-VMHost 'h178-vesx-04.lab.enterpriseadmins.org').ExtensionData.MoRef.Value

# Initialize hosts to updates, hostids can be specified comma separated.
$SettingsClustersSoftwareApplySpec = Initialize-SettingsClustersSoftwareApplySpec -Hosts $vmhostid -AcceptEula $true 

# Apply the specification object to the cluster
$taskId = Invoke-ApplyClusterSoftwareAsync -Cluster $clusterid -SettingsClustersSoftwareApplySpec $SettingsClustersSoftwareApplySpec

# The apply task runs async so we need to watch tasks for it to complete.
$task=Get-Task |?{$_.Id -eq "CisTask-$taskId"} 

# Loop until the task finishes
While ($task.State -eq "Running") {
    "sleeping..."
    Start-Sleep -Seconds 60 
    $task=Get-Task |?{$_.Id -eq "CisTask-$taskId"} 
}

In the example above we get the ID value of the cluster and host, these are the values like domain-c3146330 and host-3146220 and are the values required by the cmdlets we are using.

We then create a spec to apply. The parameters are described here: https://developer.vmware.com/apis/vsphere-automation/latest/esx/api/esx/settings/clusters/cluster/softwareactionapplyvmw-tasktrue/post/. I was originally confused by the -commit parameter until I found this documentation. I had assumed the property was similar to a git commit message and was passing in random strings of text. The property is optional, so in the above example I do not pass in the argument at all. However, if you’d like to find the current/expected value, you can get it from the cluster object like this: (Invoke-GetClusterSoftwareCompliance -Cluster 'domain-c3146214').commit

Next we pass our clusterid and above spec to the ‘Invoke’ function. The command returns a task ID, so we are capturing the output to a variable so we can use it later. The Invoke command runs asynchronously, so we will use this ID to check on the status until the task completes.

At the end we are checking for the task to complete. The Get-Task cmdlet has an -ID parameter, but in my testing I would get an error of The identifier CisTask-525d3596-4a7a-60ab-df10-ab97999b8511:com.vmware.esx.settings.clusters.software resulted in no objects. when I tried to pass the ID to Get-Task. Using where-object instead worked reliably, so I used it instead.

Hopefully this helps if you have the need to remediate a single host in a cluster using a vLCM Image.