Enterprise Admins.org

I was recently made aware of a KB article titled “Tag associations are not removed from vCenter Server database when associated objects are removed or deleted” (https://knowledge.broadcom.com/external/article?articleNumber=344960). The article includes a script that removes orphaned tag assignments left behind in the vCenter Server database after object deletion.

Investigating the Issue

After reviewing this article, I checked the vpxd.log file on one of my lab vCenter Server instances and noticed frequent entries like the following:

2025-06-07T17:33:50.765Z error vpxd[06442] [Originator@6876 sub=Authorize opID=4be68587-f898-41b0-bbd4-2764f0941eaa Authz-7c] MoRef: vim.Datastore:datastore-4936 not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl.fault.ManagedObjectNotFound

To quantify this, I ran:

cat /var/log/vmware/vpxd/vpxd.log | grep -i vmodl.fault.ManagedObjectNotFound | wc -l
13738

cat /var/log/vmware/vpxd/vpxd.log | wc -l
210258

This showed that roughly 6.5% of the log entries were related to this specific fault, which strongly suggested lingering tag associations.

Reproducing the Issue

To test further, I moved to a clean vCenter environment with no history of tag usage. I created and tagged 10 virtual machines:

$newCat = New-TagCategory -Name 'h378-category' -Cardinality:Multiple -EntityType:VirtualMachine

0..9 |%{ New-Tag -Name "h378-tag$_" -Category $newCat }

new-vm -VMHost test-vesx-71* -Name "h378-vm0" -Datastore vc3-test03-sdrs -Template template-tinycore-160-cli-cc
1..9 | %{ new-vm -VMHost test-vesx-71* -Name "h378-vm$_" -Datastore vc3-test03-sdrs -Template template-tinycore-160-cli-cc }

New-TagAssignment -Tag (Get-Tag "h378*") -Entity (Get-VM "h378*")

Get-VM "h378*" | Remove-VM -DeletePermanently:$true -Confirm:$false

After deletion, there were no log entries related to orphaned tags. I queried the database using a modified version of the cleanup script in read-only mode and confirmed that no orphaned tag rows existed. This led me to revisit the KB and note that:

In vSphere 7 and 8, tag associations are automatically removed for Virtual Machines and Hosts when the associated object is deleted.

Confirming with Cluster Objects

I then repeated the test using cluster objects, which are not automatically cleaned up:

$newCat = New-TagCategory -Name 'h378-category-Cluster' -Cardinality:Multiple -EntityType:ClusterComputeResource
0..9 |%{ New-Tag -Name "h378-cluster-tag$_" -Category $newCat }

0..9 |%{ New-Cluster -Name "h378-cluster-$_" -Location (Get-Datacenter h378-test) }

New-TagAssignment -Tag (Get-Tag "h378-cluster*") -Entity (Get-Cluster "h378*")

get-cluster "h378*" | remove-cluster -Confirm:$false

Shortly after deletion, the vpxd.log showed ManagedObjectNotFound errors. I verified the orphaned rows using the following SQL query:

${VMWARE_POSTGRES_BIN}/psql -U postgres VCDB -h /var/run/vpostgres <<EOF
select * from cis_kv_keyvalue where kv_provider like 'tagging:%'
and
kv_key like 'tag_association urn:vmomi:ClusterComputeResource:%'
and
regexp_replace(kv_key, 'tag_association urn:vmomi:ClusterComputeResource:domain-c([0-9]+).*', '\1')::bigint
not in (select id from vpx_entity where type_id=3);
EOF

This confirmed 100 orphaned tag associations, which I then cleaned up using the provided tags_delete_job_all.sh script.

Returning to the initial vCenter Server with ~6% of vpxd.log entries coming from this issue, I proceeded to create a snapshot and run the same script. It only removed about 30 orphaned associations. However, now I’m not seeing the new vmodl.fault.ManagedObjectNotFound entries showing up every few seconds.

Cleanup Results

Back on the original vCenter Server where the log showed high volumes of these errors, I took a snapshot and ran the cleanup script. It only removed around 30 entries, but new ManagedObjectNotFound messages have stopped appearing.

This reduction is easy to monitor in Aria Operations for Logs, especially across multiple vCenter environments.

Conclusion

In my environments, VM and Host deletions are the most common, and these objects now clean up their tag associations automatically in recent vSphere versions. However, orphaned associations from cluster or other object types may remain, especially in environments upgraded over time.

By reviewing your vpxd.log and using the methods shown here, you can identify and remediate these issues efficiently.

When building or troubleshooting infrastructure, it is often useful to simulate high CPU or memory usage withou deploying full production workloads. To assist with this, I previously created a few purpose built, like cpubusy.sh and memfill.sh.

Historically, I created multiple Tiny Core Linux templates, each designed for a specific purpose, a generic one for troubleshooting, one that would fill mem and a another to load up the CPU. I’d then place these scripts in /opt/bootlocal.sh, allowing each VM to run its designated script automatically at startup. I’d then control load by simply powering VMs on or off.

The Problem

That setup works fine for simple use cases, but it doesn’t scale well. What if I want:

• One VM to simulate CPU load,
• Another to test download speeds,
• A third to run a custom test script—all using the same base image?

The Common Control Script

This script runs at boot and checks for specific instructions to execute. The idea is simple: deploy a single generic VM template that decides what to do based on either:

• Metadata (via guestinfo)
• Network identity (IP, MAC, hostname)
• Shared config (via GuestStore or a web server)

This common control script can be found here: code-snips/cc.sh.

Where It Looks for Instructions

When a VM boots, the script checks for commands in the following order:

1. Web server:
   • http://<web>/<macaddress>.txt
   • http://<web>/<ipaddress>.txt
   • http://<web>/<hostname>.txt
   • http://<web>/all.txt
2. VM Guest Store (using VMware tools):
   • guestinfo.ccScript (specified via advanced VM setting)
   • /custom/cc/cc-all.txt

This layered approach gives flexibility:

• Set a global script via all.txt
• Override per host via metadata or identifiers
• Or push custom scripts directly via GuestInfo

Example: Setting the GuestInfo Property

Using PowerCLI, we can set the script filename per VM like this:

Get-VM h045-tc16-02 | New-AdvancedSetting -Name 'guestinfo.ccScript' -Value 'memfill.sh' -confirm:$false

We can also modify this via the vSphere Web Client.

Demonstration

Here’s a test case from my lab:

• I set guestinfo.ccScript to memfill.sh
• The all.txt file includes a simple command to print system time

Upon boot, the VM fills 90% of available RAM using a memory-backed filesystem and prints the time, confirming that both script sources are active.

Later, I removed the guestinfo.ccScript setting and added a <hostname>.txt script to download a file repeatedly from a test web server. After reboot, the VM behaved differently, now acting as a network test client. No changes to the template required.

Sample Scripts

Here are a few lightweight test scripts used in the demo:

• cpubusy.sh – uses sha1sum to keep all the configured CPU cores busy
• download.sh – uses wget to get the same webserver file x times and save it to /dev/null
• memfill.sh – creates a memory backed filesystem using 90% of RAM, then uses dd to fill it

Conclusion

This ‘common config’ approach provides template reuse, easier script management, and dynamic testing control, all without modifying the template.

Whether testing CPU, memory, or network load across dozens of VMs, the common control script simplifies the process and reduces maintenance overhead.

In future iterations, this setup could be extended to include conditional logic (based on boot time, VM tags, or other metadata), or integration with CI pipelines for even more powerful automation.

When working with VMs that lack network connectivity, transferring files can be tricky. I recently explored the GuestStore feature a built-in vSphere feature that solves this challenge by allowing file delivery via VMware Tools—even without a network. This post walks through how I used GuestStore to push a script into a TinyCore Linux VM with no network, CD drive, or external storage options.

The vSphere documentation does a good job explaining what GuestStore is and how it can be used: Distributing Content with GuestStore – https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-virtual-machine-administration-guide-8-0/managing-virtual-machinesvsphere-vm-admin/distributing-content-with-gueststorevsphere-vm-admin.html

Configuring ESXi Hosts for GuestStore

To begin, I configured all of the hosts in a cluster to use the same NFS datastore as a gueststore repository. The official docs linked above show how to do this per host with esxcli so I used that example to write a PowerCLI equilivant. This example uses one host has a reference to create the arguments, populates the URL value, then sets the value for each host in cluster NestedCluster03.

$setRepo = (Get-VMHost test-vesx-71* | Get-EsxCli -v2).system.settings.gueststore.repository.set.CreateArgs()
$setRepo.url = 'ds:///vmfs/volumes/ebb8ed5e-48fb2f0b/h045-gueststore'

foreach ($thisHost in (Get-Cluster NestedCluster03 | Get-VMHost | Sort-Object Name )) {
  ($thisHost | Get-EsxCli -v2).system.settings.gueststore.repository.set.Invoke($setRepo)
}

Preparing the Content / sample script

This sets a datastore folder named h045-gueststore as the base folder for my custom content. I then created a script file to demonstrate how to get this file to the VM. The full path of the test file would be [nfs-datastore-a] h045-gueststore/custom/myscript.sh. This script will write the current date/time value from the system, dot source the os-release file, and then write the ‘pretty name’ of the OS to the screen:

#!/bin/sh
echo "The current system time is $(date)"
. /etc/os-release
echo "This system is running ${PRETTY_NAME}."

This is a basic shell script. It is just a sample, we could use this same method to distribute any script or binary file that is 512MB or less.

Guest VM Retrieval with VMware Tools

I then deployed a TinyCore Linux VM, which is super small (my OVA is less than 30MB, including open-vm-tools) and perfect for this type of testing as it deploys very quickly. In this example, the TinyCore VM has no network adapter, CD/DVD drive, or floppy drive that could be used to access script files or other binaries. It does run in the NestedCluster03 which has GuestStore configured.

Inside the guest OS we’ll run the following command to retrieve the file:

/usr/local/bin/vmware-toolbox-cmd gueststore getcontent /custom/myscript.sh /tmp/myscript.sh

Since this file is only a few KB in size, we should see a complete progress bar almost immediately, with confirmation that ‘getcontent’ succeeded, as pictured below.

Running the Script

From here we can make our script executable (chmod +x /tmp/myscript.sh) and then run it (/tmp/myscript.sh).

This script creates a very basic output, but its just a starting point. The real key here isn’t the script, its the process of getting the script to the virtual machine which does not have an alternate method of file transfer.

Conclusion

Configuring GuestStore on ESXi hosts wasn’t difficult. Using VMware Tools to get these files into the guest OS was also straightforward. While there are many ways to get files into a virtual machine, this worked well in this specific case where the VM didn’t have a CD drive or functioning network connectivity.

vCenter Server 8.0 allows administrators to federate identity with Entra ID (formerly Azure AD), enabling seamless SSO and MFA. However, integrating this setup with automation tools like PowerCLI introduces a few challenges. This guide walks through enabling and using PowerCLI with federated logins.

After enabling this federated identity feature, a few additional considerations are required when connecting using PowerCLI. In most Entra ID environments multifactor authentication is enforced, for example via conditional access policy. As such, attempting to login with just a username and password will fail. Here is a sample error response:

> Connect-VIServer vc3.example.com -User h163-user2@lab.enterpriseadmins.org -Password VMware1!

Connect-VIServer : 4/29/2025 6:29:26 PM Connect-VIServer                Cannot complete login due to an incorrect user name or password.
At line:1 char:1
+ Connect-VIServer vc3.example.com -User h163-user2@lab.enterpriseadmin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Connect-VIServer], InvalidLogin
    + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_SoapException,VMware.VimAutomation.ViCore.Cmdlets.Command
   s.ConnectVIServer

The good news is we can still allow clients/end users to login via their federated identities with a little setup.

Administrator / Grant access to PowerCLI User

As an administrator, we’ll create a new OAuth2 client. We will share this client details with the client who wishes to use PowerCLI. In the codeblock below we’ll use splatting to make the code a bit more readable.

$newOAuthArguments = @{
  ClientID     = 'h366-powercli-native-Brian'
  Name         = 'h366-PowerCLI Client2'
  Scope        = @("openid", "user", "group")
  GrantTypes   = @("authorization_code", "refresh_token")
  RedirectUris = @("http://localhost:8844/authcode")
  PkceEnforced = $true
  AccessTokenTimeToLiveMinutes      = 30
  RefreshTokenTimeToLiveMinutes     = 43200
  RefreshTokenIdleTimeToLiveMinutes = 28800
}
$newClient = New-VIOAuth2Client @newOAuthArguments

In the above example, we assigned the output of New-VIOAuth2Client to a variable and did not specify a Secret parameter. With this configuration, a secret will be automatically generated, but that value is not returned in the default output of the cmdlet. We’ll use $newClient.secret to view the new secret of: s1A9RxZ0FbBEGoMplD0HcbQITBODtX85. We’ll need to share the ClientID and Secret value with the person wishing to authenticate.

Client / PowerCLI User

In the step above, our administrator created a New-VIOAuth2Client for us and shared the following details:

• ClientID = h366-powercli-native-Brian
• Secret = s1A9RxZ0FbBEGoMplD0HcbQITBODtX85

We’ll now use those values to login to our vCenter Server using PowerCLI.

$newOAuthArguments = @{
  TokenEndpointUrl         = 'https://test-vcsa-03.lab.enterpriseadmins.org/acs/t/CUSTOMER/token'
  AuthorizationEndpointUrl = 'https://test-vcsa-03.lab.enterpriseadmins.org/acs/t/CUSTOMER/authorize' 
  RedirectUrl              = 'http://localhost:8844/authcode'
  ClientId                 = 'h366-powercli-native-Brian'
  ClientSecret             = 's1A9RxZ0FbBEGoMplD0HcbQITBODtX85'
}

$oauthSecContext = New-OAuthSecurityContext @newOAuthArguments

This results in our default web browser opening to an Azure / Entra AD login page. After successfully entering our credentials, we are redirected to a page that looks like the following image:

The text states: PowerCLI authenticated successfully. Please continue in the PowerShell console. You can close this window now. If you look closely at the URL, you’ll note the page is the RedirectUrl we specified above.

We’ll now take the $ouathSecContext return from the previous codeblock and use it to create a $samlSecContext and use that to connect to our vCenter.

$samlSecContext = New-VISamlSecurityContext -VCenterServer 'test-vcsa-03.lab.enterpriseadmins.org' -OAuthSecurityContext $oauthSecContext
Connect-VIServer -Server 'test-vcsa-03.lab.enterpriseadmins.org' -SamlSecurityContext $samlSecContext

The above commands will return a successful login prompt:

Name                           Port  User
----                           ----  ----
test-vcsa-03.lab.enterprisead… 443   LAB.ENTERPRISEADMINS.ORG\h163…

We can now run PowerCLI cmdlets using our federated identity.

Conclusion

Using Entra ID federation with vCenter Server 8.0 is a great way to step up your security game, especially with MFA in the mix. As we’ve seen, it can trip up tools like PowerCLI if you’re expecting username and password logins.

Thankfully, with a little setup (like creating an OAuth client and using the right security contexts), we can still automate tasks and scripts using federated identity. If this is something your team will do often, it’s worth putting together a quick internal guide or template for setting up new PowerCLI clients. It’ll save time and keep everyone on the same page.