Centralized Startup Scripting for Automated VM Load Testing

When building or troubleshooting infrastructure, it is often useful to simulate high CPU or memory usage withou deploying full production workloads. To assist with this, I previously created a few purpose built, like cpubusy.sh and memfill.sh.

Historically, I created multiple Tiny Core Linux templates, each designed for a specific purpose, a generic one for troubleshooting, one that would fill mem and a another to load up the CPU. I’d then place these scripts in /opt/bootlocal.sh, allowing each VM to run its designated script automatically at startup. I’d then control load by simply powering VMs on or off.

The Problem

That setup works fine for simple use cases, but it doesn’t scale well. What if I want:

  • One VM to simulate CPU load,
  • Another to test download speeds,
  • A third to run a custom test script—all using the same base image?

The Common Control Script

This script runs at boot and checks for specific instructions to execute. The idea is simple: deploy a single generic VM template that decides what to do based on either:

  • Metadata (via guestinfo)
  • Network identity (IP, MAC, hostname)
  • Shared config (via GuestStore or a web server)

This common control script can be found here: code-snips/cc.sh.

Where It Looks for Instructions

When a VM boots, the script checks for commands in the following order:

  1. Web server:
    • http://<web>/<macaddress>.txt
    • http://<web>/<ipaddress>.txt
    • http://<web>/<hostname>.txt
    • http://<web>/all.txt
  2. VM Guest Store (using VMware tools):
    • guestinfo.ccScript (specified via advanced VM setting)
    • /custom/cc/cc-all.txt

This layered approach gives flexibility:

  • Set a global script via all.txt
  • Override per host via metadata or identifiers
  • Or push custom scripts directly via GuestInfo

Example: Setting the GuestInfo Property

Using PowerCLI, we can set the script filename per VM like this:

Get-VM h045-tc16-02 | New-AdvancedSetting -Name 'guestinfo.ccScript' -Value 'memfill.sh' -confirm:$false

We can also modify this via the vSphere Web Client.

Demonstration

Here’s a test case from my lab:

  • I set guestinfo.ccScript to memfill.sh
  • The all.txt file includes a simple command to print system time

Upon boot, the VM fills 90% of available RAM using a memory-backed filesystem and prints the time, confirming that both script sources are active.

Later, I removed the guestinfo.ccScript setting and added a <hostname>.txt script to download a file repeatedly from a test web server. After reboot, the VM behaved differently, now acting as a network test client. No changes to the template required.

Sample Scripts

Here are a few lightweight test scripts used in the demo:

  • cpubusy.sh – uses sha1sum to keep all the configured CPU cores busy
  • download.sh – uses wget to get the same webserver file x times and save it to /dev/null
  • memfill.sh – creates a memory backed filesystem using 90% of RAM, then uses dd to fill it

Conclusion

This ‘common config’ approach provides template reuse, easier script management, and dynamic testing control, all without modifying the template.

Whether testing CPU, memory, or network load across dozens of VMs, the common control script simplifies the process and reduces maintenance overhead.

In future iterations, this setup could be extended to include conditional logic (based on boot time, VM tags, or other metadata), or integration with CI pipelines for even more powerful automation.

Posted in Scripting, Virtualization | Leave a comment

Using GuestStore to Deliver Content to Network-Isolated VMs in vSphere

When working with VMs that lack network connectivity, transferring files can be tricky. I recently explored the GuestStore feature a built-in vSphere feature that solves this challenge by allowing file delivery via VMware Tools—even without a network. This post walks through how I used GuestStore to push a script into a TinyCore Linux VM with no network, CD drive, or external storage options.

The vSphere documentation does a good job explaining what GuestStore is and how it can be used: Distributing Content with GuestStore – https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-virtual-machine-administration-guide-8-0/managing-virtual-machinesvsphere-vm-admin/distributing-content-with-gueststorevsphere-vm-admin.html

Configuring ESXi Hosts for GuestStore

To begin, I configured all of the hosts in a cluster to use the same NFS datastore as a gueststore repository. The official docs linked above show how to do this per host with esxcli so I used that example to write a PowerCLI equilivant. This example uses one host has a reference to create the arguments, populates the URL value, then sets the value for each host in cluster NestedCluster03.

$setRepo = (Get-VMHost test-vesx-71* | Get-EsxCli -v2).system.settings.gueststore.repository.set.CreateArgs()
$setRepo.url = 'ds:///vmfs/volumes/ebb8ed5e-48fb2f0b/h045-gueststore'

foreach ($thisHost in (Get-Cluster NestedCluster03 | Get-VMHost | Sort-Object Name )) {
  ($thisHost | Get-EsxCli -v2).system.settings.gueststore.repository.set.Invoke($setRepo)
}

Preparing the Content / sample script

This sets a datastore folder named h045-gueststore as the base folder for my custom content. I then created a script file to demonstrate how to get this file to the VM. The full path of the test file would be [nfs-datastore-a] h045-gueststore/custom/myscript.sh. This script will write the current date/time value from the system, dot source the os-release file, and then write the ‘pretty name’ of the OS to the screen:

#!/bin/sh
echo "The current system time is $(date)"
. /etc/os-release
echo "This system is running ${PRETTY_NAME}."

This is a basic shell script. It is just a sample, we could use this same method to distribute any script or binary file that is 512MB or less.

Guest VM Retrieval with VMware Tools

I then deployed a TinyCore Linux VM, which is super small (my OVA is less than 30MB, including open-vm-tools) and perfect for this type of testing as it deploys very quickly. In this example, the TinyCore VM has no network adapter, CD/DVD drive, or floppy drive that could be used to access script files or other binaries. It does run in the NestedCluster03 which has GuestStore configured.

Inside the guest OS we’ll run the following command to retrieve the file:

/usr/local/bin/vmware-toolbox-cmd gueststore getcontent /custom/myscript.sh /tmp/myscript.sh

Since this file is only a few KB in size, we should see a complete progress bar almost immediately, with confirmation that ‘getcontent’ succeeded, as pictured below.

Running the Script

From here we can make our script executable (chmod +x /tmp/myscript.sh) and then run it (/tmp/myscript.sh).

This script creates a very basic output, but its just a starting point. The real key here isn’t the script, its the process of getting the script to the virtual machine which does not have an alternate method of file transfer.

Conclusion

Configuring GuestStore on ESXi hosts wasn’t difficult. Using VMware Tools to get these files into the guest OS was also straightforward. While there are many ways to get files into a virtual machine, this worked well in this specific case where the VM didn’t have a CD drive or functioning network connectivity.

Posted in Scripting, Virtualization | Leave a comment

Comparing Installed Packages on Photon OS Using PowerShell and SSH

When debugging inconsistencies between Photon OS systems, say one is failing and another is stable, it’s useful to compare their installed package versions. In one recent case, I needed a quick way to do just that from my admin workstation. Here’s how I solved it using PowerShell and the Posh-SSH module.

In this test case, both hosts have a user account with the same name/password, so only one credential was created.

# Prompt for SSH credentials
$creds = Get-Credential

# Connect to Host 1 and get package list as JSON
$host1        = '192.168.10.135'
$host1session = New-SSHSession -ComputerName $host1 -Credential $creds -AcceptKey
$host1json    = (Invoke-SSHCommand -Command 'tdnf list installed -json' -SessionId $host1session.SessionId).Output | ConvertFrom-Json

# Connect to Host 2 and get package list as JSON
$host2        = '192.168.127.174'
$host2session = New-SSHSession -ComputerName $host2 -Credential $creds -AcceptKey
$host2json    = (Invoke-SSHCommand -Command 'tdnf list installed -json' -SessionId $host2session.SessionId).Output | ConvertFrom-Json

# Compare the resulting package lists
$compared = Compare-Object -ReferenceObject $host1json -DifferenceObject $host2json -Property Name, Evr

# Group the results by package name and build tabular results for side-by-side compare
foreach ($thisPackage in ($compared | Group-Object -Property Name)) {
  [pscustomobject][ordered]@{
    Name = $thisPackage.Name
    $host1 = ($thisPackage.Group | ?{$_.SideIndicator -eq '<='}).Evr
    $host2 = ($thisPackage.Group | ?{$_.SideIndicator -eq '=>'}).Evr
  }
}

The script gets a list of all installed packages from each host as JSON (using tdnf list installed -json), then converts the JSON output to a powershell object.
The two list of installed packages are then compared using Compare-Object.
Finally, we loop through each unique package and create a new object to compare the versions side by side.

I’ve included the first 10 rows of output below for reference.

Name                           192.168.10.135       192.168.127.174
----                           --------------       ---------------
cloud-init                     24.3.1-1.ph4         25.1-1.ph4
curl                           8.7.1-4.ph4          8.12.0-1.ph4
curl-libs                      8.7.1-4.ph4          8.12.0-1.ph4
elfutils                       0.181-7.ph4          0.181-8.ph4
elfutils-libelf                0.181-7.ph4          0.181-8.ph4
expat                          2.4.9-3.ph4          2.4.9-4.ph4
expat-libs                     2.4.9-3.ph4          2.4.9-4.ph4
gettext                        0.21-4.ph4           0.21-5.ph4
glib                           2.68.4-2.ph4         2.68.4-4.ph4
glibc                          2.32-19.ph4          2.32-20.ph4

Looking at this output, we can see which packages are different between our two hosts.

Conclusion

Comparing installed packages across Photon OS systems can be an invaluable troubleshooting and auditing tool – especially when dealing with configuration drift, unexpected behavior, or undocumented changes. By using PowerShell and the Posh-SSH module, you can quickly automate the comparison process without needing to log in to each system manually. Hopefully, this gives you a solid starting point for your own comparisons and debugging tasks.

Posted in Scripting | Leave a comment

How to Use PowerCLI with Entra ID Federated vCenter Logins

vCenter Server 8.0 allows administrators to federate identity with Entra ID (formerly Azure AD), enabling seamless SSO and MFA. However, integrating this setup with automation tools like PowerCLI introduces a few challenges. This guide walks through enabling and using PowerCLI with federated logins.

After enabling this federated identity feature, a few additional considerations are required when connecting using PowerCLI. In most Entra ID environments multifactor authentication is enforced, for example via conditional access policy. As such, attempting to login with just a username and password will fail. Here is a sample error response:

> Connect-VIServer vc3.example.com -User h163-user2@lab.enterpriseadmins.org -Password VMware1!

Connect-VIServer : 4/29/2025 6:29:26 PM Connect-VIServer                Cannot complete login due to an incorrect user name or password.
At line:1 char:1
+ Connect-VIServer vc3.example.com -User h163-user2@lab.enterpriseadmin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Connect-VIServer], InvalidLogin
    + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_SoapException,VMware.VimAutomation.ViCore.Cmdlets.Command
   s.ConnectVIServer

The good news is we can still allow clients/end users to login via their federated identities with a little setup.

Administrator / Grant access to PowerCLI User

As an administrator, we’ll create a new OAuth2 client. We will share this client details with the client who wishes to use PowerCLI. In the codeblock below we’ll use splatting to make the code a bit more readable.

$newOAuthArguments = @{
  ClientID     = 'h366-powercli-native-Brian'
  Name         = 'h366-PowerCLI Client2'
  Scope        = @("openid", "user", "group")
  GrantTypes   = @("authorization_code", "refresh_token")
  RedirectUris = @("http://localhost:8844/authcode")
  PkceEnforced = $true
  AccessTokenTimeToLiveMinutes      = 30
  RefreshTokenTimeToLiveMinutes     = 43200
  RefreshTokenIdleTimeToLiveMinutes = 28800
}
$newClient = New-VIOAuth2Client @newOAuthArguments

In the above example, we assigned the output of New-VIOAuth2Client to a variable and did not specify a Secret parameter. With this configuration, a secret will be automatically generated, but that value is not returned in the default output of the cmdlet. We’ll use $newClient.secret to view the new secret of: s1A9RxZ0FbBEGoMplD0HcbQITBODtX85. We’ll need to share the ClientID and Secret value with the person wishing to authenticate.

Client / PowerCLI User

In the step above, our administrator created a New-VIOAuth2Client for us and shared the following details:

  • ClientID = h366-powercli-native-Brian
  • Secret = s1A9RxZ0FbBEGoMplD0HcbQITBODtX85

We’ll now use those values to login to our vCenter Server using PowerCLI.

$newOAuthArguments = @{
  TokenEndpointUrl         = 'https://test-vcsa-03.lab.enterpriseadmins.org/acs/t/CUSTOMER/token'
  AuthorizationEndpointUrl = 'https://test-vcsa-03.lab.enterpriseadmins.org/acs/t/CUSTOMER/authorize' 
  RedirectUrl              = 'http://localhost:8844/authcode'
  ClientId                 = 'h366-powercli-native-Brian'
  ClientSecret             = 's1A9RxZ0FbBEGoMplD0HcbQITBODtX85'
}

$oauthSecContext = New-OAuthSecurityContext @newOAuthArguments

This results in our default web browser opening to an Azure / Entra AD login page. After successfully entering our credentials, we are redirected to a page that looks like the following image:

The text states: PowerCLI authenticated successfully. Please continue in the PowerShell console. You can close this window now. If you look closely at the URL, you’ll note the page is the RedirectUrl we specified above.

We’ll now take the $ouathSecContext return from the previous codeblock and use it to create a $samlSecContext and use that to connect to our vCenter.

$samlSecContext = New-VISamlSecurityContext -VCenterServer 'test-vcsa-03.lab.enterpriseadmins.org' -OAuthSecurityContext $oauthSecContext
Connect-VIServer -Server 'test-vcsa-03.lab.enterpriseadmins.org' -SamlSecurityContext $samlSecContext

The above commands will return a successful login prompt:

Name                           Port  User
----                           ----  ----
test-vcsa-03.lab.enterprisead… 443   LAB.ENTERPRISEADMINS.ORG\h163…

We can now run PowerCLI cmdlets using our federated identity.

Conclusion

Using Entra ID federation with vCenter Server 8.0 is a great way to step up your security game, especially with MFA in the mix. As we’ve seen, it can trip up tools like PowerCLI if you’re expecting username and password logins.

Thankfully, with a little setup (like creating an OAuth client and using the right security contexts), we can still automate tasks and scripts using federated identity. If this is something your team will do often, it’s worth putting together a quick internal guide or template for setting up new PowerCLI clients. It’ll save time and keep everyone on the same page.

Posted in Scripting, Virtualization | 3 Comments

How to Increase VMXNET3 Link Speed for Nested ESXi Hosts

I recently came across a KB article (https://knowledge.broadcom.com/external/article/368812/how-to-change-the-vmxnet3-link-speed-of.html) that describes how to set the link speed of a vmxnet3 network adapter higher than the default 10 Gbps speed. In my lab I have several nested ESXi hosts and wanted to give this a try.

I powered down a nested host, edited the virtual machine settings, and added the advanced option ethernet0.linkspeed, setting its value to 25000 as pictured below.

ethernet0.linkspeed with the value 25000

After powering the virtual machine back on, I checked the ESXi network settings in vCenter and confirmed that the updated link speed was reflected, pictured below.

Expanding the vmnic0 details, I could see the adapter was still of type vmxnet3, and the link speed was correctly set to the specified value.

Conclusion

Updating virtual machine link speeds work for nested ESXi hosts as well. The valid range for linkspeed is 10000 to 65000. This makes common speeds, like 25 Gbps and 40 Gbps possible.

Posted in Lab Infrastructure, Virtualization | Leave a comment