Migrating Salt Minions to a New Master

This is a follow-up to a previous post, where I stood up a new single-node VMware Salt deployment to replace my old Aria Automation Config setup. This post covers the actual cutover: moving every existing minion in the homelab from the old master (cm-config-01) to the new one (cm-vcfsalt-01).

The Salt minion can be manually installed on a system or included as part of the VMware Tools install. When deployed through VMware Tools, settings are typically configured by advanced settings on the VM, instead of maintaining configuration files inside of the VM. From my prior setup, I had a mismatch of configurations, with some VMs using manually installed, older versions of the minion, and other VMs using the VMware Tools option. As part of this migration, I wanted to standardize on one option and decided on the VMware Tools path.

Step 1: Inventory what’s currently configured

Before changing anything, I pulled a report of every VM’s current master setting via PowerCLI. This is limited to only the existing VMs that were configured to use the VMware Tools salt minion option.

$saltSettings = Get-VM | Get-AdvancedSetting -Name "guestinfo./vmware.components.salt_minion.*"
$groupedSettings = $saltSettings | Group-Object -Property Entity
$results = foreach ($group in $groupedSettings) {
    [PSCustomObject]@{
        VMName                 = $group.Name
        SaltMinionArgs         = ($group.Group | Where-Object Name -match 'salt_minion.args$').Value
        SaltMinionDesiredState = ($group.Group | Where-Object Name -match 'salt_minion.desiredstate$').Value
    }
}
$results | Format-Table -AutoSize

Step 2: Bulk-update the advanced setting

For VMs that already had the setting defined, updating in bulk is straightforward. We get the current setting and set its value. This can be done with mainly one change, assuming the ‘DesiredState’ value is already set to ‘present’. Similar syntax can be used to change the desired state setting if needed.

Get-VM net-wanrtr-02,svcs-mongo-1* | Get-AdvancedSetting -Name 'guestinfo./vmware.components.salt_minion.args' | Set-AdvancedSetting -Value 'master=cm-vcfsalt-01.lab.enterpriseadmins.org' -Confirm:$false

For VMs where the setting didn’t exist yet (never previously managed by Salt and/or used guest OS manual salt minion installs), Set-AdvancedSetting will fail silently on that host as the existing setting is not found to set. Instead, we need to create the advanced setting using New-AdvancedSetting instead:

Get-VM h331-minion-01 | New-AdvancedSetting -Name 'guestinfo./vmware.components.salt_minion.args' -Value 'master=cm-vcfsalt-01.lab.enterpriseadmins.org' -Confirm:$false

Get-VM h331-minion-01 | New-AdvancedSetting -Name 'guestinfo./vmware.components.salt_minion.desiredstate' -Value 'present' -Confirm:$false 

Step 3: Re-trigger the minion install

For new VMs where salt was not previously configured, the installation/configuration of the minion happened automatically (as the salt-minion was already installed as part of VMware Tools, toggling the ‘desiredstate’ key triggered the configuration. However, on most of the other previoulsy configured VMs, the configuration didn’t occur automatically. I suspect removing the desiredstate key, waiting a bit, and then re-adding the key may have triggered an uninstall/reinstall, but I didn’t test that path. Instead I manually triggered the component script inside the guest. For example, on some Ubuntu Linux VMs I ran the following:

Get-VM $vmList | Invoke-VMScript -ScriptText "sudo /usr/lib/x86_64-linux-gnu/open-vm-tools/componentMgr/saltMinion/svtminion.sh" -GuestUser '<user>' -GuestPassword '<pass>'

A few things I ran into here:

• Powered-off VMs caused error messages to appear. I could have filtered them out with a filter like ?{$_.PowerState -eq 'PoweredOn'} first.
• -RunAsync is possible for the Invoke-VMscript cmdlet. If running this for a long list of VMs, the code above will process one VM at a time. For a lab this is fine, but for a longer list of VMs, adding -RunAsync will kick the commands off much faster.

“Invalid master key” errors

After repointing several minions, some did not show up in my Salt console. Checking those minions with systemctl status salt-minion showed the error Unable to sign_in to master: Invalid master key. The fix was removing the cached master public key and restarting the service:

sudo rm /etc/salt/pki/minion/minion_master.pub
sudo systemctl restart salt-minion

Windows minion troubleshooting

On Linux, reconfiguring VMs used a svtminion.sh script. When troubleshooting a couple of Windows VMs, I found a: C:\Program Files\VMware\VMware Tools\componentMgr\saltMinion\svtminion.ps1 that has similar functionality. Even after a clean reinstall (svtminion.ps1 -Remove then -Install), a couple of Windows minions kept showing repeated warnings in C:\ProgramData\Salt Project\Salt\var\log\salt\minion:

The minion failed to return the job information for job <id>. This is often due to the master being shut down or overloaded.

Deleting the stale minion_master.pub, restarting the service, even running salt-minion.exe -l debug to watch debug level logs didn’t shed much light. A simple reboot of the guest OS resolved this issue in both cases. If you hit the same “reinstalled fine, keys accepted, but still no data” symptom on Windows, try a reboot before going deeper down the debugging rabbit hole.

Conclusion

Setting up the new Salt service didn’t take long, and moving minions over took less time than one round of manual patching. With all my VMs moved over to a new Salt instance, I can now enable automated patch again. To set this up, I used the same steps that I documented ~3 years ago in this blog post. Other than a more frequent/automated patch cycle, on additional benefit is that I now have documentation showing prior/current package versions and a date/time stamp when they were updated.