{"id":2436,"date":"2026-09-24T08:51:35","date_gmt":"2026-09-24T12:51:35","guid":{"rendered":"https:\/\/enterpriseadmins.org\/blog\/?p=2436"},"modified":"2026-09-24T08:51:35","modified_gmt":"2026-09-24T12:51:35","slug":"migrating-salt-minions-to-a-new-master","status":"publish","type":"post","link":"https:\/\/enterpriseadmins.org\/blog\/scripting\/migrating-salt-minions-to-a-new-master\/","title":{"rendered":"Migrating Salt Minions to a New Master"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">This is a follow-up to a <a href=\"https:\/\/enterpriseadmins.org\/blog\/virtualization\/replacing-aria-automation-config-with-vmware-salt-single-node-in-my-homelab\/\">previous post<\/a>, where I stood up a new single-node VMware Salt deployment to replace my old Aria Automation Config setup. This post covers the actual cutover: moving every existing minion in the homelab from the old master (<code>cm-config-01<\/code>) to the new one (<code>cm-vcfsalt-01<\/code>).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Salt minion can be manually installed on a system or included as part of the VMware Tools install.  When deployed through VMware Tools, settings are typically configured by advanced settings on the VM, instead of maintaining configuration files inside of the VM.  From my prior setup, I had a mismatch of configurations, with some VMs using manually installed, older versions of the minion, and other VMs using the VMware Tools option.  As part of this migration, I wanted to standardize on one option and decided on the VMware Tools path.  <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1: Inventory what&#8217;s currently configured<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before changing anything, I pulled a report of every VM&#8217;s current master setting via PowerCLI.  This is limited to only the existing VMs that were configured to use the VMware Tools salt minion option.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$saltSettings = Get-VM | Get-AdvancedSetting -Name \"guestinfo.\/vmware.components.salt_minion.*\"\n$groupedSettings = $saltSettings | Group-Object -Property Entity\n$results = foreach ($group in $groupedSettings) {\n    &#91;PSCustomObject]@{\n        VMName                 = $group.Name\n        SaltMinionArgs         = ($group.Group | Where-Object Name -match 'salt_minion.args$').Value\n        SaltMinionDesiredState = ($group.Group | Where-Object Name -match 'salt_minion.desiredstate$').Value\n    }\n}\n$results | Format-Table -AutoSize<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2: Bulk-update the advanced setting<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For VMs that already had the setting defined, updating in bulk is straightforward.  We get the current setting and set its value.  This can be done with mainly one change, assuming the &#8216;DesiredState&#8217; value is already set to &#8216;present&#8217;.  Similar syntax can be used to change the desired state setting if needed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Get-VM net-wanrtr-02,svcs-mongo-1* | Get-AdvancedSetting -Name 'guestinfo.\/vmware.components.salt_minion.args' | Set-AdvancedSetting -Value 'master=cm-vcfsalt-01.lab.enterpriseadmins.org' -Confirm:$false<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For VMs where the setting didn&#8217;t exist yet (never previously managed by Salt and\/or used guest OS manual salt minion installs), <code>Set-AdvancedSetting<\/code> will fail silently on that host as the existing setting is not found to set.  Instead, we need to create the advanced setting using <code>New-AdvancedSetting<\/code> instead:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Get-VM h331-minion-01 | New-AdvancedSetting -Name 'guestinfo.\/vmware.components.salt_minion.args' -Value 'master=cm-vcfsalt-01.lab.enterpriseadmins.org' -Confirm:$false\n\nGet-VM h331-minion-01 | New-AdvancedSetting -Name 'guestinfo.\/vmware.components.salt_minion.desiredstate' -Value 'present' -Confirm:$false <\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3: Re-trigger the minion install<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For new VMs where salt was not previously configured, the installation\/configuration of the minion happened automatically (as the salt-minion was already installed as part of VMware Tools, toggling the &#8216;desiredstate&#8217; key triggered the configuration.  However, on most of the other previoulsy configured VMs, the configuration didn&#8217;t occur automatically.  I suspect removing the desiredstate key, waiting a bit, and then re-adding the key may have triggered an uninstall\/reinstall, but I didn&#8217;t test that path.  Instead I manually triggered the component script inside the guest.  For example, on some Ubuntu Linux VMs I ran the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Get-VM $vmList | Invoke-VMScript -ScriptText \"sudo \/usr\/lib\/x86_64-linux-gnu\/open-vm-tools\/componentMgr\/saltMinion\/svtminion.sh\" -GuestUser '&lt;user&gt;' -GuestPassword '&lt;pass&gt;'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A few things I ran into here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Powered-off VMs caused error messages to appear.  I could have filtered them out with a filter like <code>?{$_.PowerState -eq 'PoweredOn'}<\/code> first.<\/li>\n\n\n\n<li><code>-RunAsync<\/code> is possible for the <code>Invoke-VMscript<\/code> cmdlet.  If running this for a long list of VMs, the code above will process one VM at a time.  For a lab this is fine, but for a longer list of VMs, adding <code>-RunAsync<\/code> will kick the commands off much faster.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">&#8220;Invalid master key&#8221; errors<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After repointing several minions, some did not show up in my Salt console.  Checking those minions with <code>systemctl status salt-minion<\/code> showed the error <code>Unable to sign_in to master: Invalid master key<\/code>. The fix was removing the cached master public key and restarting the service:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo rm \/etc\/salt\/pki\/minion\/minion_master.pub\nsudo systemctl restart salt-minion<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Windows minion troubleshooting<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">On Linux, reconfiguring VMs used a <code>svtminion.sh<\/code> script.  When troubleshooting a couple of Windows VMs, I found a: <code>C:\\Program Files\\VMware\\VMware Tools\\componentMgr\\saltMinion\\svtminion.ps1<\/code> that has similar functionality.  Even after a clean reinstall (<code>svtminion.ps1 -Remove<\/code> then <code>-Install<\/code>), a couple of Windows minions kept showing repeated warnings in <code>C:\\ProgramData\\Salt Project\\Salt\\var\\log\\salt\\minion<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>The minion failed to return the job information for job &lt;id>. This is often due to the master being shut down or overloaded.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Deleting the stale <code>minion_master.pub<\/code>, restarting the service, even running <code>salt-minion.exe -l debug<\/code> to watch debug level logs didn&#8217;t shed much light.  A simple reboot of the guest OS resolved this issue in both cases. If you hit the same &#8220;reinstalled fine, keys accepted, but still no data&#8221; symptom on Windows, try a reboot before going deeper down the debugging rabbit hole.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Setting up the new Salt service didn&#8217;t take long, and moving minions over took less time than one round of manual patching.  With all my VMs moved over to a new Salt instance, I can now enable automated patch again.  To set this up, I used the same steps that I documented ~3 years ago <a href=\"https:\/\/enterpriseadmins.org\/blog\/scripting\/keeping-linux-up-to-date-with-aria-automation-config-part-2\/\">in this blog post<\/a>.  Other than a more frequent\/automated patch cycle, on additional benefit is that I now have documentation showing prior\/current package versions and a date\/time stamp when they were updated.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a follow-up to a previous post, where I stood up a new single-node VMware Salt deployment to replace my old Aria Automation Config setup. This post covers the actual cutover: moving every existing minion in the homelab from &hellip; <a href=\"https:\/\/enterpriseadmins.org\/blog\/scripting\/migrating-salt-minions-to-a-new-master\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[9,3],"tags":[],"class_list":["post-2436","post","type-post","status-publish","format-standard","hentry","category-lab-infrastructure","category-scripting"],"_links":{"self":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2436","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/comments?post=2436"}],"version-history":[{"count":2,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2436\/revisions"}],"predecessor-version":[{"id":2440,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2436\/revisions\/2440"}],"wp:attachment":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/media?parent=2436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/categories?post=2436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/tags?post=2436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}