vCenter Migration

I had a big day at the office a couple of days ago.  I moved my vCenter application from one dedicated server to another.  That doesn’t sound like a huge ordeal, but it was a pain in the rear.  I stumbled along with a lot of help from Google and I wanted to outline some lessons learned for everyone else.

I actually had a few objectives I wanted to achieve:

• Migrate from Windows 2003 x32 to Windows 2008 x64
• Replace my out-of-warranty/physical vCenter server with newer hardware
• Change the authentication model for the remote database from SQL to Windows Integrated
• Re-IP vCenter to be on my management VLAN

I attempted this upgrade a little over a week ago and near the end of my change control I realized that something was wrong with the Windows 2008 server which prevented the event log from opening and the service from starting.  Since I didn’t want to leave this in production I backed out my change and rescheduled.

The second attempt at this install went much better.  I had my 2008 x64 server rebuilt by my server team and the event log was working fine (I verified before attempting my install).  The vCenter install went well and I was able to achieve the first three of my objectives.  The final one was a bit more work — since I changed my vCenter IP I had to manually reconnect each of my hosts.  It was expected and only took fifteen minutes or so.

After I had visibility into each of my hosts and basic vCenter functionality I started digging a bit deeper to verify my change was successful.  When I clicked on the “vCenter Service Status” button I found a problem.  All of the metrics had errors that they couldn’t connect to the old name of my vCenter server.  I poked around the console and changed my VirtualCenter.VimApiUrl and VirtualCenter.VimWebServicesUrl in the advanced vCenter settings but was unsuccessful.  Luckily I was able to find this VMware Communities article that suggested using ADSIEdit to delete an entire container from the vCenter ADAM instance (CN=<GUID_of_vCenter>, OU=ComponentSpecs, OU=Health, DC=virtualcenter, DC=vmware, DC=int).  Thanks, bgarner!

Once I was able to see the service status I realized that my SQL agent stat rollup tasks (weekly/monthly) were not running properly.  Part of my install included changing the way I access my remote database from using MS-SQL authentication to Windows Integrated.  During the first attempt of the install I received an error that one of the rollup tasks in the SQL agent was owned by the SQL user and had to be owned by my Windows user to continue.  I decided the smartest thing to do would be to delete all three rollup tasks and simply let the install re-create them.  Turns out this wasn’t the best idea — for some reason the install only recreated the daily rollup task and not the weekly or monthly.  I didn’t save the link (shame on me) but I found a blog entry suggesting that I re-run job_schedule2_mssql.sql and job_schedule3_mssql.sql from my Program Files\VMware\Infrastructure\VirtualCenter Server directory.  I ran the scripts and verified that the jobs existed in my SQL Agent.  They were scheduled to run several hours in the future — so my vCenter Service Status still showed a warning.  (For the record, I checked back the next morning and everything was fine.)

Finally, another annoyance I’ve been dealing with for awhile was a plugin from a CapacityIQ proof-of-concept that showed up in Plugins > Manage Plungins.  I was no longer using the product but I couldn’t seem to get rid of that plugin.  I have removed plugins from the client machine several times but this one seemed to follow me around.  Fortunately another VMware Communities article came to the rescue.  The article specifically references vCenter Chargeback but the process worked for CapIQ.  The key was finding the correct methods in the https://vcenterserver.mydomain.local/mob web service — that I didn’t really understand or know about.

I hope these links help if you ever encounter a similar problem.