Troubleshooting vCenter permission errors

I was recently helping troubleshoot an issue where a service account was configured with the least privileges possible. When the service attempted to perform a specific operation, an access denied message was encountered. The service performing this action immediately cleaned up after itself, deleting the virtual machine that was created.

Typically in the UI we can see a warning event on an object when a required privilege is missing. For example, in the following screenshot a read only service account attempted to change the CPU Count for a virtual machine. This operation failed due to a missing permission, but we can clearly see the missing privilege is VirtualMachine.Config.CPUCount.

However, in our specific case the affected object was destroyed automatically, and we didn’t have an opportunity to view this event in the UI on our specific VM. We could have likely found this event on a parent object, but the environment had a lot of events occurring, making it difficult to find in the UI. Instead, we used PowerCLI to filter the logs for what we needed. In this sample we are using Get-VIEvent to query for all events in the last 15 minutes, then filter on the client side where the event text contains my service account and the Event Type is the specific NoPermission event we were interested in.

Get-VIEvent -Types Warning -Start (Get-Date).AddMinutes(-15) -Finish (Get-Date) | 
Where-Object {$_.FullFormattedMessage -match 'svc-vspherero' -AND 
       $_.EventTypeId -eq 'com.vmware.vc.authorization.NoPermission'} |
Select-Object FullFormattedMessage, ObjectName

This worked for our case. Another option would be to use the Get-VIEventPlus function from here: https://www.lucd.info/2013/03/31/get-the-vmotionsvmotion-history/. Using this custom function, a EventFilterSpec is created where we can have vCenter only return the NoPermission events. This is more efficient than doing the query on the client side and lets us return more applicable events. Using the sample below, we can group our NoPermission events and see how many times they occurred.

$allNoPerm = Get-VIEventPlus -EventType com.vmware.vc.authorization.NoPermission
$allNoPerm | Group-Object -Property FullFormattedMessage | 
  select-object Name, Count | Sort-Object -Property Count -Descending

For example, using the above code in my lab I found some additional service accounts that were missing required permissions. I was able to review documentation and confirm that the required privileges were updated between when I initially created the custom role and the current version of the service.

I hope these sample queries help identify missing privileges if needed.