Enterprise Admins.org

In a prior post (https://enterpriseadmins.org/blog/scripting/event-driven-automation-with-aria-operations-for-logs-webhooks-and-jenkins/), I wrote about using Aria Operations for Logs to call Jenkins using a webhook to automate a process. In that post, Aria Operations for Logs is watching for a specific vSphere event and when that occurs (VM deleted), we check a couple other systems (Active Directory & Aria Automation Config) and make sure the cleanup activates cascade to those other systems.

In this article we are going to do the same type of thing (one event triggers another), but instead of looking a vSphere events, we are going to use an event that comes from an agent deployed to Windows systems, specifically domain controllers, when a flexible single-master operator (FSMO) role is moved. As with the previous example, we could really look for any event, this is just something that is easy to trigger for demonstration purposes.

In my experience, these FSMO roles are only rarely moved in a stable production environment. However, when the PDC Emulator role is moved it is possible that some firewall rules may need to be modified. This is because all clients will send requests to the system holding this role in the event of a bad password attempt (https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/fsmo-roles). Since this type of change occurs rather infrequently it is something that can be easily forgotten. To help with this, we are going to create a similar event driven approach to alert on these events and start the appropriate actions.

Aria Operations for Logs – webhook payload

This webhook will be identical to the payload used for our previous example, with one important change. We’ve updated the ‘token’ portion of the URL we will be posting to.

• Endpoint Type: Custom
• Log Payload: Individual Logs
• Webhook URL: http://jenkins.example.com/generic-webhook-trigger/invoke?token=dcFsmoRoleMoved
• Action: POST
• Content Type: JSON
• Webhook body: ${messages}

Aria Operations for Logs – alert definition

The Windows event entry for this is event ID 1458, so we’ll look for that happening. Our alert definition will be very simple:

For testing, I’m using the same ‘Real Time’ query which executes every minute. Due to the infrequent nature of this event, we could likely scale this back quite a bit. However, for building the rule & testing its execution having it run more frequently was helpful.

Jenkins – New Freestyle project

As with the previous example, Jenkins is going to receive this alert via webhook and then run our custom script.

With the FSMO role moves, we will actually receive two events — one from the domain controller which previously held the role, and another event from the domain controller receiving the role. To prevent duplicate actions from triggering, the logic of our code is only going to take action for the event received from the domain controller receiving the role. This will help reduce alert fatigue.

Aria Operations for Logs is sending an event message which contains the event text as well as some additional extracted fields. In our previous post, we only needed to read the text of the message, but in this case we also need the extracted field showing which system was the source of our event. To make this happen, our post content parameter we’ll call logmsg but make the expression be the JSONPath $[0], which is the full event message and not just the text field used in our previous example.

As an example, our code looks like this… again it is just an example for illustration purposes. We could do anything we want/need with this event.

$jsonEvent = $env:logmsg | ConvertFrom-JSON

$previousOwnerCN = [regex]::Match( $jsonEvent.text.split("`n")[7], ",CN=(.*?),CN=Servers").Groups[1].Value.Trim()
$newOwnerCN = [regex]::Match( $jsonEvent.text.split("`n")[5], ",CN=(.*?),CN=Servers").Groups[1].Value.Trim()
$eventSourceHost = ($jsonEvent.fields | ?{$_.name -eq 'hostname'}).content.Trim()
$requestuser = ($jsonEvent.fields | ?{$_.name -eq 'userid'}).content.Trim()

 # Both domain controllers involved in the transfer will send an event message.  To eliminate duplicate actions, 
 # we will only action the event which comes from the new owner of the FSMO role.
if ($eventSourceHost -match $newOwnerCN) {
  $msgObject = $jsonEvent.text.split("`n")[3]
  $dnAsFqdn  = ($msgObject -replace '^.*?,DC=' -replace ',DC=','.').Trim()

  $movedRole = switch ($msgObject) {
    { $_ -match '^DC=' } { 'PDC Emulator - domain: ' + $dnAsFqdn }
    { $_ -match '^CN=RID Manager'} { 'RID Master - domain: ' + $dnAsFqdn }
    { $_ -match '^CN=Infrastructure'} { 'Infrastructure Master - domain: ' + $dnAsFqdn }
    { $_ -match '^CN=Schema'} { 'Schema Master - forest: ' + $dnAsFqdn }
    { $_ -match '^CN=Partitions'} { 'Domain Naming Master - forest: ' + $dnAsFqdn }
  }
  
  $friendlyMessage = "The role $movedRole has been moved to $newOwnerCN from $previousOwnerCN by $($requestuser)."
  if ($friendlyMessage -match 'PDC Emulator') { 
    $destIP = ([System.Net.Dns]::GetHostAddresses($eventSourceHost) | ?{$_.AddressFamily -eq 'InterNetwork'})[0].IPAddressToString
    $fwcrSubmitt = New-FirewallChangeRequest -requestor $requestuser -hostGroup 'Win-AD-Services' -changeType 'add' -ipAddress $destIP
    $friendlyMessage += '  For this role, additional firewall changes may be needed.  To help ensure this has not been overlooked, firewall change request '+$fwcrSubmitt+' has been submitted.'
  }
  $friendlyMessage
  
  Send-MailMessage -to 'bwuchner@example.com' -from 'jenkins@example.com' -Subject 'Domain Controller: FSMO Role moved' -Body $friendlyMessage -SmtpServer 'mail.example.com'
} else {
  "This event is being skipped as the eventSourceHost was $eventSourceHost which is not the newOwner $($newOwnerCN)."; exit 1
}

Note: the New-FirewallChangeRequest function is not covered by this post, but it is an example custom function to submit an internal form

Testing

Testing this process was rather easy in a lab. From Active Directory Users and Computers, you can change the operations master, moving it from one domain controller to another. Each move should trigger two Jenkins builds with only one sending an email notification. The other build will exit as an error before a message occurs. In the case of a PDC Emulator move, the email text has a bit more detail and a firewall change request gets submitted. Here is a sample of the email notification sent for a PDC Emulator move. This text is also logged as ‘console output’ for the Jenkins build.

The role PDC Emulator - domain: enterpriseadmins.org has been moved to DR-CONTROL-21 from CORE-CONTROL-21 by LAB\bwuchner.  For this role, additional firewall changes may be needed.  To help ensure this has not been overlooked, firewall change request 123d1fd1-895a-47e9-9779-88dc33a32a16 has been submitted.
Finished: SUCCESS

Conclusion

This is another example of using Aria Operations for Logs to trigger a specific event to drive automation. With this webhook alert allowing us to connect the two systems, any event that occurs, whether vSphere, Windows agent, syslog stream, or anything else, can be used as the starting point to take action or send very specific notifications to any system we choose.