A customer recently asked me for assistance with using the alert functionality in vRLI to generate alerts in vROps based on events that appear in the logging data in vRLI. The use case was to generate an alert based on an event that occurs on any ESXi host configured to send its logs to vRLI. While there is documentation on how to configure the alerts they were not getting the expected results, and were unsure how to use it to generate alerts across multiple objects in vROps from a single alert definition.
Some prerequisites:
- vRLI must be configured with an integration for the vROps instance you want to send alerts to
- The option for Enable Alerts integration must be selected on the integration in vRLI
- The user account you are logged in to vRLI with must have sufficient permissions to create/edit alerts
For this blog post, I will use an event frequently appearing in the logs of my ESXi hosts, which is related to NTP configuration to demonstrate the steps to forward an alert to vROps. All of the ESXi hosts I am using are configured to forward logs to vRLI and are monitored by vROps.
The first step is identifying the query you need to use in vRLI to generate the alert. There are different ways to do this. You may already know the filters you want to apply for the query. For my example, I added the filter hostname contains <ESXi hostnames> for the time period of the past hour. I then used the Event Types page to scroll through the different events to locate an example that had repeated a small number of times in the past hour. I did this so that I could be sure an alert would be triggered in vRLI without too many alerts being generated as my lab environment is resource constrained.
With the query identified I can now configure my alert.
- On the Alerts>Alert Definitions page select Create New Alert
- Name the Alert, provide a description and configure the Query to match the filters required for the target event
- Under the Trigger Conditions section I have chosen to trigger the alert when the event happens in real time rather than on a schedule. I have left the notification section of Trigger Conditions empty as I don’t want vRLI to send an email when the alert is triggered.
- I have selected to foward the alert to vROps and selected my first ESXi host as the fallback object. I’ll explain this option in a bit more detail later in this post.
- I have set my criticality level to Warning, selected not to Auto cancel the alert and added a small amount of text to be used in the alert recommendation within vROps.
Once the Alert is Saved, it will automatically be enabled. It is important to understand that there will be a time delay between the event being triggered in vRLI (which is also dependent on how frequently vRLI polls or receives data from the selected endpoints) to generate an alert, and the alert appearing on the object in vROps. This is because vROps monitors the connection with vRLI as a scheduled job and not in real time, therefore approximately every 10 minutes (from my testing, not a documented value as far as I could find) vROps will check if vRLI has triggered any new alerts, or if any existing alerts have been triggered against additional objects. Once this job is complete vROps will generate the required alerts against the relevant objects.
When the alert is created in vROps it will have an alert name that starts Notification event <vRLI Alert Name>. If the alert has been triggered against multiple objects you will see multiple alerts, as shown in my example below with the two ESXi hosts.
When you select the alert and view the details you will notice that the text I entered as the recommendation in vRLI is actually shown in the Description field and not the Recommendations field.
Within my alert I selected two ESXi host names as part of the trigger conditions, however you will notice that when configuring the vROps forwarding I only selected esx-01a.corp.local as the Fallback object. You may be wondering why you can only select one object, or why you shouldn’t leave this field empty. This is the part that my customer found the most confusing, and I spoke with the VMware product management team to get further clarification. Firstly why didn’t I just leave the field empty. In vRLI 8.10 and earlier the field is mandatory, even though it is not marked as such within the GUI. If you leave the field empty and save your alert the configuration for forwarding to vROps will show up as empty when you review the alert. This will hopefully be resolved in a future release.
Secondly why did I select esx-01a.corp.local as the Fallback object when I have two ESXi hosts in my event filters, why didn’t I select the vCenter object for example. The documentation states When integrated with vRealize Operations 6.0 and above, alerts are sent as notifications to the virtual machines, ESXi hosts, or vCenter Server objects that caused the alert. Alerts raised by other entities are sent to the selected fallback object.
This means that the fallback object would not be used for my example alert definition. This is because my ESXi hosts are monitored by vROPs, and so the alerts would be generated against those ESXi host objects. Therefore I could have selected any vROps object as my Fallback object. A key point to understand here is how vRLI knows that the object is monitored by vROps. Within the Interactive Analytics query results there is a field named vmw_vr_ops_id. The value of this field is a unique identifier assigned to the source of the event, in my example this is the ESXi host esx-01a.corp.local which has an id value of a197925b-d1e1-499f-aa57-16bd361bd2a2.
This value should match to the guid of the object in vROps. You can check the guid using the Object Browser in vROps to select the object. The guid will be displayed in the URL of the page.
If vRLI and vROps have a matching vmw_vr_ops_id and guid value then the alert will be generated on the correct object in vROps, even if multiple objects are part of the event trigger. If the value does not match, or the object is not monitored by vROps then the fallback object will be used as the source of the alert by vROps.
Another thing to note from my testing is that sometimes the alert does not appear in vROps on the Troubleshoot>Alerts page event once it has been successfully triggered in vRLI and created in vROps. If this happens you will still be able to see the Alert on the different objects that triggered the alert, esx-01a.corp.local or esx-02a.corp.local in my example by browsing to those objects in vROps.
