Chapter 20 Events and Event Handling

Table of Contents

20.1 Events
20.2 Event Handling
20.2.1 Event Handlers
20.2.2 Event Handling Page
20.3 Creating Event Handlers
20.3.1 Event Action Log
20.3.2 Suspending an Event Handler

This chapter describes Events and Event Handlers.

Events are displayed if an Advisor Threshold is crossed, and are used to inform you of errors or potential problems with your implementation.

Event handlers define who is notified, and how they are notified, when the thresholds on Advisors are breached and how the event is treated after the status changes.

20.1 Events

Advisors generate events if one, or more, of the defined thresholds are crossed by the monitored value.

Events are displayed on the Events page. Emergency and Critical events also appear on the Overview dashboard. The notification group or groups associated with a specific advisor receive a notification when an alert is triggered. For more information about creating notification groups, see Chapter 20, Events and Event Handling.

To view open events, click the Events link. The tree-view Asset Selector on the left enables you to choose which group's or asset's events are displayed.

Important

The events displayed are dependent on the permission of the role to which you are assigned. If you are a member of a group-specific role, the events displayed are generated by the members of that group, only. Your ability to close events is also governed by the permissions of your role.

For more information, see Chapter 23, Access Control.

The following image shows the Events page with filter enabled:

Figure 20.1 Events Page with Filter

Events page with filter

Event Filter

The Event filter enables you to filter your events based on criteria.

Table 20.1 Events Filter Controls

Name Description

Time Range

Enables you to choose a time range. Possible values are:

  • All Time: filter on all events.

  • Interval: displays the Interval drop-down list, enabling you to filter on a defined range from the current time. For example, if you select 15 minutes, the filter returns events generated in the last 15 minutes, only.

  • From/To: displays From and To fields enabling you to define a date and time range to filter on.

State:

Enables you to choose the events states to filter on. The available choices are Any, Open, or Closed.

Current Status

Enables you to filter on specific current status.

Worst Status

Enables you to filter on specific worst status.

Advisors

Enables you to filter events based on the Advisors which generated them.

Filter

Click to filter the events list on the defined criteria.

Save as Default

Saves the current filter as the default view of the events list.

Reset to Default

Reverts any changes, and restores the saved default criteria.


Events List

The Events list displays all events for the selected group or asset.

Table 20.2 Events List Columns

Name Description

Multi-select checkbox

Enables you to select all events.

Current

Displays an icon indicating the current status of the event.

Worst

Displays an icon indicating the worst status of the event.

Subject

Displays the hostname and location of the problem. For example, if the issue is low disk space on a monitored host, the Subject displays the hostname and the path to the drive which is running out of space. If the issue is related to an instance running on that host, the hostname, and the name and port number of the instance are displayed.

Topic

Displays the short description of the event.

Time

Displays the approximate time the event was generated.

Actions

Displays the possible actions. Click X to open the Close Events dialog.


The Event severities are:

  • Emergency: The event is an emergency and requires immediate attention.

  • Critical: The event is critical and requires immediate attention. Critical events indicate that a serious problem has occurred or is about to occur.

  • Warning: The event is something to investigate and schedule for correction, but does not immediately affect the operation of your server, such as free space on a disk, or a table cache is inadequately sized.

  • Notice: The event is for informational purposes. Notice events call attention to issues that do not affect the operation of your server, such as minor configuration issue.

  • Success: The rule executed successfully with no issues. It also indicates an event, previously in a Critical or Failure state, has returned to normal.

  • Unknown: The current status of the event/rule cannot be determined.

  • Closed: The issue has been corrected and marked closed.

IconDescription
Red and orange flame icon indicates an emergency alert.
Round red icon indicates a critical alert.
Triangular yellow icon indicates a warning.
Blue octagon with the letter "i" indicates an informational alert.
Green check indicates that the Advisor ran successfully and no alert was generated.
Skull icon indicates that the status of the Advisor is unknown.

Closing Events

Advisor's generate events when the threshold defined on the Advisor is breached. Investigate the issue that triggered the event; rectify the issue or problem (or choose to ignore it); then close the event when you are satisfied it does not have a significant impact on your servers.

Some of the advisors identify transient or temporary issues, such as a replication slave being unavailable. For these advisors, you can schedule events to automatically be closed when the event moves from notification status back to the OK state.

When auto-close is enabled, the event remains open while the condition that triggered the event is still in effect. When the condition is no longer in effect, the event is automatically closed. You can also manually close such events before resolving the issue. Events can also be closed by event handlers. For more information on handling events, see Chapter 20, Events and Event Handling.

Important

Not all Advisors generate events which can be auto-closed. See Chapter 21, Expression-Based Advisor Reference and Chapter 22, GUI-Based Advisor Reference for more information on the Advisors which support auto-close.

Events which support auto-close are closed by the Default Auto-close Policy after the event which triggered them is no longer in effect. For more information on Default Auto-close Policy, see Default Auto-close Policy. It is possible to override the Default Auto-close Policy by setting Auto-Close Events to No in an Event Handler

To close an individual event, click the [X] icon in the Actions column. Document the resolution using the Notes text area and choose the Close Events button. During the closing operation, you can also reconfigure the rule scheduling that triggered this event by selecting the checkbox After closing, take me to the page for adjusting schedules of Advisor(s) that reported these events. This option opens the Advisors page and selects the relevant Advisors.

For more information on configuring advisor scheduling and auto closing, see Table 19.3, “Advisor Edit Menu Controls”.

To close a number of alerts simultaneously, select the checkbox beside each event to close and click the Close Selected Events button.

When closing individual or multiple events, a notification window indicates what operations have been completed. The events remain in the displayed event list, but the close link is replaced by a link to the resolution notes. You can update the active list by clicking filter to re-filter the event display.

A historical list of all events, including closed events, is available by setting the Current Severity to Closed. The list shows all of the closed events for a given time range and the servers selected in the server tree. Historical data is limited by the data purge settings. For more information, see Section 25.4, “Data Purge Behavior”

Automatic Closing of Events

If a custom advisor is deleted, or one of the default advisors is made redundant and removed as part of an upgrade, their events can be orphaned. The system automatically closes events which have no advisor linked to them. A note is added to the event stating why it was closed.

Auto-closed events send a notification only if notifications were sent for any previous state transtitions. If no other notifications were sent, no notification is sent for the auto-close.

20.2 Event Handling

This section describes the Event Handling of MySQL Enterprise Service Manager.

20.2.1 Event Handlers

Event handlers are conditions associated with actions. If the condition is met, the action is performed.

Event handler conditions are comprised of the following elements:

  • Groups of assets or individual assets.

    Important

    It is not currently possible to select both groups and individual assets, you must select one or the other.

  • Advisors you want to raise notifications for.

  • Event statuses to trigger the notifications (WARNING, CRITICAL, EMERGENCY, and so on).

The conditions are constructed in the following way:

Group AND Advisor AND Status

while the contents of the elements are OR clauses. For example:

        (Group A OR Group B) AND 
        (Advisor= MySQL Process OR Advisor=CPU Utilization Advisor) AND 
        (status=Warning OR status=Critical)

If the MySQL Process advisor generates a Warning event for one of the contents of Group A, the condition is true and the associated action is triggered. The action can be one of the following:

  • Send an email or SNMP notification if one of the following occurs:

    • The condition evaluates as True.

    • The condition evaluates as True and the status changes to any other status.

    • The condition evaluates as True and the status escalates.

  • Auto-close the event if the current status of the event is OK, but the prior status matched one of those defined in the condition.

20.2.2 Event Handling Page

To display the Event Handling page, select Event Handling from the Configuration drop-down menu.

The Event Handler page is grouped in the following sections:

  • Event Handlers: Lists the event handlers defined on the system. The Default Auto Close Policy is present by default and cannot be edited.

  • Email Notification Groups: lists the email notification groups defined on the system.

  • Email Settings: enables you to define the email configuration, such as SMTP server, username and password to use for all outgoing emails.

  • Email Notification Status: displays the success or failure of the last email sent.

  • SNMP Settings: enables you to define the SNMP trap configuration, such as SNMP version, SNMP targets, and so on.

  • SNMP Notification Status: displays the success or failure of the last SNMP trap sent.

20.2.2.1 Event Handlers List

The Event Handlers section lists all event handlers defined on the system and enables you to create Event Handlers.

Figure 20.2 Event Handlers section

Event handler list section of Event Handling page.

Event Handlers section contains the following controls:

Table 20.3 Event Handler List Controls

Name Description

Create Event Handler

Opens the Create Event Handler dialog. For more information, see Section 20.3, “Creating Event Handlers”.

Show * Entries

Select the maximum number of event handlers to display.

Handler Name

Lists the names of the event handlers.

State

Lists the state of the event handler. Possible states are:

  • Active: the event handler is running.

  • Suspended: the event handler is not running.

Groups

Lists the groups assigned to the event handler.

Assets

Lists the assets assigned to the event handler.

Advisors

Lists the Advisors assigned to the event handler.

Statuses

Lists the statuses assigned to the event handler.

Actions

Lists the SMTP or SNMP actions assigned to the event handler.

Search

Enables you to search for specific event handlers.


Default Auto-close Policy

The Default Auto-close Policy closes events after they change status. If a threshold is defined for an advisor, and the threshold is breached, an event is displayed in the Events page. If it changes status to a lower priority status, or to a status without a defined threshold, the default auto-close policy closes the event.

Note

The Default Auto-close Policy event handler is the only event handler created by default.

This policy does not apply to all Advisors. Some Advisors, such as MySQL Server Has Been Restarted, are too important to auto-close.

Important

It is not possible to edit this Event Handler, but it is possible to override it using the Auto-Close Events option in the Create Event Handler dialog.

20.2.2.2 Email Notification Group Controls

This section describes the controls on the Email Notification Group section.

Figure 20.3 Email Notification Groups section

Email notification groups section on Event Handling page.

The Email Notification Groups contains the following controls:

Table 20.4 Email Notification Groups Controls

Name Description

Create Notification Group button

Opens the Create Notification Group dialog. For more information, see

Group Name

Lists the names of the notification groups.

Recipients

Lists the recipients’ email addresses.

Subject Line

The subject line of the notification emails.

SMS

Status of SMS encoding. The following values are possible:

  • true: SMS encoding is enabled.

  • false: SMS encoding is not enabled.

MEM Admin

Status of emails regarding MySQL Enterprise Monitor. The following values are possible:

  • true: critical MySQL Enterprise Monitor emails will be sent to this notification group.

  • false: no email related to MySQL Enterprise Monitor will be sent to this notification group.


Creating an Email Notification Group

You can define email notification groups using the Create Group dialog. To open the Create Group dialog, click Create Notification Group in the Email Notification Groups section of the Event Handling page.

Figure 20.4 Create Group Dialog

Create notification group dialog.

To create a notification group, do the following:

  1. On the Event Handlers page, select Create Notification Group.

    The Create Group dialog is displayed.

  2. In the Group Name field, specify a group name to uniquely identify this notification group.

  3. In the Recipients field, add a comma-separated list of email addresses. These are the addresses to which the notifications will be sent.

  4. In the Subject Line field, specify the subject line which will be added to every email sent by this notification group.

  5. If required, select SMS (Use SMS encoding for this notification group).

  6. If you want to send information regarding the status of MySQL Enterprise Monitor to the recipients of this notification, select the MEM Admin checkbox. Only critical system messages will be included.

  7. Click Save Notification Group.

    The notification group is available for use in event handlers.

20.2.2.3 Email Settings

The Email Settings section enables you to define the email configuration, such as SMTP server, username and password to use for all outgoing emails.

Figure 20.5 Email Settings section

Email settings section on Event Handling page.

The Email Settings pane contains the following controls:

Table 20.5 Email Settings Controls

Name Description

Enable Email Notifications

Select to activate the email settings controls.

From Address

The email address added to the From field of all emails sent from MySQL Enterprise Monitor.

SMTP server

The outgoing email server address

SMTP Server Login

The username for the SMTP server

Update Password on Save

Select to activate the password fields.

Disable JavaMail TLS/SSL

Select if the SMTP server does not require an encrypted connection.

On Save, Send Test Email Message To

Enter an email address if you want to send a test email when the changes are saved.

Save Email Settings

Saves the Email Settings and sends a test email if an address is defined in the On Save, Send Test Email Message To field.


20.2.2.4 Email Notification Status

The Email Notification Status section displays the success or failure of the last email sent, and an error message describing why the sending failed.

20.2.2.5 SNMP Settings

The SNMP Settings section enables you to define the SNMP trap configuration, such as SNMP version, SNMP targets, and so on.

Figure 20.6 SNMP Settings section

SNMP settings section on Event Handling page.

The SNMP Settings pane contains the following controls:

Table 20.6 SNMP Settings Controls

Name Description

Enable SNMP Notifications

Activates the SNMP configuration fields.

Use SNMP v1/v2

Choose the version of SNMP you intend to use.

Target and Port Number

IP address and Port number of the system which will receive the SNMP Traps.

Community String

SNMP community string. Default value is public.

Use the remote MySQL agent host IP address as the SNMP trap agent address for Advisor traps (optional)

Defines the source IP address included in the trap.

  • Disabled: the trap uses the IP address of the service manager.

  • Enabled: the trap uses the IP address of the agent monitoring the host for which the advisor was triggered.

SNMP trap agent address for internally generated traps (optional)

Defines the source IP address included in traps generated by MySQL Enterprise Service Manager

On Save send test trap

Send a test trap message when Save is clicked. Select one, or more, of the trap types from the list. One trap is sent for each option selected.


20.2.2.6 SNMP Notification Status

The SNMP Notification Status section displays the success or failure of the last trap sent, and an error message describing why the sending failed.

20.3 Creating Event Handlers

Event handlers enable you to create a condition which, when met, triggers notifications to concerned parties such as DBAs, System Administrators and so on.

The following condition criteria can be defined:

  • Assets and Groups: enables you to select multiple assets or multiple groups to monitor.

    Important

    It is possible to define both Assets and Groups in an event handler, but is not recommended. It is recommended that you create the event handler using either Assets or Groups, not both. If you define Assets and Groups in an event handler, notifications will only be sent for the defined Assets which also exist in the defined Groups.

  • Advisors: enables you to select multiple Advisors to evaluate.

  • Event Statuses: enables you to select multiple statuses to monitor.

To create an event handler, click Create Event Handler in the Event Handlers section on the Event Handlers page.

The Create Event Handler dialog is displayed.

Table 20.7 Create Event Handler Controls

Name Description

Event Handler Name

Specify a name which uniquely identifies the new event handler.

Filters

Assets

Select the individual assets to monitor from the Assets drop-down list. If this field is left blank, all assets are included in the event handler's condition, unless one or more groups are defined. If groups are defined, and the asset field is blank, the event handler's condition includes groups only.

The Assets drop-down list displays the Assets in their groups, if groups are defined. If no groups are defined, it lists the assets. It is not possible to select groups in the Assets field. You must expand the group to select individual assets.

Note

If you select the top-level of the asset, all assets are selected. This includes network interfaces, file systems, MySQL instances, and so on. You must expand the asset's entry to select individual assets.

Groups

Select the groups of assets to monitor. If this field is left blank, all groups are included in the event handler's condition, unless one or more assets are defined. If assets are defined, and the group field is blank, the event handler's condition includes assets only.

Advisors

Select the Advisors. If this field is left blank, all advisors are included in the event handler's condition.

Event Statuses

Select the statuses for which you want to receive notifications.

Event Handling

SMTP Notification Groups

Select the groups you want to notify.

SMTP/SNMP Notification Policy

Select one of the following:

  • Notify on event escalation: sends a notification only if the event changes to a higher priority. From Warning to Critical, for example. No notification is sent if the event changes to a lower priority. From Critical to Warning, for example.

  • Notify on any status change: sends a notification if the status changes to any other status.

  • Always notify: sends a notification every time the defined event status(es) are broken. For example, if Critical and Emergency are defined in the Event Status field, and Always notify is selected, a notification will be sent every time the Critical and Emergency events are triggered for the selected Advisors.

SMTP Rate Limit

Defines the maximum number of SMTP notifications which can be sent per minute. The default is 10.

Send SNMP Traps

Defines whether SNMP Traps are used for notifications.

Auto-Close Events

Defines whether the events are closed after the trigger status changes. The following values are possible:

  • Yes: the default auto-close policy is honored and the events are closed when the conditions defined are no longer met.

  • No: the default auto-close policy is ignored and the events remain open in the Events page even after the conditions are no longer met.


Important

If you leave the Assets, Groups, and Advisors fields empty, the event handler can generate an extremely high volume of emails, depending on the number of assets monitored. It is recommended to create event handlers which address specific requirements and contain strictly defined criteria.

Important

Do not define both Assets and Groups, use one or the other.

20.3.1 Event Action Log

If an event handler is triggered, the action taken is displayed in the Action Log [n] section at the bottom of the expanded event, where [n] represents the number of actions logged for that event.

The Action Log [n] lists the time the action was taken, the type of action (SMTP or SNMP), the failure or success of the action, and the triggering policy used to trigger the event. The Triggering Policy column lists the names of the event handlers which triggered the actions.

20.3.2 Suspending an Event Handler

To stop an event handler, select Suspend Event Handler from the event handler's drop-down menu. A prompt is displayed enabling you to enter notes on why the event handler was suspended, and confirm the suspension.

Note

The rights to suspend event handlers depend on the Event Blackout permission. If this right is not granted to your role, it is not possible to suspend an event handler.