Alerts Notification Setup Guide
This document provides a detailed overview of how to create alert rules for effective monitoring and alerting.
Introduction
An alert rule defines the conditions under which a notification should be triggered. It includes queries to select and evaluate specific datasets, a threshold-based alert condition that determines when the alert should fire, an evaluation interval and duration that specify how often the rule runs and how long the condition must be met, and additional customizable options such as handling missing data, configuring notification messages, and more. Alert rules form the foundation for generating meaningful alerts based on monitored metrics or system behavior.
Alert Notification Setup Workflow Overview
Setting up alerts allows you to proactively monitor your system and get notified when specific conditions occur. The process involves two main components that need to be configured in sequence:
Notification Channel: This defines how and where alert notifications will be sent (e.g., to a specific email address, a Slack channel, etc.). Think of this as setting up the "delivery method" for your alerts.
Alert Rule: This defines what condition should be monitored (e.g., CPU usage exceeding 90%) and when an alert should be triggered. This rule will then use a pre-configured Notification Channel to send the actual notification.
Creating an Email Notification Channel
Navigate to Notification Channels
Look at the left-hand sidebar menu.
Click on Notification Channels.
Start Creating a New Channel
On the 'Notification Channels' page, click the blue Add Channel button in the top-right corner.
Select Channel Type and Name
You'll be on the 'Select Notification Channel' screen.
In the Channel Name field, type a descriptive name, like Email Notification Channel or Email.
In the Channel dropdown on the right, select Email (or any other communication mode listed like Slack).
Configure Email Details
A new section 'Configure Notification Channel' will appear.
In the Recipient Addresses separated by ';' field, enter the email address (or multiple addresses separated by semicolons ;) where you want to receive alerts. For example: [email protected];[email protected].
(Optional) You can add a standard message in the Optional Message field if you like.
Test the Channel (Recommended)
Click the Test Channel button.
A pop-up 'Test Notification Channel' will appear.
Click Send Message.
Check the email inbox you provided. You should receive a test message confirming the integration works.
Click the Add Notification Channel button at the bottom.
Publish the Channel
You'll be taken back to the 'Notification Channels' list.
Find the channel you just created. It will have a DRAFT status next to it.
Click the Publish button associated with your new draft channel.
Your channel status should change to LIVE.
Creating the Alert Rule
Navigate to Alerts
Look at the left-hand sidebar menu.
Click on Alerts.
Make sure the Custom Alerts tab is selected.
Click the blue Add Alert Rule button in the top-right corner.
Select Notification Channel for this Rule(Optional)
On the 'Add Alert Rule' page, the first section is 'Notification Channel'.
Click the dropdown menu under Notification Channels.
Select the Notification Channel you created.
Build the Query (Define What to Monitor)
Go to the 'Query Builder' section.
Click the Component dropdown and select the system you want to monitor. Example Infra.
Click the Metric dropdown that appears. Select the specific pre-registered metric you care about. Example: [KAFKA]: High CPU Usage Detected. System Under Heavy Load.
Choose the Operator (how to compare the metric). Example: Greater than (>).
Enter the Threshold value that triggers the alert. Example: 90 (meaning alert if CPU usage is greater than 90%).
Preview the Metric (Optional but Recommended)
Click the Run Query button.
A graph will appear showing the historical data for the metric you selected. This helps confirm you're monitoring the right thing.
Configure Rule Details
Scroll down to the 'Rule Configuration' section.
Enter a descriptive Name for your alert rule. Example: Kafka High CPU Warning.
Enter a Description explaining what this alert means. Example: High CPU usage in Kafka may impact data processing and system performance.
Select a Severity level (e.g., Warning, Critical). Example: Warning.
Adjust the Frequency and Interval sliders.
Interval: How far back the data is checked (e.g., check the last 5 minutes of data).
Frequency: How often the rule is checked (e.g., check every 5 minutes).
Add Labels (Optional but useful for organization)
Go to the 'Labels' section.
You can add tags (key-value pairs) to categorize your alerts.
Click the Create Rule button at the bottom to save the alert rule.
Publish the Alert Rule
You'll be taken back to the 'Alert Rules' list.
Find the alert rule you just created. It will have a DRAFT status.
Click the Publish button associated with your new draft rule.
The status should change (likely to ACTIVE or similar, though the image shows the list before publishing the second rule).
Viewing Alert Rule Details
On the 'Alert Rules' page (ensure Custom Alerts tab is selected), find the specific alert rule you want to view in the list.
In the row for that alert rule, look under the Actions column.
Click the eye icon.
You will be taken to a detailed 'View' page for that alert rule.
This page displays all the configured information like Labels, Status, Description, Severity, Frequency, Interval, the Notification Channel it uses, and more.
You cannot directly edit fields on this page, but you can use the buttons at the top right (like Edit, Mute) to perform further actions if needed. Click Back to return to the list.
Silencing an Alert Rule
On the 'Alert Rules' page (ensure Custom Alerts tab is selected), find the specific alert rule you want to silence.
In the row for that alert rule, look under the Actions column, Click the bell icon.
Click the bell icon.
A pop-up window titled 'Silence Alert for:' will appear.
Select how long you want to stop receiving notifications for this specific alert(Example for 1 hour, for 2 hour), click on your desired duration
The alert manager will stop sending notifications to that alert rule for the specified duration, even if the alert condition is met.
The alert continues to be evaluated in the background, but no messages will be sent to contact points.
After the silencing period ends, notifications will resume automatically if the alert condition is still active.
Note: For instructions on modifying an existing rule, see Modify Existing Alerts.
Last updated