Get started with Grafana Alerting

By Antonio Calero Merello

Last update on May 13, 2024

Beginner

Introduction

In this guide, we’ll walk you through the process of setting up your first alert in just a few minutes. You’ll witness your alert in action with real-time data, as well as sending alert notifications.

In this tutorial you will:

Set up an Alert
Send an alert notification to a public webhook.

Before you begin

Ensure you have the following applications installed.

Docker Compose (included in Docker for Desktop for macOS and Windows)
Git

Set up a sample application

The sample application generates real data and exposes metrics, which are stored in Prometheus. In Grafana Alerting, you can then build an alert rule based on the data generated.

Download the files to your local machine.

Clone the tutorial environment repository.

git clone https://github.com/grafana/tutorial-environment.git

Change to the directory where you cloned the repository:
```
cd tutorial-environment
```
Make sure Docker is running:
```
docker --version
```
This command will display the installed Docker version if the installation was successful.
Start the sample application:
```
docker compose up -d
```
The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.
Note
If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
Ensure all services are up-and-running:
```
docker compose ps
```
In the State column, it should say Up for all services.

The Grafana News app should be live on localhost:8081.

Generate data

To enhance the hands-on aspect of this tutorial, you can actively participate in the Grafana News application to simulate web traffic and interactions. This enables you to observe data within Grafana and set up alerts accordingly.

Grafana News

Grafana News is an application created to demonstrate the observation of data using the Grafana stack. It achieves this by generating web traffic through activities such as posting links and voting for your preferred ones.

To add a link:

Enter a Title
Enter a URL
Click Submit to add the link. The link will appear listed under the Grafana News heading.
To vote for a link, click the triangle icon next to the name of the link.

Create a contact point for Grafana Managed Alerts

Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.

In this step, we’ll set up a new contact point. This contact point will use the webhooks channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert. We will use requestbin.com to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.

In your browser, navigate to localhost:3000, where Grafana is running. You should get logged in automatically.
In another window, go to requestbin.com.
Under the Create Request Bin button, click the link to create a public bin instead.
From Request Bin, copy the endpoint URL.

Your Request Bin is now waiting for the first request.

Next, let’s configure a Contact Point in Grafana’s Alerting UI to send notifications to our Request Bin.

Return to Grafana. In Grafana’s sidebar, hover over the Alerting (bell) icon and then click Contact points.
Click + Add contact point.
In Name, write RequestBin.
In Integration, choose Webhook.
In URL, paste the endpoint to your request bin.
Click Test, and then click Send test notification to send a test alert to your request bin.
Navigate back to the Request Bin you created earlier. On the left side, there’s now a POST / entry. Click it to see what information Grafana sent.
Return to Grafana and click Save contact point.

We have created a dummy webhook endpoint and created a new Alerting Contact Point in Grafana. Now, we can create an alert rule and link it to this new channel.

Create an alert

Next, we’ll establish an alert within Grafana Alerting to notify us whenever our sample app experiences a specific volume of requests.

In Grafana, navigate to Alerting > Alert rules. Click on New alert rule.

Enter alert rule name Make it short and descriptive as this will appear in your alert notification. For instance, server-requests-duration

Define query and alert condition

In this section, we define queries, expressions (used to manipulate the data), and the condition that must be met for the alert to be triggered.

Select the Prometheus data source from the drop-down menu.
Note
To visualize this data in Grafana, we need time-series metrics that we can collect and store. We can do that with Prometheus, which pulls metrics from our sample app.
In the Query editor, switch to Code mode by clicking the button at the right.
Enter the following query:
promql
```
sum(rate(tns_request_duration_seconds_count[1m])) by(method)
```
This PromQL query calculates the sum of the per-second average rates of increase of the tns_request_duration_seconds_count metric over the last 1 minute, grouped by the HTTP method used in the requests. This can be useful for analyzing the request duration trends for different HTTP methods.
Remove expression “B” (click the bin icon). The Reduce expression comes by default, and in this case, it is not needed since the queried data is already reduced.
In expression “C”, Threshold, change the Input to “A”. By selecting the input, you’re telling Grafana which data source and which specific metric within that data source the expression should evaluate. The Threshold expression is now the alert condition. Enter 0.2 as threshold value. You can read more about queries and conditions here.
Click Preview to run the queries.
You should see the request duration for different HTTP methods.
Note
If it returns “No data,” or an error, you are welcome to post questions in our Grafana Community forum.

Set evaluation behavior

An evaluation group defines an evaluation interval - how often a rule is checked. Alert rules within the same evaluation group are evaluated sequentially

In Folder, click + New folder and enter a name. For example: grafana-news. This folder will contain our alerts.
In the Evaluation group, repeat the above step to create a new evaluation group. We will name it 1m-evaluation.
Choose an Evaluation interval (how often the alert will be evaluated). For example, every 1m (1 minute).
Set the pending period (the “for” period). This is the time that a condition has to be met until the alert enters into Firing state and a notification is sent. For example, 0s (zero seconds) so the alert rule fires the moment the condition is met.

Configure labels and notifications

Add labels to ease searching or route notifications to a policy.

Add a label.Add app as the label key, and grafana-news as the value.
Add a notification recipient. Under Contact point, select RequestBin from the drop-down menu.
Add an annotation (optional).
To provide more context on the alert, you can link a dashboard and panel to our Alert. To do this, click Link Dashboard and panel button.
Linking an alert rule to a panel adds an annotation to the panel when the status of your alert rulechanges. If you don’t have a panel already, and since this is optional, you can skip this step for now and link it after you have finished configuring the alert rule.
Click Save rule and exit at the top right corner.

Trigger an alert

We have now configured an alert rule and a contact point. Now let’s see if we can trigger an alert by generating some traffic on our sample application.

Browse to localhost:8081.
Add a new title and URL.
Repeatedly click the vote button or refresh the page to generate a traffic spike.

Once the query sum(rate(tns_request_duration_seconds_count[1m])) by(method) returns a value greater than 0.2, Grafana will trigger our alert. Browse to the Request Bin we created earlier and find the sent Grafana alert notification with details and metadata.

Summary

In this tutorial, you have learned how to set up an alert, send alert notifications to a public webhook, and generate some sample data to observe your alert in action. By following these steps, you’ve gained a foundational understanding of how to leverage Grafana’s alerting capabilities to monitor and respond to events of interest in your data. Happy monitoring!

Learn more

Check out the links below to continue your learning journey with Grafana’s LGTM stack.