DocumentationGrafana CloudAlerts and IRMAlertingIntroductionNotificationsTemplates

Templates

Use templating to customize, format, and reuse alert notification messages. Create more flexible and informative alert notification messages by incorporating dynamic content, such as metric values, labels, and other contextual information.

In Grafana, there are two ways to template your alert notification messages:

1. Labels and annotations

   • Template labels and annotations in alert rules.
   • Labels and annotations contain information about an alert.
   • Labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.

2. Notification templates

   • Template notifications in contact points.
   • Add notification templates to contact points for reuse and consistent messaging in your notifications.
   • Use notification templates to change the title, message, and format of the message in your notifications.

This diagram illustrates the entire process of templating, from the creation of labels and annotations in alert rules or notification templates in contact points, to what they look like when exported and applied in your alert notification messages.

In this diagram:

• Monitored Application: A web server, database, or any other service generating metrics. For example, it could be an NGINX server providing metrics about request rates, response times, and so on.
• Prometheus: Prometheus collects metrics from the monitored application. For example, it might scrape metrics from the NGINX server, including labels like instance (the server hostname) and job (the service name).
• Grafana: Grafana queries Prometheus to retrieve metrics data. For example, you might create an alert rule to monitor NGINX request rates over time, and template labels or annotations based on the instance label.
• Alertmanager: Part of the Prometheus ecosystem, Alertmanager handles alert notifications. For example, if the request rate exceeds a certain threshold on a particular NGINX server, Alertmanager can send an alert notification to, for example, Slack or email, including the server name and the exceeded threshold (the instance label will be interpolated, and the actual server name will appear in the alert notification).
• Alert notification: When an alert rule condition is met, Alertmanager sends a notification to various channels such as Slack, Grafana OnCall, etc. These notifications can include information from the labels associated with the alerting rule. For example, if an alert triggers due to high CPU usage on a specific server, the notification message can include details like server name (instance label), disk usage percentage, and the threshold that was exceeded.

Labels and annotations

Labels and annotations contain information about an alert. Labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.

Template labels

Label templates are applied in the alert rule itself (i.e. in the Configure labels and notifications section of an alert).

Templating can be applied by using variables and functions. These variables can represent dynamic values retrieved from your data queries.

Here are some commonly used built-in variables to interact with the name and value of labels in Grafana alerting:

• The $labels variable, which contains all labels from the query.

  For example, let’s say you have an alert rule that triggers when the CPU usage exceeds a certain threshold. You want to create annotations that provide additional context when this alert is triggered, such as including the specific server that experienced the high CPU usage.

    The host {{ index $labels "instance" }} has exceeded 80% CPU usage for the last 5 minutes
  

  The outcome of this template would print:

    The host instance 1 has exceeded 80% CPU usage for the last 5 minutes
  

• The $value variable, which is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule.

  In the context of the previous example, $value variable would write something like this:

    CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ $value }}
  

  The outcome of this template would print:

    CPU usage for instance1 has exceeded 80% for the last 5 minutes: [ var='A' labels={instance=instance1} value=81.234 ]
  

• The $values variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs (i.e. the id that identifies the query or expression. By default the Red ID of the query is “A”).

  Given an alert with the labels instance=server1 and an instant query with the value 81.2345, would write like this:

    CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "A" }}
  

  And it would print:

    CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345
  

Template annotations

Both labels and annotations have the same structure: a set of named values; however their intended uses are different. The purpose of annotations is to add additional information to existing alerts.

There are a number of suggested annotations in Grafana such as description, summary, runbook_url, dashboardUId and panelId. Like labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired.

Here is an example of templating an annotation in the context of an alert rule. The text/template is added into the Add annotations section.

    CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes

The outcome of this template would print

    CPU usage for Instance 1 has exceeded 80% for the last 5 minutes

Template notifications

Notification templates represent the alternative approach to templating designed for reusing templates. Notifications are messages to inform users about events or conditions triggered by alerts. You can create reusable notification templates to customize the content and format of alert notifications. Variables, labels, or other context-specific details can be added to the templates to dynamically insert information like metric values.

Here is an example of a notification template:

Go Copy

{ define "alerts.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s)
{{ template "alerts.summarize" .Alerts.Firing }}
{{- end }}
{{- if .Alerts.Resolved -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ template "alerts.summarize" .Alerts.Resolved }}
{{- end }}
{{- end }}

{{ define "alerts.summarize" -}}
{{ range . -}}
- {{ index .Annotations "summary" }}
{{ end }}
{{ end }}

This is the message you would receive in your contact point:

        1 firing alert(s)
        - The database server db1 has exceeded 75% of available disk space. Disk space used is 76%, please resize the disk size within the next 24 hours

        1 resolved alert(s)
        - The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes

Once the template is created, you need to make reference to it in your Contact point (in the Optional [contact point] settings) .