Use Terraform to provision alerting resources
Use Terraform’s Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code.
This guide outlines the steps and references to provision alerting resources with Terraform. For a practical demo, you can clone and try this example using Grafana OSS and Docker Compose.
To create and manage your alerting resources using Terraform, you have to complete the following tasks.
- Create an API key to configure the Terraform provider.
- Create your alerting resources in Terraform format by
- exporting configured alerting resources
- or writing the Terraform Alerting schemas.
By default, you cannot edit provisioned resources. Enable
disable_provenance
in the Terraform resource to allow changes in the Grafana UI.
- Run
terraform apply
to provision your alerting resources.
Before you begin, you should have available a Grafana instance and Terraform installed on your machine.
Create an API key and configure the Terraform provider
You can create a service account token to authenticate Terraform with Grafana. To create an API key for provisioning alerting resources, complete the following steps.
- Create a new service account.
- Assign the role or permission to access the Alerting provisioning API.
- Create a new service account token.
- Name and save the token for use in Terraform.
You can now move to the working directory for your Terraform configurations, and create a file named main.tf
like:
terraform {
required_providers {
grafana = {
source = "grafana/grafana"
version = ">= 2.9.0"
}
}
}
provider "grafana" {
url = <grafana-url>
auth = <api-key>
}
Replace the following values:
<grafana-url>
with the URL of the Grafana instance.<api-key>
with the API token previously created.
This Terraform configuration installs the Grafana Terraform provider and authenticates against your Grafana instance using an API token. For other authentication alternatives including basic authentication, refer to the auth
option documentation.
For Grafana Cloud, refer to the instructions to manage a Grafana Cloud stack with Terraform. For role-based access control, refer to Provisioning RBAC with Terraform and the alerting provisioning roles (fixed:alerting.provisioning.*
).
Create Terraform configurations for alerting resources
Grafana Terraform provider enables you to manage the following alerting resources.
This section provides examples of Terraform configurations for each alerting resource and demonstrate how to link them together.
Add alert rules
Alert rules enable you to receive alerts by querying any backend Grafana data sources.
First, create a data source to query and a folder to store your rules in.
In this example, the TestData data source is used.
terraformresource "grafana_data_source" "<terraform_data_source_name>" { name = "TestData" type = "testdata" } resource "grafana_folder" "<terraform_folder_name>" { title = "My Rule Folder" }
Replace the following field values:
<terraform_data_source_name>
with the terraform name of the data source.<terraform_folder_name>
with the terraform name of the folder.
Create or find an alert rule you want to import in Grafana.
Export the alert rule group in Terraform format. This exports the alert rule group as
grafana_rule_group
Terraform resource.You can edit the exported resource, or alternatively, consider creating the resource from scratch.
terraformresource "grafana_rule_group" "<terraform_rule_group_name>" { name = "My Alert Rules" folder_uid = grafana_folder.<terraform_folder_name>.uid interval_seconds = 60 org_id = 1 rule { name = "My Random Walk Alert" condition = "C" for = "0s" // Query the datasource. data { ref_id = "A" relative_time_range { from = 600 to = 0 } datasource_uid = grafana_data_source.<terraform_data_source_name>.uid // `model` is a JSON blob that sends datasource-specific data. // It's different for every datasource. The alert's query is defined here. model = jsonencode({ intervalMs = 1000 maxDataPoints = 43200 refId = "A" }) } // The query was configured to obtain data from the last 60 seconds. Let's alert on the average value of that series using a Reduce stage. data { datasource_uid = "__expr__" // You can also create a rule in the UI, then GET that rule to obtain the JSON. // This can be helpful when using more complex reduce expressions. model = <<EOT {"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"last"},"type":"avg"}],"datasource":{"name":"Expression","type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"last","refId":"B","type":"reduce"} EOT ref_id = "B" relative_time_range { from = 0 to = 0 } } // Now, let's use a math expression as our threshold. // We want to alert when the value of stage "B" above exceeds 70. data { datasource_uid = "__expr__" ref_id = "C" relative_time_range { from = 0 to = 0 } model = jsonencode({ expression = "$B > 70" type = "math" refId = "C" }) } } }
Replace the following field values:
<terraform_rule_group_name>
with the name of the alert rule group.
Note that the distinct Grafana resources are connected through
uid
values in their Terraform configurations. Theuid
value is randomly generated when provisioning.To link the alert rule group with its respective data source and folder in this example, replace the following field values:
<terraform_data_source_name>
with the terraform name of the previously defined data source.<terraform_folder_name>
with the terraform name of the previously defined folder.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add contact points
Contact points are the receivers of alert notifications.
Create or find the contact points you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the contact point in Terraform format. This exports the contact point as
grafana_contact_point
Terraform resource—edit it if necessary.In this example, notifications are muted on weekends.
terraformresource "grafana_contact_point" "<terraform_contact_point_name>" { name = "My contact point email" email { addresses = ["<email_address>"] } }
Replace the following field values:
<terraform_contact_point_name>
with the terraform name of the contact point. It is used to reference the contact point in other Terraform resources.<email_address>
with the email to receive alert notifications.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add and enable templates
Notification templates allow customization of alert notifications across multiple contact points.
Create or find the notification template you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the template as
grafana_message_template
Terraform resource.This example is a simple demo template defined as
custom_email.message
.terraformresource "grafana_message_template" "<terraform_message_template_name>" { name = "custom_email.message" template = <<EOT {{ define "custom_email.message" }} Lorem ipsum - Custom alert! {{ end }} EOT }
In the previous contact point, enable the template by setting the
email.message
property as follows.terraformresource "grafana_contact_point" "<terraform_contact_point_name>" { name = "My contact point email" email { addresses = ["<email_address>"] message = "{{ template \"custom_email.message\" .}}" } }
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add mute timings
Mute timings pause alert notifications during predetermined intervals.
Create or find the mute timings you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the mute timing in Terraform format. This exports the mute timing as
grafana_mute_timing
Terraform resource—edit it if necessary.This example turns off notifications on weekends.
terraformresource "grafana_mute_timing" "<terraform_mute_timing_name>" { name = "No weekends" intervals { weekdays = ["saturday", "sunday"] } }
Replace the following field values:
<terraform_mute_timing_name>
with the name of the Terraform resource. It is used to reference the mute timing in the Terraform notification policy tree.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add the notification policy tree
Notification policies defines how to route alert instances to your contact points.
Warning
Since the policy tree is a single resource, provisioning thegrafana_notification_policy
resource overwrites a policy tree created through any other means.
Find the default notification policy tree. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the notification policy tree in Terraform format. This exports it as
grafana_notification_policy
Terraform resource—edit it if necessary.terraformresource "grafana_notification_policy" "my_policy_tree" { contact_point = grafana_contact_point.<terraform_contact_point_name>.name ... policy { contact_point = grafana_contact_point.<terraform_contact_point_name>.name matcher {...} mute_timings = [grafana_mute_timing.<terraform_mute_timing_name>.name] } }
To configure the mute timing and contact point previously created in the notification policy tree, replace the following field values:
<terraform_data_source_name>
with the terraform name of the previously defined contact point.<terraform_folder_name>
with the terraform name of the previously defined mute timing.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Enable editing resources in the Grafana UI
By default, you cannot edit resources provisioned via Terraform in Grafana. This ensures that your alerting stack always stays in sync with your Terraform code.
To make provisioned resources editable in the Grafana UI, enable the disable_provenance
attribute on alerting resources.
resource "grafana_contact_point" "my_contact_point" {
name = "My Contact Point"
disable_provenance = true
}
resource "grafana_message_template" "my_template" {
name = "My Reusable Template"
template = "{{define \"My Reusable Template\" }}\n template content\n{{ end }}"
disable_provenance = true
}
...
Provision Grafana resources with Terraform
To create the previous alerting resources in Grafana with the Terraform CLI, complete the following steps.
Initialize the working directory containing the Terraform configuration files.
shellterraform init
This command initializes the Terraform directory, installing the Grafana Terraform provider configured in the
main.tf
file.Apply the Terraform configuration files to provision the resources.
shellterraform apply
Before applying any changes to Grafana, Terraform displays the execution plan and requests your approval.
shellPlan: 4 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value:
After you have confirmed to proceed with the changes, Terraform creates the provisioned resources in Grafana.
shellApply complete! Resources: 4 added, 0 changed, 0 destroyed.
You can now access Grafana to verify the creation of the distinct resources.
More examples
For more examples on the concept of this guide:
- Try the demo provisioning alerting resources in Grafana OSS using Terraform and Docker Compose.
- Review all the available options and examples of the Terraform Alerting schemas in the Grafana Terraform Provider documentation.
- Review the tutorial to manage a Grafana Cloud stack using Terraform.
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)