Managing OnCall on Grafana Cloud using Terraform
Learn how to connect an integration to Grafana OnCall, configure escalation policies, and add your on-call schedule to Grafana OnCall using Terraform.
In this guide, We will be using two hypothetical teams named Devs
and SREs
for showcasing how Grafana OnCall can be used by more than one team in your organization.
We will also connect Slack as the ChatOps tool for Grafana OnCall in this guide.
Prerequisites
Before you begin, you should have the following available:
- A Grafana Cloud account, as shown in Get started
- Terraform installed on your machine
- Admin permissions in your Grafana instance as well as the Slack workspace
Note
All of the following Terraform configuration files should be saved in the same directory.
Connect Slack to Grafana OnCall
To connect Slack to Grafana OnCall, use the Slack integration for Grafana OnCall
Terraform configuration for Grafana provider
This Terraform configuration configures the Grafana provider to provide necessary authentication when managing Grafana OnCall.
Create an API Token in Grafana OnCall. You’ll need this key to configure Terraform.
- Log into your Grafana Cloud instance.
- Click Alerts & IRM and then select OnCall.
- Click Settings.
- Under API Tokens, Click +Create.
- Enter a name for your API Token and click Create Token.
- Click Copy Token to save it for later use.
Create a file named
main.tf
and add the following code block:terraformterraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.9.0" } } } provider "grafana" { alias = "oncall" oncall_access_token = "<OnCall-API-Token>" oncall_url = "<OnCall-URL>" }
Replace the following field values:
<OnCall-API-Token>
with API Token created in the first step.<OnCall-URL>
with the API URL found on the OnCall Settings page.
Add on-call schedule
This Terraform configuration creates two on-call schedules named SREs
and Devs
in Grafana OnCall using grafana_oncall_schedule (Resource).
It also configures Slack channels to be notified about on-call schedules for both Devs
and SREs
.
See more about managing on-call schedules.
Create two new calendars in your calendar service, one for Devs and one for SREs.
Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar.
Create a file named
schedule.tf
and add the following:terraform# Name of the Slack channel to notify about on-call schedules for Devs data "grafana_oncall_slack_channel" "Devs" { provider = grafana.oncall name = "<Devs-channel-name>" } # Name of the Slack channel to notify about on-call schedules for SREs data "grafana_oncall_slack_channel" "SREs" { provider = grafana.oncall name = "<SREs-channel-name>" } resource "grafana_oncall_schedule" "schedule_Devs" { provider = grafana.oncall name = "Devs" type = "ical" ical_url_primary = "<secret-iCal-URL-for-devs-calendar>" slack { channel_id = data.grafana_oncall_slack_channel.Devs.slack_id } } resource "grafana_oncall_schedule" "schedule_SREs" { provider = grafana.oncall name = "SREs" type = "ical" ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>" slack { channel_id = data.grafana_oncall_slack_channel.SREs.slack_id } }
Replace the following field values:
<Devs-channel-name>
with name of the Slack channel to notify about on-call schedules for Devs<SREs-channel-name>
with name of the Slack channel to notify about on-call schedules for SREs<secret-iCal-URL-for-devs-calendar>
with the secret iCal URL created in the first step for Devs Calendar<secret-iCal-URL-for-SREs-calendar>
with the secret iCal URL created in the first step for SREs Calendar
Add Escalation chains
This Terraform configuration creates two escalation chains named SREs
and Devs
in Grafana OnCall using grafana_oncall_escalation_chain (Resource).
The configuration also adds the following three steps to the two escalation chains using grafana_oncall_escalation (Resource):
- Notify users from on-call schedule
- Wait for 5 minutes
- Notify default Slack channel
Create a file named
escalation-devs.tf
and add the following:terraformresource "grafana_oncall_escalation_chain" "Devs" { provider = grafana.oncall name = "Devs" } // Notify users from on-call schedule resource "grafana_oncall_escalation" "notify_schedule_step_Devs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.Devs.id type = "notify_on_call_from_schedule" notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id position = 0 } // Wait step for 5 Minutes resource "grafana_oncall_escalation" "wait_step_Devs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.Devs.id type = "wait" duration = 300 position = 1 } // Notify default Slack channel step resource "grafana_oncall_escalation" "notify_step_Devs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.Devs.id type = "notify_whole_channel" important = true position = 2 }
Create a file named
escalation-sre.tf
and add the following:terraformresource "grafana_oncall_escalation_chain" "SREs" { provider = grafana.oncall name = "SREs" } // Notify users from on-call schedule resource "grafana_oncall_escalation" "notify_schedule_step_SREs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.SREs.id type = "notify_on_call_from_schedule" notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id position = 0 } // Wait step for 5 Minutes resource "grafana_oncall_escalation" "wait_step_SREs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.SREs.id type = "wait" duration = 300 position = 1 } // Notify default Slack channel step resource "grafana_oncall_escalation" "notify_step_SREs" { provider = grafana.oncall escalation_chain_id = grafana_oncall_escalation_chain.SREs.id type = "notify_whole_channel" important = true position = 2 }
Connect an integration to Grafana OnCall
This Terraform configuration connects AlertManager to Grafana OnCall using grafana_oncall_integration (Resource).
It also adds the Devs
escalation chain as the default route for alerts.
Create a file named
integrations.tf
and add the following:terraformresource "grafana_oncall_integration" "AlertManager" { provider = grafana.oncall name = "AlertManager" type = "alertmanager" default_route { escalation_chain_id = grafana_oncall_escalation_chain.Devs.id } }
To configure AlertManager see Connect AlertManager to Grafana OnCall
Set up a route to configure escalation behavior for alert group notifications
This Terraform configuration sets up a route to the AlertManager integration using grafana_oncall_route (Resource).
This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\"
in the payload will be escalated to the SREs
escalation chain.
Create a file named routes.tf
and add the following:
resource "grafana_oncall_route" "route_SREs" {
provider = grafana.oncall
integration_id = grafana_oncall_integration.AlertManager.id
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
routing_regex = "\"namespace\" *: *\"ops-.*\""
position = 0
}
Validation
Once you apply the changes in the Terraform configurations, you can verify the following:
Two new Schedules named
Devs
andSREs
are created in Grafana OnCall:New Escalation chain named
SREs
is created in Grafana OnCall:New Escalation chain named
Devs
is created in Grafana OnCall:The AlertManager integration is added and configured with escalation policies:
Conclusion
In this guide, you connected an integration to Grafana OnCall, configured escalation policies, and added your on-call schedule to Grafana OnCall using Terraform. We also connected Slack as the ChatOps tool for Grafana OnCall.
To learn more about managing Grafana Cloud using Terraform, see Grafana provider’s documentation.
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)