Managing OnCall on Grafana Cloud using Terraform

Learn how to connect an integration to Grafana OnCall, configure escalation policies, and add your on-call schedule to Grafana OnCall using Terraform. In this guide, We will be using two hypothetical teams named Devs and SREs for showcasing how Grafana OnCall can be used by more than one team in your organization. We will also connect Slack as the ChatOps tool for Grafana OnCall in this guide.

Prerequisites

Before you begin, you should have the following available:

A Grafana Cloud account, as shown in Get started
Terraform installed on your machine
Admin permissions in your Grafana instance as well as the Slack workspace

Note
All of the following Terraform configuration files should be saved in the same directory.

Connect Slack to Grafana OnCall

To connect Slack to Grafana OnCall, use the Slack integration for Grafana OnCall

Terraform configuration for Grafana provider

This Terraform configuration configures the Grafana provider to provide necessary authentication when managing Grafana OnCall.

Create an API Token in Grafana OnCall. You’ll need this key to configure Terraform.
1. Log into your Grafana Cloud instance.
2. Click Alerts & IRM and then select OnCall.
3. Click Settings.
4. Under API Tokens, Click +Create.
5. Enter a name for your API Token and click Create Token.
6. Click Copy Token to save it for later use.

Create a file named main.tf and add the following code block:

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.9.0"
    }
  }
}

provider "grafana" {
  alias = "oncall"

  oncall_access_token = "<OnCall-API-Token>"
  oncall_url = "<OnCall-URL>"
}

Replace the following field values:
- <OnCall-API-Token> with API Token created in the first step.
- <OnCall-URL> with the API URL found on the OnCall Settings page.

Add on-call schedule

This Terraform configuration creates two on-call schedules named SREs and Devs in Grafana OnCall using grafana_oncall_schedule (Resource). It also configures Slack channels to be notified about on-call schedules for both Devs and SREs. See more about managing on-call schedules.

Create two new calendars in your calendar service, one for Devs and one for SREs.
Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar.

Create a file named schedule.tf and add the following:

# Name of the Slack channel to notify about on-call schedules for Devs
data "grafana_oncall_slack_channel" "Devs" {
  provider = grafana.oncall

  name = "<Devs-channel-name>"
}

# Name of the Slack channel to notify about on-call schedules for SREs
data "grafana_oncall_slack_channel" "SREs" {
  provider = grafana.oncall

  name = "<SREs-channel-name>"
}

resource "grafana_oncall_schedule" "schedule_Devs" {
  provider = grafana.oncall

  name             = "Devs"
  type             = "ical"
  ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
  slack {
    channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
  }
}

resource "grafana_oncall_schedule" "schedule_SREs" {
  provider = grafana.oncall

  name             = "SREs"
  type             = "ical"
  ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
  slack {
    channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
  }
}

Replace the following field values:
- <Devs-channel-name> with name of the Slack channel to notify about on-call schedules for Devs
- <SREs-channel-name> with name of the Slack channel to notify about on-call schedules for SREs
- <secret-iCal-URL-for-devs-calendar> with the secret iCal URL created in the first step for Devs Calendar
- <secret-iCal-URL-for-SREs-calendar> with the secret iCal URL created in the first step for SREs Calendar

Add Escalation chains

This Terraform configuration creates two escalation chains named SREs and Devs in Grafana OnCall using grafana_oncall_escalation_chain (Resource). The configuration also adds the following three steps to the two escalation chains using grafana_oncall_escalation (Resource):

Notify users from on-call schedule
Wait for 5 minutes
Notify default Slack channel

Create a file named escalation-devs.tf and add the following:

resource "grafana_oncall_escalation_chain" "Devs" {
  provider = grafana.oncall

  name = "Devs"
}

// Notify users from on-call schedule
resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id          = grafana_oncall_escalation_chain.Devs.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
  position                     = 0
}

// Wait step for 5 Minutes
resource "grafana_oncall_escalation" "wait_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  type                = "wait"
  duration            = 300
  position            = 1
}

// Notify default Slack channel step
resource "grafana_oncall_escalation" "notify_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  type                = "notify_whole_channel"
  important           = true
  position            = 2
}

Create a file named escalation-sre.tf and add the following:

resource "grafana_oncall_escalation_chain" "SREs" {
  provider = grafana.oncall

  name = "SREs"
}

// Notify users from on-call schedule
resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id          = grafana_oncall_escalation_chain.SREs.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
  position                     = 0
}

// Wait step for 5 Minutes
resource "grafana_oncall_escalation" "wait_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  type                = "wait"
  duration            = 300
  position            = 1
}

// Notify default Slack channel step
resource "grafana_oncall_escalation" "notify_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  type                = "notify_whole_channel"
  important           = true
  position            = 2
}

Connect an integration to Grafana OnCall

This Terraform configuration connects AlertManager to Grafana OnCall using grafana_oncall_integration (Resource). It also adds the Devs escalation chain as the default route for alerts.

Create a file named integrations.tf and add the following:

resource "grafana_oncall_integration" "AlertManager" {
  provider = grafana.oncall

  name = "AlertManager"
  type = "alertmanager"
  default_route {
    escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  }
}

To configure AlertManager see Connect AlertManager to Grafana OnCall

Set up a route to configure escalation behavior for alert group notifications

This Terraform configuration sets up a route to the AlertManager integration using grafana_oncall_route (Resource). This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\" in the payload will be escalated to the SREs escalation chain.

Create a file named routes.tf and add the following:

resource "grafana_oncall_route" "route_SREs" {
  provider = grafana.oncall

  integration_id      = grafana_oncall_integration.AlertManager.id
  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  routing_regex       = "\"namespace\" *: *\"ops-.*\""
  position            = 0
}

Validation

Once you apply the changes in the Terraform configurations, you can verify the following:

Two new Schedules named Devs and SREs are created in Grafana OnCall:
New Escalation chain named SREs is created in Grafana OnCall:
New Escalation chain named Devs is created in Grafana OnCall:
The AlertManager integration is added and configured with escalation policies:

Conclusion

In this guide, you connected an integration to Grafana OnCall, configured escalation policies, and added your on-call schedule to Grafana OnCall using Terraform. We also connected Slack as the ChatOps tool for Grafana OnCall.

To learn more about managing Grafana Cloud using Terraform, see Grafana provider’s documentation.