Outgoing Webhook for Grafana Incident
The Outgoing Webhook integration allows you to receive real-time notifications and updates where you need them. It can trigger events when an incident is created, updated, or closed, and Grafana Incident will notify the specified URL.
Use the Outgoing Webhook integration to create custom workflows and actions based on the details of an Incident.
Note: Outgoing Webhooks are useful in conjunction with the other Grafana Incident APIs.
Example use case
Consider the following use case of the Outgoing Webhook integration:
Multiple teams at your organization use Grafana Incident, so you use labels to identify which incidents belong to which teams. Your current incident response workflow includes a manual step to create an issue in the related teams’ Jira project when an incident is declared.
To automate the issue creation in Jira, use the Outgoing Webhook integration to trigger a custom workflow that reads the Incident event labels and uses them to create an issue in the related Jira project for the right team.
Install the Outgoing Webhook integration
An Admin can install the integration:
- From the Grafana Incident Integrations tab, select Outgoing Webhook.
- Click Install Integration.
- Once you install the integration, a signing secret is automatically generated. You will use this secret when verifying the signature in your webhook handler code.
Configure actions
An admin can wire-up event actions to indicate when the Webhook should be fired.
- Click Run when an event fires.
- Enter a target endpoint URL for the outgoing webhook POST request.
- Click Add event action to save your webhook - see options below:
Trigger webhooks based on events
Once configured, you can trigger outgoing webhooks based on different actions. The following actions are supported:
- Incident declared: The webhook triggers when an incident is declared.
- Incident change: Triggered when there is a change in the incident status or details.
- Incident closed: Triggers when an incident is closed
- Incident matches a filter: This trigger activates when an incident matches specific filter criteria.
Define filters for event triggers
When setting up filters for event triggers, you can utilize various conditions to specify the criteria. Each condition consists of a key-value pair separated by a colon. Additionally, you can employ logical operators such as and()
, or()
, and -
(negation) to create filter combinations.
Supported filter conditions:
status
:active
|resolved
isdrill
:true
|false
severity
:critical
|major
|minor
role
:investigator
|commander
label
:"<label-name>"
Quotes are optionaland()
: Used to combine multiple conditions with the logical AND operator (default behavior if no operator is specified).or()
: Combines conditions with the logical OR operator.-
: Negates a condition.
Examples:
-severity:critical
: Matches incidents with severity other than critical.and(label:customer severity:critical)
: Matches critical incidents with thecustomer
label.(label:customer severity:critical)
: Simplified form of the above, asand
is the default operator.and(severity:critical or(label:"Customer A" label:"Customer B"))
: Matches critical incidents with eitherCustomer A
orCustomer B
label.
Event actions will trigger at most once per incident. For instance, if a label:customer
filter is applied, adding, removing, and re-adding a customer
label will result in only one webhook call.
Available event types
Event type | ||||
---|---|---|---|---|
Incident is declared | grafana.incident.created | |||
Incident changes | grafana.incident.updated.role grafana.incident.updated.status grafana.incident.updated.title grafana.incident.updated.severity grafana.incident.added.label grafana.incident.removed.label | |||
Incident is resolved | grafana.incident.closed |
Verify webhook requests
The GI-Signature
header is included with requests and has a Unix-seconds timestamp (t=
) followed by one or more versioned signatures (e.g. v1=
) for example:
GI-Signature:t=1677589543,v1=12d6e3e06e66f2a32b4027827fbb95ce139ee8381a1cec1c40a02ec5f877797c
The v1
signature is calculated from the HMAC-SHA256 value of a “signing string” follows:
signing-string = body-hash + ":" +timestamp + ":" + signature-version
For example, given a payload of { 'somejson': true }
, a secret of some-secret-value
, a target endpoint URL of https://myendpoint.com/some/path
sent at 1677589543
:
signature-version = "v1"
body-hash = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM="
signing-string = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1"
hmac = a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3
GI-Signature:t=1677589543,v1=a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3
Verifying on the command line using openssl
:
echo -n "{ 'somejson': true }" | openssl dgst -sha256 -binary | openssl enc -base64
Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=
echo -n "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1" | openssl dgst -sha256 -hmac "some-secret-value"
a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3
Sample Outgoing Webhook
Below is a sample for an incident with an updated severity (grafana.incident.updated.severity
).
Request headers
As described above, the signature found in GI-Signature
can be used to verify the event integrity.
Header | Value |
---|---|
gi-signature | t=1677589543,v1=12d6e3e06e66f2a32b4027827fbb95ce139ee8381a1cec1c40a02ec5f877797c |
content-type | application/json |
To conform with the CloudEvents specification
we also provide the following headers (prefixed with ce-
):
Header | Value |
---|---|
ce-type | incident.webhook |
ce-time | 2023-03-02T14:12:00Z |
ce-subject | grafana.incident.updated.severity |
ce-specversion | 1.0 |
ce-source | /grafana/incident |
ce-id | webhook-out-6400aeb030670e95 |
ce-dataschema | v1.0.0 |
Webhook payload
The body of the webhook request will be the OutgoingWebhookPayload JSON object:
{
"version": "v1.0.0",
"id": "webhook-out-6400aeb030670e95",
"source": "/grafana/incident",
"time": "2023-03-02T14:12:00Z",
"event": "grafana.incident.updated.severity",
"incident": {
"incidentID": "5",
"severityID": 1
"labels": [],
"isDrill": false,
"createdTime": "2023-02-27T15:06:25.243788Z",
"modifiedTime": "2023-03-02T14:12:00.730047Z",
"createdByUser": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
},
"closedTime": "",
"durationSeconds": 860734,
"status": "active",
"title": "55",
"overviewURL": "/a/grafana-incident-app/incidents/5/55",
"roles": [
{
"role": "investigator",
"description": "Leads the investigation (has their full-time attention)",
"maxPeople": 1,
"mandatory": true,
"important": true,
"user": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
}
},
{
"role": "observer",
"description": "Watching the incident",
"maxPeople": 0,
"mandatory": false,
"important": false,
"user": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
}
},
{
"role": "commander",
"description": "Owns the incident (has their full-time attention)",
"maxPeople": 1,
"mandatory": true,
"important": true,
"user": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
}
}
],
"taskList": {
"tasks": [
{
"taskID": "assign-role-investigator",
"immutable": true,
"createdTime": "2023-02-27T15:06:25.232016511Z",
"modifiedTime": "2023-03-01T09:33:24.566202085Z",
"text": "Assign INVESTIGATOR role",
"status": "done",
"authorUser": null,
"assignedUser": null
},
{
"taskID": "assign-role-commander",
"immutable": true,
"createdTime": "2023-02-27T15:06:25.23202747Z",
"modifiedTime": "2023-03-01T09:33:34.67823384Z",
"text": "Assign COMMANDER role",
"status": "done",
"authorUser": null,
"assignedUser": null
},
{
"taskID": "must-choose-severity",
"immutable": true,
"createdTime": "2023-02-27T15:06:25.232032261Z",
"modifiedTime": "2023-03-02T14:12:00.729522804Z",
"text": "Specify incident severity",
"status": "done",
"authorUser": null,
"assignedUser": null
},
{
"taskID": "task-63ff1bf465deb38c",
"immutable": false,
"createdTime": "2023-03-01T09:33:40.759215968Z",
"modifiedTime": "2023-03-01T09:33:55.286168752Z",
"text": "111",
"status": "done",
"authorUser": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
},
"assignedUser": {
"userID": "grafana-incident:user-63f8b6204887f793",
"name": "admin",
"photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
}
}
],
"todoCount": 0,
"doneCount": 4
},
"summary": "Resolved with no comment",
"heroImagePath": "/api/hero-images/default_org/mC0B9YIgrBH3Hf9jvnKsvalHT90QtlODeiXDd2aLgRsflEZrRHGS5ld2KfIAbndglMIHiKSR04FPRPBpS34DA182rpeBxF9MSypELoU7nyOzXS09YenaCUtBZEzCi1xd/v1128/5.png",
"incidentStart": "2023-02-20T15:06:26Z",
"incidentEnd": ""
}
}
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)