Incoming Webhooks for Grafana Incident
The Incoming Webhook integration allows you to wire up any third-party system to trigger an Incident.
Details of the webhook are attached to the Incident.
Get started
It takes only a few steps to configure a third-party tool to automatically declare an Incident.
Enable Incoming Webhooks
In the Grafana Incident web app, an admin can go to Integrations to enable incoming webhooks.
Installing the Incoming Webhooks integration will generate a token which you will use to authorize the requests.
- Go to Alerts & Incidents and under Incident choose Integrations
- Select the Incoming Webhooks integration
- Click Install integration
- Make note of the Token and URL fields
Get the URL
Copy the URL and the token from the integration details.
There are two ways to pass the token
in the request:
- You can set the Authorization header (recommended) to
Bearer {token}
- You can add the
token
as a URL parameter
Use URL parameters to customize the incident
You can add the following parameters to the URL to further customize the Incident.
drill=true
(boolean) Iftrue
will declare a drill (test) incident (recommended for testing)title
- (string) Required. The title of the incidentseverity
- (string) The severity of the incident to declare (default:pending
)status
- (string) The status of the incident, eitheractive
orresolved
(default:active
)labels
- (list of string) A comma-separated list of labels to add to the incidentroomprefix
- (string) The chat room prefix for platforms that support it (e.g.incident
)url
- (string) A link to some context, usually the place (or thing) where the incident was declaredcaption
- (string) An optional caption to go along withurl
You can also decide what the response should include:
includes
- (comma separated list) A list of fields to include in the response, e.g.?includes=incident
will return the incident object.
Using special values, you can extract fields from the Webhook’s JSON request bodies.
Paste the URL into the third-party tool
Configure the third party system to issue a POST
request to the api/experimental/incoming-webhooks/grafana.incident.create
endpoint from the previous step.
POST .../api/v1/incoming-webhooks/grafana.incident.create
&drill=true // make it a drill
&title=json(message.shortMessage) // extract the field from the body
&severity=minor // minor severity assumed
&labels=autogenerated // label it as 'autogenerated' so we know
Content-Type: application/json
Authorization: Bearer token_goes_here
{
"message": {
"shortMessage": "A short description might appear here"
}
}
Only share the Incoming Webhook token with the third-party system. If it leaks, anybody would be able to declare incidents in your system by making a POST request. Learn more about Disabling an Incoming Webhook
Understanding the response
In most cases, the handler will return a 200 OK
indicating success. The body of the webhook request will be the IncomingWebhookResponse JSON object:
{
"incident": {
"incidentID": "incident-1",
"title": "title of the incident"
// more incident fields
},
"processingErrors": ["something went wrong"]
}
It is possible for errors to occur during processing, but the handler errs on the side of always declaring the incident, regardless. You can check the processingErrors
field in the response for any issues during development. You should resolve all errors before shipping to production.
Test the endpoint using curl
The integration details provides you with an example curl command to test the endpoint.
Or you can write your own, remembering to use your own token:
curl ".../api/v1/incoming-webhook/grafana.incident.create?drill=true&title=json(report.title)" \
--request POST
--header 'Authorization: Bearer use_real_token'
--data '{"report":{"title":"testing the new incoming webhooks integration"}}'
Notice that a drill incident has been declared, with details of the request included in the timeline.
Lookup JSON values from the body
If your Webhook sports a Content-Type: application/json
with an object in the body, you can access the data using the json()
function in the URL parameters.
- Only JSON payloads that are within the 1MB limit will be parsed
For example, a Webhook body might contain some data like this:
Content-Type: "application/json"
{
"reportID": "abc123",
"report": {
"title": "Unable to access public website"
}
}
By adding the title=json(report.title)
parameter to our Incoming Webhook URL, we are able to extract the nested report.title
:
.../api/v1/incoming-webhooks/grafana.incident.create?title=json(report.title)
When the incident is declared via this route, the incident’s title will match the report.title
value “Unable to access public website”.
If it is not possible to parse the JSON and find the value, a sensible default will be used.
Disabling Incoming Webhooks
To stop an Incoming Webhook from working, you just reinstall the integration and a new token will be generated, rendering the previous one useless.
- Go to Alerts & Incidents and under Incident choose Integrations
- Select the Incoming Webhooks integration
- Click Uninstall - this will stop accepting any incoming webhook requests
- Click Install integration - a new token will be generated
- Make note of the Token and URL fields
When should you reinstall Incoming Webhooks?
You should reset your tokens by reinstalling your Incoming Webhooks integration if:
- the URL has leaked or been otherwise compromised
- you are no longer using the Incoming Webhook URL
- you notice any suspicious activity
- the third-party system starts declaring too many incidents
Add metadata to your incoming webhooks
You can optionally add metadata, which will be included in the incident, via the following HTTP headers:
User-Agent
Origin
Referer
Link
These values are not processed, they are just repeated inside the incident, but can be useful ways to identify the source of the webhook.
Usage limits
- Only the first 1MB of the webhook body will be read - so JSON payloads must be smaller than 1MB
- You may only declare an incident via an incoming webhook once per minute.
429 Too Many Requests
will be returned if you exceed this limit.
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)