Use images in notifications
Note
Grafana Cloud users can request this feature by opening a support ticket in the Cloud Portal.
Images in notifications helps recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.
Note
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
When an alert is fired or resolved Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.
Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert, instead you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.
Once a screenshot has been taken Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to it’s internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.
Refer to the table at the end of this page for a list of contact points and their support for images in notifications.
Requirements
To use images in notifications, Grafana must be set up to use image rendering. You can either install the image rendering plugin or run it as a remote rendering service.
When a screenshot is taken it is saved to the [data][paths] folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder otherwise screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt.
You should use a cloud storage service unless sending alerts to Discord, Email, Pushover, Slack or Telegram. These integrations support either embedding screenshots in the email or attaching screenshots to the notification, while other integrations must link screenshots uploaded to a cloud storage bucket. If a cloud storage service has been configured then integrations that support both will link screenshots from the cloud storage bucket instead of embedding or attaching screenshots to the notification.
If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; then image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using integrations which support uploading images or disabling images in notifications altogether.
When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess but not impossible.
Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.
If Grafana is configured to upload screenshots to its internal web server, and accessing Grafana requires logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using integrations which support uploading images or disabling images in notifications altogether.
Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from
static_root_path/images/attachments
after a certain amount of time we recommend setting up a CRON job.
Configuration
Note
Grafana Cloud users can request this feature by opening a support ticket in the Cloud Portal.
Having installed either the image rendering plugin, or set up Grafana to use a remote rendering service, set capture
in [unified_alerting.screenshots]
to true
:
# Enable screenshots in notifications. You must have either installed the Grafana image rendering
# plugin, or set up Grafana to use a remote rendering service.
# For more information on configuration options, refer to [rendering].
capture = false
If screenshots should be uploaded to cloud storage then upload_external_image_storage
should also be set to true
:
# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false
Restart Grafana for the changes to take effect.
Advanced configuration
We recommended that max_concurrent_screenshots
is less than or equal to concurrent_render_request_limit
. The default value for both max_concurrent_screenshots
and concurrent_render_request_limit
is 5
:
# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5
Supported contact points
Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.
Name | Upload from disk | Reference from cloud storage |
---|---|---|
DingDing | No | No |
Discord | Yes (Maximum of 10 per notification) | Yes (Maximum of 10 per notification) |
Yes (Embedded in the email) | Yes | |
Google Chat | No | Yes |
Kafka | No | No |
Line | No | No |
Microsoft Teams | No | Yes |
Opsgenie | No | Yes |
Pagerduty | No | Yes |
Prometheus Alertmanager | No | No |
Pushover | Yes (Maximum of 1 per notification) | No |
Sensu Go | No | No |
Slack | Yes (when using Bot tokens, maximum of 5 per notification) | Yes (when using webhooks, maximum of 1 per notification) |
Telegram | Yes | No |
Threema | No | No |
VictorOps | No | No |
Webhook | No | Yes |
Limitations
- This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
- A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
- When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
- If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN.
Troubleshooting
If Grafana has been set up to send images in notifications, however notifications are still being received without them, follow the troubleshooting steps below:
- Check that images in notifications has been set up as per the instructions.
- Enable debug logging in Grafana and look for logs with the logger
ngalert.image
. - If the alert is not associated with a dashboard there will be logs for
Cannot take screenshot for alert rule as it is not associated with a dashboard
. - If the alert is associated with a dashboard, but no panel in the dashboard, there will be logs for
Cannot take screenshot for alert rule as it is not associated with a panel
. - If images cannot be taken because of mis-configuration or an issue with image rendering there will be logs for
Failed to take an image
including the Dashboard UID, Panel ID, and the error message. - Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.
Metrics
Grafana provides the following metrics to observe the performance and failure rate of images in notifications.
For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter grafana_screenshot_failures_total
is updated.
grafana_alerting_image_cache_hits_total
grafana_alerting_image_cache_misses_total
grafana_screenshot_duration_seconds
grafana_screenshot_failures_total
grafana_screenshot_successes_total
grafana_screenshot_upload_failures_total
grafana_screenshot_upload_successes_total
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)