Query forecast metrics
Once you have trained a model, you can run predictions using PromQL. A new grafanacloud-ml-metrics Prometheus data source will be present in your Grafana Cloud instance. This data source understands PromQL, and translates specific queries into running models.
The two main series to be queried are <forecast_metric_name>:predicted
which contains series related to the predictions of a specific forecast, and <forecast_metric_name>:actual
which provides the actual values retrieved from the query.
All available series have a resolution equal to the training data interval of their corresponding forecast.
Predictions
The :predicted
metric contains three series differentiated by the ml_forecast
label. For example, a forecast with a metric name of request_rate
the resulting series will be:
request_rate:predicted{ml_forecast="yhat"}
request_rate:predicted{ml_forecast="yhat_lower"}
request_rate:predicted{ml_forecast="yhat_upper"}
Where yhat
is the calculated the predicted value, and yhat_lower
and yhat_upper
are the confidence bounds of the prediction.
Actual
The :actual
metric will query the datasource for up-to-date data of the real values. The results for this query do not have an ml_forecast
label, so to compare the :predicted
and :actual
series you will likely want to use ignoring (ml_forecast)
.
For example, to calculate the residual, or error, of a prediction you can use the PromQL:
abs(request_rate:predicted{ml_forecast="yhat"} - ignoring (ml_forecast) request_rate:actual)
Anomalous
Another common use case is determining anomalies, defined as when the :actual value is outside of the :predicted upper or lower bounds. The :anomalous metric will query the data source for the presence of any anomalous values. It has a value of 1 when :actual is anomalously high, a value of 0 when :actual is not an anomaly, and a value of -1 when :actual is anomalously low. To query for all points where there are anomalies, high or low, you will want to query for :anomalous != 0.
:actual > :predicted{ml_forecast=“yhat_upper”} | :actual < :predicted{ml_forecast=“yhat_lower”} | :anomalous |
---|---|---|
No | No | 0 |
Yes | No | 1 |
No | Yes | -1 |
The results for this query do not have an ml_forecast
label, so to combine :anomalous with either the :predicted or :actual series you will want to use ignoring (ml_forecast).
Querying the Values of Anomalous Points
You may want to query the values of anomalous points, rather than only querying whether or not they are anomalous. This can be done, for all anomalies whether low or high, with this query:
request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous != 0)
For only anomalies below the lower bound:
request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous == -1)
or
request_rate:actual < ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_lower"}
And for only anomalies above the upper bound:
request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous == 1)
or
request_rate:actual > ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_upper"}
The primary use of the queries using greater or less than, instead of the queries checking the ‘anomalous’ flag, is if you want to add something else to your criteria. For example, if you want only results more than 10% above the upper bound, you can add 10% to the expression on the right.
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)