DocumentationGrafana CloudAlerts and IRMMachine LearningDynamic alertingQuery results

Query forecast metrics

Once you have trained a model, you can run predictions using PromQL. A new grafanacloud-ml-metrics Prometheus data source will be present in your Grafana Cloud instance. This data source understands PromQL, and translates specific queries into running models.

The two main series to be queried are <forecast_metric_name>:predicted which contains series related to the predictions of a specific forecast, and <forecast_metric_name>:actual which provides the actual values retrieved from the query.

All available series have a resolution equal to the training data interval of their corresponding forecast.

Predictions

The :predicted metric contains three series differentiated by the ml_forecast label. For example, a forecast with a metric name of request_rate the resulting series will be:

Copy

request_rate:predicted{ml_forecast="yhat"}
request_rate:predicted{ml_forecast="yhat_lower"}
request_rate:predicted{ml_forecast="yhat_upper"}

Where yhat is the calculated the predicted value, and yhat_lower and yhat_upper are the confidence bounds of the prediction.

Actual

The :actual metric will query the datasource for up-to-date data of the real values. The results for this query do not have an ml_forecast label, so to compare the :predicted and :actual series you will likely want to use ignoring (ml_forecast).

For example, to calculate the residual, or error, of a prediction you can use the PromQL:

Copy

abs(request_rate:predicted{ml_forecast="yhat"} - ignoring (ml_forecast) request_rate:actual)

Anomalous

Another common use case is determining anomalies, defined as when the :actual value is outside of the :predicted upper or lower bounds. The :anomalous metric will query the data source for the presence of any anomalous values. It has a value of 1 when :actual is anomalously high, a value of 0 when :actual is not an anomaly, and a value of -1 when :actual is anomalously low. To query for all points where there are anomalies, high or low, you will want to query for :anomalous != 0.

The results for this query do not have an ml_forecast label, so to combine :anomalous with either the :predicted or :actual series you will want to use ignoring (ml_forecast).

Querying the Values of Anomalous Points

You may want to query the values of anomalous points, rather than only querying whether or not they are anomalous. This can be done, for all anomalies whether low or high, with this query:

Copy

request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous != 0)

For only anomalies below the lower bound:

Copy

request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous == -1)

or

Copy

request_rate:actual < ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_lower"}

And for only anomalies above the upper bound:

Copy

request_rate:actual and ignoring (ml_forecast) (request_rate:anomalous == 1)

or

Copy

request_rate:actual > ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_upper"}

The primary use of the queries using greater or less than, instead of the queries checking the ‘anomalous’ flag, is if you want to add something else to your criteria. For example, if you want only results more than 10% above the upper bound, you can add 10% to the expression on the right.