Adaptive Metrics HTTP API
The following topic describes the Adaptive Metrics HTTP API.
Before you begin
To interact with the API, gather the following information:
URL
: In the formhttps://<your-grafana-cloud-prom-url>.grafana.net
. To find yourURL
value, go to yourgrafana.com
account and check the Details page of your hosted Prometheus endpoint.TENANT
: The numeric instance ID where you want to use Adaptive Metrics. To find yourTENANT
value, go to your grafana.com account and check the Details page of your hosted Prometheus endpoint for Username / Instance ID.TOKEN
: A token from a Grafana Cloud Access Policy, make sure the access policy hasmetrics:read
andmetrics:write
scopes for the stack ID where you want to use Adaptive Metrics.
Note
You must go to the Grafana portal to create an access policy for Adaptive Metrics.
Warning
Grafana discourages creating access policies with bothread
andwrite
permissions. You can still create such a policy, but a warning will appear.
The format of an aggregation rule
Each aggregation rule looks similar to this:
{
"metric": "agent_request_duration_seconds_sum",
"drop_labels": [
"container",
"instance",
"method",
"namespace",
"pod",
"provider",
"status_code",
"ws"
],
"aggregations": [
"sum:counter"
]
}
In the preceding example:
metric
is the name of the metric to be aggregated.drop_labels
is an array of the labels that will be aggregated and their value will be set to<aggregated>
.aggregations
is an array of the aggregation types to calculate for this metric.
You can use an aggregation rule file to define multiple rules simultaneously.
The following example rule file is an array of one or more aggregation rules:
[
{
"metric": "agent_request_duration_seconds_sum",
"drop_labels": ["namespace", "pod"],
"aggregations": ["sum:counter"]
},
{
"metric": "prometheus_request_duration_seconds_sum",
"drop_labels": ["container", "instance", "ws"],
"aggregations": ["sum:counter"]
}
]
List recommendations
Download recommendations for metrics to aggregate using the command below. TOKEN
and TENANT
are variables defined within the requirements section.
curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations"
TOKEN
must belong to an access policy with the metrics:read
scope.
You can use an optional verbose flag to retrieve more information about each recommendation:
curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations?verbose=true"
Any existing aggregation rules that have been previously applied will be included in the downloaded list of recommendations. This simplifies the process of viewing the recommended aggregation rules and applying them. By default, you can apply recommended rules and still maintain all of your existing rules.
Note
Recommendations are generated asynchronously from the API calls; the API call only returns the most recently generated set of recommendations.
List current recommendations configuration
Download the current configuration of the recommendations service using the following command:
curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations/config"
TOKEN
must belong to an access policy with the metrics:read
scope.
The only tunable parameter exposed by the recommendations service is the keep_labels
parameter.
This parameter allows the user to define a list of labels that they never want recommended for aggregation.
This can be useful at organizations where certain labels are always expected on metrics, regardless of whether or not those labels have been recently queried.
Note
Thekeep_labels
parameter only applies to new recommendations; existing rules that contain labels specified inkeep_labels
will remain unchanged.
An example response from the /recommendations/config
endpoint would look as follows:
{
"keep_labels": ["instance", "pod"]
}
The preceding response indicates that the recommendations service has been configured to never recommend aggregating the instance
or pod
labels.
Recommendation exemptions
The Adaptive Metrics API provides the possibility to exempt certain metrics, labels, and metric-label combinations from generated recommendations.
The recommendation exemption format looks like this:
Field name | Type | Description |
---|---|---|
id | identifier | Unique identifier of the exemption (set by the server). |
metric (optional) | metric name | Metric name to be exempted from recommendations. |
keep_labels (optional) | string array | List of labels to not be considered for aggregation by the recommendation system. |
Note
For an exemption to be considered valid, either themetric
orkeep_labels
field must be defined.
Below are examples of exemptions and how they would be reflected in the preceding mentioned format:
Completely exclude metric
api_http_requests_total
from being recommended for aggregation (keep full cardinality).json{ "metric": "api_http_requests_total" }
Exclude labels
namespace
andstatus
for metricapi_http_requests_total
from being recommended for aggregation.json{ "metric": "api_http_requests_total", "keep_labels": ["namespace", "status"] }
Exclude the
pod
label for all metrics from being recommended for aggregation.json{ "keep_labels": ["pod"] }
Create a recommendation exemption
curl -u "$TENANT:$TOKEN" --request POST --data @exemption.json "$URL/v1/recommendations/exemptions
TOKEN
must belong to an access policy with the adaptive-metrics-exemptions:write
scope.
If successful, the server returns a JSON object including the id
of the newly created exemption.
List recommendation exemptions
curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions
TOKEN
must belong to an access policy with the metrics:read
scope.
Get a recommendation exemption
curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions/<exemption_id>
TOKEN
must belong to an access policy with the adaptive-metrics-exemptions:read
scope.
The exemption_id
is the identifier of the exemption to be fetched, as returned by the server when creating the exemption.
Update recommendation exemption
curl -u "$TENANT:$TOKEN" --request PUT --data @exemption.json "$URL/v1/recommendations/exemptions/<exemption_id>
TOKEN
must belong to an access policy with the adaptive-metrics-exemptions:write
scope.
The exemption_id
is the identifier of the exemption to be updated, as returned by the server when creating the exemption.
Delete a recommendation exemption
curl -u "$TENANT:$TOKEN" --request DELETE "$URL/v1/recommendations/exemptions/<exemption_id>
TOKEN
must belong to an access policy with the adaptive-metrics-exemptions:write
scope.
The exemption_id
is the identifier of the exemption to delete, as returned by the server when creating the exemption.
Update recommendations configuration
Upload new recommendations configuration using the following command:
curl -u "$TENANT:$TOKEN" --request POST --data @config.json "$URL/aggregations/recommendations/config"
TOKEN
must belong to an access policy with the metrics:write
scope.
This command uses the same endpoint described in List current recommendations configuration and expects the same JSON format.
List currently applied aggregation rules
Download your existing aggregation rules:
curl -u "$TENANT:$TOKEN" "$URL/aggregations/rules"
TOKEN
must belong to an access policy with the metrics:read
scope.
Upload new aggregation rules
Uploading new aggregation rules is a multi-step process:
- Fetch the currently applied rules.
- Modify rules locally.
- Upload rules back.
There is a delay between uploading new aggregation rules and those metrics aggregations taking effect in your environment. In most cases, the delay is approximately 5-10 minutes, but we currently have no mechanism to let you know precisely when new aggregations take effect.
Grafana currently limits how often new aggregation rules are applied. Although you can upload as many new versions of your aggregation rules as you like, those updates are only applied once every 10 minutes. If you make multiple updates in quick succession, the system applies your first received (oldest) update. Then, 10 minutes later, the most recently received update is applied. The intermediate updates never get applied.
Fetch the currently applied rules
Use this command:
curl -u "$TENANT:$TOKEN" -D headers.txt "$URL/aggregations/rules" > rules.json
TOKEN
must belong to an access policy with the metrics:read
scope.
The preceding command uses an additional -D headers.txt
argument, which stores the headers in a file called headers.txt.
This step is required if you want to then upload a new rule file, for example if you want to update the existing aggregation rules you have in place. The information in these headers ensures there are no update collisions. An update collision is the scenario where multiple users try to edit the rules file at the same time and overwrite one another’s changes.
Modify the rules locally
Use your editor of choice to modify the rules.json
file downloaded in the prior step.
Upload rules back
The API supports uploading an entire rules file.
Warning
This action will overwrite your existing rule file.
To upload your modified rules.json
file from the previous step, use the following shell script:
TMPFILE=$(mktemp)
trap 'rm "$TMPFILE"' EXIT
cat headers.txt | grep -i '^etag:' | sed 's/^ETag:/If-Match:/i' > "$TMPFILE"
curl --request POST --header @"$TMPFILE" --data-binary @$1 -u "$TENANT:$TOKEN" "$URL/aggregations/rules"
TOKEN
must belong to an access policy with the metrics:write
scope.
The cat headers.txt
command modifies the headers.txt
file created in the previous curl call that pulled down the existing aggregation rules.
The curl --request POST
command uploads your new rules file, as well as the updated headers.
Save the shell script as rules_upload.sh.
To run that script, use the following command:
./rules_upload.sh <your_new_rules_file.json>
Replace <your_new_rules_file.json>
with the name of the rules file you wish to upload.
Note
If, upon trying toPOST
the new rules file, you see the errorthe Etag supplied in the 'If-Match' header does not match the Etag of the rules you are trying to replace
, the headers you provided are either missing or stale. To fix, re-fetch the rules file and headers, being careful to look for any changes that may have been introduced since your last edits. For more information on Etag headers, see Etag.
Note
After you configure aggregation rules, the active series count might increase temporarily. Aggregated and unaggregated series will be considered active at the same time. After a short period of time, the unaggregated series will no longer be considered active, and you will see a net reduction in active series.
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)