Grafana Cloud Metrics: A guide to what metrics to monitor and best practices

• 16 Sep, 2022 • 6 min

Metrics are the cornerstone of an observable system – they tell you a system’s measured outputs, granting visibility into what your customers are experiencing and when there’s a problem. However, not all methods for recording and saving metrics from a system’s output are alike. The best method for shipping your system’s metrics to Grafana Cloud depends on many factors, varying from the source of your metrics data to your familiarity with observability tools.

Grafana Cloud customers typically consider one or more of the following types of questions before determining their preferred method for shipping metrics:

What is my preferred time series database?
How familiar am I with using programs for collecting and aggregating metrics?
What is the quantity and scale of metrics I’d like to monitor?
What is my application’s runtime environment?

This guide will help you better understand the possibilities and the trade-offs. By the end, you should be able to identify what’s the best way to ship metrics from your system to Grafana Cloud.

Why ship metrics to Grafana Cloud?

Creating, deploying, and managing your applications and core infrastructure are complicated enough without having to worry about hosting a redundant metrics or time series database to allow resilient monitoring and data observability.

Grafana Cloud is a managed service, letting you plug in and start playing around almost instantly with open source tools such as Grafana, Grafana Mimir, and Grafana Loki – skipping the time-consuming installation and maintenance required when deploying services on your own.

Once you’re up and running with Grafana Cloud, you’ll be able to enjoy the peace of mind knowing that we’ll automatically scale with your usage. That’s not to mention the security fixes and release upgrades that we handle on behalf of your organization on a regular basis.

Chart comparing costs for self-managed metrics vs. Grafana Cloud

What metrics should you ship to Grafana Cloud?

The metrics you choose to monitor depend on your goals and the outputs of your system. With an endless list of possibilities, it may be helpful to focus on metrics that tell you insights into the “The Four Golden Signals” of monitoring:

Latency: The time it takes to service a request.
Traffic: How much demand is being placed on your system.
Errors: The rate of failures in your system.
Saturation: How “full” is your system — and critically, how much capacity remains.

Once you’ve narrowed down the metrics you want to ship, we recommend reviewing the typical measurement structure of a metric. That way you’ll get a sense of just how granular (or not) you can go with your metrics and a feel for how you’ll query your metrics using Grafana.

For our examples in this guide, we’ll go with Graphite, which is one of the easiest and most flexible methods for pushing metrics to Grafana Cloud.

Each metric is composed of the following properties:

name // Graphite style name (required)

interval // the resolution of the metric in seconds (required)

value // float64 value (required)

time // unix timestamp in seconds (required)

tags // list of key=value pairs of tags (optional)

Metric name syntax

<namespace>.<instrumented section>.<target (noun)>.<action (past tense verb)>

E.g. accounts.authentication.password.succeeded

Credit: Practical Guide to Graphite Monitoring

Metric tags (optional)

["key1=value1", "key2=value2"]

E.g. ["location=usa", "environment=production"]

Once you select relevant metric names and tags, you’ll want to add:

The time - a Unix timestamp in seconds
The interval - the time between metric data points
The value - a float that tells us how much or how little was measured

Bringing it together, we get the following example Graphite metric:

{

    "name": "accounts.authentication.password.succeeded ",

    "interval": 10,

    "value": 1.0,

    "tags": \["location=usa", "environment=production"],

    "time": 1660850859

}

Now it’s time for you to start shipping metrics that are relevant to you and your systems. Read on for additional pointers on how to ship metrics to Grafana Cloud.

How can you ship metrics to Grafana Cloud?

The most commonly used methods for shipping metrics to Grafana Cloud are outlined below:

Most flexible: HTTP Write

Recommended for people who want to ship metrics regarding their systems over HTTP. This is a great option for application developers, Raspberry Pi hackers, Linux administrators, and more given the universal nature of the HTTP interface.

Graphite examples

Prometheus examples (via InfluxDB Line Protocol)

Easiest: Agent

Dozens of integrations for popular server tools like Linux, Kubernetes, Nginx, and Docker. Simply copy-paste a command on your server to get started and access prebuilt dashboards to quickly visualize your metrics within minutes.

Recommended for people wanting to know performance metrics of their infrastructure.

Best for Prometheus power users: Remote write

Most difficult. Suitable for use cases with a high volume of metrics (> 100k active series) sent over a dedicated socket.

What’s next? Once your metric data is flowing into Grafana Cloud, the next step is to begin building visualizations so that you can monitor those metrics over time. We recommend visiting the Explore tool in your Grafana instance.

Grafana Explore

The Explore tool lets you quickly and easily build queries that return your metrics in the form of graphs and tables, which you can then add to dashboards that show the trends in your data. It’s a perfect place to start digging into your data and figure out which parts are best suited for dashboards, alerting, and more.

Using Explore

From within Explore, below are the components you’ll need to adjust based on your needs.

Select a data source
You’ll see a dropdown in the top-left corner to select your data source. If you’re following the HTTP Write examples above, you’ll want to select either the grafanacloud--graphite data source or the grafancloud--prom data source.
Write a query
Explore offers a prebuilt query builder below the data source selector. Click the dropdowns beside Series to quickly select a metric series for your query. Or, if you’re already experienced with the query language for your data source, Explore lets you write queries from scratch, too.
Visualize
Click Run query in the top-right corner of Explore to execute a query. Explore will show your metrics in a graph and table so that you can quickly view your metric data. And finally, click to Add to dashboard once you’re ready to monitor a query over time.

Learn more about Grafana Explore.

Learn more about Grafana Cloud Metrics

If you have any questions about shipping metrics to Grafana Cloud, please don’t hesitate to reach out to our team!

Below are some additional resources on best practices, configurations, and use cases.

Grafana Cloud resources

Graphite resources

Grafana Cloud documentation

Blog posts

Prometheus resources