How to use the Grafana Operator: Managing a Grafana Cloud stack in Kubernetes

When deploying an application using Kubernetes, you get used to all your resources being manageable by describing them to the Kubernetes API. Whether it’s deployments, secrets, configurations, or entire machines, everything exists as code somewhere. Introducing a cloud service into such an environment often means introducing additional ways to configure it, which can become cumbersome, given the rising number of cloud services modern applications depend on.

By combining Grafana Cloud with the Grafana Operator, however, you get the best of both worlds: peace of mind by not having to host telemetry yourself and the ability to keep all your configurations in one place. 

The Grafana Operator is a Kubernetes operator to help you manage your Grafana instances and their resources from within Kubernetes. In this blog post, we’ll show you how to connect Grafana Operator with your Grafana Cloud stack, as well as how to configure folders, dashboards, and alerts as Kubernetes objects.

Getting started with the Grafana Operator

To begin using the Grafana Operator with Grafana Cloud or on-prem Grafana instances, follow these steps:

Quick installation

The quickest way to install the Grafana Operator in your cluster is by using Helm. Run the following command in your terminal:

Copy

helm upgrade -i grafana-operator oci://ghcr.io/grafana/helm-charts/grafana-operator --version v5.7.0

This command uses Helm to fetch the latest Grafana Operator chart from the specified repository and install it in your cluster. You can also specify a particular version using the –version attribute in the above command. And if there are particular configurations you want to alter, peek into the values.yaml of the Helm chart for customization options.

For other ways to install, check out our Grafana Operator installation documentation.

Connecting to Grafana Cloud

The Grafana Operator supports Grafana Cloud instances using external instances. These are Grafana instances that are not deployed by the operator – meaning, they are not running in your Kubernetes environment – but should still be configured by it. To create an external instance, you will need:

• The URL of your Grafana Cloud instance
• A service account token for authentication

Creating a service account

To create a service account, navigate to the Service accounts page, located in the Administration section of your Grafana instance.

Click the Add service account button in the top right corner and give your service account a descriptive name like “Grafana Operator in cluster XYZ”.

The role you assign to the service account will restrict what the operator can do. If you’re unsure what to pick, editor is a good starting point, but you will need to add additional permissions if you want to configure data sources.

After creating the service account, create a token for this account. Save this token in a Kubernetes secret in a namespace of your choice.

Copy

apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud
  namespace: default
type: Opaque
stringData:
  SERVICE_ACCOUNT_TOKEN: glsa_.....

Creating the external instance

Now that we have a secret containing our authentication information, the next step is to create an external instance. The YAML for it looks like this:

Copy

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: my-grafana-cloud-stack
  labels:
    instance: "my-grafana-cloud-stack"
spec:
  external:
    url: https://<your-stack-name>.grafana.net
    apiKey:
      name: grafana-cloud
      key: SERVICE_ACCOUNT_TOKEN

It is important to set a meaningful label on this resource, as labels are used to match resources to instances.

After applying this to your cluster, you can verify that everything works by running kubectl get grafanas:

Copy

NAME           			STAGE  		STAGE STATUS   AGE
my-grafana-cloud-stack	complete   	success    	4d22h

Creating resources

Now that the operator knows about your instance, you can begin adding resources, including folders, dashboards, and alert rules. 

Folders

Folders are the easiest resource to configure. In their simplest form, they only require a name to be set.

Copy

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: test-folder
spec:
  instanceSelector:
    matchLabels:
      instance: "my-grafana-cloud-stack"
  # If title is not defined, the value will be taken from metadata.name
  title: custom title

This will create a top level folder called “custom title” in your cloud stack. If you want to configure multiple instances simultaneously, you can create more instances and set the instanceSelector accordingly.

Dashboards

The GrafanaDashboard resource looks similar to the folder resource.

Copy

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-sample
spec:
  instanceSelector:
    matchLabels:
      instance: "my-grafana-cloud-stack"
  json: >
	{
  	"id": null,
  	"title": "Simple Dashboard",
  	"tags": [],
  	"style": "dark",
  	"timezone": "browser",
  	"editable": true,
  	"hideControls": false,
  	"graphTooltip": 1,
  	"panels": [],
  	"time": {
    	"from": "now-6h",
    	"to": "now"
  	},
  	"timepicker": {
    	"time_options": [],
    	"refresh_intervals": []
  	},
  	"templating": {
    	"list": []
  	},
  	"annotations": {
    	"list": []
  	},
  	"refresh": "5s",
  	"schemaVersion": 17,
  	"version": 0,
  	"links": []
	}	

In a standard scenario, the operator would use the custom resource name to deploy and manage the folder in Grafana. The spec.folder field can be used to set a custom folder name.

Alert rules

Alert rules in Grafana are heavily inspired by Prometheus rules. They are grouped in alert rule groups, which have a name and evaluation interval. Alert rule groups are stored in folders, the same way dashboards are. As alert rules can get quite complex, the recommended approach is to build the alert rules in the UI and then use the Modify export functionality.

If you want to test out this functionality, but don’t have any data in your instance yet, you can use one of our demo data sources.

Copy the rules section from the resulting YAML and use it to fill out the GrafanaAlertRuleGroup as follows:

Copy

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: temperature-alerts
spec:
  folderRef: 'test-folder'
  instanceSelector:
    matchLabels:
      instance: "my-grafana-cloud-stack"
  interval: "5m"
  rules:
    - uid: cde8n0peq8rnkd
      title: "Something is happening"
      #...

You will find your created alert rule in the alerting section of our Grafana instance.

Next steps 

By using the Grafana Operator for resource management within Kubernetes, you make it easier to incorporate Grafana Cloud into your Kubernetes-driven environment. This method also ensures uniform and effective configuration management, adhering to the infrastructure-as-code principle, which streamlines your workflows and increases efficiency.

To learn more about the Grafana Operator, you can refer to our technical documentation or checkout our GitHub Repo. You can also reach out to the maintainers in the #grafana-operator channel in our Grafana Community Slack, or join the weekly community calls on Tuesdays at 12:30 PM UTC.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, and dashboards. We have a generous forever-free tier and plans for every use case. Sign up for free now!