Overview of Grafana Kubernetes Monitoring Helm chart
When you use the easy configuration process in Kubernetes Monitoring, the Grafana Kubernetes Monitoring Helm chart is adjusted based on your choices for configuration. The Helm chart deploys software as well as a tuned configuration to monitor your Cluster. As improvements are made to Kubernetes Monitoring, the Helm chart is updated as well. You can also customize the chart for your specific needs and tailor it to specific cluster environments. For example:
- Your configuration might already have an existing kube-state-metrics in your Cluster, so you don’t want the Helm chart to install another one.
- Enterprise clusters with many workloads running can have specific requirements.
Charts and subcharts
The Grafana Kubernetes Monitoring Helm chart deploys a complete monitoring solution for your Cluster. It includes all of the backing systems listed in Tools installed with Helm chart, and all the configuration to make these systems run. The Grafana Alloy Helm chart, which is deployed by the Kubernetes Monitoring Helm chart, simply deploys Alloy.
Default installation with Helm chart
The Helm chart installs the following items by default for infrastructure and application monitoring.
Tools installed with Helm chart
The Helm chart installs the following for monitoring infrastructure if you choose to collect metrics, Pod logs, Cost metrics, and Cluster Events. For applications, the chart installs telemetry receivers. All elements installed by the Helm chart are automatically updated.
kube-state-metrics, to listen to the API server and generate metrics on the health of objects such as deployments, Nodes, and Pods
Node Exporter, to provide hardware device and kernel-related metrics
OpenCost, to calculate cost data
An Alloy Deployment for Cluster Events, to get logs from the API server
An Alloy DaemonSet to retrieve Pod logs
An Alloy StatefulSet for metrics and application data, to get metrics from:
- Kubelet, the primary Node agent which ensures containers are running and healthy
- Node Exporter
- OpenCost
- cAdvisor, which provides container CPU, memory, and disk usage
- kube-state-metrics
This StatefulSet also opens up receivers for accepting application metrics, logs, and traces.
An Alloy DaemonSet to gather profiles
The chart creates configuration files for Grafana Alloy.
Deployment
After you have made configuration choices, the values.yaml
file is altered to reflect your selections for configuration.
When you deploy the chart with the command available in the configuration wizard, the chart installs the appropriate elements.
Four separate instances of Alloy deploy so that there are no issues with scaling.
After deployment, you can check the Metrics status tab under Configuration. This page provides a snapshot of the overall health of the metrics being ingested.
![Metrics status tab Descriptions and statuses for each item chosen to be configured and whether they are online](/media/docs/grafana-cloud/k8s/screenshot-metrics-status.png)
Customization
Refer to the Examples directory of the Helm chart to view the values.yaml
file when you have a specific modification you want to make.
You can also look at the output.yaml
and other files included in the examples.
Each of these examples is tested to ensure they work properly.
Troubleshooting tips
Some general troubleshooting techniques include:
- Checking for any changes with the command
helm template ...
. This produces an `output.yaml’ file to check the result. - Checking the configuration with the command
helm test --logs
. This provides a configuration validation, including all phases of metrics gathering through display. - Check the
extraConfig
section of the Helm chart to ensure this section is not used for modifications. This section is only for additional configuration not already in the chart, and not for modifications to the chart.
Two common issues often occur when a Helm chart is not configured correctly:
- Duplicate metrics
- Missing metrics
Duplicate metrics
For duplicate metrics, determine whether some metric data sources (such as Node Exporter or kube-state-metrics) already exist on the Cluster. If so, remove them or adjust the Helm chart values to use the existing ones and skip deploying another instance.
Missing metrics
It’s helpful to keep in mind the different phases of metrics gathering when debugging:
- Discovery: Find the metric source. In this phase, find out whether the tool to gather metrics is working. For example, is Node Exporter running? Can Alloy find Node Exporter? Perhaps there’s configuration that’s not correct because Alloy is looking in a namespace or for a specific label.
- Scraping: Were the metrics gathered correctly? As an example, most metric sources use HTTP, but the metric source you are trying to find uses HTTPS. Identify whether the configuration is set for scraping HTTPS.
- Processing: Were metrics correctly processed? With Kubernetes Monitoring, metrics are filtered to a small subset of the useful metrics.
- Delivering: In this phase, metrics are sent to Grafana Cloud. While this is an uncommon problem, if there is an issue, there are likely no metrics being delivered.
- Displaying: A metric is not showing up in the Kubernetes Monitoring GUI. If you’ve determined the metrics are being delivered but some are not displaying, there may be a missing or incorrect label for the metric. Each metric has a Cluster label.
Also refer to Missing metrics
Was this page helpful?
Related resources from Grafana Labs
![Getting started with the Grafana LGTM Stack](/static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=200 200w, /static/assets/featured/getting-started-with-grafana-lgtm-stack-amer_featured.png?w=400 400w)
![Intro to Kubernetes monitoring in Grafana Cloud](/media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=200 200w, /media/webinar/Screenshot-2024-02-12-at-4.09.30-PM.png?w=400 400w)
![Building advanced Grafana dashboards](/media/webinar/AMER_buildadv_notxt.png?w=200 200w, /media/webinar/AMER_buildadv_notxt.png?w=400 400w)