Menu
DocumentationGrafana CloudMonitor infrastructureKubernetes MonitoringConfigure Kubernetes MonitoringHelm chart overview
Grafana Cloud

Overview of Grafana Kubernetes Monitoring Helm chart

When you use the easy configuration process in Kubernetes Monitoring, the Grafana Kubernetes Monitoring Helm chart is adjusted based on your choices for configuration. The Helm chart deploys software as well as a tuned configuration to monitor your Cluster. As improvements are made to Kubernetes Monitoring, the Helm chart is updated as well. You can also customize the chart for your specific needs and tailor it to specific cluster environments. For example:

  • Your configuration might already have an existing kube-state-metrics in your Cluster, so you don’t want the Helm chart to install another one.
  • Enterprise clusters with many workloads running can have specific requirements.

Charts and subcharts

The Grafana Kubernetes Monitoring Helm chart deploys a complete monitoring solution for your Cluster. It includes all of the backing systems listed in Tools installed with Helm chart, and all the configuration to make these systems run. The Grafana Alloy Helm chart, which is deployed by the Kubernetes Monitoring Helm chart, simply deploys Alloy.

Default installation with Helm chart

The Helm chart installs the following items by default for infrastructure and application monitoring.

Items installed by Helm chart and their function
Items installed by Helm chart and their function

Tools installed with Helm chart

The Helm chart installs the following for monitoring infrastructure if you choose to collect metrics, Pod logs, Cost metrics, and Cluster Events. For applications, the chart installs telemetry receivers. All elements installed by the Helm chart are automatically updated.

  • kube-state-metrics, to listen to the API server and generate metrics on the health of objects such as deployments, Nodes, and Pods

  • Node Exporter, to provide hardware device and kernel-related metrics

  • OpenCost, to calculate cost data

  • An Alloy Deployment for Cluster Events, to get logs from the API server

  • An Alloy DaemonSet to retrieve Pod logs

  • An Alloy StatefulSet for metrics and application data, to get metrics from:

    • Kubelet, the primary Node agent which ensures containers are running and healthy
    • Node Exporter
    • OpenCost
    • cAdvisor, which provides container CPU, memory, and disk usage
    • kube-state-metrics

    This StatefulSet also opens up receivers for accepting application metrics, logs, and traces.

  • An Alloy DaemonSet to gather profiles

The chart creates configuration files for Grafana Alloy.

Deployment

After you have made configuration choices, the values.yaml file is altered to reflect your selections for configuration. When you deploy the chart with the command available in the configuration wizard, the chart installs the appropriate elements. Four separate instances of Alloy deploy so that there are no issues with scaling.

After deployment, you can check the Metrics status tab under Configuration. This page provides a snapshot of the overall health of the metrics being ingested.

Descriptions and statuses for each item chosen to be configured and whether they are online
Metrics status tab

Customization

Refer to the Examples directory of the Helm chart to view the values.yaml file when you have a specific modification you want to make. You can also look at the output.yaml and other files included in the examples. Each of these examples is tested to ensure they work properly.

Troubleshooting tips

Some general troubleshooting techniques include:

  • Checking for any changes with the command helm template .... This produces an `output.yaml’ file to check the result.
  • Checking the configuration with the command helm test --logs. This provides a configuration validation, including all phases of metrics gathering through display.
  • Check the extraConfig section of the Helm chart to ensure this section is not used for modifications. This section is only for additional configuration not already in the chart, and not for modifications to the chart.

Two common issues often occur when a Helm chart is not configured correctly:

  • Duplicate metrics
  • Missing metrics

Duplicate metrics

For duplicate metrics, determine whether some metric data sources (such as Node Exporter or kube-state-metrics) already exist on the Cluster. If so, remove them or adjust the Helm chart values to use the existing ones and skip deploying another instance.

Missing metrics

It’s helpful to keep in mind the different phases of metrics gathering when debugging:

  • Discovery: Find the metric source. In this phase, find out whether the tool to gather metrics is working. For example, is Node Exporter running? Can Alloy find Node Exporter? Perhaps there’s configuration that’s not correct because Alloy is looking in a namespace or for a specific label.
  • Scraping: Were the metrics gathered correctly? As an example, most metric sources use HTTP, but the metric source you are trying to find uses HTTPS. Identify whether the configuration is set for scraping HTTPS.
  • Processing: Were metrics correctly processed? With Kubernetes Monitoring, metrics are filtered to a small subset of the useful metrics.
  • Delivering: In this phase, metrics are sent to Grafana Cloud. While this is an uncommon problem, if there is an issue, there are likely no metrics being delivered.
  • Displaying: A metric is not showing up in the Kubernetes Monitoring GUI. If you’ve determined the metrics are being delivered but some are not displaying, there may be a missing or incorrect label for the metric. Each metric has a Cluster label.

Also refer to Missing metrics