Monitoring containers with cAdvisor
Published on

Monitoring containers with cAdvisor

Author
Written by Peter Jausovec
    Contents
  1. Running cAdvisor in Kubernetes
  2. Connecting cAdvisor to Prometheus and Grafana
  3. Installing Prometheus on Kubernetes
  4. Grafana dashboards
  5. What's next?
  6. Resources
Discuss on Discord
Monitoring with cAdvisor allows you to gather information about individual Docker containers running on your host - be it a virtual machine, Kubernetes cluster, or any other host capable of running containers.
cAdvisor (short for "Container Advisor") is a daemon that collects the data about the resource usage and performance of your containers.
In addition to container usage metrics, cAdvisor can also collect metrics from your applications. If your applications are already emitting metrics, you can configure cAdviser to scrape the endpoint and include which metrics you want to extract.
cAdvisor also features a built-in UI, and it also allows you to export the collected data to different storage driver plugins.
cAdvisor UI
cAdvisor UI
For example, you can export the collected data to:
The easiest way to get started and see the data that gets collected is to run the cAdvisor Docker image locally:
sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:v0.37.5

Note

The latest version of cAdvisor at the time of writing this was v0.37.5. Make sure you're always using the latest bits.
In case you're wondering about all those volumes... These are the folders you need to mount inside the cAdvisor image so that the cAdvisor can analyze all the data from them.
Once the cAdvisor is running, it will collect data about all containers running on the same host. Note that there are options you can set to limit which containers get monitored.

Running cAdvisor in Kubernetes

cAdvisor is integrated with the kubelet binary, and it exposes the metrics on /metrics/cadvisor endpoint.
Therefore we don't need to install cAdvisor on the Kubernetes cluster explicitly.
Here's an example of how we can use kubectl to retrieve the cluster node metrics and Pod metrics:
$ kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/[node-name]
{
  "kind": "NodeMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "[node-name]",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/[node-name]]",
    "creationTimestamp": "2021-08-26T22:12:26Z"
  },
  "timestamp": "2021-08-26T22:11:53Z",
  "window": "30s",
  "usage": {
    "cpu": "39840075n",
    "memory": "487200Ki"
  }
}
Similarly, we can use the following URL /apis/metrics.k8s.io/v1beta1/namespaces/<NAMESPACE>/pods/<POD_NAME> to get the metrics about a specific pod.
Let's create an httpbin deployment:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml
To retrieve the metrics from the httpbin pod, run the command below (make sure you replace the pod name with the name of your pod running in your cluster):
$ kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/httpbin-74fb669cc6-xs74p
{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "httpbin-74fb669cc6-xs74p",
    "namespace": "default",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/httpbin-74fb669cc6-xs74p",
    "creationTimestamp": "2021-08-26T22:15:40Z"
  },
  "timestamp": "2021-08-26T22:15:16Z",
  "window": "30s",
  "containers": [
    {
      "name": "httpbin",
      "usage": {
        "cpu": "316267n",
        "memory": "38496Ki"
      }
    }
  ]
}

Connecting cAdvisor to Prometheus and Grafana

By default, cAdvisor exposes the Prometheus metrics on the /metrics endpoint.
# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="de117632",cadvisorVersion="v0.39.0",dockerVersion="20.10.3",kernelVersion="5.4.104+",osVersion="Alpine Linux v3.12"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{container_env_ARG1="",container_env_ARG2="",container_env_CADVISOR_HEALTHCHECK_URL="",container_env_DEFAULT_HTTP_BACKEND_PORT="",container_env_DEFAULT_HTTP_BACKEND_PORT_80_TCP="",container_env_DEFAULT_HTTP_BACKEND_PORT_80_TCP_ADDR="",container_env_DEFAULT_HTTP_BACKEND_PORT_80_TCP_PORT="",container_env_DEFAULT_HTTP_BACKEND_PORT_80_TCP_PROTO="",
...
Because metrics are already in Prometheus format and cAdvisor exports them automatically on a well-known endpoint, we don't need to change the existing cAdvisor deployment. Instead, we can install and configure Prometheus to scrape the metrics from the /metrics endpoint.

Installing Prometheus on Kubernetes

I'll use Prometheus Operator to install Prometheus on Kubernetes. We'll install the complete monitoring bundle, including Prometheus, Grafana, and Alert manager.
Start by cloning the kube-prometheus repository:
git clone https://github.com/prometheus-operator/kube-prometheus.git
Then, go to the kube-prometheus folder and deploy the CRDs first:
kubectl apply -f manifests/setup
Wait for a bit for the CRDs to be applied and then create the deployments:
kubectl apply -f manifests/
Once you've deployed everything (you can run kubectl get pod -A to check all pods are up and running), you can open the Prometheus UI:
kubectl port-forward svc/prometheus-k8s 9090 -n monitoring
If you open http://localhost:9090, you can now query for any metrics collected by the cAdvisor - e.g., metrics starting with container_* as shown in the figure below.
Prometheus metrics from cAdvisor
Prometheus metrics from cAdvisor

Grafana dashboards

Grafana gets installed as part of the kube-prometheus operator. We can open the Grafana UI by port-forwarding to port 3000:
kubectl port-forward svc/grafana 5000:3000 -n monitoring
If you open Grafana on http://localhost:5000 you'll notice there's already a set of pre-created dashboards that came with the kube-prometheus operator.
Grafana dashboards
Grafana dashboards
The dashboards show you the information about Kubernetes resources - memory usage, CPU usage, quotas, and so on. These metrics are coming from the node-exporter component.
The node-exporter exports the hardware and OS metrics to Prometheus while cAdvisor collects the metrics about containers.
To get the cAdvisor metrics pulled into Grafana, we'll install the Kubernetes cluster monitoring (via Prometheus) dashboard from Grafana.
Installing a dashboard is straightforward.
  1. In Grafana, go to the "+" button on the sidebar.
  2. Click Import.
Grafana import
Grafana import
  1. Paste the dashboard ID (315 in our case) to the ID text field
  2. Click the Load button.
  3. From the Prometheus drop-down list, select "prometheus".
  4. Click the Import button.
Grafana import screen
Grafana import screen
When Grafana imports the dashboard, it will automatically open it. The dashboard features high-level metrics about the total CPU and memory usage and detailed metrics about each specific container.
aAdvisor dashboard
aAdvisor dashboard

What's next?

As the next step, you should familiarize yourself with the graphs and data displayed in Grafana and learn how to read them. Find which metrics and dashboards are valuable to you and your system.
Once you've decided that, you might want to set up the alerting. The Prometheus operator includes the Alert manager. You can use the alert manager and configure it to send alerts when specific metrics are not within the defined thresholds. For example, you could configure the system to send a notification to PagerDuty whenever the cluster memory or CPU usage is above a certain threshold.

Resources

Join the discussion
SHARE THIS ARTICLE
Peter Jausovec

Peter Jausovec

Peter Jausovec is a platform advocate at Solo.io. He has more than 15 years of experience in the field of software development and tech, in various roles such as QA (test), software engineering and leading tech teams. He's been working in the cloud-native space, focusing on Kubernetes and service meshes, and delivering talks and workshops around the world. He authored and co-authored a couple of books, latest being Cloud Native: Using Containers, Functions, and Data to Build Next-Generation Applications.

Related posts

How to use kwatch to detect crashes in Kubernetes clusters
Kubernetes

How to use kwatch to detect crashes in Kubernetes clusters

kwatch is an open-source tool that detects crashes in Kubernetes Pods in real-time and sends the notifications to supported backends. In this post, we'll show you how to deploy and configure kwatch to send Kubernetes Pod crash notifications to PagerDuty.

Amgad Ramses

Amgad Ramses

Running Hugo on free Ampere VM (Oracle Cloud Infrastructure)
General

Running Hugo on free Ampere VM (Oracle Cloud Infrastructure)

In this article, I'll take you on a journey of setting up a free-for-life virtual machine instance running on OCI. We'll be creating the account, a virtual machine instance, creating a Github repository with Hugo, setting up Nginx on the VM, and obtaining a free SSL certificate.

Peter Jausovec

Peter Jausovec

Attach multiple VirtualServices to Istio Gateway
Service Mesh

Attach multiple VirtualServices to Istio Gateway

In this post, you'll learn how to expose multiple Kubernetes services running inside your cluster using Istio' Gateway and VirtualService resources.

Peter Jausovec

Peter Jausovec

GetIstio - CLI, training, and community
Service Mesh

GetIstio - CLI, training, and community

GetIstio CLI was released this week. GetIstio features aCLI that offers an easier way to install and upgrade Istio, a free Istio Fundamentals training, and a community, with the first meeting coming up on February 18th.

Peter Jausovec

Peter Jausovec

Build Netlify-like deployment for React app using Kubernetes pods
Kubernetes

Build Netlify-like deployment for React app using Kubernetes pods

Ever wondered how you can build your own system that automatically updates your React app each time you push changes to the repository where your app is hosted? In this article I explain how you can use build a Netlify-like deployment for React apps using a multi-container Kubernetes pod.

Peter Jausovec

Peter Jausovec

Rapid microservices development with Signadot
Service mesh

Rapid microservices development with Signadot

While developing microservices locally is possible, running and testing them in a production-like Kubernetes environment is complex. A typical development workflow while developing service in Kubernetes can significantly slow you down - from building a Docker image, pushing it, restarting the deployments, and testing the changes in a shared cluster. And all that, assuming you manage to keep the shared cluster up to date! In this article, I'll look at a tool called Signadot. Signadot introduces a concept of sandboxes that allow you to considerably shorten your developer workflow and go from minutes to mere seconds! The sandbox concept will enable you to build and run a service locally using the upstream and downstream dependencies inside a shared cluster.

Peter Jausovec

Peter Jausovec

;