How to use kwatch to detect crashes in Kubernetes clusters
kwatch is an open-source tool that detects crashes in Kubernetes Pods in real-time and sends the notifications to supported backends. In this post, we'll show you how to deploy and configure kwatch to send Kubernetes Pod crash notifications to PagerDuty.
In this post, you will learn how to set up and deploy kwatch in your Kubernetes cluster to monitor and receive real-time crash alerts through PagerDuty.
kwatch helps you monitor all changes in your Kubernetes cluster. It detects crashes in real-time and publishes notifications to your favorite channels (Slack, Discord, PagerDuty, Telegram, Microsfot Teams and others) instantly.
To go through this tutorial, you'll need a free PagerDuty account and access to a Kubernetes cluster. To try this out, you can use minikube, kind or any other Kubernetes cluster.
Configure PagerDuty
We'll configure a new PagerDuty service through Events API. kwatch will use the events API endpoint and the integration key to send events to PagerDuty.
Follow these steps to configure the PagerDuty service:
After you've logged into your PagerDuty account, click the Service menu and then click the New Service button.
Name your service test, add a brief description, and click Next.
Creating a new PagerDuty service
New service
On the next screen, leave the Generate a new Escalation Policy selected and click Next.
Generate a new Escalation Policy
Using the escalation policy, you can connect services to individual users and/or schedules so correct people get notified in case of an escalation.
On the Alert Grouping page, we can leave the Intelligent option selected and click Next. The alert grouping allows you to combine similar alerts into a single incident to reduce notifications.
On the Integrations page, we'll select the Events API V2 integration by clicking on the check box.
Events API V2 integration
Events API V2 integration
Click Create Service to create the service.
Once the service has been created you'll be presented with the page that shows the integration key and the integration URLs as shown in the image below.
Events API V2 overview
Events API V2 overview
We'll use the integration key to configure kwatch.
Configure kwatch
To configure kwatch we'll download a template ConfigMap and then configure it to use the PagerDuty integration key.
Let's start by obtaining the configuration template first:
Once the Pod is up and running, we can edit the deployment and change the image name to a non-existent image – this is just to force the Pod to crash by failing to pull the (non-existent) image.
We can run kubectl edit deploy httpbin and change the docker.io/kennethreitz/httpbin image name to BLAHdocker.io/kennethreitz/httpbin.
As expected, the Pod will fail, and if you look on your PagerDuty dashboard, you'll notice an open incident that contains the details about the failure.
PagerDuty dashboard with an alert
PagerDuty dashboard with an alert
Conclusion
In this blog post, we've introduced a way to use kwatch and PagerDuty to trigger alerts whenever a Pod fails.
We've used the most straightforward configuration to watch all namespaces in the Kubernetes cluster. We could also provide specific namespaces we want to monitor using the' namespaces' field.
In addition to PagerDuty, kwatch also offers integration with Slack, Telegram, Microsoft Teams, Rocket Chat, and Mattermost.