Monitoring Kubernetes

Check_MK Manual
Last updated: February 5 2019

Search in the manual

1. Introduction

The great success of Docker has led to people using Docker on an ever-larger scale. In contrast to virtual machines such as VMWare, its very low overhead makes the container ‚cheap’ and thus almost a mass-product. It goes without saying that a good tool for orchestrating the containers is essential. For the majority, the open source tool Kubernetes will be the tool of choice.

Check_MK supports monitoring of Kubernetes from version 1.5.0p12. The focus is currently on states and metrics that are especially interesting for the administrator. The following Check plug-ins are available in version 1.5.0p12:

This is only a first step however. Future versions of Check_MK will further expand and refine the monitoring of Kubernetes. It could also be possible that we change the architecture of the monitoring again and you will thus need to adjust your configuration.

2. Setting up the monitoring

2.1. Service-Account

To set up a Kubernetes cluster in Check_MK you first need to have a service account and a related Cluster Role in Kubernetes so that Check_MK can access the API. We will create the file check_mk_rbac.yaml for you as a ready template which you will find in the ‚Treasures, in the share/doc/check_mk/treasures/kubernetes directory. Their first part looks something like this:

share/doc/check_mk/treasures/kubernetes/check_mk_rbac.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: check-mk
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: check-mk
  namespace: check-mk
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: check-mk
rules:

We use check-mk here as Name and Namespace respectively.

Load this file onto your Kubernetes cluster with the kubectl command:

user@host:~$ kubectl apply -f check_mk_rbac.yaml
namespace/check-mk created
serviceaccount/check-mk created
clusterrole.rbac.authorization.k8s.io/check-mk created
clusterrolebinding.rbac.authorization.k8s.io/check-mk created

If you use the Google Kubernetes engine, it may be that you receive an "Error from server (Forbidden): error when creating get "check_mk_rbac.yaml": response. In this case you must first extend your user’s permissions. This is done with the following command (replacing MYNAME with your Google login name):

user@host:~$ kubectl create clusterrolebinding MYNAME-cluster-admin-binding --clusterrole=cluster-admin --user=MYNAME@example.org

If all has gone well, you can query the new service account with kubectl get serviceaccounts:

user@host:~$ kubectl get serviceaccounts check-mk -n check-mk -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},
"name":"check-mk","namespace":"check-mk"}}
  creationTimestamp: "2019-01-23T08:16:05Z"
  name: check-mk
  namespace: check-mk
  resourceVersion: "4004661"
  selfLink: /api/v1/namespaces/check-mk/serviceaccounts/check-mk
  uid: 218179a3-1ee7-11e9-bf43-080027a5f141
secrets:
- name: check-mk-token-z9hbp

There you will also find the name of the associated Secrets. This has the form ‚check-mk-token-ID‘ (here in the example check-mk-token-z9hbp). The ID for the Secret is generated automatically by Kubernetes. You can then use the contents of the Secrets with the get secrets query:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQVRBTkJna3Foa2lHO...
  namespace: Y2hlY2stbWs=
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObG...
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: check-mk
    kubernetes.io/service-account.uid: 218179a3-1ee7-11e9-bf43-080027a5f141
  creationTimestamp: "2019-01-23T08:16:06Z"
  name: check-mk-token-z9hbp
  namespace: check-mk
  resourceVersion: "4004660"
  selfLink: /api/v1/namespaces/check-mk/secrets/check-mk-token-z9hbp
  uid: 2183cee6-1ee7-11e9-bf43-080027a5f141
type: kubernetes.io/service-account-token

The output will include the base64 encoded CA certificate (ca.crt), and the base64 encoded tokens (token) for the account. You can choose the certificate from the output of get secret – e.g. with the following command cut it out, and immediately convert it to the form you need to import into Check_MK:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "ca.crt" | cut -f4 -d' ' | base64 --decode
-----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTE4MDkxMDE2MDAwMVoXDTI4MDkwODE2MDAwMVowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK9Z
iG0gNZK5VU94a0E6OrUqxOQRdkv6S6vG3LnuozdgNfxsEetR9bMGu15DWaSa40JX
FbC5RxzNq/W9B2pPmkAlAguqHvayn7lNWjoF5P+31tucIxs3AOfBsLetyCJQduYD
jbe1v1/KCn/4YUzk99cW0ivPqnwVHBoMPUfVof8yA00RJugH6lMZL3kmOkD5AtRH
FTThW9riAlJATBofLfkgRnUEpfb3u1xF9vYEDwKkcV91ealZowJ/BciuxM2F8RIg
LdwF/vOh6a+4Cu8adTyQ8mAryfVPDhFBhbsg+BXRykhNzNDPruC+9wAG/50vg4kV
4wFpkPOkOCvB8ROYelkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQW
MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBCwUAA4IBAQAeNwON8SACLl2SB8t8P4/heKdR3Hyg3hlAOSGjsyo396goAPS1
t6IeCzWZ5Z/LsF7o8y9g8A7blUvARLysmmWOre3X4wDuPvH7jrYt+PUjq+RNeeUX
5R1XAyFfuVcWstT5HpKXdh6U6HfzGpKS1JoFkySrYARhJ+MipJUKNrQLESNqdxBK
4gLCdFxutTTFYkKf6crfIkHoDfXfurMo+wyEYE4Yeh8KRSQWvaKTdab4UvMwlUbO
+8wFZRe08faBqyvavH31KfmkBLZbMMM5r4Jj0Z6a56qZDuiMzlkCl6rmKynQeFzD
KKvQHZazKf1NdcCqKOoU+eh6q6dI9uVFZybG
-----END CERTIFICATE-----

2.2. Importing a certificate into Check_MK

For Check_MK to accept the Kubernetes CA certificate, you must add it to WATO at Global Settings ➳ Site Management ➳ Trusted certificate authorities for SSL.

Without the correct import of the CA, the Check_MK service of the Kubernetes cluster will fail with and certificate verify failed:

2.3. Entering a password (Token) in Check_MK

The best way to save the service account token is to use WATO's password storage. This is the safest option, since the deposit and the use of the passwords is organizationally separate. Alternatively, enter the password directly in plain text when creating the rule (see below).

The following command line truncates the password directly from the output of get secrets:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode
 eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjaGVjay1tayIsI
mt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjaGVjay1tay10b2tlbi16OWhicCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5
hbWUiOiJjaGVjay1tayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjIxODE3OWEzLTFlZTctMTFlOS1iZjQzLTA4MDAyN2E1ZjE0MSIsInN1YiI6I
nN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjaGVjay1tazpjaGVjay1tayJ9.gcLEH8jjUloTeaAj-U_kRAmRVIiETTk89ujViriGtllnv2iKF12p0L9ybT1fO-1Vx7XyU8jneQRO9lZw8JbhVmaPjrkEc8
kAcUdpGERUHmVFG-yj3KhOwMMUSyfg6wAeBLvj-y1-_pMJEVkVbylYCP6xoLh_rpf75JkAicZTDmhkBNOtSf9ZMjxEmL6kzNYvPwz76szLJUg_ZC636OA2Z47qREUtdNVLyutls7ZVLzuluS2rnfoP
JEVp_hN3PXTRei0F5rNeA01wmgWtDfo0xALZ-GfvEQ-O6GjNwHDlsqYmgtz5rC23cWLAf6MtETfyeEJjRqwituhqUJ9Jp7ZHgQ%

If you are working directly under Linux, you can also enter |xsel--clipboard. Then the password is not output, but copied directly to Clipboard (as if you had copied with the mouse):

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode | xsel --clipboard

The ‚password‘ really is that long. Add it, for example, under the ID kubernetes in the password storage:

2.4. Adding a Kubernetes-Cluster to the Monitoring

The monitoring under Check_MK functions in two levels. The Kubernetes Cluster itself is monitored as a host. For the individual Kubernetes nodes we use the piggyback principle. That means each node is monitored as a separate host in Check_MK. The monitoring data from these hosts are not retrieved separately from Kubernetes, but instead derived from the data from the Kubernetes cluster.

Because Kubernetes are not queried through the regular Check_MK agent you may need the Kubernetes Special Agent, which is also called data source program. Hereby Check_MK does not contact the destination host as usual over TCP port 6556, but instead calls a utility that communicates with the target system via the application-specific API in Kubernetes.

This is why you need one more step: set a rule which assigns this special agent to your Kubernetes host. These can be found in WATO at Host & Service Parameters ➳ Datasource Programs ➳ Kubernetes. In the rule’s properties you either enter the password in plain text or select it from the password storage if you have already saved it there.

You do not normally need any further information. The functions of the other options are best found in the Online Help ICON [icon_help.png].

Now when you query the Kubernetes cluster’s service configuration, you should already find some of its services:

2.5. Monitoring the nodes

So that the nodes are also monitored, you must also create them as hosts in WATO You can do this (from Check_MK version 1.6.0) with the new Dynamic Configuration Daemon (DCD). Or you simply create these as hosts by hand.

It is important that the hostnames in Check_MK exactly match the names of the Kubernetes nodes. You can easily get these names from the Kubernetes host’s Nodes service.

Unless you have a Check_MK agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

3. Hardware/software inventory

The Kubernetes integration in Check_MK also supports the [Inventory|hardware/software inventory]. In version 1.5.0p12 this is limited to the Kubernetes roles. More plug-ins are planned.

4. Removing Check_MK

If you want to remove Check_MK’s service account and cluster role from Kubernetes, this can be performed with the following commands:

user@host:~$ kubectl -f check_mk_rbac.yaml
namespace "check-mk" deleted
serviceaccount "check-mk" deleted
clusterrole.rbac.authorization.k8s.io "check-mk" deleted
clusterrolebinding.rbac.authorization.k8s.io "check-mk" deleted