Monitoring of dockers

Navigation

Contents

Search in the user manual

This article is just a draft and not yet finished!

1. Introduction

In recent years the container concept has taken the IT world by storm. That has of course thrown up questions about the monitoring of such containers. From Version 1.5.0 Check_MK can monitor docker containers directly over the Linux agents. In this way not only framework data – such as the status of the daemons or the container can be monitored – but also the container itself. A full list of the elements that can currently be monitored can, as usual, be found in Catalogue of the Check plug-ins.

Alongside the status and inventory information which Check_MK can determine over the Node (docker-jargon for 'the host on which the containers are running'), Check_MK can also determine detailed status information for the containers. For this every container will be installed as a self-contained host in Check_MK if the container is to be monitored. Its data will be piggybacked to this host. In the future the creation and deletion of such hosts will be automated. Until then these hosts must be created manually, or with the aid of your own scripts.

2. Creation

2.1. Creating a docker node

For the host on which the containers are running no other action is required other than to install the normal Linux agent – this then automatically detects whether a docker node is present.

Install the newly-detected services in the usual manner:

Once the changes have been activated, monitoring of the the docker's framework data will commence.

2.2. Creating the container hosts

Monitoring a container can be performed in two different ways. The simplest method is to use the piggyback data which can be installed between the docker host and every container. Install the appropriate Plug-in on the docker node as well. If you use the Agent Bakery, also create a rule in Piggybacked docker containers. Further configuration is not needed for the plug-in to function. The data will subsequently be retrieved from the docker node automatically.

The next step is to define the container(s) as a Host in Check_MK, and in the host set the Check_MK Agent host tag to No agent, and the IP address family host tag to No IP. In addition you can set the docker host as a Parent. The container's ID will be the host's name.

Here as well the the new services will then be automatically detected and need only to be activated:

Monitoring the host's status

Since a container's Host status cannot really be verified using TCP-Packets or ICMP, this must be determined in another way. The Docker container status service facilitates this – in any case it checks whether or not the container is running, and can thus be used as a secure tool for detecting the host's status. Define a rule in the Host Check Command rule set for this purpose, and set the Use the status of the service option to the mentioned service. Don't forget to set the conditions so that only containers are affected. In our example all containers are located in a folder with the same name:

Operating the agent directly in the container

Especially for self-created docker images you might want to implement the agents themselves in the container. In this case the data will no longer be retrieved from the docker's host's agents as described above. Instead, a separate agent will run in each container. Check_MK detects completely automatically whether an agent is present in the container, so that the data is always retrieved only once.

This however only functions if all necessary commands are present in the container. Especially with minimally-constructed containers based on Alpine-Linux, it is quite possible that elementary things such as the Bash are not present. In this case the container should be monitored from the docker node.

The use of the Host Check Command rule set will in this case only be required if the container is not pingable – but it will otherwise function exactly as described above.

3. Diagnostic options

3.1. Diagnosis of a docker node

Should the setup not be successful, there are a number of options for analysing the problem. The Check_MK-Agent supports docker monitoring from Version 1.5.0. Verify therefore that an agent with at least this or a later version is installed on the host.

If the version of the agent on the host is suitable, next check if the data is present in the output. The output can be downloaded as text data using the Download agent output option of the Host Dropdown menu in the GUI:

Alternatively, you could search the Agent-Cache directly. For clarity the output in the following example is abreviated to the output for the node:

OMD[mysite]:~$ grep "<<<docker" tmp/check_mk/cache/mydockerhost
<<<docker_node_info>>>
<<<docker_node_disk_usage:sep(44)>>>
<<<docker_node_images>>>
<<<docker_node_network:sep(0)>>>

If the sections are not managed here, the docker installation will not be recognised. The following command is used for the Docker node info service. This command must be executable in exactly this form on the host system. If necessary, check your docker installation:

root@linux# docker info --format "{{json .}}" 2>&1

3.2. Diagnosis for a container host

If the container host receives no data, or respectively, no services are detected, first check if piggyback data is available for this host. The host's name must be identical to the ID of the container. Alternatively, using the hostname translation for piggybacked hosts rule set you can also perform a mapping manually. Here, however, only the Explicit hostname mapping option is available:

To verify whether piggyback data will be created for an ID, you can check the following directory:

OMD[mysite]:~$ ls -l tmp/check_mk/piggyback/
76adfc5a7794  f0bced2c8c96  bf9b3b853834

4. Files and directories

Path Function
tmp/check_mk/piggyback/ WATO stores the piggyback data here. For each host a subfolder with the host's name will be generated. This contains a text file with the host's data. The filename is the host that supplied the data.
tmp/check_mk/cache/ Here the most recent agent output from all hosts is saved temporarily. The contents of a host's file is identical to that from the cmk -d myserver123 command.