Monitoring agents


1. Introduction

For a monitoring system to receive more information from an endpoint, help is required from the target system. For example - how else can Check_MK know how full a server's storage volume is without that system somehow providing the information? The component that provides this information is always an active piece of software - namely an agent.

There are situations where one does not need to install an addtional agent - since one that can be used is already present. The best example here is SNMP. All manageable network devices and appliances have a built-in SNMP agent.

As always, there is an exception: the monitoring of network services. For example - by its nature a web application can be accessed via HTTP and be monitored over the same connection. But even in this case there is usually an additional agent in use which provides further server data to the monitoring.

The following table shows the four different ways that Check_MK can access services to be monitored:

Method Description Prerequisite
SNMP Check_MK accesses the target device's existing SNMP-agent. With active queries (GET) it collects details on the system's condition. SNMP-Agent enabled
Check_MK-Agent Check_MK has its own agents for servers and workstations. Eleven different operating systems are supported - from the commonplace like Windows and Linux to exotics such as OpenVMS. The agents are passive and connect to TCP Port 6556. Only on receiving a Check_MK server query will they be activated and respond with the required data. Check_MK-Agent installed
Special Agent Some systems allow neither an agent installation nor do they support SNMP in a usable form. Instead of these they offer Management-APIs based on TELNET, SSH or HTTP/XML. Check_MK queries these interfaces via special agents that run on the Check_MK-Server Create an API account for Check_MK
Active Checks Network-based services such as HTTP, SMTP or IMAP can by their very nature be queried via the network. For this Check_MK sometimes uses its own, sometimes existing plug-ins originally developed for Nagios. For example, check_http is very popular for querying websites. These are described as Aktive Checks. More details may be found in its own article None

1.1. Monitoring by events

Until now we have only discussed active monitoring - THE best example for Check_MK. There is also the reverse method: namely that by which the target system itself sends messages to the monitoring, e.g., via syslog or SNMP traps. For this entire subject Check_MK has the Event console which is described in its own article.

2. The Check_MK agent

2.1. Download and installation

As described, you require a Check_MK agent in order to monitor servers and workstations. Agents for eleven different operating system families are currently maintained in the Check_MK-Project. All of these agents are components in Check_MK and are available for downloading via the Check_MK-Server's Web-GUI. Additionally, in the  Check_MK Enterprise Edition there is the Agent Bakery. This enables you to 'bake' your own agents containing a personalised configuration and collection of plug-ins. This can even be done individually and differently for each host.

2.2. Installation via the download page

The agents are accessed via the WATO-module Monitoring agents. In the Enterprise Editioni, you will be brought to the Agent bakery. From there the Agent files (until version 1.2.8: Unpackaged Agents) button then takes you to the download page for the agents and their plug-ins. The Raw Edition takes you directly to the download page:

The packeting agents for Windows (check_mk_agent.msi) and Linux (.deb or .rpmu files) are found right in the first section. After installing these packages the agent is basically ready to run, and you can begin monitoring. All further configuration is achieved via configurations data, and plug-ins are installed by downloading them from the same page and copying them into the appropriate folder.

2.3. Details of agents for specific operating systems

The agents' structure, configuration and capabilities vary depending on the operating system. For this reason details for specific agents can be found in their own articles:

3. Installation via the agent bakery

3.1. Introduction

While it's true that the Check_MK-agent can function 'naked' immediately, without needing configuration and without plug-ins, nonetheless in some cases the agent does need to be set up. Some examples:

  • Restriction of access to specific IP-Addresses
  • Monitoring of ORACLE data bases (plug-in and configuration are required)
  • Monitoring of text log files (plug-in, data names and text pattern required)
  • Utilization of the Check_MK-inventory system (plug-in required)

If you have a  Check_MK Enterprise Edition you can package personalised agents with the Agent Bakery. In this way, alongside the existing agents, you can also create agent packages that contain configurations and extra plug-ins. These packages are ideally for being automatically distributed, however, they can also be installed manually. You can even create personalized agents for specific groups of hosts. This allows great flexibility through the use of the automatic agent deployment.

The bakery is accessed via WATO ➳ Monitoring agents:

If you have not yet made settings for specific hosts, there is only a single agent configuration. This is the Default configuration. With the Bakery, Check_MK version 1.2.8 supports the Windows, Linux and AIX operating systems. For Linux you have a choice between the packet formats RPM (SUSE, RedHat, CentOS) and DEB (Debian, Ubuntu), as well as a tarball that is simply unpacked as root under /. Likewise, a tarball is available for AIX, however, it does not include automatic inetd configuration. This must be performed as a one-off action manually.

Every agent configuration has an explicit ID: its hash. A hash's first eight characters are displayed in the GUI. This hash will be a part of the package version and included in it's file name. Whenever you change something in a package's configuration or update Check_MK, the package's hash will also be changed. This way, the operating systems package manager recognizes that this is an update. Check_MK's version number would not suffice here.

3.2. Configuration via Rules

The agent's configuration can be altered, as is so often the case in Check_MK, via rules. These offer you the possibility of equipping different hosts with diverse settings or plug-ins. Via the Rules button you can access a page which lists all rule sets that influence the agents:

Let's take the following example: you wish to limit the list of IP Adresses that are permitted to access the agent. For this you select the Generic Options ➳ Restrict agent access via IP address rules set. Enter one or more IP adresses as the rule's value:

After saving with and , return to the Agent bakery. The button ensures that the agent will be freshly baked. The result - you now have two configurations:

In the Hosts column you will find a list of hosts associated with the relevent configuration. For space reasons the list is not complete here. The VANILLA and GENERIC names have a special role. These two pseudo-hosts are always present and have the following functions:

VANILLA A virtual host whose agent contains only the default configuration, to which therefore none of the agent rules apply.
GENERICA virtual host to which ALL rules with no defined additional conditions apply. The GENERIC entry is especially useful for installing agents on hosts that have not yet been incorporated in the monitoring.

The more host-specific rules you deploy, the more different versions of agents will be built. The bakery makes sure that only such combinations of configurations are built that will be used by at least one of the present hosts.

By the way, in WATO you can also easily reach a host's agent packages via the host's Details and the Monitoring Agent button:

Why are packages for all operating systems offered for every host? The answer is very simple: if no agent is installed on a system Check_MK naturally cannot recognise the operating system! In any case, once automatic agent updates are activated you don't need to do anything more.

3.3. Plug-ins

Many rules are concerned with the installation of various plug-ins. These extend the agents for the monitoring of quite specific components. Most of these are special applications such as, e.g. data bases. Alongside the rule that activates a plug-in you will find the settings for configuring the plug-in. Here for example is the rule for monitoring MySQL:

3.4. Customising agents manually

Please note that you do not manually modify the configuration files of an agent that was created by the Bakery. This will work at first but the next update of the agent will cause the the changes to be lost. However, it is possible to install additional plugins without problems.

4. When should an agent be updated?

Regardless of whether you monitor only a handful - or even thousands of hosts - management of the Check_MK agents on all hosts is always a larger operation. The automatic update of the agents in the  Check_MK Enterprise Edition is however a big help. Nonetheless, you should really only update the agents when:

  • the update solves a problem affecting you, or
  • the update includes required new functions.

In order for this to be possible a general rule applies in Check_MK: newer Check_MK-versions can fundamentally handle the output of older agents.

Note: the reverse is not necessarily true. If an agent's Check_MK version is newer than that of the monitoring server it is possible that the output of the target agent's existing check plug-ins cannot be properly interpreted. In such a case the affected services go into an UNKNOWN (please do not send a Crash-report in such a situation):

5. Error diagnosis

5.1. Testing agents via the command line

Although the agents for the various operating systems were independently developed, from Check_MK's point of view they all behave in the same way by opening the TCP port 6556 for queries from the monitoring server. The query protocol is absolutely simple: the server connects to the port and the data flows in a readable text format from the agent. As soon as the data transfer is completed the agent disconnects itself from the port. The agent basically reads no data from the network!

A correctly-installed agent can be very easily queried from the command line. The best way is directly from the Check_MK instance that is also actively monitoring the agent. In this way you can be certain that the server's IP address will be accepted by agents. A suitable command is e.g. telnet:

OMD[mysite]:~$ telnet 10.1.1.2 6556
Trying 10.1.1.2...
Connected to 10.1.1.2.
Escape character is '^]'.
<<<check_mk>>>
Version: 1.2.7i1
AgentOS: linux
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins

With nc or netcat the data is returned 'naked'. This is useful for example, if you wish to use a script to process the data:

OMD[mysite]:~$ nc 10.1.1.2 6556
<<<check_mk>>>
Version: 1.2.7i1
AgentOS: linux
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins

The output always begins with the line <<<check_mk>>>. Lines included in <<< and >>> are called Section headers. These divide the agent output into sections. Each section contains related information and is usually simply the output from a diagnosis command. The check_mk section plays a special role. It contains general information about the agent such as e.g., its version number.

If the host is already being monitored you can also fetch the data with the cmk -d command. This uses the IP address configured via WATO, allows for a possibly reconfigured port number, and also the case of a special agent:

OMD[mysite]:~$ cmk -d myhost123
<<<check_mk>>>
Version: 1.2.7i1

If monitoring is already running regularly for the host in question a current copy of the output can always be found in the tmp/check_mk/cache table:

OMD[mysite]:~$ cat tmp/check_mk/cache/myhost123
<<<check_mk>>>
Version: 1.2.7i1

5.2. Diagnosis via the GUI

You can also conduct a diagnosis of the agents via the GUI. This takes all settings into consideration and also supports SNMP devices and those queried using special agents. In effect, Check_MK simply attempts to always query via TCP-Port 6556 and SNMP simultaneously. You can access the details of a host's diagnosis with the Diagnostic button in WATO:

You can try out quite a few of the settings (e.g., the SNMP community) right away, and save them when successful.