Monitoring Linux

Last updated: April 23. 2018


1. Introduction

Linux systems can be especially well monitored by Check_MK. This has less to do with the Check_MK developer team ‘feeling at home’ with Linux – rather it has more to do with Linux being a very open system, and it providing numerous very well documented and easily queried interfaces which enable a comprehensive monitoring operation.

The installation of a monitoring agent is essential, since most of the interfaces are not in themselves accessible over a network. For this reason Check_MK has its own agent for the monitoring of Linux. This is easy to install and is economical in its use of system resources.

The Check_MK agent for Linux is...

  • ... minimalistic, since it requires little CPU, RAM and hard drive space.
  • ... transparent, because is is a shell script, in which you can see which command it is executing.
  • ... secure, since it permits no access from the network whatsoever.

The agent consists of a simple shell script that is installed in /usr/bin/check_mk_agent, and which invokes available system commands sequentially in order to seek data for the monitoring. It routes its output using either xinetd or systemd to TCP-Port 6556, or it is alternatively called per SSH. Should you like neither of these procedures you can also implement your own methods for Check_MK to receive data from agents.

Monitoring of information that the standard agent doesn't provide can be added in the form of agent plug-ins. In the  Check_MK Enterprise Edition you can bundle all settings and plug-ins together with the agent into an RPM or DEB packet using the Agent bakery, this packet can then be installed with a single command and/or even be updated fully automatically.

2. Installation

2.1. Various options

Check_MK provides a number of ways to install a Linux agent – from the manual installation of its separate components, to a fully automated deployment. Some of these are only available in the  Check_MK Enterprise Edition:

Method Description CRE CEE
Included as standard RPM/DEB-Packet Simple installation of a standard agent with a manual configuration over configuration files X X
RPM/DEB-Packet from the agent bakery Configuration via the GUI – individual configuration of each host is possible X
automated updates The packet from the agent bakery is initially installed manually or by using a script, and subsequently from there updated automatically. X
Manual installation The individual files, without a packet, are copied to the target system and then xinetd, systemd, SSH or a user's own access method are manually installed. X X

2.2. Installation using a RPM/DEB-Packet

The agent files are located in the Monitoring agents WATO module. In the  Check_MK Enterprise Edition this will lead to the Agent bakery. From here, with the Unpackaged agents or Agent files (Version 1.4.0) buttons the list of the agent files can be opened:

Everything required is in the first section called Packaged Agents. Finished RPM and DEB packets for the installation of a Linux agent with standard settings can be found here (the packet for Windows with the .msi extension is also here):

Whether you require RPM or DEB is dependent on the Linux distribution on which the packet is to be installed:

Packet File extension For installation on
RPM .rpm Red Hat Enterprise Linux, Fedora, CentOS, openSUSE, SLES, and derivatives of these
DEB .deb Debian, Ubuntu, and all other DEB-based distributions

Attention: in the Linux/Unix-Agents section can be found the straight agent scripts for the manual installation to /usr/bin/. Only use these if the packeted agents do not function on your system, or if you wish to monitor other Unix derivatives as Linux.

Exotic Linux distributions

Also, if the Check_MK-server only supports Enterprise-Linux distributions that are still supported by their producer, the Check_MK-agent is in such a case much less demanding. It supports every Linux distribution – even ancient ‘dinosaurs’ on which a Version 2.4 core is running! It is however possible that not all plug-ins will run correctly in the agent, but the basic monitoring itself will function.

Installation with RPM

The installation of an agent with the RPM packet is very simple:

  1. Download the agent packet with the *.rpm file extension.
  2. With scp, WinSCP or similar, copy it to the target system to be monitored.
  3. Install it there as root using the rpm -U command.
  4. Next install the xinetd packet from your distribution, if it is not already installed.

Here is an example of a packet installation:

root@linux# rpm -U check-mk-agent-1.4.0p4-1.noarch.rpm

By the way, the -U stands for ‘Update’, but it can also perform an initial installation correctly. This means that the same command can also be used later for an update to a newer version.

The later installation of xinetd on Red Hat / CentOS is performed with:

root@linux# yum install xinetd

For SUSE Linux use zypper:

root@linux# zypper install xinetd

Following the installation an agent is immediately active and can be queried through TCP Port 6556.

Installation with DEB

This procedure is the same as for RPM, only the commands are different. An installation and a later update of the agent are both performed with:

root@linux# dpkg -i check-mk-agent_1.4.0p4-1_all.deb

A later installation of xinetd:

root@linux# apt-get install xinetd

Packet load over HTTP

Sometimes scp or WinSCP can be very laborious. The packet can also be loaded directly from the Check_MK-server to the target system over HTTP. For this purpose the agent file downloads have been deliberately made possible without needing a login. The files after all contain no secrets. Anyone can download and install Check_MK themselves and thus access the files.

The easiest method is to use wget. The URLs can be identified via the browser. If you know the packet's name you can simply assemble the URL name yourself. Then simply set /mysite/check_mk/agents/ before the file name:

root@linux# wget http://mycmkserver/mysite/check_mk/agents/check-mk-agent-1.4.0p4-1.noarch.rpm

RPM actually has an inbuilt wget. Here you can perform the download and the installation in a single action:

root@linux# rpm -U http://mycmkserver/mysite/check_mk/agents/check-mk-agent-1.4.0p4-1.noarch.rpm

The  Check_MK Enterprise Edition utilises the Agent-Bakery via a WATO-Modul to packetise individally customised agents. This process will be described in the general chapter covering the Agents.

Installation of the baked packets is performed exactly as described above.

2.3. Automatic updates

If you use the agent bakery, the agent's automatic updates can set this up. This process is described in its own article.

2.4. Manual installation

The manual installation of an agent is rarely necessary, but is not very difficult. As well as the agent files from the site, you also require the Linux/Unix agents box – here you will find the Check_MK Agent for Linux file:

Load this file to the target system, and copy it to a directory that is executable for root. /usr/local/bin/ is very well suited for this, since it is found in the search path and has been conceived for the user's own extensions. Here you can also work directly with wget:

root@linux# cd /usr/local/bin
root@linux# wget http://mycmkserver/mysite/check_mk/agents/check_mk_agent.linux
root@linux# mv check_mk_agent.linux check_mk_agent
root@linux# chmod 755 check_mk_agent

Please don't forget the last two commands – these remove the .linux file extension and make the file executable. If everything has been done correctly the agent should now simply be a command that can be executed, and which should generate its expected output. This also works if you are not in /usr/local/bin. The | head truncates everything after the 11th line:

root@linux# check_mk_agent | head
<<<check_mk>>>
Version: 1.2.8p16
AgentOS: linux
Hostname: mycmkserver
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>

In the case of a very old distribution which does not recognise the timeout command, load the small program waitmax from the agent page and install it likewise in /usr/local/bin. Both timeout and waitmax perform the same function – they force a timeout when executing a program:

root@linux# timeout --help
Usage: timeout [OPTION] DURATION COMMAND [ARG]...
  or:  timeout [OPTION]
  Start COMMAND, and kill it if still running after DURATION.

Waitmax was developed as a Check_MK component at a time when timeout was not widely used. It has almost the same call syntax:

root@linux# waitmax --help
age: waitmax [-s SIGNUM] MAXTIME PROGRAM [ARGS...]

Execute PROGRAM as a subprocess. If PROGRAM does not exit before MAXTIME
seconds, it will be killed with SIGTERM or an alternative signal.

   -s, --signal SIGNUM   kill with SIGNUM on timeout
   -h, --help            this help
   -V, --version         show version an exit

Should you wish to configure or extend the agent, you will need to create the required directories yourself. The location for the three required directories is hard coded in variables that begin with MK_, and which will also be provided to the plug-ins over the environment:

root@linux# grep 'export MK_' check_mk_agent
export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_VARDIR="/var/lib/check_mk_agent"

These three directories should be created:

root@linux# mkdir /usr/lib/check_mk_agent /etc/check_mk /var/lib/check_mk_agent

Should you want to change the file path, simply edit it in /usr/local/bin/check_mk_agent.

If you fundamentally want to call up the agents over SSH, a configuration for the xinetd is required, plus you only need the SSH-configuration. How that is achieved will be described later below.

The configuration per xinetd enables an access of the agent data via TCP Port 6556 and is the standard method in local networks. Install the packet xinetd for this, and create the following file:

/etc/xinetd.d/check_mk_agent
service check_mk
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/local/bin/check_mk_agent
        only_from      = 10.118.14.5 10.118.14.37
        disable        = no
}

Here enter the IP-Address of your Check_MK-server that will be permitted to access the agent under only_from. Then only an activation is required and the agent will be ready:

root@linux# /etc/init.d/xinetd restart

3. Testing, and error diagnosis

Once the agent has been installed, the obvious question will surely be how to test whether everything has been done correctly. All of the possibilities that are available from the Check_MK-server are described in the general chapter on the agents. There are of course further diagnostic possibilities when one is directly logged into the target system itself.

Since the ‘agent’ is basically nothing more than a simple program that obtains data from your system and outputs it as loosely-formated text, you can also invoke it as a program, and in fact do so quite easily:

root@linux# check_mk_agent
<<<check_mk>>>
Version: 1.2.8p16
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>
udev              devtmpfs     8155492         4   8155488       1% /dev
tmpfs             tmpfs        1634036      1204   1632832       1% /run
/dev/sda5         ext4       226298268 176973752  37806104      83% /
none              tmpfs              4         0         4       0% /sys/fs/cgroup

Because the output can be rather long, less can also be very practical here (you can quit it with the 'Q'-key):

root@linux# check_mk_agent | less

This output does not of course prove whether the agent is also accessible over the network. But in this way it can be tested whether all of the desired data is present in the output.

Incidentally – it is not essential to be root to be able to invoke the agent, however it is possible that the output could be missing some information which requires root-permissions to obtain (e.g., multipath information and the output from ethtool).

Debug mode

So that possible erroneus outputs from defective plug-ins or commands do not 'corrupt' the real production data, as a matter of principle the agent suppresses the standard error channel. If one is looking for a specific problem this can be reactivated by invoking the agent with a special debug mode. This is done with the -d option. In so doing all of the shell commands which the agent executes will be output.

So that you can work with less here, the standard output and error channels must be combined with 2>&1:

root@linux# check_mk_agent -d 2>&1 | less

4. Inclusion of standard check plug-ins

4.1. Executing plug-ins over MRPE

If you have migrated your monitoring from a Nagios-based solution to Check_MK, it cannot be ruled out that you have existing check plug-ins of a standard form for which no counterpart in Check_MK (yet) exists. In most cases these will be self-written plug-ins in Perl or Shell.

The Check_MK-agent offers a simple mechanism that enables such plug-ins still to be used: MK's Remote Plugin Executor or MRPE for short. The name is intentionally an analogy to NRPE in Nagios, for which it performs the same function.

The MRPE is integral in agents and is configured with a simple text file which you yourself can save in /etc/check_mk/mrpe.cfg. In this file enter one plug-in call per line – together with the name that Check_MK should use for the service that is to be automatically created. Here is an example:

/etc/check_mk/mrpe.cfg
Foo_Application /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5

If the agent is allowed to run locally, for each plug-in a new section will be found with the title <<<mrpe>>>, containing the name, exit code and output from the the plug-in. This can be verified with the following practical grep-command:

root@linux# check_mk_agent | grep -A1 '^...mrpe'
<<<mrpe>>>
(check_foo) Foo_Application 0 OK - Foo server up and running
<<<mrpe>>>
(check_bar) Bar_Extender 1 WARN - Bar extender overload 6.012|bar_load=6.012

The 0 or 2 in the output stand for the plug-in's exit codes and conform to the standard pattern: 0 = OK, 1 = WARN, 2 = CRIT and 3 = UNKNOWN.

The rest will be taken care of by Check_MK automatically. Once a service discovery has been performed for the host, the two new services will be shown as available. It will look like this (here in the new display format in version 1.4.0):

Incidentally: due to the file's syntax the name may not contain blank characters. With the help of the same syntax as used in URLs, a space can be replaced by %20 – (the ASCII-Code 32 for ‘space’ is Hexadecimal 20):

/etc/check_mk/mrpe.cfg
Foo%20Application /usr/local/bin/check_foo -w 60 -c 80
Bar%20Extender /usr/local/bin/check_bar -s -X -w 4:5

4.2. Asynchronous execution

Please note that all plug-ins running in mrpe.cfg will be executed synchronously and sequentially. The plug-ins should thus not have overly long runtimes. If a plug-in hangs, all following plug-ins will be delayed. This could lead to the agent's complete retrievals under Check_MK entering a timeout, meaning that the host can no longer be reliably monitored.

If you really require longer running plug-ins, these should be converted to asynchronous processing, thus avoiding the problem described above. For this, define a time frame in seconds for which a calculated result will be valid – for example, 300 for five minutes. To achieve this, set the expression (interval=300) following the service name in mrpe.cfg:

/etc/check_mk/mrpe.cfg
Foo_Application (interval=300) /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5

This will have a number of effects:

  • The plug-in will be executed in a background process and will no longer hinder the agent's processing.
  • Because the agent no longer waits for the execution, the result will be delivered until the next call of the agent.
  • The plug-in will not be run before the defined 300 seconds have elapsed. Until that time the old result will be recycled.

With this method tests that require very long processing times, as well as longer intervals, can also be run without needing to make changes to the configuration on the Check_MK-server.

4.3. MRPE with the agent bakery

Proud owners of the  Check_MK Enterprise Edition can also configure MRPE with the Agent bakery. The rule set Monitoring Agents ➳ Generic Options ➳ Execute MRPE Checks is responsible for this. There you can configure the same things as described above. The appropriate mrpe.cfg file will then be automatically generated by the bakery.

Baking the plug-ins

Likewise the Check plug-ins themselves can be delivered with the packet. With these the agent will be complete and will require no manual installation of additional files. This is all performed as follows:

  1. Create the directory local/share/check_mk/agents/custom on the Check_MK-server
  2. Create a subdirectory, e.g. my_mrpe_plugins in the above directory
  3. In turn create in this the subdirectory bin
  4. Copy your plug-ins to this directory
  5. Create a rule in Monitoring Agents ➳ Generic Options ➳ Deploy custom files with agent.
  6. Select my_mrpe_plugins, save and bake it!

The Check plug-ins will be installed in your agent's standard-bin-directory. By default this is /usr/bin. Then when configuring the MRPE-checks, /usr/bin/check_foo will be required instead of /usr/local/bin/check_foo.

5. Agent for extending plug-ins

5.1. What are plug-ins?

The standard agent /usr/bin/check_mk_agent contains a whole series of sections which provide monitoring data for various checks which will then be found automatically by a service discovery. These include all of an operating system's important monitorings.

Additionally, there is also the possibility of augmenting an agent with plug-ins. These are small scripts or programs which are invoked by an agent to include further sections with additional monitoring data. The Check_MK-Project delivers a whole series of such plug-ins, which – when correctly installed and configured – via a service discovery can provide new checks automatically.

Why are these plug-ins not simply built into the standard agent? For each plug-in one of the following reasons prevents such an integration:

  • The plug-in is coded in a programming language other than Shell and thus cannot implemented inline (example: mk_logwatch).
  • The plug-in in any case requires a configuration without which it will not function (example: mk_oracle).
  • The plug-in is so specialised that most users will not require it (example: plesk_domains).

5.2. Manual installation of plug-ins

The plug-ins for Linux and UNIX included with the project are all located on the Check_MK-server in local/share/check_mk/agents/plugins. Additionally, these are available via the download page for the agents in WATO (as described at the beginning of this article) in the Linux/Unix Agents - Plugins menu box:

For all of our standard agent plug-ins there are also the matching check plug-ins which can evaluate the data and create services from these. They are ready to use and do not require an extra installation.

Before installing a plug-in in an agent, please have a look at its associated file. Important tips for the correct use of the plug-in can often be found there.

The actual installation is then simple: just save the file to /usr/lib/check_mk_agent/plugins. When doing this ensure that the file is executable. If not, use a chmod 755, otherwise the agent will not run the plug-in. Especially if the files are not transferred over scp, rather via HTTP from the download page, the execution permissions will be lost!

Once a plug-in is executable and in the correct directory, it will be invoked by the agent and a new section will be generated in the agents output. This usually has the same name as the plug-in. Complex plug-ins (e.g., mk_oracle) in fact create a whole series of sections.

5.3. Configuration of plug-ins

Some plug-ins require a configuration file in /etc/check_mk/ to be able to function. With others a configuration is optional and allows special features or customisation. Others simply work as is. There are various sources of relevant information:

  • The documentation for the matching check plug-ins in the Check plugins WATO-module
  • Comments in the plug-in itself (often very helpful!)
  • A relevant article in this handbuch (e.g., on the monitoring of Oracle)

5.4. Asynchronous execution

In the same way as with MRPE plug-ins can process asynchronously. This is very useful if the plug-ins have a very long runtime, and the acquired status data in any case does not need to be refreshed every minute.

An asynchronous execution is not configured with a file. Instead, create a subdirectory in plugins whose name is a numeric: representing a count of seconds. Plug-ins in this directory will not only be executed asynchronously, at the same time with the second count a minimum waiting time is also specified before the plug-in can be run again. If the agent is again called before the specified time has elapsed, it will use cached data from the last run of the plug-in. In this way an interval longer than the typical one minute can in effect be configured for the plug-in.

The following example shows how the >my_foo_plugin plug-in can be changed from synchronous to asynchronous execution, with a 5 minute interval.:

root@linux# cd /usr/lib/check_mk_agent/plugins
root@linux# mkdir 300
root@linux# mv my_foo_plugin 300

Please note that a few plug-ins are set up internally to inherently execute asynchronously. Among these is mk_oracle. Always install such plug-ins directly in /usr/lib/check_mk_agent/plugins!

5.5. Installing plug-ins using the bakery

Plug-ins included with Check_MK can be configured using the Agent Bakery. This not only takes care of the installation of the plug-ins themselves, but also for the correct generation of the configuration file should one be required.

Each plug-in is configured with an agent rule. The appropriate rule set can be found in Monitoring agentes ➳ Agent plugins:

5.6. Manual execution of plug-ins

Since agent plug-ins are executable programs, for testing and diagnosis they can be manually started. There are however plug-ins which require specific environment variables to be set by agents – so that they can find their configuration file, for example. Set these variables manually before execution:

root@linux# export MK_LIBDIR=/usr/lib/check_mk_agent
root@linux# export MK_CONFDIR=/etc/check_mk
root@linux# export MK_VARDIR=/var/lib/check_mk_agent
root@linux# /usr/lib/check_mk_agent/plugins/mk_foobar
<<<foobar>>>
FOO BAR BLA BLUBB 17 47 11

Some plug-ins have special invocation options for debugging. Just have a look in the plug-in!

6. Security

6.1. Considerations

Nowadays everything must be safe – and of course monitoring is no exception. Because the monitoring agent is installed on every server being monitored, a security problem here can have serious consequences.

For this reason emphasis is placed on security in the design of Check_MK, and from the earliest days of Check_MK as been an unshakeable principle: An agent reads no data from the network – full stop. Consequently it is quite impossible that an attacker could sneak any type of command or script element over the monitoring port 6556.

This alone provides such a high level of security that most users do without additional measures in the LAN. If the system being monitored is only accessible over an insecure internet connection, then of course quite different precedures are necessary, and an encryption with SSH would certainly be the ideal first choice.

From Version 1.4.0 the Check_MK-agent additionally includes an inbuilt encryption, which represents a good comprise between security and complexity. In the following section we will show all of the options for protection in detail.

6.2. Restricting of access over IP-addresses

Even if an attacker is unable to execute commands, the monitoring data could be useful to it as among other info, the data includes a list of all processes running on the system. Is is therefore best that the data is accessible to nobody.

Xinetd

If the Check_MK-agents are as usual enabled over the xinetd, it is very simple and effective to restrict the access to specific IP-addresses – to those of the monitoring server of course. This is easily done, and was in fact to be seen in the example above:

/etc/xinetd.d/check_mk_agent
service check_mk
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        only_from      = 10.118.14.5 10.118.14.37
        disable        = no
}

Agent Bakery users can configure the permitted IP-addresses using the WATO rule set Monitoring agents ➳ Rules ➳ Generic options ➳ Restrict agent access via IP address.

An attacker can of course easily falsify their IP-address and thus make a connection to an agent – but it is very unlikely that it will receive an answer since this goes to the genuine monitoring server. Or it really does receive an answer, but the Check_MK-server looks down the pipe and will quickly record an error.

Systemd

Since Systemd is now the new toy that everyone uses, Linux distributors are working hard to retire the good old Xinetd. The prepackaged Linux-Agent (not one from the Bakery!) already installs itself with Systemd instruments, if the target system is based on Systemd and no Xinetd is available.

Systemd however cannot perform such simple tasks as only_from (Pöttering!!). One will be bluntly referred to iptables. Should your agent thus, without Xinetd, be invoked purely with Systemd, there is unfortunately no simpler possibility for restricting the IP-addresses, other than using a firewall configuration.

If you are not yet one of the Systemd evangelists, there is however a simple alternative: Even in Systemd-based systems it is (still) possible to use Xinetd. This can be run as a service under Systemd – and again the method with only_from as described above also works. To this end install the xinetd packet, and then reinstall the Check_MK-agent – this should then find and preferentially install Xinetd.

6.3. Invoking over SSH

The ultimate security for invoking a Check_MK-agent is offered by invoking it via Secure Shell – in Linux in the form of an implementation of >OpenSSH. This method is advisable for:

  • The monitoring of Linux servers that are only accessible over the internet.
  • The monitoring of computers in a DMZ.
  • Similar situations in which a TCP-connection from the Check_MK-Server to the agent is at all possible.

The setting-up is performed in the following steps:

  1. Create an SSH-key pair especially for this purpose
  2. Enable an access to the agent on the target system using these keys.
  3. Disconnect the access over Xinetd.
  4. Configure the Check_MK-Server so that SSH Port is used instead of the TCP-Connection.

And now the above procedure, step-by-step with all necessary details:

Creating SSH-key pairs

SSH works with a “Public-Key-Authorization”. For this purpose first create a pair of complementary keys – one of which is public, and the other private. This is performed by an instance user with ssh-keygen -t ed25519:

OMD[mysite]:~$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/omd/sites/mysite/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /omd/sites/mysite/.ssh/id_ed25519.
Your public key has been saved in /omd/sites/mysite/.ssh/id_ed25519.pub.
The key fingerprint is:
cc:87:34:d2:ed:87:ed:f7:1b:ec:58:1f:7c:23:00:e2 mysite@mycmkserver
The key's randomart image is:
+--[ED25519  256--+
|                 |
|       . .       |
|      ..+..      |
|      .=.+.o     |
|       ES +.o    |
|         . o. o  |
|            ...B.|
|             .=.*|
|             . o+|
+-----------------+

Important: do not enter a passphrase here! Encrypting the file with the secret key will not achieve anything, because you won't want to have to enter the passphrase every time you start the Check_MK-server...

This will result in two files in the .ssh directory:

OMD[mysite]:~$ ll .ssh
total 8
-rw------- 1 mysite mysite 1679 Feb 16 14:18 id_ed25519
-rw-r--r-- 1 mysite mysite  398 Feb 16 14:18 id_ed25519.pub

The private key is called id_ed25519, and is only readable for the instance user (-rw-------) and that is what we want! The public key id_ed25519.pub will look something like this:

OMD[mysite]:~$ cat .ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver

Enabling access over SSH

The next step must be carried out on every Linux server that is to be monitored over SSH. Log in to the server as root, and – if it does not already exist – create the subdirectory .ssh in the (/root) home directory:

root@linux# mkdir /root/.ssh

The permissions for this directory must be 700 so that it will be recognised by SSH. If you have created the directory yourself, you will also require:

root@linux# chmod 700 /root/.ssh

Now open the authorized_keys file with the (console-based) text editor of your choice. If the file does not already exist, it will be generated automatically by the editor:

root@linux# vim /root/.ssh/authorized_keys

Now copy the content of the public keys into this file. This can be done, for example, with the mouse and copy & paste. Be precise! Every blank character is valid. Be careful to never have two blank characters together, – AND, all of this is a single line! If the file already exists, simply add this as a new line at the end of the file.

Restricting access to agent execution

Now for something very important! The SSH-key should be used exclusively for executing the agent. SSH offers such a function under the name Command restriction. For this, set the text command="/usr/bin/check_mk_agent" at the beginning of the line which has just been created – separated from the rest with one blank character. The result will look like this:

/root/.ssh/authorized_keys
command="/usr/bin/check_mk_agent" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver

Save the changed file, and check the permissions – they must be set to 600:

root@linux# chmod 600 /root/.ssh/authorized_keys
root@linux# ll /root/.ssh/authorized_keys
-rw------- 1 root root 1304 Feb 16 14:36 authorized_keys

Now an access via SSH to the agent from the monitoring server should be possible. This can be checked with:

OMD[mysite]:~$ ssh root@myhost123
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 55:34:f9:dd:2b:db:a7:fc:5d:4c:9d:37:28:f7:69:62.
Are you sure you want to continue connecting (yes/no)? yes
<<<check_mk>>>
Version: 1.4.0p3
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>

By the way, the request for the key fingerprint is received only with the initial use. Should this not function, check the following:

  • Is the SSH-server actually installed on the target system?
  • Do the specified files and directories have the correct permissions?
  • Has the syntax in authorized_keys been coded correctly?
  • Has the correct public key been entered?
  • Have you logged in as the correct user (root@...)?
  • Have you thought of the command="...."?

Disconnecting access over Xinetd

The implementation of SSH is not much use if the access over Port 6556 remains possible as before. To close the port, set Check_MK's Xinetd-service to disabled. Do not delete the complete configuration file, as this will then reappear with the next agent update anyway!

Disabling is performed in /etc/xinetd.d/check_mk_agent:

/etc/xinetd.d/check_mk_agent
service check_mk
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        disable        = yes
}

Following the restart of Xinetd, do not forget:

root@linux# /etc/init.d/xinetd restart

A deinstallation of Xinetd is of course also possible – but with the next update of the Check_MK-agent it will reactivate itself in any case over Systemd!

Under no circumstances forget to test the setup at the end of this precedure. A connection to Port 6556 should no longer be possible:

OMD[mysite]:~$ telnet myhost123 6556
Trying 10.118.15.23...
telnet: Unable to connect to remote host: Connection refused

Migrating Check_MK's access to SSH

The target system is now ready. Now only the configuration of Check_MK itelf needs to be done. This is performed via the rule set with {{Datasource programs|Individual program call instead of agent access}}. Create a rule here for the affected hosts, and enter the command ssh -T -oStrictHostKeyChecking=no root@<IP>:

After saving and an Activate changes everything should function as intended! For diagnostic purposes the commands cmk -D and cmk -d are well suited, as explained in the Article about the command line.

Details concerning the ‘Datasource programs’ can be viewed in its own article.

Multiple SSH-keys

It is also possible to work with more than one SSH-key. Save the keys to any chosen directory. With ‘Datasource program’ the data path to the appropriate private key must include the -i option. Ideally, use $OMD_ROOT as the substitute for the path to to the instance directory (/omd/sites/mysite). The configuration will then also be runnable in an instance with a different name:

Different host groups can use different SSH-keys, if multiple different rules are used in Datasource programs.

6.4. Inbuilt encryption

From Check_MK Version 1.4.0, Linux – and also the Windows Agent – can encrypt its own data without additional tools. Strictly speaking, this is no substitute for an access control. But since an attacker cannot send commands, and can't work with encrypted output data, as a solution this is nearly as effective.

The complexity with the use of encryption and the additional CPU-load incurred are both less than with the SSH method as described above, the method which we nevertheless still recommend when transferring data over the internet.

The encryption of course requires a suitable configuration, both on the server and in the agents. This can be created manually ( Check_MK Raw Edition) or with the Agent bakery ( Check_MK Enterprise Edition).

Installation without the bakery

Even without the Agent bakery the first step is still performed in WATO: by creating a Host & Service Parameters ➳ Access to agents ➳ Encryption rule in the rule set. The rule should apply to all hosts for which an encryption will be implemented. SNMP-hosts ignore this setting, so it is not necessary to explicitly exclude them.

Important is the Encryption for agent setting. As long as the rule retains the default Disable, evrything naturally remains as in the old setup. The choice is thus between:

  • Enable: Encryption is active, but data from unencrypted agents will continue to be accepted.
  • Enforce: Encryption is active, and only encrypted data will be accepted.

It makes sense to begin with Enable initially. Once you are satisfied that all agents have been converted to encryption, switch over to Enforce, in order to identify hosts which still send data as clear text.

The encryption makes use of a shared password which must be stored on both the Check_MK-Server and on the agent as simple text (‘Shared secret’), and which will be entered here. Choose a random password and have it ready for the second step – the configuration of the agent.

Create the file /etc/check_mk/encryption.cfg with the following contents on the agent:

/etc/check_mk/encryption.cfg
ENCRYPTED=yes
PASSPHRASE='XEwks9fm'

Of course use your own password in PASSPHRASE. This file should definitely be secured against reading by other users:

root@linux# chmod 600 /etc/check_mk/encryption.cfg

Now the following tests can be made (for information on these, see the article on Check_MK's command line):

  • An invocation of check_mk_agent on the target system should produce a nonsense jumble of characters.
  • A telnet myhost123 6556 from the Check_MK-server should produce the same jumble.
  • A cmk -d myshost123 on the Check_MK-server should show the normal clear text.

Implementing with the bakery

The implementation of the encryption with the agent bakery is very simple. By creating the rules as just described, one is basically finished. Now only new agents need to be baked and distributed. The file /etc/check_mk/encryption.cfg will be automatically generated and added to the agent packet.

7. Monitoring Linux over SNMP

Since an easily installed SNMP-Agenten is available for Linux, the question arises whether it is possible or even desirable to monitor Linux over SNMP? The answer is quite simple: possible yes, sensible no. Why is this so?

  • The monitoring data from the SNMP-agents is very limited, so that in any case Check_MK-agents are required for a halfway adequate monitoring.
  • The SNMP-agent delivers no useful data that isn't already provided by the Check_MK-Agent.
  • The SNMP-agent is time-consuming to install.
  • Last but not least, the SNMP protocol consumes far more CPU and network resources than a normal Check_MK monitoring.

There are however a few situations in which monitoring per SNMP in addition to the normal agents can be sensible. Such a case would be when either a user's software, or a hardware monitoring tool supplied by the server's manufacturer only delivers data via SNMP.

For such a situation, at the host's characteristics in the Host tags submenu in WATO, set the Agent type to Dual: Check_MK Agent + SNMP. Services that are accessible over SNMP as well as per Check_MK-agent (e.g. CPU-load, file systems, network cards) will be automatically retrieved by Check_MK-agent and not over SNMP. In this way a duplicated data transfer will be automatically prevented.

8. Hardware monitoring

8.1. Fundamentals

To be as comprehensive as possible the monitoring of a Linux-server must of course include the hardware. This is achieved in part directly with the Check_MK-agents, and partly also using special plug-ins. Additionally, there are also cases in which monitoring can be implemented per SNMP, or even over a separate management board.

8.2. Monitoring the SMART-values

Modern hard drives almost always use S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology). This system continuously logs data on the condition of the HDD or SSD, and with the smart plug-in Check_MK can retrieve these values and evaluate the most important of them. In order that the plug-in functions after an installation, the following preconditions must be met:

  • The smartmontools packet must be installed. This can be installed on all modern distributions with the appropriate packet manager.
  • If the drives are connected via a RAID-controller and this allows access to the SMART-values, the appropriate tool must also be installed.

tw_cli (3ware) and megacli (LSI) are supported.

If these prerequisites have been satisfied, and the plug-in has been installed, the data will be automatically read and appended to the agent's output. In Check_MK the new services can then also be activated directly:

8.3. Monitoring with the help of IPMI

IPMI (Intelligent Platform Management Interface) is an interface for hardware management which, among other functions, enables the monitoring of hardware. Check_MK uses freeipmi for this, in order to access the hardware directly and without a network. It is installed from the packet source and is ready for immediate use, so that the data will be transmitted at the next polling by Check_MK.

If freeipmi is not available, or there are other grounds preventing an installation, ipmitool can also be used. This is often already present on a system and must only served by an IPMI hardware driver – which can be provided by the openipmi packet, for example. Likewise, nothing more needs to be done here subsequently. The data will be recorded by Check_MK automatically.

For error diagnosis the tools can also be executed manually in a host shell. If the freeipmi packet has been installed it can be used to control the function:

root@linux# ipmi-sensors Temperature
32 Temperature_Ambient 20.00_C_(1.00/42.00) [OK]
96 Temperature_Systemboard 23.00_C_(1.00/65.00) [OK]
160 Temperature_CPU_1 31.00_C_(1.00/90.00) [OK]
224 Temperature_CPU_2 NA(1.00/78.00) [Unknown]
288 Temperature_DIMM-1A 54.00_C_(NA/115.00) [OK]
352 Temperature_DIMM-1B 56.00_C_(NA/115.00) [OK]
416 Temperature_DIMM-2A NA(NA/115.00) [Unknown]
480 Temperature_DIMM-2B NA(NA/115.00) [Unknown]

If ipmitool has been installed the data output can be checked with the following command:

root@linux# ipmitool sensor list
UID_Light 0.000 unspecified ok na na 0.000 na na na
Int._Health_LED 0.000 unspecified ok na na 0.000 na na na
Ext._Health_LED 0.000 unspecified ok na na 0.000 na na na
Power_Supply_1 0.000 unspecified nc na na 0.000 na na na
Fan_Block_1 34.888 unspecified nc na na 75.264 na na na
Fan_Block_2 29.792 unspecified nc na na 75.264 na na na
Temp_1 39.000 degrees_C ok na na -64.000 na na na
Temp_2 16.000 degrees_C ok na na -64.000 na na na
Power_Meter 180.000 Watts cr na na 384.00

8.4. Manufacturer-specific Tools

Many server producers offer their own tools for recording hardware information and delivering this via SNMP. The following prerequisites apply to be able to retrieve this data and provide it to Check_MK:

  • An SNMP-server has been installed on the Linux host.
  • The manufacturer's tool has been installed – e.g., Dell's OpenManage or Supermicros SuperDoctor.
  • The host has been configured in Check_MK for an additional monitoring per SNMP, with Agent type set to Dual: Check_MK Agent + SNMP.

The new services for hardware monitoring thereby supported will then be automatically recognised. No further plug-ins will be required.

8.5. Supplementary monitoring over the Management Board

From Version 1.4.0 a management board can be configured for every host and additional data collected per SNMP. The services thereby recognised will likewise be assigned to the host.

Setting up a management board is very easy. In the host's attributes, just enter the protocol, the IP-address and the access data for SNMP, and save the new settings:

With a service discovery the newly-discovered services will then be activated as usual.

9. Files and directories

9.1. Data paths on the monitored system

Pfad Function
/usr/bin/check_mk_agent Installation location of the Check_MK-agents on the target system.
/etc/check_mk Location of configuration files for the agents.
/etc/check_mk/mrpe.cfg Configuration file for MRPE – for the execution of standard, Nagios-compatible check plug-ins
/etc/check_mk/encryption.cfg Configuration for the encryption of the agent data.
/usr/lib/check_mk_agent The home directory for agent extensions
/usr/lib/check_mk_agent/plugins Plug-ins which extend the agent for collecting supplementary monitoring data. Plug-ins can be written in any available programming language.
/usr/lib/check_mk_agent/local The user's own ‘local checks’
/etc/xinetd.d/check_mk_agent Configuration for the xinetd, which connects the agent's output to TCP-Port 6556.

9.2. Data paths on the monitored system

Pfad Function
local/share/check_mk/agents/custom The home directory for own files which are to be delivered with baked agents.