Automatic agent updates

Last updated: July 01. 2016


From version 1.2.8, Check_MK can automatically update its agents on Linux and Windows. Thus, in the case of a new version of Check_MK you can not only make minor updates to the agents, but an altered agent configuration can also be deployed. At the same time you can take advantage of the Agent bakery, which enables hosts to be individually configured.

1. Setting-up automatic updates

To set up automatic updates, take the following steps: First, access the Monitoring Agents WATO module and click on the Automatic updates button.

Under Prerequisites you will find a list of preconditions that must be satisfied in order that automatic updates can take place. These can simply be checked-off in sequence. Don't forget that for every page there is an online help that explains all points in detail. Clicking on takes you directly to the location in WATO where the appropriate configuration must be actioned. In detail, these are:

1.1. Creating signature keys

The system is so-constructed that the agents can download updates from their central monitoring server via HTTP or HTTPs. Because the agents contain executable code it is particularly important to ensure that the agents cannot be falsified by an attacker. To this end signature keys are used. These consist of a pair of public and private keys (Public-key cryptography).

You can create as many signature keys as you wish. These will each be secured with a passphrase which later must be entered at every signing. With this you can prevent, for example, an attacker that has accessed a backup of the monitoring server from signing agents.

Create a signature key here and note the passphrase. This can neither be changed nor recreated at a later time. If the key is lost it can mean that all agents will need to be manually updated!

1.2. Configuring the update plug-ins

The actual update is performed by an agent plug-in named cmk-update-agent. This will be invoked by the agents in a customisable cycle - once per hour for example. It queries the deployment server (your central monitoring system) if a new packet is available for this host, and if so it executes the update.

This plug-in must naturally be available and correctly configured in the agent. To this end, add this plug-in to the agents via the Install agent updater (Linux, Windows) rule set.

Create a rule here, explicitly select all check boxes and enter the values as required (numerous tips for these are found in the online help ).

1.3. Baking agents

Freshly-bake the agents so that the new rule takes effect and the agent packets actually contain the plug-in.

1.4. Signing agents

Next, you need to sign the agents with the key created in step 1. For this you will require your passphrase for the first time. The signed agents will then be labelled with a symbol. If you have created multiple keys the signature is done separately with every key. An agent is satisfied when the new packet is signed with an existing key known to the agent.

This step must be repeated every time when you freshly-bake the agents in the future.

1.5. Registering agents

The next step must now be carried out on the individual monitored hosts themselves. For this you need to install an agent there for the last time by hand. From Baked Agents download the appropriate packet for the target system. Be certain to take the one that also contains the agent updater.

Copy these to the target system and install them as usual with rpm, deb or msiexec (or respectively with a double-click). When that has been successful the cmk-update-agent program will be available. Under Linux this is found via the search path (/usr/bin). Under Windows it will be in the plugins\ subfolder in the Check_MK agents index.

This program can now be activated with the register argument. Under Windows this must be carried out using a prompt with administrator-rights. Enter the required data in sequence (not all settings are necessary if you have installed an ready-baked agent):

root@linux# cmk-update-agent register
+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater - Registration                            |
|                                                                   |
|  Activation of automatic agent updates. Your first step is to     |
|  register this host at your deployment server for agent updates.  |
|  For this step you need an administration account on WATO for     |
|  that server.                                                     |
|                                                                   |
+-------------------------------------------------------------------+
Deployment server to connect to: moni01.servers.intern
Protocol to use for connection [http/https]: https
Check_MK Site on deployment server: mysite
Our host name in the monitoring: myhost
WATO User with admin permissions: cmkdadmin
Password: cmk
Going to register agent at deployment server
Successfully registered agent for deployment.
You can now update your agent by running 'cmk-update-agent -v'
Saved your registration settings to /etc/cmk-update-agent.state.

Hint: you can do this in scripts with the command:

/usr/bin/cmk-update-agent register -s localhost -i today -H today -p http -U 'cmkadmin' -P '***' -v

Some advice:

  • Before a host can be registered it must first be installed and activated in the monitoring (including an Activate Changes).
  • In order to be able to use HTTPS, HTTPS must be installed on your monitoring server. HTTP is much simpler, but doesn't offer encryption of the data transfer. Because the agent can theoretically contain passwords, HTTPS is the recommended solution. The agents' authentification is however secured independently using the signature.
  • The login as a WATO user is required only once. During the registration process the agent and the server exchange a secret key known only to this host. The WATO user's password is not stored anywhere.
  • During the registration process the plug-in also requires the name of the host as known to the monitoring. This name is not necessarily identical to the host computer's name. The host name will then be stored locally with the key.

When everything has been successful the key will have been stored in the /etc/cmk-update-agent.state file. This is located on the server in ~/var/check_mk/agent_deployment/myhost. From this point on the key allows the host, or respectively its own agents to download from the server without needing a password. Downloading from other host's agent is not possible as these can contain confidential data.

1.6. Master Switch

Finally, activate everything by clicking on in Master Switch. The Prerequisites table should look like this:

From now on, once per hour, the agent will query and be on the lookout for a new agent version. As soon as one is available and signed it will be automatically downloaded and installed.

2. Restricting the update to specific hosts

Before rolling out new agents on multiple hosts, you will of course want to test them first on a small number of hosts. This important measure will forestall a possible bug having a big impact.

The middle field - Host Selection - on the Automatic agent updates page provides this function:

After the conditions for the selection of hosts have been satisfied here, you can enter individual host names in the Test with this host name to check whether or not the updates for these hosts have been activated. The conditions are always linked with and.

At the same time, the Master Switch is naturally also an option to completely deactivate the updates.

3. Diagnosis

There are a number of information sources for diagnosing whether all updates function as intended:

3.1. Statistics on the Automatic agent updates page

This summary shows how the individual hosts behave in agents update. The online help provides further explanations. A click on opens a detailed list of the individual hosts. The full list of all registered hosts can be accessed via the Other > Agent update status view. There you can also search for specific hosts.

3.2. Check_MK Agent - the new check on every affected host

If you have installed the update plug-in on an agent this will regularly inform you of the status of the update in the form of monitoring data. From this the service discovery creates a new service with the name Check_MK Agent on the host. This reflects the current status of the update. In the form of monitoring alarms you can also be informed of a problem with the updates.

The state of this check is limited to WARN at worst.

You can see in this list how the hash of an agent is beginning of which is planned for a host, which is lastly downloaded by a host and which is reported as installed. That way you will have the complete overview if the guideline is satisfied or at which point the process currently is.

3.3. Log messages on the target host itself

In the case of a problem, log data for the updates can also be found on the target host. Under Linux cmk-update-agents log to syslog. Windows writes to the log/cmk-update-agent.log data set.

/var/log/syslog
Nov 27 13:21:06 Klappfisch cmk-update-agent: Check_MK Agent Updater
Nov 27 13:21:06 Klappfisch cmk-update-agent: Getting target agent configuration from deployment server
Nov 27 13:21:06 Klappfisch cmk-update-agent: Agent f9e0a488f44dffe5 already installed.

4. Deactivating automatic updates on a host

If a host is to be excluded from automatic updates, the Install agent updater (Linux, Windows) rule set should be amended so that there the update plug-in is deactivated. At the next regular update the agent itself will remove its own updater!

It goes without saying that the update can only be reactivated by manually installing a new packet! The registration is however retained and does not to be renewed.

5. Agent updates in distributed monitoring

If you operate a distributed monitoring with multiple instances, the updates are furnished exclusively via the central server. A distribution of the agents on slave servers has not (yet) been envisaged in the current implementation.