Check_MK version update


1. Introduction

Updating Check_MK is a bit different than most other software packages that you may be familiar with. Why is that?

The reason is that Check_MK not only allows multiple independent instances (sites) to run on a single server, but it also allows multiple software versions to be installed simultaneously. Thereby every instance is assigned to an installed version. To illustrate, we can take the following situation on a fictional server:

Here the instance mysite1 uses version 1.2.8p11.cee, but the instances mysite2 and mysite3 use version 1.2.6p10.cre. The Check_MK-Version 1.2.6p10.demo is actually installed, but is currently not being used.

This example makes it clear that an update does not simply mean the installation of a new Check_MK RPM/DEB-packet on a server. A more important additional step is also required. Let us take the following situation as an example:

Here the instance mysite is to be updated to the Check_MK-version 1.2.8p11.cee. The first step is to download and install the appropriate RPM/DEB-Packet. This is performed exactly as with the initial installation. At first the newly-installed version will not be used by any instance, and it will look like this:

The second step will now be an update of the instance from 1.2.6p10.cre to 1.2.8p11.cee. This is achieved with the omd update command, which we will discuss in detail below:

Following the update, the (likely) no longer required version 1.2.6p10.cre can be deleted by uninstalling the appropriate package. It is however not a bad idea to leave it installed, so that if necessary, with a renewed omd update you can quickly revert to the old version.

2. Updating of Check_MK in detail

2.1. Installation of newer versions

As described in the introduction, the first step with an update is the installation of the desired new version of Check_MK. This is achieved in exactly the same way as with the initial installation – it will however proceed somewhat more quickly since most of the dependent packages have already been installed. In the following example we are installing the package for Ubuntu 14.04 (Codename Trusty):

root@linux# gdebi omd-1.2.8p11.cee_0.trusty_amd64.deb

A list of installed Check_MK-versions can be displayed at any time with the omd versions command:

root@linux# omd versions
2016.12.18.cee
1.2.8p11.cee
1.2.6p10.cre (default)

One of these listed versions is marked with (default). This default version will be used automatically when creating new sites, as long as no other version is specified with omd -V myversion create mysite. When working with existing sites this is irrelevant. The current default version can be queried with omd version, and it can be altered with omd setversion:

root@linux# omd version
1.2.6p10.cre
root@linux# omd setversion 1.2.8p11.cee
root@linux# omd version
1.2.8p11.cee

The default version plays no role when updating or managing existing instances. The omd command always starts itself automatically in the version appropriate to the instance.

A listing of the current instances and the versions they use is provided by the omd sites command:

root@linux# omd sites
SITE             VERSION
mysite           1.2.6p10.cre
test             2016.12.18.cee

2.2. Performing the update

Once the desired new version has been installed, the instance can be updated. No root-permissions are required for this. The best way to do this is as an instance user:

root@linux# su - mysite

Ensure that the instance has been stopped:

OMD[mysite]:~$ omd stop

The update – in effect switching to a different version – can now simply be achieved with the omd update command:

OMD[mysite]:~$ omd update

If more than one target version is available, a selection list will open:

An important part of an update is the refreshing of the originally provided configuration files. Here changes that had possibly been made to these files by the user will not simply be discarded, rather they will be merged. This functions very much like version control systems which attempt to amalgamate changes made to a single file simultaneously by multiple developers.

Occasionally – when the changes affect the same location in the file – that won't function, and a conflict occurs. How you can solve these will be shown later below.

The update provides a listing of all modified files and directories:

2016-10-11 18:27:07 - Updating site 'mysite' from version 1.2.6p10.cre to 1.2.8p11.cee...

 * Unwanted       var/log/nagios.log (unchanged, deleted by you)
 * Updated        etc/nagvis/nagvis.ini.php
 * Updated        etc/mk-livestatus/nagios.cfg
 * Updated        etc/check_mk/defaults
 * Updated        etc/apache/conf.d/02_fcgid.conf
Finished update.

Once everything has been successfully processed, the instance can be switched to the new version...

OMD[mysite]:~$ omd version
1.2.8p11.cee

... and can then be started:

OMD[mysite]:~$ omd start

2.3. Incompatible changes

Software development of course consists of changes. Because we are always actively working to keep Check_MK modern, sometimes cutting dead weight and making changes that turn out to be incompatible is unavoidable. That means that when updating it may possibly be necessary to adapt your configuration, or you should at least check it.

A typical example of such a situation is with new check plug-ins which replace existing plug-ins. If you use one of the affected plug-ins, a new service discovery will be required on the affected host.

An overview of all changes in Check_MK, including a search function can be found online: here. Even more practical however is the built-in search function in the version release notes. This is accessed by clicking on the version number at the top-left of the side bar:

Check_MK tracks new changes automatically, and issues appropriate warnings if they are incompatible:

You can then inspect these ‘Werks’ (‘works’), and approve them with a mouse click. You can also find a listing covering the complete history of the changes, including a search function:

2.4. Downgrade – falling back to an old version

The process of switching back to an older version runs just like an update. To be precise, omd update does not care whether the target version is newer or older than the current version – thus you can switch ‘forwards’ or ‘backwards’ as desired.

Be aware however, that even if a downgrade to an older version functions wonderfully, an older Check_MK may not always be able to process data from newer versions. A new Check_MK-version may possibly store data and configurations in an extended format that an older version of the software may not understand.

Configurations which are managed in WATO may possibly be converted into a new format once WATO is in active use and can then store the configurations. As long as such actions have not yet occurred, a switch back to an earlier version is generally unproblematic.

Should you be uncertain whether it is necessary to fall back to an earlier version, we recommend to:

  • Perform a data backup BEFORE the update.
  • Take the time to test the new version before making alterations via WATO.

2.5. The update in detail

Are you curious about what exactly is happening ‘under the hood’ of an update? Or have data conflicts appeared when omd update is running? If so, here is some further reading.

Three actions take place during omd update:

  1. The refreshing of the default files under etc/ and var/ – i.e., files created by omd create.
  2. The switching of the active version to the target version by changing the symbolic link version which is found in the Site-directory.
  3. Post-processing by various packages (e.g., Check_MK). In particular, an Activate Changes will be automatically executed in order to generate a valid configuration for the core.

Actualising files, merging changes

The first step is by far the most comprehensive. Here Check_MK demonstrates a big advantage in comparison to the usual software installation – Check_MK helps you to adapt all of the standard configuration files to the prerequisites of the new version. This resembles the procedure for updating a Linux-Distribution, but goes further in the implementation. Check_MK can handle a multiplicity of situations, for example:

  • The merging of file changes with changes made locally by the user
  • Files, directories and symbolic links which are obsolete in the new version, or which have been deleted by the user
  • Changes to permissions
  • Changes to a file type (a symbolic link derived from a file or directory, or vice versa)
  • Changes to the target of a symbolic link

Check_MK always ensures that your local changes are retained, and that all of the changes required by the new version are simultaneously implemented.

Merging and conflicts

If the new version intends changing a configuration file on which the user has also made changes, Check_MK automatically attempts to merge both sets of changes. This is achieved using the same methods as used by version-control systems.

The fewest problems are experienced when your and Check_MK's changes have a clear physical separation in the text (at least a few lines apart). The merge will then be effected automatically, and without needing the user's intervention.

If two changes ‘collide’ because they both affect the same location in the data, Check_MK cannot and will not decide which of the changes is more important. In such a situation the user will be alerted, and be able to solve the conflict interactively:

You now have the following options:

dThis shows the differences between the new default version and your version of the file in the form of a ‘unified diff’ (diff -u).
yThis is similar to the above, but based on the preceeding default version shows which changes you have made to the file.
nThis third option in effect ‘closes the triangle’ by showing the changes which Check_MK intends making to the file.
tBy selecting t, your original file – without the already successfully-merged changes – will be opened in an editor. Now edit the file in order to bypass possible conflicts. Once the editor has been closed Check_MK will reattempt the merge.
kHere you can decide whether to take the data ‘as is’. The successfully inserted changes are retained. Apart from this the file remains as customised by the user.
rWith this you can fall back to the old version of your file, and go without Check_MK's update for this file. Any possibly required customisations must be performed manually.
iInstall the new default file: your changes in the file will be lost.
sIf you are uncertain, you can open a shell with s. You will find yourself in a directory containing the relevant file, and there can get a picture of the situation. Quit the shell with Strg-D in order to proceed with the update.
aAbort the update. The instance remains with the old version. Files that have already been changed however remain changed! A new update attempt can be started at any time.

Further conflict situations

Alongside the content-merging of files there is a whole series of further situations in which Check_MK requires your decisions. Some of these are very unusual situations, that nevertheless need to be handled correctly. In these cases Check_MK will always give you the choice of keeping your version, or of adopting the new default version. What is more, there is always the option of aborting an update, or of opening a shell. Examples of such ‘difficult’ situations are:

  • conflicting changes to file types (e.g., when a file is replaced by a symbolic link)
  • conflicting changes to file permissions
  • changed files that are not required by the new sofware version
  • files, directories or links created by a user, which conflict with a new version's files/directories/links

Explanation of the tasks in an update

The update procedure will always output a line of explanation when it makes a change to a file automatically. The following situations are possible – for links and directories as well as for files:

Updated A file has been changed with the new version. Since you have not made a change to the file, Check_MK simply installs the new default version of the file.
Merged A file has been changed with the new version, and at the same time the user has made other changes to the file. Both versions of the file can be merged into one without conflict.
Identical A file has been changed in the new version, and at the same time the user has already made identical changes to the file. Check_MK must not perform any action.
Installed The new version includes a new configuration file which has just now been installed.
Identical new The new version includes a file, an identical copy of which the user has already installed.
Obsolete The new version has obsoleted a file (also applies to a link or a directory). The user has anyway already deleted it. No action.
Vanished Another file is obsolete in the new Check_MK, and the user has neither deleted nor changed the existing version. Check_MK deletes this file automatically.
Unwanted The user has deleted a file wihich is normally present. Because the version in the new Check_MK has no changes from the last version of the file, Check_MK allows the file to be absent.
Missing The user has already deleted a file, but in the new Check_MK this file contains changes from the previous version. Check_MK installs the new file, and logs a notification of this action to the user.
Permissions Check_MK has updated a file's permissions because different permissions are set in the new version.

2.6. Updating without user interaction

Would you like to automate Check_MK's software updates? You may at first have difficulties with the interactive responses from omd update. There is a simple solution for this scenario: the command has options that have been especially conceived for use in scripts:

  • The options -f or --force directly following omd inhibit all types of “Are you sure... ?” questions.
  • The option --conflict= directly following update determines the desired behaviour if a file conflict occurs.

Possible values for --conflict= are:

--conflict=keepold In the case of a conflict, the user's own modified version of the file is retained. It is however possible that Check_MK may not be executable, and that manual rectification will be required.
--conflict=install In the event of a conflict, the new standard version of the file will be installed. With this, local changes to the file will be at least partly lost.
--conflict=abort In the event of a conflict the update is stopped. That does not necessarily mean that everything will fall back to the old state. A number of configuration files may have already been updated. The software version will however remain the old version.
--conflict=ask This is the standard procedure, so in this form the option is actually superfluous.

Below is an example of the complete command for an automated update to version 1.2.8p11 of the mysite instance:

root@linux# omd stop mysite ; omd -f update --conflict=install mysite 1.2.8p11 && omd start

Through the && before omd start a restarting of the instance will be prevented if the omd update is aborted by an error. Replace the && with a semicolon (;) if a start should definitely be attempted even in such a situation.

If you are certain that only a single Check_MK-instance is running on the server, the name to be used in a shell script can simply be trapped in a variable:

root@linux# omd sites --bare
mysite
root@linux# SITENAME=$(omd sites --bare)
root@linux# echo $SITENAME
mysite

This enables the above line to be independent of the instance's name. For example, a small shell script could look thus like this:

update.sh
!/bin/bash
SITE=$(omd sites --bare)
VERSION=1.2.8p11

omd stop $SITE
omd -f update --conflict=install $SITE $VERSION && omd start $SITE

3. Update from the demo to the full version

Was your first installation of Check_MK the demo version? As soon as you have a  Check_MK Enterprise Edition subscription you can simply update your existing instances to the full version.

The procedure is exactly the same as for a ‘normal’ update. The only difference is that a version's name with the .demo suffix is updated to a name with the .cee suffix. Simply install the desired package of the full version and switch the existing instance to this with omd update.

This can be most easily achieved if both versions are identical, apart from the .demo and .cee suffixes respectively. What this means for the functionality is that, apart from the random WARN-state, the demo version is completely identical to the full version. Thus an update makes no difference at all.

A simultaneous changeover of the actual version is however quite possible. The fundamentals remain valid as for a normal update of Check_MK.

4. Updating from the Raw Edition to the Enterprise Edition

An update of the  Check_MK Raw Edition to the  Check_MK Enterprise Edition is also possible. Here as well, the procedure is the same as before: install the desired pacakge, and update the instances with omd update.

Since a number of modules and features of the CEE are not available in the CRE, following a changeover there are a couple of points to be aware of. The decisive point is that when creating new CRE, or respectively, CRE instances, different default settings are defined.

Nagios vs. CMC

Since the CRE supports only Nagios as its core, this is preinstalled in instances created by the CRE. This is retained when an update to the CEE is made. That means that after an update, processing will initially continue with a Nagios core. A migration to the CMC is performed with omd config, and this precedure will be described in its own article.

RRD-Format

The CEE supports an alternative format for saving historic performance data, one which requires significantly less hard drive-I/O. This is preinstalled in new CEE-instances. CRE-instances will not be changed over automatically by an update. How the migration can be performed is described in its own chapter within the title covering Performance data and graphing.

Notification spooler

The CRE has no notification spooler. Thus following the changover to the CEE it is not active at first. How to activate it can be learned here.

5. Uninstalling Check_MK

The uninstallation of no longer required Check_MK-versions is performed using the operating system's package manager. To do this, enter the installed package's name – NOT the file name of the original RPM/DEB-file. Important: Only delete Check_MK-versions that are no longer being used by any instance!

Check_MK-instances that are no longer required can simply be removed with omd rm (thereby deleting all data as well!):

root@linux# omd rm mysite

SLES, RedHat, CentOS

Here is how to identify which Check_MK-Packages are being used in RPM-based systems:

root@linux# rpm -qa | grep check-mk
check-mk-enterprise-2016.05.19.0
check-mk-enterprise-2016.10.11.0
check-mk-raw-1.2.8b9.0
check-mk-raw-1.2.8p11.0

The deletion is performed with rpm -e:

root@linux# rpm -e check-mk-raw-1.2.8b9.0

Debian, Ubuntu

Use the below to identify which packets are installed:

root@linux# dpkg -l | grep check-mk
ii  check-mk-enterprise-2016.05.19  0.trusty  amd64  Check_MK is a full featured system monitoring
ii  check-mk-enterprise-2016.10.11  0.trusty  amd64  Check_MK is a full featured system monitoring
ii  check-mk-raw-1.2.8b9            0.trusty  amd64  Check_MK is a full featured system monitoring
ii  check-mk-raw-1.2.8p11           0.trusty  amd64  Check_MK is a full featured system monitoring

The uninstallation is performed with dpkg --purge:

root@linux# dpkg --purge check-mk-raw-1.2.8b9
(Read database ... 505719 Files and directories are currently installed.)
Remove from check-mk-raw-1.2.8b9 (0.trusty) ...
Delete the configuration files from check-mk-raw-1.2.8b9 (0.trusty) ...

6. Files and directories

The files and directories relevant to this article can be found here. As always, paths that do not begin with ‘/’ apply after the home directory of the (/omd/sites/mysite) instance:

Pfad Function
version Symbolic link to the installation of the Check_MK-version used by this instance.
/omd/versions Within this directory a subdirectory exists for every installed Check_MK-version. The files belonging to root and are never changed.
/omd/sites Within this directory, for every instance there is a home directory containing its configuration files and variable data. This data belongs to the instance's user, and can be changed by configuration and operations.
/usr/bin/omd Management command for Check_MK-instances. This is a symbolic link to the default version's bin-directory. When a particular instance is accessed the omd-command substitutes itself with that of the appropriate version.