Inventory - How check_mk finds services to check

Introduction

Configuring which check should be done on which host is a tedious work in Nagios. The fact the it is much work is not the biggest problem. A greater issue is to keep your configuration up-to-date. Your colleges introduce new filesystems, new network interfaces and new database instances without always informing you. How can you be sure that every important item is really being monitored?

Check_mk helps you not only to scan new hosts for items to check but also to keep track of your existing hosts. It can do so because of the special nature of its agents: They always send all interesting data about the host regardless of which items are checked with Nagios.

Not all but most services can be detected automatically. This holds not only for filesystems and network interaces but for many other things as well. Calling check_mk with the option -L will show all checks and wether they support inventory or not (the list is abbreviated here):

root@linux# check_mk -L
Available check types:

                      plugin   perf-  in-
Name                  type     data   vent.  service description
-------------------------------------------------------------------------
cpu.threads           tcp      yes    yes    Number of threads
df                    tcp      yes    yes    fs_%s
df_netapp             snmp     yes    yes    fs_%s
diskstat              tcp      yes    yes    Disk IO %s
ipmi                  tcp      yes    yes    IPMI Sensor %s
ironport_misc         snmp     yes    yes    %s
kernel                tcp      yes    no     Kernel %s
kernel.util           tcp      yes    yes    CPU utilization
local                 tcp      yes    yes    %s
mem.used              tcp      yes    yes    Memory used
multipath             tcp      no     yes    Multipath %s
netctr                tcp      yes    no     NIC %s
netctr.combined       tcp      yes    yes    NIC %s counters
ps                    tcp      no     no     proc_%s
services              tcp      no     yes    service_%s
snia_sml              snmp     no     yes    SNMP %s
vms_sys.util          tcp      yes    yes    CPU utilization
vmstat_aix            tcp      yes    no     vmstat %s
winperf.cpuusage      tcp      yes    yes    CPU Usage
winperf.diskstat      tcp      yes    yes    Disk IO

Performing an inventory

Inventory is not done automatically (well, you could write a cronjob doing this). You perform it with the option -I and a comma separated list of check types or simply the keyword alltcp - which means selects all non-SNMP check types.

You also can append one or more hostnames. That will restrict inventory to that hosts. Otherwise all hosts will be scanned. So the easiest way is doing a complete inventory over all hosts:

root@linux# check_mk -I alltcp

The following call will just scan two hosts. This can be useful after adding new hosts to all_hosts:

root@linux# check_mk -I alltcp newhost1 hewhost2

The following call scans for new filesystems on a Netapp filer:

root@linux# check_mk -I df_netapp filer01 filer02

Why is inventory so fast?

In order to speed up inventory (greatly), check_mk does not retrieve the data from the hosts if they already have been checked at least once. Each time a check is running a cache file is kept in /var/lib/check_mk/cache. Inventory information is drawn from there if available. From that follows:

• If you Nagios is not running, the cache files might not be current
• It might take some for a new item to get into inventory.

The cache can be disabled with --no-cache:

root@linux# check_mk --no-cache -I alltcp newhost1 hewhost2

SNMP checks

SNMP based checks can also be inventorized as the upper example has shown. Since check_mk does never retrieve the complete SNMP database of a host (that can last minutes) you always have to specify which check types you are looking for. This might change in a future version. But currently there is no such thing as allsnmp.

What happens with the items found?

All new items check_mk finds are saved in configuration files similar - but not quite compatible - to main.mk. They are created in a separate directory which defaults to /var/lib/check_mk/autochecks. At setup.sh you have been asked for a "working directory of check_mk". autochecks will be created as a subdirectory of that.

Each time you call check_mk it reads in all files in that directory and appends the entries to your checks variable. Let's look at such a file:

# /var/lib/check_mk/autochecks/df-2009-05-20_19.21.44.mk
[
  # === zwin17 ===
  ("zwin17", "df", 'C:/', filesystem_default_levels), # 36

  # === zsrv01 ===
  ("zsrv01", "df", '/', filesystem_default_levels), # 24
  ("zsrv01", "df", '/home', filesystem_default_levels), # 17
]

Changing and removing inventorized checks

Check_mk's inventory does never remove any checks. If a filesystem previously found is now missing, it is either a critical problem or it has been removed by the host's administrator. If the latter is the case simply delete the line from the autochecks file. Or delete the whole file if all checks it contains are obsolete.

Check_mk never overwrites files in autochecks. It is completely save to edit them and change levels or other check parameters. You might want to allow the filesystem /home on zsrv01 to get 98% full until a warning is notified or 99% full before it will get critical. Simply edit the file:

# /var/lib/check_mk/autochecks/df-2009-05-20_19.21.44.mk
[
  # === zwin17 ===
  ("zwin17", "df", 'C:/', filesystem_default_levels), # 36

  # === zsrv01 ===
  ("zsrv01", "df", '/', (98, 99) ), # 24
  ("zsrv01", "df", '/home', filesystem_default_levels), # 17
]

Cleaning up autochecks

The fact that Check_MK creates new files for each inventory is handy if you want to revert or modify the results of recent inventories. As time goes by there are quite a lot of files in the autochecks directory, however.

As of version 1.1.7i1, Check_MK offers the new option -u or --cleanup-autochecks, which reads in all files in /var/lib/check_mk/autochecks, creates one new file per host and removes the exceeding files afterwards. That greatly reduces the number of files in the directory and also makes the removal of all data of a host an easy task. This option can either be used stand alone...

root@linux# check_mk -u

... or as a modifier to -I:

root@linux# check_mk -uI tcp host123

If called that way, the cleanup is done right after the inventory. If you like that feature, you can make Check_MK always cleanup immediately after each inventory by setting in your main.mk:

always_cleanup_autochecks = True

Updating your Nagios configuration

Please do not forget to update your Nagios configuration with a

root@linux# check_mk -U

... after every inventory (that found something new) or manual change in the autochecks. That will not only update your Nagios configuration files but also recompile all host checks.

Inventorized versus manual checks

Even when checks can be found via inventory it is allowed to configure them manually. You can have various reasons for that. One is that you want to define levels others than those the inventory sets.

Whenever a check is defined manually in main.mk the inventory will never find that item agin.

Excluding items from the inventory

Sometimes the inventory finds things that you do not want to check. Removing that items from the files in autochecks is not a perfect idea: At the next inventory those items will reappear again.

It is better to explicitely exclude them. Check_mk provides two configuration variables for doing that:

Config variable	Meaning
ignored_checktypes	Simple list of checktypes to exclude from inventory
ignored_services	Host specifig configuration list of service names to exclude

In ignored_checktypes you can switch off inventory for certain check types completely and globally. Lets assume, that you do not want to monitor network interface throughput and link settings at all. Simply list the according check types (see check_mk -L) in this list:

ignored_checktypes = [ "netctr.combined", "netif.params" ]

If you want to control inventory more specific you need ignored_services. This is a configuration list with the following values in each entry:

1. Optional: List of host tags
2. List of hosts
3. List of service patterns

The following example will exclude the Eventlog Security from the two hosts win01 and win01:

ignored_services = [
  ( [ "win01", "win02" ], [ "LOG Security" ] )
]

Note that the list of services is interpreted as regular expressions matching the beginning of the service description as displayed in Nagios. The following example will not only one but all Logfiles, i.e. all services beginning with LOG, as well as the drives with the letter D::

ignored_services = [
  ( [ "win01", "win02" ], [ "LOG", "fs_C:" ] )
]

If you are unsure about the correct spelling of a service you can call check_mk -D to dump all services.

If you have tagged all your windows host with win the following configuration snippet will do the same but for all Windows hosts:

ignored_services = [
  ( [ "win" ], ALL_HOSTS, [ "LOG", "fs_C:" ] )
]

Please note, that the two ignore_... variables only affect future invetories. They have no effect on the checking or on previously inventorized services.