Implementing naming conventions using labels in checkmk

Most IT organizations have strict naming conventions for their hosts (servers, network devices and others) representing valuable information about:

  • country
  • region
  • customer
  • application
  • server
  • client
  • network
  • environment (test/production)

This list might be longer or any kind of subset of these attributes. As a sample I use my devices available in my home environment. (A review of my naming convention seems to be very urgent…)

Often naming conventions offer the ability to understand the usage of a given device, the location, or whatever. This information is also required to make your monitoring journey a success.

checkmk offers two key features to attach information of this kind to discovered hosts, labels and tags. In my opinion, the most valuable advantage of labels, compared to tags, is the ability to assign labels dynamically, without any dependency to folders or other mechanism.

This qualifies labels to dynamically apply “attributes” to hosts, using the currently valid naming conventions.

My home naming conventions are:

The following things I’d like to identify for monitoring:

  • the owner of the system
  • the type of the system (virtual system or physical)
  • the application, which is hosted on the system
  • the classification of the physical systems (e.g.: always online or not)

To do so, follow these steps in checkmk:

First Step: Open the setup dialog

  1. Type “host labels” in the search area
  2. In the navigation bar click “Setup”
  3. Click on host labels to enter the rules area for labels

Second Step: Use the “Create rule in folder” button
As naming conventions should apply across all systems in my monitoring environment, I create these rules in the “Main directory”.


Fill in your settings

  1. A description of your Rule
  2. The label you want to attach to this set of devices (how labels work)
  3. Select explicit hosts, to write down a regular expression, selecting the focused host names. The “~” indicates, that a regular expression will follow.
  4. Don’t forget to save your changes.

After repeating the above steps for all of my weak naming convention entries, I have the following rule set.

Please note, that my label names follow some conventions, to make sure, that there is no chaos introduced in the setup of checkmk.

Summary:

Labels can be implied dynamically to the hosts in your checkmk monitoring environment. Tags can’t be applied in this way. Labels can be used later on, while defining monitoring rules, views, dashboard, filters and so on. Labels are a good mechanism to group hosts in different dimensions.

Eight steps to migrate IBM Agent Builder solutions

With the introduction of IBM Monitoring V8 a complete new user interface has been introduced and the agents also changed in the way how communicating with the server.

That implies, that all existing Agent Builder solutions you created have also to change. Agents created for your ITM V6 deployment have also to be adopted.

In this article I want to give a sample for one of my agent builder solutions, I’ve created for OpenVPN. I extend the solution in that way, that I can use it under ITM V6 as well as under APM V8.

The work to be performed is pretty limited. The documentation describes a few major prerequisites, to successfully deploy your agent to a IBM Performance Management infrastructure:

  • A minimum of one attribute group to be used generate an agent status overview dashboard.

  • These single row attribute group (or groups) will be used to provide the four other required fields for this overview

    • A status indicator for the overall service quality

    • A port number, where the service is provided

    • A host name where the agent service located

    • An IP address where the monitored service resides

For more details, please consult the documentation.

In my situation, these attributes were not provided with my ITM V6 agent builder solution. So I expanded my existing solution:

  1. Changing the version number of my agent builder solution (optional)

  2. Create a new data source “openvpnstatus_sh”, which is a script data provider delivering one single line with all attributes defined to it.
    Frame1

  3. The attribute “ReturnCode” will be used later on to describe the overall status of my OpenVPN server. So I have to define the good value and the bad value (see documentation for more details)

  4. Make sure, that the check box under Self Describing Agent is activated.

  5. Run the Dashboard Setup Wizard to produce the dashboard views

    Please make sure, that you checked “Show agent components in the dashboard”! Otherwise…

  6. Now you can select the different values:

    1. Status

    2. Additional attributes for the summary widget

    3. Select the attribute groups to be displayed in the details view for the agent

    4. Define the name of the monitored component

    5. Define the hostname where the software is running

    6. Define the IP address

    7. Define the Port Number

    8. Click “Finish”

Now your agent is almost ready. Start the Agent Builder generation process using the icon

You can generate ITM V6 and IPM V8 compatible agent packages in one step. If you do so, and you want to support Cognos Reporting, then leave “Cognos reporting” checked. While Cognos Reporting isn’t supported for new V8 agents at this time, leave it unchecked when you don’t want to support ITM V6. The agent package size will be much smaller without Cognos Report support.

After successful generating the agent package you will find the following files and directories:

  • smai-openvpnserver-08.13.00.00.tgz

  • subdirectory k00 containing fixlets for BixFix deployment of the new generated agent

After expanding the archive file you find a well known set of files:

The install routines are a now able to verify which kind of agent deployment (V6 or V8) is required. Depending on the given agent install directory, the routine determines automatically, which agent framework has to be installed.

Remark:

The Agent Builder agent has no info about where the monitoring infrastructure server is running. It will connect to the same server as the previously installed OS agent. In turn, any Agent Builder generated agent cannot run without a previously installed APM V8 agent.

Wrap-Up:

It is pretty simple to create agents for ITM V6 and APM V8 at the same time. The Agent Builder supports the two very different user interfaces in one generation process.

Only for the Dashboard definition new required data must be available, but it is simple to gather.

If you have any further questions regarding the IBM Agent Builder, drop me a message.

OMEGAMON XE for Messaging — ITM Sample Situation Package

This solution presents ITM V6.x usage scenarios for OMEGAMON XE for Messaging V7 Tivoli Enterprise Monitoring Agents.

OMEGAMON XE for Messaging V7 comes with a bunch of good, predefined situations, which have been extended in some areas. Others are defined completely new from scratch, to help customers to quickly identify and resolve production-relevant issues.

It should highlight the capabilities of the ITM V6 infrastructure and the power of using ITM situations to identify potential upcoming problems in WebSphere MQ infrastructures. It gives multiple examples for useful situations and potential solutions.

All situations have proven their value in real customer environments and have been created to show our customers the benefit of using ITM monitoring for production environments.

The attached archive file includes a detailed description of the MQ Situation Sample and the situations itself.

WebSphere MQ Monitoring — Comprehensive Workspace Sample Using Navigator Views

This solution presents ITM V6.x enhanced comprehensive workspaces in a custom navigator view for OMEGAMON XE for Messaging V7 (aka ITCAM for Application, MQ Agent).

OMEGAMON XE for Messaging V7 delivers a lot of useful workspaces with very detailed information on a single WebSphere MQ server. This solution presents a complete new approach to navigate to the details of a single MQ resources. The inspection of single objects is more context driven and spans WebSphere MQ server bounds.

The structure of the new navigator is inherited from the original product, so that the user will feel comfortable with the solution. When installed, situations are associated to the new navigation tree.

This solution should highlight the capabilities of the ITM V6 infrastructure and the power of using ITM navigator views in a production environment to identify potential upcoming problems in WebSphere MQ infrastructures.

The linking capability enables the users to follow the path of the message flows across system borders and get a more comprehensive view of the entire object chain making up the communication path in WebSphere MQ. It enables users to quickly identify the root cause of message flow problems.

The PDF document gives you all required details.

The archive file contains the export of the ITM Navigator, which could be used for a quick implementation.

Implementing a 32-bit ITM Agent under Enterprise Linux 64-bit

Implementing the ISM Agent (and other 32-bit ITM Agents) under Linux 64-bit is a little bit tricky, because it requires a 32-bit framework for the ITM base libraries.

The documentation covers all required steps, but is a litlle bit spread across IBM’s website.

In the prerequisites documentation it is stated, that the Tivoli Agent Framework is required in 32-bit architecture.

The installation is a little bit more effort, because it requires the ITM base image to be mounted and the installation of the “Tivoli Enterprise Services User Interface Extensions” has to be performed. The following modules where required on the system I used:

yum install compat-libstdc++-33.i686

yum install libXp

yum install ksh

yum install libstdc++.i686

See the general support matrix for the ISM agent. In the footnote area the required modules are listed.

Because I already installed the OS agent before, the Korn Shell has been installed before.

From the ITM install image (here: ITM_V6.3.0.6_BASE_Linux_x64) I started install.sh:

I accepted the shutdown of any running ITM Agent, and requested to install products to the local host.

After agreeing to the license agreement, the challenge is, to pick the choice to install for another operating system. As you can see from the list of the already installed components, the “Tivoli Enterprise Services User Interface” is already installed. But it is the 64-bit version only. For the ISM agent the 32-bit version is required. So we pick the option 6.

I picked option 3, “Linux x86_64 R2.6, R3.0 (32 bit)” as the OS platform.

In the next step I picked option 2, “Tivoli Enterprise Services User Interface Extensions”.

I used the ITM provided prerequisite checker to verify the software requirements again, and then accepted the installation of the new component.

I accepted all subsequent default values for the setup process and ended with the installation of the 32-bit Tivoli Agent Framework.

After having this successfully installed, the subsequent ISM Agent works really straight forward.

After successfully executing the install.sh from the ISM install image, you simply could accept the default values, if no special requirements have to be met.

After a successful installation the cinfo output should look somehow like this:

If you have further questions, please do not hesitate to drop me a reply below. I will come back to you as soon as possible.

Follow me on Twitter @DetlefWolf to continue the conversation.

Traveling To The Cloud – Predict

As I stated in my previous post, traditional monitoring approaches focusing on named systems do no longer make sense. In an agile cloud environment the system name does not matter, so in turn the performance values from this system don’t.

In such a situation the prediction approach also has to change. The data flowing into IBM Operations Analytics – Predictive Insights should no longer identify a single system nor a single instance of a resource. It should represent the sum of resources or the average usage value. So let us review a few simple examples:

While we monitor the key performance metrics of the system instance with our monitoring agents like

  • Disk I/O per second

  • Memory Usage in Megabytes

  • Network Packages sent and received

  • CPU percentage used

we feed the following values into our prediction tool:

  • SUM(Disk I/O per second) across all used OS images

  • SUM(Memory Usage in Megabytes) across all used OS images

  • SUM(Network Packages sent and received) across all used OS images

  • AVG(CPU percentage used) across all used OS images

IBM Monitoring stores historical data in the Tivoli Data Warehouse. A traditional system setup might directly leverage the data stored in the data warehouse to feed the prediction tool. With the elastic cloud approach we should add some new views to the database, which enable the required summarized data view as described above.

To ensure that a single operating system instance isn’t overloaded a traditional resource monitoring has to be deployed to each cloud participant. Distribution lists from IBM Monitoring will help to do this automatically.

These list of systems are also important to maintain the efficiency of the view’s introduced for the prediction.

The following table is required in the WAREHOUS database:

This table represents the distribution list known from IBM Monitoring.

Based on this table we can create views like the one below: With this new view we are now able to feed data regarding the disk usage into the IBM Operations Analytics – Predictive Insights tool.

The column “CloudName” is useful to identify records for streams. The “TimeFrame” column works as time dimension.

Five streams are the result from the table above:

  • AllReadRequestPerSecond

  • AllWriteRequestPerSecond

  • AvgWaitTimeSec

  • AllReadBytesPerSec

  • AllWriteBytesPerSec

All streams are generate for each single instance “CloudName”.

 In the Predictive Insights Modeling Tool the view is selectable (as a table), so that the generation of the data model is pretty straight forward.

 The SQL line



makes sure that TimeFrame is a Candle Time Stamp which is known to IBM Operations Analytics – Predictive Insights tool.

This sample shows how a data model for the cloud might look like.

With moving more and more systems to the cloud and becoming more and more agile while serving the IT workload, the monitoring approach has to become more agile as well. Also the point of view which key performance metrics matter have to change. But as you can see, the data is there, we only have to change the perspective a little bit.

So what is your approach? What requirements do you see arising while moving your monitoring and prediction tools to the cloud?

Follow me on Twitter @DetlefWolf, or drop me a discussion point below to continue the conversation.

In my next blog I will share a few ideas how to automate the implementation of IT monitoring in a cloud environment.

SCAPI: Preparing your system — Software Packages

Before you can install SmartCloud Analytics Insight (SCAPI) you have to meet the software prerequisites on the RedHat Enterprise Linux Server system you are using to host the SCAPI Data Server Components. Currently only RHEL 6 64-bit is supported.

The documentation names the requirements in several locations of the installation brochure.

I’m using the following command stack to make sure that all software packages are installed:

  • yum -y install libstdc++.i686
  • yum -y install *libstdc++-33*.i*86
  • yum -y install openmotif22*.*86
  • yum -y install pam.i686
  • yum -y install libXpm.i686
  • yum -y install libXtst.i686
  • yum -y install freetype.i686
  • yum -y install libmcpp.i686
  • yum -y install libXdmcp.i686
  • yum -y install libxkbfile.i686
  • yum -y install libpciaccess.i686
  • yum -y install libXxf86misc
  • yum -y install libXm.so.4
  • yum -y install ksh*
  • yum -y install libstdc++.*
  • yum -y install *libstdc++-33*
  • yum -y install openmotif22*
  • yum -y install compat-glibc
  • yum -y install pam
  • yum -y install libXpm
  • yum -y install libXtst
  • yum -y install freetype
  • yum -y install xorg-x11-xinit
  • yum -y install Xorg
  • yum -y install firefox
  • yum -y install openmotif
  • yum -y install atlas
  • yum -y install compat-libgfortran-41
  • yum -y install blas
  • yum -y install lapack
  • yum -y install dapl
  • yum -y install sg3_utils
  • yum -y install libstdc++.so.6
  • yum -y install libstdc++.so.5
  • yum -y install java-1.7*-openjdk.x86_64
    Java is required to get the prerequisite checker executed delivered with the IBM InfoSphere Streams software.
    The packages below are installed because the InfoSphere Streams checker.
  • yum -y install libcurl-devel.i686
  • yum -y install libcurl-devel.x86_64
  • yum -y install fuse-curlftpfs.x86_64
  • yum -y install libcurl.i686
  • yum -y install libcurl.x86_64
  • yum -y install perl-Time-HiRes
  • yum -y install perl-XML-Simple*
  • yum -y install gcc-c++

This command stack includes only those packages, which are provided by RedHat Satellite server.

After you installed all the packages above you have to add the provided RPM package as documented in the installation manual.

I’ve used the following command:

#
# Install provided InfoSphere RPM Prerequisite
rpm -Uvh <streams_unpack_folder>/rpm/*.rpm

Having all this packages installed, allows you to install all SCAPI software components.

Implementing the OMNIbus WebGUI in silent mode

After installing the OMNIbus WebGUI you have to implement the properties to direct the WebGUI server to the correct OMNIbus server and the correct authorization source.

To run the configuration wizard in silent mode, which is essential for repeated installations, use the ws_ant.sh shell script as documented in the manual.

The documentation suggests that you can locate the OMNIbusWebGUI.properties file in any location. This is not really true.

Inside this file other files are referenced which are provided in the same directory as the properties file.

The following approach worked for me:

I’ve installed the OMNIbus WebGUI server with default values, so it ended up in /opt/IBM/netcool/omnibus_webgui/ directory. In the subdirectory bin I’ve found the OMNIbusWebGUI.properties file.

Change the file accordingly to your installation and leave it in this directory.  As documented in the manual execute the ws_ant.sh script from this directory, adding the required activity.

The ws_ant.sh requires the build.xml in the working directory. build.xml file describes the different callable activities. These are

  • configureOS
  • resetVMM
  • configureVMM
  • restartServer

Through this build.xml file the WebSphere script execution is controlled.

In this xml file, the OMNIbusWebGUI.properties file is referenced with hard coded path names. Additionally, other XML files are referenced which are expected in the same directory.

So, edit the properties file in the location where you find it. And then execute the ws_ant shell script from this directory…