Now that I have setup my DHCP and DNS servers with dnsmasq my next target is to add some monitoring software to my arch linux RPi box.

Icinga is an open source host, service and network monitoring server. It has a nice web interface where you can quickly check the status of all the hosts. The screenshot below shows the status screen after installation. Only the localhost is present at this stage. It looks heathy. 🙂

I will start my experiment by installing the icinga version 1. I will basically follow the instructions on the Arch Wiki pages.

First let’s bring the operating system up-to-date.

# pacman --sync –-sysupgrade –-refresh # pacman-db-upgrade

I will add a dedicated user account for running the icinga services, let’s call it icinga.

# groupadd -g 667 icinga # useradd -u 667 -g icinga -G http -d /dev/null -s /bin/false icinga

The icinga tool can be build using the build description (PKGBUILD)  available in the Arch User Repository (AUR). After building the tool we can install it with pacman. As a prerequisite we will need to install the base-devel group package to get all the needed build tools.

# pacman -S –needed base-devel

Then let’s create a build directory where we will download the source code for building. Note that the building must be done as an ordinary user that just has the right to run the ‘sudo’ command (edit /etc/sudoers if needed with visudo).

$ mkdir ~/builds

The package description file (icinga.tar.gz) can be downloaded from AUR with curl. The version I’m building seems to be 1.11.7.

$ cd ~/builds $ curl -L -O https://aur.archlinux.org/packages/ic/icinga/icinga.tar.gz $ tar xvf icinga.tar.gz

Unpacking the tar file creates a subdirectory called icinga. Before starting the build we need to edit the PKGBUILD file and change the target architecture for RPi.

$ nano icinga/PKGBUILD arch=(‘armv6h’)

Next we will just call the makepkg tool to build the package. The -s option makes sure that all the dependecies are also installed.

$ cd icinga $ makepkg -s

As the first step the build process will run configure. Below is the summary file.

*** Configuration summary for icinga-core 1.11.7 09-03-2014 ***: General Options: ------------------------- Icinga executable: icinga Icinga user/group: icinga,icinga Command user/group: icinga,icinga Apache user/group: http,http Embedded Perl: yes, with caching Event Broker: yes Enable compressed logs: yes Enable Performance Data: no ido2db lockfile: /run/ido2db.pid ido sockfile: /var/spool/icinga/ido2db.sock idomod tempfile: /run/idomod.tmp Build IDOUtils: libdbi, instance_name=default libdbi driver dir: /usr/local/lib/dbd Install ${prefix}: /usr/share/icinga Lock file: /run/icinga.pid Temp file: /tmp/icinga.tmp Chk file: /var/spool/icinga/icinga.chk HTTP auth file: /etc/icinga/htpasswd.users Lib directory: /usr/lib/icinga Bin directory: /usr/bin Plugin directory: /usr/lib/monitoring-plugins Eventhandler directory: ${exec_prefix}/libexec/eventhandlers Log directory: /var/log/icinga Check result directory: /var/spool/icinga/checkresults Temp directory: /tmp State directory: /var/spool/icinga Ext Cmd file directory: /var/spool/icinga/rw Init directory: /etc/rc.d/init.d Apache conf.d directory: /etc/httpd/conf/extra Apache config file: /etc/httpd/conf/extra/icinga.conf Mail program: /usr/bin/mail Host OS: linux-gnueabihf Environment Prefix: ICINGA_ Web Interface Options: ------------------------ HTML URL: http://localhost/icinga/ CGI URL: http://localhost/icinga/cgi-bin/ Main URL: http://localhost/icinga/cgi-bin/tac.cgi

The build phase takes some time so this may be a good time to grab a cup of coffee…

When the build is ready there should be an installation package in the build directory (in my case it is called icinga-1.11.7-1-armv6h.pkg.tar.xz).

Let’s proceed and install the package.

$ sudo pacman -U icinga-1.11.7-1-armv6h.pkg.tar.xz [sudo] password for eb: loading packages... resolving dependencies... looking for conflicting packages...Packages (1) icinga-1.11.7-1Total Installed Size:  30.61 MiB:: Proceed with installation? [Y/n] (1/1) checking keys in keyring                                               [############################################] 100% (1/1) checking package integrity                                             [############################################] 100% (1/1) loading package files                                                  [############################################] 100% (1/1) checking for file conflicts                                            [############################################] 100% (1/1) checking available disk space                                          [############################################] 100% (1/1) installing icinga                                                      [############################################] 100%--> Sample config files are installed with .sample extension. --> Remember, these are *SAMPLE* config files.  You'll need to read --> the documentation for more information on how to actually define --> services, hosts, etc. to fit your particular needs. --> Remenber to include the the http icinga configuration file in the --> /etc/httpd/conf/httpd.conf file.Optional dependencies for icinga     monitoring-plugins: plugins needed for icinga checks

Let’s also install the recommended monitoring-plugins package.

# pacman -S monitoring-plugins

As I intend to monitor some SNMP enabled devices I will also install the snmp and mrtg packages.

# pacman -S net-snmp # pacman -S mrtg

Configuring Icinga

During the installation a set of sample configuration files (with .sample extension) were copied into the /etc/icinga directory. We can use those sample files as a starting point for the configuration.

# cd /etc/icinga # cp cgi.cfg.sample cgi.cfg # cp resource.cfg.sample resource.cfg # cp icinga.cfg.sample icinga.cfg # cp objects/commands.cfg.sample objects/commands.cfg # cp objects/contacts.cfg.sample objects/contacts.cfg # cp objects/localhost.cfg.sample objects/localhost.cfg # cp objects/templates.cfg.sample objects/templates.cfg # cp objects/timeperiods.cfg.sample objects/timeperiods.cfg

The objects folder contains configuration files for the objects to be monitored. By default only the localhost will be configured with the file localhost.cfg. I will enable notifications for all the defined services, e.g. this is how the HTTP service is defined:

define service{     use                     local-service         ; Name of service template to use     host_name               localhost     service_description     HTTP     check_command           check_http     notifications_enabled   1     }

Configuring the web server

I’m going to use the nginx web server here. However, also apache must be installed to get the htpasswd command.

# pacman -S nginx # pacman -S apache

Let’s start by creating a username and password for web access. I will call the admin user icingaadmin.

# htpasswd -c /etc/icinga/htpasswd.users icingaadmin

For nginx we need to edit the file /etc/nginx/nginx.conf and define the document root and authentication for icinga.

http {      ...      server {          listen       80;          server_name  arch-linux;      ...      location /icinga/ {          alias                   /usr/share/webapps/icinga/;          auth_basic              "Restricted";          auth_basic_user_file    /etc/icinga/htpasswd.users;     }     ...

Next we will need to configure CGI by adding the following section to nginx.conf inside the server section.

location ~ ^/icinga/(.*)\.cgi$ {      root           /usr/share/webapps/;      fastcgi_pass   unix:/var/run/fcgiwrap.sock;      include        fastcgi.conf;      fastcgi_param  AUTH_USER          $remote_user;      fastcgi_param  REMOTE_USER        $remote_user;    }

The fcgiwrap utility enables the web server to call external applications. Let’s install it. After installation the fcgiwrap.socket must be enabled and started.

# pacman -S fcgiwrap # systemctl enable fcgiwrap.socket # systemctl start fcgiwrap.socket # systemctl status fcgiwrap.socket

Configuring IDOUtils

Icinga Data Out Utilities (IDOUtils) contain tools for  storing the network monitoring data into a database.

First we will need to install the database software, for icinga we will use mariadb. After installation we will need to create the database and start the service. After the service has been started (check with systemctl status mysqld.service) run the mysql_secure_installation utility to set the database root user password and to adjust other security settings.

# pacman -S mariadb # mysql_install_db –user=mysql –basedir=/usr –datadir=/var/lib/mysql # systemctl enable mysqld.service # systemctl start mysqld.service # systemctl status mysqld.service # /usr/bin/mysql_secure_installation

Let’s continue with the IDOUtils. Create the configuration files by copying the sample files in the /etc/icinga folder.

# cd /etc/icinga # cp idomod.cfg-sample idomod.cfg # cp ido2db.cfg-sample ido2db.cfg # cd /etc/icinga/modules # cp idoutils.cfg-sample idoutils.cfg

Then create the database and tables for icinga use.

$ mysql -u root -p > CREATE USER 'icinga'@'localhost' IDENTIFIED BY 'icinga'; > CREATE DATABASE icinga; > GRANT USAGE ON icinga.* TO 'icinga'@'localhost' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0; > GRANT SELECT , INSERT , UPDATE , DELETE, DROP, CREATE VIEW, INDEX ON icinga.* TO 'icinga'@'localhost'; > FLUSH PRIVILEGES; > quit $ mysql -u root -p icinga < /usr/share/icinga/idoutils/db/mysql/mysql.sql

• Define the database socket connection for ido2db in the file /etc/icinga/ido2db.cfg.

# DATABASE SOCKET
# Optional db_socket allows to specify a different socket location.
# This will be passed to libdbi MySQL as mysql_unix_socket, while
# PostgeSQL overrides the port, ocilib Oracle ignores this setting.
#
# Note: This setting overrules db_port, making it useless!
db_socket=/var/run/mysqld/mysqld.sock

• Enable and start the Icinga Data Out Utilities

# systemctl enable ido2db # systemctl start ido2db

Check that the service starts.

# systemctl status ido2db * ido2db.service – Icinga Data Out Utilities (IDOUtils) Loaded: loaded (/usr/lib/systemd/system/ido2db.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2015-05-17 20:20:24 EEST; 3s ago Main PID: 18522 (ido2db) CGroup: /system.slice/ido2db.service `-18522 /usr/bin/ido2db -f -c /etc/icinga/ido2db.cfg May 17 20:20:24 arch-linux ido2db[18522]: libpq.so.5: cannot open shared object file: No such file or directory May 17 20:20:24 arch-linux ido2db[18522]: libdbi: Failed to load driver: /usr/lib/dbd/libdbdpgsql.so May 17 20:20:24 arch-linux ido2db[18522]: libsqlite.so.0: cannot open shared object file: No such file or directory May 17 20:20:24 arch-linux ido2db[18522]: libdbi: Failed to load driver: /usr/lib/dbd/libdbdsqlite.so May 17 20:20:24 arch-linux ido2db[18522]: libsqlite3.so.0: cannot open shared object file: No such file or directory May 17 20:20:24 arch-linux ido2db[18522]: libdbi: Failed to load driver: /usr/lib/dbd/libdbdsqlite3.so May 17 20:20:24 arch-linux ido2db[18522]: libpq.so.5: cannot open shared object file: No such file or directory May 17 20:20:24 arch-linux ido2db[18522]: libdbi: Failed to load driver: /usr/lib/dbd/libdbdpgsql.so May 17 20:20:24 arch-linux ido2db[18522]: Successfully connected to mysql database May 17 20:20:24 arch-linux ido2db[18522]: Successfully disconnected from mysql database

During system startup the ido2db service should be started after mysqld service. To guarantee this we can edit the ido2db.service definition in the folder /etc/systemd/system/multi-user.target.wants.

[Unit] Description=Icinga Data Out Utilities (IDOUtils) Requires=mysqld.service After=syslog.target network.target mysqld.service

• Enable and start the icinga monitor.

# systemctl enable icinga # systemctl start icinga

Check the status:

# systemctl status icinga * icinga.service – Icinga Open-Source Monitoring System Loaded: loaded (/usr/lib/systemd/system/icinga.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2015-05-17 20:28:08 EEST; 4s ago Process: 18541 ExecStartPre=/usr/bin/icinga $ICINGA_VERIFY_OPTS (code=exited, status=0/SUCCESS) Main PID: 18544 (icinga) CGroup: /system.slice/icinga.service `-18544 /usr/bin/icinga /etc/icinga/icinga.cfg May 17 20:28:09 arch-linux icinga[18544]: Icinga 1.11.7 May 17 20:28:09 arch-linux icinga[18544]: Copyright (c) 2009-2014 Icinga Development Team (http://www.icinga.org) May 17 20:28:09 arch-linux icinga[18544]: Copyright (c) 2009-2013 Nagios Core Development Team and Community Contributors May 17 20:28:09 arch-linux icinga[18544]: Copyright (c) 1999-2009 Ethan Galstad May 17 20:28:09 arch-linux icinga[18544]: Last Modified: 09-03-2014 May 17 20:28:09 arch-linux icinga[18544]: License: GPL May 17 20:28:09 arch-linux icinga[18544]: Warning: config setting ‘event_profiling_enabled’ unknown. Remove it from y…ation! May 17 20:28:09 arch-linux icinga[18544]: Icinga 1.11.7 starting… (PID=18544) May 17 20:28:09 arch-linux icinga[18544]: Local time is Sun May 17 20:28:08 EEST 2015 May 17 20:28:09 arch-linux icinga[18544]: Event loop started..

Looks like there is one obsolete configuration setting (event_profiling_enabled) in the icinga configuration file /etc/icinga/icinga.cfg. Let’s comment that out.

Before proceeding let’s also change the owner for the icinga command file to be the web server user (http:http). This makes it possible to change the icinga settings from the web interface.

# chown http:http /var/spool/icinga/rw/icinga.cmd

Also we need to allow ordinary users to execute the ping command so that icinga can use it for detecting whether hosts are up.

# chmod u+s /usr/bin/ping

• Enable and start the web server

# systemctl enable nginx # systemctl start nginx

Check the status:

# systemctl status nginx * nginx.service – A high performance web server and a reverse proxy server Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2015-05-17 20:41:46 EEST; 2min 57s ago Process: 18719 ExecStart=/usr/bin/nginx -g pid /run/nginx.pid; error_log stderr; (code=exited, status=0/SUCCESS) Main PID: 18720 (nginx) CGroup: /system.slice/nginx.service |-18720 nginx: master process /usr/bin/nginx -g pid /run/nginx.pid; error_log stderr; `-18721 nginx: worker process May 17 20:41:45 arch-linux systemd[1]: Starting A high performance web server and a reverse proxy server… May 17 20:41:45 arch-linux systemd[1]: PID file /run/nginx.pid not readable (yet?) after start. May 17 20:41:46 arch-linux systemd[1]: Started A high performance web server and a reverse proxy server.

I can now open the Icinga status page by pointing my web browser to the address http://arch-linux/icinga/.

To get nice equipment icons into the status map you need to edit the host definition e.g. /etc/icinga/objects/localhost.cfg and add the icon names. There are a lot of ready made icons in the folder /usr/share/webapps/icinga/images/logos. I’ll use the computer icon found in the equipment sub-folder.

define host{         use   linux-server      ; Name of host template to use                                 ; This host definition will inherit all variables that are defined                                 ; in (or inherited by) the linux-server host template definition.         host_name     localhost         alias         localhost         address       127.0.0.1         icon_image    equipment/computer.png         statusmap_image equipment/computer.gd2         }

There is also a fancier web user interface available called icinga-web. It is based on PHP with a database backend. However, I think the classic web interface is quite ok for my purposes.