Monitoring Notifications via Telegram

2016-01-10 (updated 2021-01-06) Admin Checkmk Monitoring Telegram

After a self-inflicted downtime of my server that also happens to handle my email I decided that it’s time to have monitoring notifications delivered out-of-band instead of foolishly trying to send them via the most important resource I monitor.

Since I recently started using Telegram I decided to give it a try for monitoring notifications.

Please note that normal Telegram message contents are unencrypted so they can possibly be read on the Telegram server side!

Telegram Bot

The first step is to create a bot that can send notifications by following the Telegram Bot instructions. By using a Telegram bot you don’t have to use a real Telegram client or reuse your Telegram account. Instead all that has to be done is triggering a REST interface via HTTPS including an authorization token that you receive when creating said bot (all following snippets will use ${TOKEN} as a placeholder for the actual token).

To do all the initial HTTP requests for setup and testing I’ll be using curl. In addition to make the answers coming from the Telegram Bot API a bit more readable for this blog post they will be piped through jq. The latter is of course totally optional.

Where to send?

To send messages to a Telegram contact, the contact first needs to start a chat with the bot. Clicking on the bot link after creation should be enough, it will automatically send a message of /start to the bot.

Afterwards the bot can be queried for updates which will, among other information, provide us with a chat id which is later used to send messages to. To fetch the updates a request to the getUpdates method has to be made.

$ curl --silent "https://api.telegram.org/bot${TOKEN}/getUpdates" | jq
{
  "ok": true,
  "result": [
    {
      "update_id": 909090909,
      "message": {
        "message_id": 39,
        "from": {
          "id": 424242424,
          "first_name": "User"
        },
        "chat": {
          "id": 424242424,
          "first_name": "User",
          "type": "private"
        },
        "date": 1452179894,
        "text": "test"
      }
    }
  ]
}

In this example the chat id to look out for is 424242424.

How to send?

Sending messages to the chat id fetched before works very similar to fetching updates. This time a request to the sendMessage method has to be made. Additionally this request needs to contain the information where to send the message to as well as the actual message text. These should be passed in the request as form data (very similar to the request a browser makes when submitting a standard HTML form). The following call to curl does exactly that.

$ curl --silent \
  --data-urlencode "chat_id=424242424" \
  --data-urlencode "text=Message" \
  "https://api.telegram.org/bot${TOKEN}/sendMessage" | jq
{
  "ok": true,
  "result": {
    "message_id": 41,
    "from": {
      "id": 101010101,
      "first_name": "Name of created bot",
      "username": "name_of_created_bot"
    },
    "chat": {
      "id": 424242424,
      "first_name": "User",
      "type": "private"
    },
    "date": 1452181242
    "text": "Message"
  }
}

And this is actually all there is to sending messages via Telegram, at least as far as commandline access goes.

Telegram notifications in Icinga 2

I’ve been experimenting with monitoring using Icinga 2 for a few days and therefore wanted to try out Telegram notifications for that system as well.

Since the example configuration already contains a flexible setup for sending emails I decided to copy that approach and adapt it to Telegram.

There are several parts to receive Icinga 2 notifications via Telegram:

Shell scripts to send messages for host/service notifications
NotificationCommand entries that call the shell scripts
Notification templates for global properties applied to the above commands
Apply rules to instantiate Notification objects for users or groups the have a Telegram Chat ID set

Icinga 2 shell scripts

The shell scripts are basically a modified version of email-host-notification.sh and email-service-notification.sh that ship with Icinga 2. Instead of calling sendmail they rather call curl and instead of the contact email address they expect the Telegram bot token and the chat id as environment variables.

Here is telegram-host-notification.sh for sending host notifications.

#!/bin/sh
template=$(cat <<TEMPLATE
*$NOTIFICATIONTYPE - $HOSTDISPLAYNAME is $HOSTSTATE*

Notification Type: $NOTIFICATIONTYPE

Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $HOSTSTATE

Date/Time: $LONGDATETIME

Additional Info:
\`$HOSTOUTPUT\`

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
)

/usr/bin/curl --silent --output /dev/null \
    --data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
    --data-urlencode "text=${template}" \
    --data-urlencode "parse_mode=Markdown" \
    "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage"

And telegram-service-notification.sh is of course for sending notifications about service problems.

#!/bin/sh
template=$(cat <<TEMPLATE
*$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE*

Notification Type: $NOTIFICATIONTYPE

Service: $SERVICEDESC
Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $SERVICESTATE

Date/Time: $LONGDATETIME

Additional Info:
\`$SERVICEOUTPUT\`

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
)

/usr/bin/curl --silent --output /dev/null \
    --data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
    --data-urlencode "text=${template}" \
    --data-urlencode "parse_mode=Markdown" \
    "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage"

Icinga 2 Configuration

Since the Telegram bot token is a secret that should not be duplicated or be openly visible, let’s declare a constant which can later be forwarded as an environment variable to the notification shell scripts (replace ${TOKEN} with the one received on bot registration)

const TelegramBotToken = "${TOKEN}"

Next up is the pair of NotificationCommand objects for calling the shell scripts and preparing a suitable environment.

object NotificationCommand "telegram-host-notification" {
  import "plugin-notification-command"

  command = [ SysconfDir + "/icinga2/scripts/telegram-host-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    HOSTSTATE = "$host.state$"
    LONGDATETIME = "$icinga.long_date_time$"
    HOSTOUTPUT = "$host.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"
    TELEGRAM_BOT_TOKEN = TelegramBotToken
    TELEGRAM_CHAT_ID = "$user.vars.telegram_chat_id$"
  }
}

object NotificationCommand "telegram-service-notification" {
  import "plugin-notification-command"

  command = [ SysconfDir + "/icinga2/scripts/telegram-service-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"
    SERVICEDESC = "$service.name$"
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    SERVICESTATE = "$service.state$"
    LONGDATETIME = "$icinga.long_date_time$"
    SERVICEOUTPUT = "$service.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"
    SERVICEDISPLAYNAME = "$service.display_name$"
    TELEGRAM_BOT_TOKEN = TelegramBotToken
    TELEGRAM_CHAT_ID = "$user.vars.telegram_chat_id$"
  }
}

As you can see here, the TelegramBotToken constant is passed as the TELEGRAM_BOT_TOKEN environment variable. That way the token is not openly visible in the process list while sending a notification.

The Telegram chat id in turn is taken from the user that is being notified, more on that later.

Next is to link these NotificationCommand objects to actual Notification objects which are in turn attached to certain hosts.

template Notification "telegram-host-notification" {
  command = "telegram-host-notification"
  period = "24x7"
}

template Notification "telegram-service-notification" {
  command = "telegram-service-notification"
  period = "24x7"
}

apply Notification "telegram-notification" to Host {
  import "telegram-host-notification"

  user_groups = host.vars.notification.telegram.groups
  users = host.vars.notification.telegram.users

  assign where host.vars.notification.telegram
}

apply Notification "telegram-notification" to Service {
  import "telegram-service-notification"

  user_groups = host.vars.notification.telegram.groups
  users = host.vars.notification.telegram.users

  assign where host.vars.notification.telegram
}

Again this is an almost 100% copy of the logic used for email notifications, only the object names and variables have been renamed.

Finally the contacts and hosts that should get notified via Telegram need to be defined. For the hosts I opted to do this in the generic-host template to make it apply to all of my hosts to keep the config DRY:

template Host "generic-host" {
  [...]
  vars.notification["telegram"] = {
    users = [ "icingaadmin" ]
  }
}

And for the individual users it’s just a matter of setting another property

object User "icingaadmin" {
  [...]
  vars.telegram_chat_id = "101010101"
}

Seeing it in action

After reloading the Icinga 2 configuration and manually triggering one of the services from Icinga Web 2, the Telegram client greeted me with a new message coming from the bot.

Icinga 2 Telegram Service Notification

Telegram Notifications in Checkmk

Update 2021-01-06: Checkmk Telegram Notifications is now an actual project available as Checkmk package.

After having tried Icinga 2 I decided to stick to my previous setup of Checkmk which I have been using for the last few years. Icinga 2 has a very powerful and flexible configuration system that is indeed a big improvement over the Nagios configuration but having to write all these rules manually is not really an option for me since for the few hosts and services I monitor I’d have to write all of it manually.

So here we go again, another setup for sending notifications using Telegram, this time for Checkmk. Most of it is configured via the great Multisite and WATO web frontend that comes with Checkmk so less typing but a bit more clicking.

For Checkmk there are a few steps less to end up with Telegram notifications. Also, apart from the first step everything can/should be done in the WATO web frontend, no need to edit configuration files.

Put a notification script into the Checkmk notifications directory
Add a custom attribute for users to set the Telegram chat id
Set attribute for users that should receive notifications
Add a notification rule

Checkmk Notification Script

The notification script is very similar to those used in a standard Nagios or Icinga installation. The environment passed to it may be a bit different but in the end they are very much alike. The basics are even documented.

Since Checkmk is mostly written in Python I opted to do the HTTP requests using the urllib2 Python module, there is no need for curl to send notifications in this case. The script is based on the asciimail notification script that ships with Checkmk, the message formatting and text template are very close to what I wanted to have.

The telegram notification script can be put into PREFIX/share/check_mk/notifications/ (with PREFIX being wherever Checkmk got installed to) and will automatically be available as a notification method in the WATO Notification configuration.

NOTE: The Telegram bot token currently needs to be set at the top of the file, the above linked version only contains a placeholder. Once I have figured out a good place where to configure the token I will probably create a proper Checkmk Package out of it.

Checkmk Custom Attribute

Next up is adding a custom attribute in WATO so the Telegram chat id can be set for every user that should get notified. I chose TELEGRAM_CHAT_ID for the name since that fits in very well with the other environment variables passed to the notification script.

The value of this field will then end up as NOTIFY_CONTACT_TELEGRAM_CHAT_ID in the environment when checking the “Add as Custom Macro” setting and when the Nagios/Icinga core exports custom variables to the environment (this may be disabled by default on an OMD setup!).

Also make sure to check “Editable by Users”, at least my installation did not show any text input at all unless that was checked (maybe a bug?).

The complete User Attribute configuration is visible in the following screenshot.

Checkmk Custom Attribute

Checkmk Notification Rule

To get notified one has to add a rule that triggers the notification script. The actual circumstances when to trigger and whom to actually send a notification is of course different for everybody, since my installation only has one user to notify a rule that triggers for everything and everybody was just fine for me. I simply copied the default email rule and changed the “Notification Method” to “Telegram” and disabled the email rule temporarily.

Checkmk Notification Rule

Seeing it in action (again)

After applying the configuration changes in WATO and manually triggering as service notification from the web frontend, the Telegram client again rang, due to the slightly different text formatting the result of course looks a bit different (plus I dropped the “Perfdata” line later on, some checks return lots of performance data, making the message unnecessarily long).

Checkmk Telegram Service Notification