DevOps Zone is brought to you in partnership with:

Sys engineer with a passion for finding new ways to use Operations to enable business to move faster and more effectively. Key buzzwords and areas of interest: linux, ruby, cloud, webops, devops, kanban! Joe is a DZone MVB and is not an employee of DZone and has posted 19 posts at DZone. You can read more from them at their website. View Full User Profile

Sensu Monitoring Handler Sets

04.27.2012
| 4220 views |
  • submit to reddit

In past articles we have covered some of basics of Sensu handlers. A nice feature we haven't touched on yet is handler "sets". Handler sets were added around v0.9.2 and can be quite useful for saving time when modifying your handler.

For example, consider you have a standard set of handlers that you assign to most of your checks — pagerduty, irc, campfire. Now, suppose you want to add the GELF (graylog2) handler to all of your monitors as well. If each of your checks is defined as such:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["pagerduty", "irc", "campfire"]
    }
  }
}

… you would need to modify every check's "handlers" attribute to include your new "gelf" handler. If you have a lot of checks this can be a little bit of a burden.

Handler sets save a lot of time in this situation. If you instead define your checks like this:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["default"]
    }
  }
}

And then define your handlers on your Sensu server like the following. Notice we have converted the 'default' handler to a set.

"handlers": {
      "default": {
        "type": "set",
        "handlers": ["pagerduty", "irc", "campfire", "gelf"]
      },
      "pagerduty": {
        "type": "pipe", 
        "command": "/etc/sensu/handlers/pagerduty"
      },
      "irc": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/irc"
      },
      "campfire": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/campfire"
      },
      "gelf": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/gelf.rb"
      }   
    },

You only need to modify the "default" handler set to add the new "gelf" handler and no need to change any of your check definitions.

It's a simple feature, but it can save a bunch of time. You could also setup a handler set specifically for metrics and another set for notifications. In the case of metrics, you could easily ship metrics to multiple systems – Graphite, Librato, Cube, etc. And adding a new system would be as simple as creating the handler and adding it to the handler set. Happy Sensu'ing.

Published at DZone with permission of Joe Miller, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)