Nagios alerts that only wake people up when they have to

Two kinds of reader

Every check result is meant for one of two readers. A person who has to act now, or a report somebody reads later. Monitoring goes wrong when everything goes to the first one. After a month of messages about a disk at 81% and timeouts that cleared on their own, people stop reading. Including the message that mattered.

Our rule is simple. If a notification wakes someone up, there has to be something for that person to do. Everything else goes into a daily summary or the monthly report.

Soft and hard states

Nagios doesn't alert on the first failed check. A failure first puts the service in a soft state and schedules retries. Only after max_check_attempts failures in a row does it become hard, and only then do notifications go out.

define service {
    use                     generic-service
    host_name               web-01
    service_description     HTTPS checkout
    check_command           check_https_url!/checkout/!3!6
    check_interval          1      ; minutes between normal checks
    retry_interval          1      ; minutes between retries while soft
    max_check_attempts      3      ; 3 failures in a row -> hard state
    notification_interval   30     ; repeat every 30 minutes until fixed
    first_notification_delay 0
    notification_options    c,r    ; page on critical and on recovery only
    contact_groups          paging
}

With those numbers, a checkout that's really down reaches someone in about three minutes. A single lost probe reaches nobody.

One group that pages, one that doesn't

Define two contact groups with different notification commands and time periods:

define contactgroup {
    contactgroup_name   paging
    alias               Needs someone now
    members             oncall
}
define contactgroup {
    contactgroup_name   reporting
    alias               Daily summary
    members             ops-digest
}

A disk at 85% goes to reporting as a warning. A disk at 95%, or one that will be full within a day at the current rate, goes to paging as critical. It's the same metric with two thresholds, and two different people read the result.

Dependencies

When a switch, a hypervisor or an uplink goes, every host behind it fails its checks too. Without dependencies that's forty messages about one fault. Tell Nagios which hosts sit behind which, and which services need which:

define host {
    host_name   web-01
    parents     core-switch-01
    ...
}

define servicedependency {
    host_name                       db-01
    service_description             MySQL
    dependent_host_name             web-01
    dependent_service_description   HTTPS checkout
    notification_failure_criteria   w,u,c
    execution_failure_criteria      n
}

When a parent is down, the hosts behind it show as unreachable instead of down. The u flag in the host's notification_options decides whether that pages anyone, and usually it shouldn't. (For services, the same letter means unknown, which catches people out.)

Flapping

A service that keeps changing state, usually a check sitting right on its threshold, can send a stream of messages. Flap detection holds them back while the state keeps changing. We treat flapping as a bug in the check or its threshold. Fix the check, then consider turning flap detection off again for that service, because it can also hide real changes.

Planned work

Before planned work, schedule downtime for the hosts and services involved, with a comment pointing to the change log entry. Nothing pages during the window. Anything still failing when it ends does.

The monthly clean-up

Once a month we go through every message that went to paging and put each one in a box:

The aim is that every page belongs in the first box. The share that does is one of the figures in the monthly report.