Notification destinations
Where an alert is delivered, and how to route it to the right one.
What a destination is #
A destination is a place an alert can be delivered: a channel, an on-call rota, or a system of your own. Each one is configured once with its credentials, given a name, and then referred to by that name.
The kinds available #
- Slack — post to a channel through an incoming webhook.
- Microsoft Teams — post an Adaptive Card through a channel workflow.
- Google Chat — post to a space through an incoming webhook.
- Discord — post an embed to a channel through a channel webhook.
- Mattermost — post to a channel through an incoming webhook.
- PagerDuty — open an incident on trigger and resolve it on recovery.
- Opsgenie — create an alert on trigger and close it on recovery, in either region.
- Webhook — any endpoint you choose, with your own method, headers, authentication and payload.
The two incident-response destinations behave differently from the chat ones on purpose. An incident is opened when the monitor fires and resolved when it recovers, so a flapping monitor opens and closes incidents rather than leaving one open. Opsgenie additionally keys the alert by monitor and by group, so two groups failing on one monitor arrive as two alerts.
The webhook destination #
The webhook is the one that takes real configuration, and the one worth knowing how to set up.
- Method — POST, PUT or PATCH.
- Payload template — your own JSON, with the alert's fields available as variables. Leaving it empty sends the standard event.
- Headers — a JSON object, sealed at rest like any other credential.
- Authentication — none, basic or bearer.
- Encoding — send the payload as a form rather than JSON.
Every webhook delivery is signed. The signature is an HMAC over the body, sent alongside it in a header, with the version prefix in the value so the scheme can change later. Your endpoint can verify the body it received rather than trusting that it came from us.
Two ways to route #
An alert reaches a destination in one of two ways, and they combine.
- Naming a destination in the message — put
@namein the monitor's message and it is delivered there. The simplest arrangement, and the one to start with. - A workflow — match on the alert itself and route by what it is, rather than repeating the destination on every monitor that needs it.
Workflows #
A workflow says which alerts go where. It matches on the state of the alert, its priority and its tags, and every filter you set has to hold for it to fire.
- State — alert, warning, recovered or no data.
- Priority — any of the priority levels, or leave it empty to match all of them.
- Tags — all of the tags you name must be present.
- Destinations — up to 20.
The tags an alert carries are the monitor's own tags plus its service, environment and team, which is what makes routing by ownership work without any extra tagging.
Which one wins
A workflow that matches takes precedence over a name written in the message. Where both would deliver, the same destination appears once.
The common arrangement is one workflow for the whole team on low-priority alerts, and one that routes high-priority alerts to an on-call rota. You then stop naming destinations in monitor messages entirely, and every monitor inherits the routing.
Delivery #
Delivery is at-least-once, so a destination that has already received an alert may receive it again. A delivery that does not succeed is retried with a widening gap between attempts, and is given up on after an hour. Every delivery is kept for 30 days, whether it succeeded or not, so you can tell an alert that never arrived from one that arrived and was ignored.
Onward #
Once a destination is receiving alerts, Monitors covers what to watch and Monitor quality covers the monitors that are firing without telling you anything.