Workflows and webhooks
Route alert changes to the tools your team works in.
Workflows #
A workflow connects monitor state changes to a destination. When a monitor transitions, the workflow runs and delivers to the destination you configured.
{
"id": "checkout-critical",
"name": "Checkout critical alerts",
"enabled": true,
"handle": "@pagerduty-oncall",
"monitorFilter": "service:checkout priority:1",
"onTransition": "alert"
}- Monitor filter — which monitors the workflow covers.
- Destination — where deliveries go.
- Trigger — the transition that starts a delivery.
- Enabled — whether it is currently running.
Delivery #
Deliveries are retried with backoff so a destination that is briefly unavailable does not lose an alert.
- Retries start at 30 seconds and back off to 15 minutes.
- A delivery is given one hour to succeed.
- Delivery history is kept for 30 days.
A rejection is final
A destination that answers with a client error is treated as delivered, because retrying will not change the answer. Rate limiting is the exception and is retried.
Webhook signing #
Webhooks are signed so the receiver can verify the request came from Sentinel and was not modified in transit.
signature = hex(hmac_sha256(secret, timestamp + "." + body))
- `x-sentinel-signature` — the signature, prefixed with its version.
- `x-sentinel-timestamp` — when the delivery was made.
- `x-sentinel-delivery` — a unique id for the delivery.
Verify by recomputing the signature over the timestamp and body, and rejecting a delivery whose timestamp is outside your tolerance window.
Check the timestamp as well as the signature. A signature that verifies but is an hour old is a replay, not a delivery.
Testing a destination #
Send a test delivery before relying on a destination. The test goes to the real endpoint and is recorded beside your real deliveries.