Monitors

Monitor types #

• Metric — alert on a metric crossing a threshold.
• Log — alert on a count of matching logs.
• APM — alert on a service's error rate or latency.
• Error tracking — alert on new or recurring errors.
• Process — alert when a process stops reporting on a host.
• Kubernetes — alert on pod state.
• Anomaly — alert when a metric departs from its own baseline.
• Outlier — alert when a host diverges from its peers.
• Forecast — alert when a metric is heading for a threshold.
• Composite — alert on an expression over other monitors.

Building a monitor #

• Name — what this alert is about.
• Type and source — what is being watched.
• Query — the filter or metric selecting the data.
• Aggregation — how the values are combined.
• Warning and critical thresholds — when to notify.
• Window — how long a condition must hold. From 60 seconds to 7 days.
• Priority — 1 to 5, used for routing.
• Delay — how far back to shift the evaluation, up to a day. This is how you alert on something that takes time to become visible.
• Destination — where the alert goes. Name it in the message, or let a workflow route it.

How a monitor is evaluated #

Monitors are evaluated on a fixed schedule, roughly once a minute, and each evaluation takes the last value across the monitor's window and compares it against the thresholds. That has two consequences worth knowing before you tune one.

• A condition shorter than the evaluation interval can pass between evaluations. Short spikes are found with a shorter window or a count-based monitor rather than by checking more often.
• A group that stops reporting becomes a group in the no-data state, rather than quietly staying where it was.

The delay setting shifts the end of the window backwards without changing its length. It exists for signals that are always a little behind the event you care about, such as a metric your reporting pipeline publishes on an interval.

Aggregation #

• count, rate, error_count, error_rate — for logs.
• avg, min, max, sum — for numeric values.
• p50, p75, p90, p95, p99 — latency percentiles.

Thresholds #

A monitor compares its aggregated value against the warning and critical thresholds. The critical threshold is evaluated first, so a value past critical never reports as a warning.

• Comparators: >, >=, <, <=.
• Priority runs from 1 (highest) to 5 (lowest).
• Windows run from 60 seconds to 7 days.

Status #

• OK — the condition is not met.
• Warning — the warning threshold is met.
• Alert — the critical threshold is met.
• No data — nothing has been reported in the window.

A monitor's status is the worst status across the groups it evaluates, so one failing group alerts the whole monitor.

Anomaly and outlier monitors #

These learn from your own data rather than a fixed threshold. An anomaly monitor builds a baseline from recent history and alerts on a departure from it; an outlier monitor compares a host against its peers.

Muting #

Mute a monitor when the alert is known and not actionable — during maintenance, or while a fix is being deployed. Muting is permanent until you unmute.

Downtimes #

A downtime suppresses notifications for the monitors it covers while it is active. Evaluation continues, so the monitor still records what happened.

• Scope — an environment, service, team or monitor name.
• Recurrence — repeat daily or weekly.
• End — when it stops applying.

A downtime covers the monitors matching its scope. Use * to cover everything.

Triggered alerts #

The Triggered view lists every monitor currently in alert, so you have one place to see what is firing. A status change is what creates an entry, so a condition that holds steady stops producing alerts and the row settles.

Onward #

Where an alert goes is Notification destinations. The monitors firing without telling you anything are in Monitor quality, and the place to put the numbers behind them is Dashboards.