Alerts

An alert is one message about an incident: it opened, it got worse, it crossed another expiry threshold, it is still open, it resolved. Alerts go to alert channels. Which channels a monitor alerts, and for which events, is its routing: the organization's defaults, or routes of its own.

Channels

Alert channels in the sidebar lists every channel with its kind, target, whether it is a default, whether it is enabled and whether it is verified. New channel creates one; an organization can have up to 100 channels.

Voice calls are not offered.

Every channel has two switches in the list:

• Enabled: a disabled channel gets no alerts.
• Default: a default channel gets the alerts of every monitor that uses the organization defaults.

Editing a channel replaces its settings as a whole. A Slack or Discord URL has to be given again (blank keeps the stored one in the dashboard); a webhook keeps its secret unless you rotate it; a new address or phone number has to be verified again. Deleting a channel removes it from every monitor's routes.

Verification

An email address or a phone number gets no alerts until its owner confirms a code. The Verify dialog opens after you create such a channel, and the row shows Verify until it is done:

• Send code sends a 6-digit code by email or text message.
• The code works for 10 minutes and allows 5 attempts.
• Send a new code after one minute. A contact gets at most 5 codes a day, across organizations.
• An organization can have at most 3 channels awaiting verification at a time.

Every code sent by SMS takes one unit of the organization's daily SMS cap.

Who gets alerted

Organization defaults

A monitor without alert settings of its own alerts the organization's default channels with the events down, expiring, expired and recovered. Slow-response warnings are not sent by default. If there is no enabled default channel (an SMS channel counts only once it is verified), every member of the organization gets an email, so a new organization is covered from the first monitor.

Routes per monitor

The Alerts tab of a monitor (also Alert settings in its menu) switches between Organization defaults and Custom for this monitor. Custom shows:

• When
  • Open after failures: problems in a row before an incident opens, 1 to 10. Default 2. Expiry conditions open on the first.
  • Resolve after successes: ok checks in a row before it resolves, 1 to 10. Default 1.
  • Remind while down: Never, or every 15 or 30 minutes, 1, 2, 4 or 12 hours, or 1 day. Default every 1 hour. Reminders repeat a critical incident's own event until the incident is acknowledged or resolved.
• Channels: a matrix with one row per channel and a tick box per event. Only the events the monitor's kind can send are offered: an HTTP monitor offers Down, Slow or warning and Recovered, plus Expiring soon and Expired or invalid when a watch is on; SSL and domain monitors offer the expiry events, recovery and, for SSL, Down for an unreachable host. A channel that isn't verified is marked "not verified, gets no alerts". No ticks at all mutes the monitor. A monitor can have up to 20 routes.

When you create a monitor, Who gets alerted on the create page offers the same choice in short: Organization defaults, or Choose channels, which routes the default events to the channels you tick. Fine-tune on the Alerts tab afterwards.

Events

A warning that becomes critical sends the condition's event once, when the incident first reaches critical. Critical easing to warning sends nothing. See Incidents for the whole lifecycle.

Test sends

Send test in a channel's row menu, or in its delivery log, pushes a test alert through the real delivery path and reports what the provider answered. On a monitor's page, Test notification sends to the channels that monitor alerts: one click with one channel, a menu to pick one or all with several. Unverified channels can't be tested.

A delivered test counts like an alert. Tests and verification codes are rate limited to 20 per user and 60 per organization an hour.

Delivery

Each alert becomes one delivery per channel, so a failing channel retries on its own and a delivered one is never re-sent.

• A failed send is retried with backoff: 1 minute, then 2, 4, 8, 16, 32, and 64 minutes, up to 8 attempts.
• Some failures are final and are not retried: an endpoint answering a 4xx other than 408 or 429, a rejected phone number, no recipients, or a daily cap reached. The failure goes on the incident's timeline as Alert failed.
• Webhook, Slack and Discord sends never follow redirects and only reach public addresses over https.

SMS limits

An SMS is always one segment: the text is rewritten in the GSM-7 alphabet (diacritics stripped, typographic quotes and dashes made ASCII, emoji dropped) and cut to 160 characters so that the link to the dashboard stays whole. Every SMS, alerts, codes and tests alike, takes one unit of the organization's daily cap: 10 a day by default. A send the provider never took gives its unit back. Emails have a daily cap too (500 a day by default).

Delivery log

Click a channel's row (or Delivery log in its menu) for every alert it was given in the last 90 days, newest first: the event, the monitor, a link to the incident, and the status with the attempts, the next try while retrying and, for a failure, the provider's error with URLs, addresses and numbers redacted. Filter by All, Failed, Retrying or Sent, and Send test from the same drawer to check a fix.

The same deliveries appear on each incident's page as chips under the timeline entry that sent them.

Metering

Every sent delivery counts as one notification: email (a member email counts once however many members it reaches), webhook (Slack and Discord included) or SMS. Attempts that fail, retries and dead deliveries count nothing. Verification codes are not billed.

From the CLI

nrc monitor channels create "Ops" --kind email --address [email protected] --default
nrc monitor channels verify "Ops"
nrc monitor channels test "Ops"
nrc monitor channels deliveries list "Ops" --status dead
nrc monitor monitors alerts set "Shop" --route "Ops=down,recovered"

Alert channels have a CI token scope of their own, monitor.channels, which monitor.write does not include. See CLI and the CLI reference.