Incidents

An incident is a confirmed problem of one aspect of one monitor. It is what alerts are about: every alert belongs to an incident, and the incident keeps a timeline of what happened and every alert sent for it.

Lifecycle

A monitor has at most one open incident per aspect (availability, certificate, domain).

1. Opened. An availability problem opens an incident after Open after failures problems in a row (2 by default, 1 to 10). An expiry condition (a certificate or domain within a threshold, expired, invalid or not registered) opens one at once. The opening sends the condition's event: down, warning, expiring or expired.
2. Escalated. A warning that becomes critical (a slow site that goes down, an expiring certificate that expires) escalates in place and alerts once, the first time the incident reaches critical.
3. Eased to a warning. Critical that becomes warning stays open at warning, sends nothing and keeps its peak severity. A later flap back to critical goes on the timeline, not out as another alert.
4. Expiry threshold crossed. An expiring certificate or domain alerts expiring again at each lower threshold (30, then 14, then 7 days). The timeline records each threshold and whether an alert was queued for it.
5. Reminder sent. While a critical incident is open and nobody has acknowledged it, its event is sent again every Remind while down interval (60 minutes by default; 15 minutes to 24 hours, or never).
6. Acknowledged. Someone took it on; reminders stop.
7. Resolved. After Resolve after successes ok checks in a row (1 by default, 1 to 10) the incident resolves and one recovered alert goes out. An incident also closes, quietly, when the monitor is paused or when a settings change alters what the monitor checks; the resolved entry says so.

Inconclusive checks change nothing. An Alert failed entry marks a delivery that gave up.

Severity and aspect

An incident's severity is its current one; its peak severity is the worst it reached. Lists and the Severity filter read the current severity of an open incident and the peak of a resolved one, so "Critical" finds every incident that ever paged.

The aspect says what the incident is about and picks its words: Availability (the target answering; Down or Slow), Certificate (Expiring or Expired) or Domain (Expiring, Expired, Not registered). An HTTP monitor's certificate and domain incidents come from its expiry watches and never touch its uptime.

Acknowledge

Acknowledge on an open incident's page records who took it on and stops the reminders until it resolves. It changes nothing else: the incident resolves on its own when the checks pass. Acknowledging twice is harmless. The incident shows who acknowledged it and when; lists mark acknowledged incidents with a small icon.

From the CLI: nrc monitor incidents acknowledge <incident>.

The incidents page

Incidents in the sidebar lists the organization's incidents. The summary band at the top shows a headline (anything critical open is red, else a warning amber, else "No open incidents"), chips that filter the list (Critical, Warning, Unacknowledged, Resolved this week) and the last 30 days: the mean time to resolve of the incidents resolved in them, and how many were resolved.

The toolbar has Open, Resolved and All tabs, a search box (monitor name or target), Severity and Type (aspect) filters, an Unacknowledged toggle and, for resolved and all incidents, a Started range (last 24 hours, 7, 30 or 90 days, any time). Open incidents are listed whenever they began. The view lives in the URL.

Open incidents come first, critical before warning, then resolved ones, newest first within each. Each row shows the incident in words ("Down · HTTP 503", "Certificate expires in 4 days"), the monitor, the cause, when it started and how long it lasted or has lasted. A monitor's own Incidents tab shows the same list for that monitor.

An incident's page

The header names the monitor and the aspect, with Open monitor and, while it is open and unacknowledged, Acknowledge. The hero shows what is wrong in a few words (or "Resolved after 2h 14m"), and four figures: Started, Lasted or Ongoing for, Alerts sent (an alert to two channels is one; hover for the count per channel) and Acknowledged (by whom and when).

Under it, a sentence says what is wrong and what to do about it, from what the check found: which names the certificate is for when the hostname doesn't match, what status the server answered and what was expected, how slow it was against the limit, when the domain expires. For a resolved incident it says how it ended: checks passed again, the monitor was paused, or its settings changed.

The Timeline lists every entry, oldest first, each with the alerts it sent as chips: Sent, Retrying (with the attempt number and next try) or Failed (with the provider's error on hover). A run of reminders folds into one row ("Reminder sent 6 times, hourly") that opens to show each. The side cards show the evidence, the check that found the problem (status and response time; or the certificate's names, issuer, validity, hostname match and chain; or the domain's registrar, expiry and nameservers), and the monitor's current state.

Status pages

A monitor's incidents are internal: their summaries, error messages and targets are never published. A status page component that follows the monitor changes status with them, and the page names that component under "Current problems" until a post covers it. Nothing more goes public unless you post it.

When a monitor shows on a status page, its incident's page lists those pages and, with the permission to post, offers Post to status page. That opens the post dialog with fixed public wording, the monitor's component affected and the incident linked, for you to check and send. See Post incidents.

Retention

Resolved incidents are kept for 1 year. Alert deliveries (the chips and the channel delivery log) are kept for 90 days, so an incident open longer than that loses its oldest alert history. Status changes are kept a year as well, so "Down for 4 min, then Up" outlives the raw checks (about 30 days).

From the CLI

nrc monitor incidents list                       # open first, then resolved
nrc monitor incidents list --status open --severity critical
nrc monitor incidents get <incident>             # timeline and alerts sent
nrc monitor incidents acknowledge <incident>
nrc monitor incidents summary                    # open by severity, resolved, time to resolve

Every flag is in the CLI reference.