> ## Documentation Index
> Fetch the complete documentation index at: https://docs.novacula.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts

> Watch your fleet with preset conditions and deliver notifications to Slack, Telegram, Teams, or a webhook

Alerts watch your fleet and notify you through your own channels when something needs attention. Open **Alerts** in the sidebar — the page has four tabs, in this order: **Rules**, **Channels**, **Presets**, and **Incidents**.

The pieces fit together like this:

* **Rules** — a preset + a target + a threshold + a channel. A rule is what actually fires.
* **Channels** — where a notification is delivered (Slack, Telegram, Microsoft Teams, or a webhook).
* **Presets** — the ready-made conditions Novacula supports. You pick from these; there's no raw query authoring.
* **Incidents** — the feed of firing and resolved occurrences.

Set them up in the reverse of that order: add a **channel** first, then author a **rule** against a **preset**.

## Delivery channels

On the **Channels** tab, select **New channel**. Give the channel a **name** (it's what rules refer to — for example `Production operations`), pick a **type**, and fill in what that type needs:

| Type                  | Destination field                                    | Credential field                                                  |
| --------------------- | ---------------------------------------------------- | ----------------------------------------------------------------- |
| **Webhook** (default) | —                                                    | **Webhook URL** — the HTTPS endpoint that receives alert payloads |
| **Slack**             | **Slack channel**, e.g. `#operations`                | **Slack API URL**                                                 |
| **Telegram**          | **Telegram chat ID**, numeric, e.g. `-1001234567890` | **Telegram bot token** from BotFather                             |
| **Microsoft Teams**   | —                                                    | **Microsoft Teams webhook URL** (incoming webhook)                |

Credentials are **encrypted at rest and never returned to the browser** — they're write-only. To rotate one, enter a new value; leave it blank to keep the stored credential. The **channel type can't be changed** after creation, and a channel that rules still use can't be deleted until you move or delete those rules — the delete dialog tells you so and the channel's detail view lists the rules holding it.

## Presets

The **Presets** tab lists the conditions the connected Hub supports. Each row shows the preset's **severity**, its **default condition**, its **threshold range** (or *Fixed condition*), the scopes it's **available for**, and how many rules currently use it. The catalogue:

| Preset                     | Severity | Default condition                                                     | Threshold range | Scopes                 |
| -------------------------- | -------- | --------------------------------------------------------------------- | --------------- | ---------------------- |
| **High CPU usage**         | Warning  | Host CPU above **90%** for 2 minutes                                  | 0–100%          | Organization, Executor |
| **Memory pressure**        | Warning  | Host memory used above **90%** for 5 minutes                          | 0–100%          | Organization, Executor |
| **Disk capacity pressure** | Critical | Host disk used above **85%** for 5 minutes                            | 0–100%          | Organization, Executor |
| **Reconcile errors**       | Critical | Executor reconcile errors above **0 / second** over a 5-minute window | 0–1,000,000     | Organization, Executor |
| **Low peer count**         | Warning  | Fewer than **3** connected peers                                      | 0–10,000        | Organization, Node     |
| **Node not synced**        | Warning  | Sync progress below **99%** for 10 minutes                            | 0–100%          | Organization, Node     |
| **Block height stalled**   | Critical | No block height increase for **15 minutes**                           | Fixed condition | Organization, Node     |
| **Node offline**           | Critical | No height telemetry for **1 minute**                                  | Fixed condition | **Node only**          |
| **High RPC error rate**    | Warning  | RPC errors above **5%** of requests for 5 minutes                     | 0–100%          | Organization, Node     |
| **High RPC latency**       | Warning  | p99 RPC latency above **1 s** for 5 minutes                           | 0–3600 s        | Organization, Node     |

Two things to note when planning coverage:

* **Block height stalled** and **Node offline** are **fixed-condition** presets — there's no threshold to tune, only which target they watch.
* **Node offline** is the one preset with no organization scope: you author it **per node**, so a new node isn't covered until you add a rule for it.

Every preset today is **Warning** or **Critical**; the Info severity exists but no current preset uses it.

Authoring is **preset-only** — custom PromQL or LogQL rules and email delivery aren't available.

## Rules

On the **Rules** tab, select **New rule** (or **Create rule** from a preset row, which pre-selects it):

* **What to monitor** — pick a preset. Its severity comes from the preset and isn't editable.
* **Target scope** — **Organization**, **Node**, or **Executor**, then the specific target. Only the scopes that preset allows are offered (see the table above), so picking **Node offline** leaves **Node** as the only choice.
* **Threshold** — keep the preset default or switch on **Custom threshold** and set a value within the preset's range. Fixed-condition presets have no threshold field.
* **Delivery channel** — choose a configured channel. You need at least one: without a channel the form tells you *"A delivery channel is required"* and offers to create one first.
* **Rule enabled** — a disabled rule stays configured but isn't applied. You can also flip this from the row's **Enable rule** / **Disable rule** action later.

A live **Rule preview** shows the effective condition before you save; it tells you what's still missing (a target, a channel, a valid threshold) instead of previewing a half-filled rule.

Each tab has its own search box and filters — rules by status, preset, severity, scope, target, and channel; channels by type, destination, and whether anything uses them; presets by severity, scope, and configurable-vs-fixed.

<Note>
  If a rule references a preset the connected Hub no longer publishes, it's shown as **unsupported** and you must pick a supported preset before the rule can be saved again. Existing rules aren't deleted for you.
</Note>

## What a notification looks like

Slack, Telegram, and Microsoft Teams messages are written to be read by a person, not to reproduce the underlying monitoring payload. Each message covers one rule and one affected target, and carries:

* **A title** with a status marker and the preset's plain-language summary — 🔴 Critical, 🟠 Warning, 🔵 Info while firing, ✅ Resolved once it clears. On Slack the message colour matches the severity.
* **Links back into the UI** — to the rule that fired, to the executor, and to the affected node.
* **The observed value**, formatted to suit the condition: a percentage, a plain number, a duration, or `errors/s`. A preset that fires on missing telemetry reads **No telemetry** instead of a number.
* **The threshold** it crossed, in the same units, including the window and the "for" duration where the preset has one (for example `> 85% for 5 min`). Absence conditions have no threshold and omit the line.
* **Duration** — the UTC time it started while firing; once resolved, how long it lasted plus the start → end window.

A resolved notification keeps the value from the last firing evaluation and marks it **(at trigger)**, so you can tell a historical reading from a current one.

**Webhook** channels are the exception: they receive the full payload, including raw labels and the generator URL, so you can route and process it yourself. Incidents in the UI keep that raw context too — open an incident for its labels and annotations.

## Incidents

The **Incidents** tab is the feed of alert occurrences:

* Each incident is **Firing** or **Resolved** — filter to **All incidents**, **Firing**, or **Resolved** — with **started**, **last seen**, and, once resolved, **ended** times.
* **Acknowledge** a firing incident to signal someone's on it; the row then shows **Acknowledged by** and when.
* Open an incident for its full detail: **Lifecycle**, **Alert context** (severity, status, target, rule, channel), **Technical identity** (incident ID, fingerprint, alert name, created), and the raw **Labels** and **Annotations**. **Open rule** jumps to the rule that produced it.
* The target is resolved from the linked rule where possible; an incident whose target no longer exists reads **Target unavailable**.
* The feed updates by **polling**, and **Load more** pages through history.

An older Hub may not expose the incident feed at all — the tab then says so, and rules and channels keep working.

## Permissions

* **View** alerts, channels, and incidents — any organization member.
* **Create, edit, delete** rules and channels, and **acknowledge** incidents — Owners and Admins.

Changes to rules and channels, and incident acknowledgements, are recorded in the [Audit log](/docs/audit/audit-log).

## Related

* [Node monitoring](/docs/nodes/monitoring) — the metrics these conditions watch.
* [Audit log](/docs/audit/audit-log) — who changed alert configuration, and when.