Alert rules and webhook

An alert is a message Nexus Telemetry Fleet sends by email or to a webhook when a terminal changes state. This page is the reference behind the dashboard’s Alerts page and the alert settings in Settings: every kind of alert, the rules that hold one back, and the exact shape of a webhook post.

# Kinds

An alert is one of 8 kinds. The first 4 are the tick boxes under Alerts in Settings. The fleet server raises the other 4 on its own.

An unticked kind is sent to neither channel.

An offline alert waits 2 minutes before it is sent. A recovered inside those 2 minutes cancels it, and that recovered is not sent either.

# Suppression and reminders

None of them can be changed.

The 15 minutes count from the last alert queued for delivery, not from one held by the cooldown, so a terminal in constant trouble raises one alert every 15 minutes. A different kind, or a different terminal, is never held. A held alert is recorded in the delivery log with the reason. The cooldown never holds a test alert. The hourly cap can.

When several alerts are due on one pass, they are sent as one email. Webhook posts are never combined: there is one post per alert.

# Email

Email is sent through the SendGrid API, one request per delivery, to up to 10 recipients. SMTP is not supported. NF_SENDGRID_API_KEY, or NF_SENDGRID_API_KEY_FILE, holds the key, and NF_ALERT_FROM holds the sender address. Both are read at every send. A running process cannot see a change to its own environment, so only the file named by NF_SENDGRID_API_KEY_FILE can be changed without a restart.

The dashboard link is left out of a test alert, and out of every message when the fleet server has no NF_SERVER_URL. A send that fails is recorded in the delivery log with SendGrid’s reason.

# Webhook

The fleet server sends one JSON post per alert, to one address per organisation.

# Address

The check applies to an IP address in the URL. A host name that resolves to a private address is accepted.

# Signing

A signing secret is optional, up to 128 characters. With one saved, every post has a signature header. With none, the post is sent without one.

A receiver verifies a post by computing the HMAC over the raw body, before parsing it, and comparing the result with the header in constant time. Example post under Alerts in Settings shows a sample post and its signature to test against.

printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET"

The result must equal the 64 hex characters that follow sha256= in the header.

# Delivery

A 4xx response is retried the same as a 5xx. An alert counts as delivered only when every channel accepted it. A retry sends the alert to every channel again, so a webhook that keeps failing means repeated emails.

The delivery log records when the alert was raised, its kind, the terminal, the number of attempts, whether it was held and why, and the error from the last attempt. The error records the response status, and the first 240 characters of a SendGrid error body. It does not record a webhook’s response body, or the signature.

# Payload

The body of a post is one JSON object with 7 fields. Every field is always present and never null. The JSON has no whitespace, and those exact bytes are what the signature covers.

{"at_ms":1756800000000,"collector_id":"c-example","collector_name":"Head office","detail":"","event":"offline","org_id":"org-example","suppression":"none"}

For recovered, suppression is none or maintenance. A silence or an acknowledgement never holds back a recovery.

# One example per kind

{"at_ms":1756800420000,"collector_id":"c-4f21","collector_name":"Depot","detail":"IN_OCEAN","event":"restricted","org_id":"org-acme","suppression":"none"}

detail holds the restriction code the terminal reports.

{"at_ms":1756800900000,"collector_id":"c-4f21","collector_name":"Depot","detail":"active carrier Vodafone NL","event":"on_backup","org_id":"org-acme","suppression":"silenced"}

detail names the network traffic is now on.

{"at_ms":1756803600000,"collector_id":"c-4f21","collector_name":"Depot","detail":"returned to service","event":"recovered","org_id":"org-acme","suppression":"none"}

detail reads returned to service when the collector reconnected, back online when a stale collector reported again, and "" when the terminal’s state returned to healthy.

{"at_ms":1756804200000,"collector_id":"c-9a02","collector_name":"Site 3","detail":"5 state changes in the last 30 minutes; further changes are held until stable","event":"flapping","org_id":"org-acme","suppression":"none"}

{"at_ms":1756806000000,"collector_id":"c-9a02","collector_name":"Site 3","detail":"no state changes for 30 minutes","event":"stable","org_id":"org-acme","suppression":"none"}

{"at_ms":1756818000000,"collector_id":"c-4f21","collector_name":"Depot","detail":"still offline after 4h 0m","event":"reminder","org_id":"org-acme","suppression":"none"}

detail names the state and how long the terminal has been in it.

{"at_ms":1756800000000,"collector_id":"test","collector_name":"test","detail":"Test alert to webhook requested by admin from the dashboard","event":"test","org_id":"org-acme","suppression":"none"}

detail names the user who pressed the button.

# Receiver

A receiver answers 2xx first, then processes the alert. It verifies the signature over the raw body, before parsing the JSON. The example below does both.

node receiver.js

import { createServer } from 'node:http';
import { createHmac, timingSafeEqual } from 'node:crypto';

const SECRET = process.env.WEBHOOK_SECRET;

createServer((req, res) => {
  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    const body = Buffer.concat(chunks);
    if (SECRET) {
      const want = 'sha256=' + createHmac('sha256', SECRET).update(body).digest('hex');
      const got = req.headers['x-nexus-telemetry-signature'] || '';
      const a = Buffer.from(want);
      const b = Buffer.from(got);
      if (a.length !== b.length || !timingSafeEqual(a, b)) {
        res.writeHead(401).end();
        return;
      }
    }
    res.writeHead(200).end();
    const alert = JSON.parse(body.toString('utf8'));
    console.log(alert.event, alert.collector_name, alert.detail);
  });
}).listen(8787, () => console.log('listening on 8787'));

WEBHOOK_SECRET is the receiver’s own variable, not a fleet server setting. The fleet server stores the secret with the organisation.

To test a receiver, save its address under Alerts in Settings and press Send test alert on the Webhook row. The test is sent on the same 30 second pass as a real alert, and the Alerts page records whether it was delivered.

The address http://127.0.0.1:8787 can be saved only on a fleet server with NF_ALLOW_PRIVATE_WEBHOOKS set. See Server configuration.