Log events
The Nexus Telemetry Fleet server and collector write their logs as one JSON
object per line. Every event line has an event field with a fixed
name, and this page lists those names. The names are stable. Log pipelines and alert rules can key on them. A name
is only renamed or removed as a documented change. The names are the
product’s own, and some use older internal words: board for the
dashboard, beat for a summary, verdict for a state. The text on this
page uses the dashboard’s words.
Each event has a level. ERROR is a failure that requires action. WARN
is a fault the product has worked around. Read its action column.
INFO is a lifecycle step, a state change or a heartbeat. DEBUG is
per-reading detail. TRACE is every message sent and received. At the default NF_LOG=info
a healthy fleet server logs its startup, state changes, and an hourly
heartbeat, and little else.
No log ever contains a position, a token, key material, a licence key, or anything that identifies a person at a site. Readings are not logged.
See Hardening for reading the fleet server’s log, and Files and repositories for where each log is.
Process and configuration
| Event | Level | Fires when | Action |
|---|---|---|---|
process.panic |
ERROR | The process crashed. It then exits | The service manager restarts it. Report the location field, the place in the code, to support. This is always a bug |
log.initialised |
INFO | Logging started | None. The line records the log format, level and file |
log.config_fallback |
WARN | NF_LOG, NF_LOG_FORMAT or NF_HEARTBEAT_SECS was invalid, or the log directory could not be used, and the default applied |
Fix the variable or the directory |
config.fallback |
WARN | An invalid configuration variable, NF_POLL_INTERVAL for example, was ignored |
Fix the variable |
server.started |
INFO | The fleet server started | None. The line records the version, the data directory, the addresses it listens on and the number of collectors |
collector.started |
INFO | The collector started | None. The line records the version, the collector id, the data directory and the fleet server address |
collector.dish_addr |
INFO | At startup. The line records the address the collector uses to read the terminal | None. Read it when a terminal never appears on the dashboard |
server.heartbeat |
INFO | Every hour, or every NF_HEARTBEAT_SECS |
None. Alert when no heartbeat has arrived for twice the interval |
collector.heartbeat |
INFO | Every hour, or every NF_HEARTBEAT_SECS |
None. Alert when no heartbeat has arrived for twice the interval |
health.degraded_entered |
WARN | The fleet server became degraded: a health check failed | Investigate the check named in the line. /healthz answers 503 until the check passes |
health.degraded_cleared |
INFO | Every health check passes again | None |
Readings received by the fleet server
| Event | Level | Fires when | Action |
|---|---|---|---|
ingest.bucket_stored |
DEBUG | A 5-minute summary was stored | None |
ingest.bucket_replayed |
DEBUG | A summary arrived that the fleet server already had | None. A collector sends the current summary again each time it connects |
ingest.bucket_rejected_stale |
WARN | A summary arrived for a period the fleet server already holds, with less in it than the stored one | If one collector repeats this more than once a minute, check the host’s clock and local store |
ingest.batch_oversized |
WARN | A collector sent more in one batch than the fleet server accepts | Investigate the collector’s host |
ingest.schema_version_skew |
WARN | A collector’s readings use a different schema version from the fleet server’s. They are still read | Update the collector, or the fleet server, so that the versions match |
ingest.samples_dropped |
WARN | Individual readings were dropped because their time was in the future, or more than 10 years in the past | Fix the host’s clock |
ingest.bucket_malformed |
WARN | Summaries were dropped because they were not whole 5-minute periods | Investigate the collector’s host |
ingest.bucket_ancient |
WARN | Whole summaries were dropped because their period was more than 10 years in the past | Fix the host’s clock |
ingest.dwell_over_span |
WARN | A summary recorded more time in one state than the period it covered. The excess was dropped | Investigate the collector’s host |
Collector
| Event | Level | Fires when | Action |
|---|---|---|---|
collector.beat_shipped |
DEBUG | A summary was sent | None |
collector.beat_skipped |
WARN | A whole summary was skipped because one part of it, the terminal, the connection or the location, could not be read | Investigate the part named in the line. The collector tries again at the next interval |
collector.beat_send_failed |
INFO | The connection closed while a summary was being sent. It is sent again after reconnecting | None |
collector.heartbeat_send_failed |
INFO | The connection closed while a heartbeat was being sent | None |
collector.domain_poll_failed |
WARN | One part of the reading, the terminal, the connection or the location, could not be read this interval | If sustained, check that part, for example whether the terminal is reachable |
collector.domain_recovered |
INFO | A part that was failing was read again | None |
collector.event_pushed |
INFO | A state change was sent ahead of the next summary | None |
collector.decommissioned |
INFO | The fleet server completed the collector’s decommission | None |
collector.local_status_started |
INFO | The onboard status page started serving on the site network | None. The line records its address |
collector.local_status_bind_failed |
ERROR | The onboard status page setting was invalid, or its port could not be opened. The page is off | Fix NF_LOCAL_STATUS, or free the port, and restart the collector |
collector.local_status_denied |
WARN | A request to the onboard status page from an address outside the site network was refused | If repeated, something outside the site network can reach the collector. Check the routing and the firewall |
collector.local_status_stopped |
INFO | The onboard status page stopped, because it was turned off or its setting changed | None |
collector.local_status_remote |
INFO | The onboard status page setting arrived from the dashboard | None. applied=false means NF_LOCAL_STATUS on the collector overrides the dashboard |
location.override_loaded |
INFO | At startup, a location source set from the dashboard on an earlier run was read from the data directory | None |
location.reader_started |
INFO | An external GPS source started: serial, NMEA over TCP or UDP, or gpsd | None |
location.serial_detected |
INFO | Serial auto-detect found a port with valid GPS sentences | None |
location.serial_open_failed |
WARN | A serial port could not be opened while looking for a GPS receiver. Logged once for each port | For a permission error, add the service user to the group that owns serial ports, dialout or uucp |
location.override_applied |
INFO | A location source set from the dashboard arrived and was applied | None |
location.locked_ignored |
INFO | A location source arrived from the dashboard and was not applied, because NF_LOCATION_LOCK is set |
Remove the lock if the dashboard is to set the location |
location.dish_backoff |
WARN | The terminal has returned no position for several readings. The collector keeps the last known position | If sustained, the terminal’s Starlink subscription may not include GPS. Set a fixed position or an external source |
location.dish_recovered |
INFO | The terminal returned a position again | None |
Connections, enrolment, sign-in and certificates
| Event | Level | Fires when | Action |
|---|---|---|---|
stream.connected |
DEBUG | A collector connected | None. Not logged at the default level |
stream.closed |
DEBUG | A collector’s connection ended | None |
enrol.accepted |
INFO | A collector enrolled, or a collector enrolled earlier connected again | None |
enrol.ip_cap_override |
INFO | At startup, NF_ENROL_MAX_PER_IP set the enrolment limit |
None |
enrol.ip_cap_invalid |
WARN | NF_ENROL_MAX_PER_IP is not a positive whole number. The default applies |
Fix or unset the variable |
enrol.rejected |
WARN | An enrolment was refused: a bad, expired or used token, a suspended organisation, the rate limit, or the licence. The reason field says which |
Check the source address if one address repeats this. A licence_ reason means the licence requires a key, a renewal or a higher collector limit |
custody.sealed_at_birth |
INFO | A new organisation’s root key was encrypted for the provider’s custody and removed from the fleet server | None |
provision.rejected |
WARN | A request to the provisioning API was refused, for a bad credential or the rate limit | If repeated, check the provisioning secret, or investigate the source address |
org.destroyed |
INFO | An organisation was erased outright, with its database, keys, certificate authority and users. Only on a test fleet server started with the destroy unlock, which a production fleet server never is | None |
provision.org_created |
INFO | An organisation was created through the provisioning API | None |
board.settling |
INFO | The fleet server started and is allowing collectors time to reconnect. A terminal that has not reconnected is in the planned outage state until the time ends | None. Expected on every restart. Not logged with NF_SETTLE_SECS=0 |
board.settling_cleared |
INFO | The reconnection time ended | None |
stream.handshake_shed |
WARN | The fleet server refused a collector’s connection because its limit of connections in one second was reached. The collector retries | Expected after a restart. If sustained at other times, investigate the source, or raise NF_STREAM_HANDSHAKE_MAX on a larger host |
collector.stopping |
INFO | The collector was told to stop and is telling the fleet server before it exits | None. The terminal’s state is stopped, not an outage |
collector.goodbye_sent |
INFO | The stop notice was sent | None |
collector.stopped_announced |
INFO | The fleet server received a collector’s stop notice | None. An outage alert is held while the stop lasts |
licence.refreshed |
INFO | The daily licence check returned a newer licence and applied it | None |
licence.census_sent |
DEBUG | The hourly census for an organisation, its enrolled and active collector counts, was recorded by Nexus Telemetry | None |
licence.census_unknown |
DEBUG | Nexus Telemetry answered the census with a status other than success. The status field says which |
None. The next hour’s census is sent as usual |
licence.census_unreachable |
DEBUG | The census could not reach Nexus Telemetry | None. The next hour’s census is sent as usual |
auth.login_succeeded |
INFO | A user signed in | None |
auth.login_failed |
WARN | A sign-in failed | Check the source address if one address repeats this. The rate limit applies |
auth.login_ambiguous |
WARN | A username matched accounts in more than one organisation, and the user was asked for their organisation’s short name | None. Frequent for one username means the same name is in use in several organisations |
auth.rate_limited |
WARN | The sign-in rate limit refused an attempt | Check the source address |
auth.password_changed |
INFO | A user set their own password. Their other sessions ended | None |
auth.two_factor_challenged |
INFO | A password was accepted for an account with two-factor authentication, and the code is awaited | None |
auth.two_factor_passed |
INFO | The two-factor code, or a recovery code, was accepted | None |
auth.two_factor_failed |
WARN | A two-factor code was refused | None. 5 failures end the sign-in |
auth.two_factor_abandoned |
WARN | A sign-in ended without the two-factor code | None |
auth.two_factor_enrolled |
INFO | A user set up an authenticator app. Their other sessions ended | None |
auth.two_factor_cleared |
INFO | Two-factor authentication was removed from an account, by the user, an administrator or the command line | None |
auth.email_code_sent |
INFO | A verification code was emailed to a user | None |
auth.email_code_echoed |
INFO | A verification code was written to the log instead of sent, because NF_AUTH_EMAIL_ECHO is set or there is no email key |
Set the email key, or unset NF_AUTH_EMAIL_ECHO. A production fleet server must not log codes |
auth.email_send_failed |
WARN | A verification code could not be sent | Check NF_SENDGRID_API_KEY and the sender address |
auth.email_verified |
INFO | A user verified their email address | None |
auth.sign_in_policy_set |
INFO | An organisation’s sign-in policy was saved | None |
demo.refused |
INFO | A visitor to the demo organisation tried a change the demo does not allow | None |
oidc.client_loaded |
INFO | At startup, the Microsoft sign-in registration was read | None |
oidc.client_invalid |
ERROR | The Microsoft sign-in registration is set but its key or certificate could not be read. Microsoft sign-in is off | Fix the files: a PKCS#8 key and a PEM certificate |
oidc.started |
INFO | A user was sent to Microsoft to sign in | None |
oidc.refused |
WARN | A Microsoft sign-in did not complete. The reason field names the check that failed |
If repeated, check the tenant id and the registration’s redirect address |
oidc.no_account |
WARN | A Microsoft identity was verified and matched no account, because no account has a verified email address equal to it | Add the address to the user’s account and have them verify it |
oidc.ambiguous |
WARN | A Microsoft identity matched accounts in more than one organisation, and the user was asked to sign in from their organisation’s own address | None |
oidc.tested |
INFO | An administrator’s test sign-in through Microsoft succeeded, and the registration can be turned on | None |
oidc.connected |
INFO | A Microsoft identity was connected to an account at its first sign-in | None |
oidc.disconnected |
INFO | A user removed the Microsoft identity from their account | None |
pki.renewal_flagged |
INFO | A device certificate is near expiry and its renewal is queued | None |
pki.renewal_completed |
INFO | A device certificate was renewed | None |
pki.stream_cert_renewed |
INFO | The fleet server’s own certificate was renewed | None |
pki.intermediate_promoted |
INFO | The next CA certificate took over from the expiring one | None |
pki.promotion_blocked |
WARN | The current CA certificate is near expiry and there is no next one ready | Restore the root key and extend the signing cover. See Best practice |
pki.runway_low |
WARN | The signing cover kept on the fleet server has dropped below its target. The check runs daily, and the target leaves about a year before the cover runs out | Extend the signing cover |
pki.renewal_window_clipped |
WARN | cert_renewal_window_secs was set to half of cert_lifetime_secs or more, and was reduced to half |
Set the renewal window well below the lifetime. The defaults are a lifetime of 365 days and a window of 30 days |
update.manifest_checked |
INFO | The collector release manifest was read and names a newer version | None |
update.requested |
INFO | A user started an update from the dashboard, for one collector or for all that are behind | None |
update.command_sent |
INFO | The update command was sent to a collector | None |
update.acked |
INFO | The collector acknowledged the update. The detail field says how it will apply. unsupported-platform means it cannot |
For unsupported-platform, update the collector by hand. See Release channels |
collector.update_command |
INFO | The collector received the update command. The outcome field says what it did |
None |
collector.update_applying |
INFO | On macOS, the collector passed the verified package to the installer | None |
collector.update_failed |
WARN | An update failed. The running version is unchanged | Read the error. The update can be started again from the dashboard |
stream.connected and stream.closed are DEBUG because they fire on every
connection and disconnection. At the default level a collector’s connection
and disconnection are recorded in the terminal’s event history instead. To see them in the log, set NF_LOG=debug, or
NF_LOG=nf_server::stream=debug for those two events only.
Alerts, states and maintenance
| Event | Level | Fires when | Action |
|---|---|---|---|
alert.state_changed |
INFO | A terminal’s state change queued an alert | None |
alert.suppressed |
INFO | A maintenance window, a silence or an acknowledgement stopped an alert’s email. The webhook was still sent, with the reason | None |
alert.acknowledged |
INFO | A user acknowledged an alert | None |
alert.silenced |
INFO | A user silenced a terminal’s alerts | None |
alert.delivered |
INFO | An alert was delivered by email or webhook | None |
alert.shaped |
INFO | A noise rule stopped an alert. The rule field says which: the confirmation wait, a recovery for an outage that was never sent, or a terminal changing state repeatedly |
None |
alert.digest_delivered |
INFO | Several alerts due together were sent as one email | None |
alert.capped |
INFO | The organisation reached its hourly limit of alerts. Further alerts were not sent | If frequent, find the terminal that is changing state repeatedly |
alert.reminder |
INFO | A reminder was sent for an alert still open | None |
alert.stable |
INFO | A terminal that was changing state repeatedly has held one state for 30 minutes | None |
alert.delivery_failed |
WARN | An alert could not be delivered. The fleet server retries, or has given up | Check the email key and the webhook address |
verdict.transition |
INFO | A terminal’s state changed, after the change was confirmed | None. Alerts that share a correlation_key are one incident |
maintenance.window_created |
INFO | A user scheduled a maintenance window | None |
maintenance.window_cancelled |
INFO | A user cancelled a maintenance window | None |
fleet.members_updated |
INFO | A user saved a group’s membership. Groups were fleets in an earlier release | None |
weather.fetch |
INFO | A weather forecast was fetched. stored=false means the request failed and the previous forecast was kept |
None |
weather.fetch_failed |
WARN | The weather service refused a request or timed out. Logged with the weather.fetch line for the same request |
Occasional is normal. If persistent, check the fleet server’s internet access and the weather settings |
weather.budget_exhausted |
WARN | The daily limit of weather requests was reached. Forecasts are not refreshed until the next UTC day | Raise NF_WEATHER_DAILY_CAP only with a paid weather key |
board.slow_build |
INFO | Building the dashboard’s view of the fleet took over 250 milliseconds. The fields say which part | Occasional after a restart is normal. If sustained, the host is too small for the fleet. Read db_ms first |
maint.slow_step |
INFO | One step of the fleet server’s maintenance pass took over a second | Occasional after a restart is normal. A step that grows on every pass requires support |
maint.pass_end |
DEBUG | A maintenance pass completed | None. Its absence for many minutes means the pass is stuck. See which maint.slow_step came before |
rollup.truncated |
INFO | The pass that summarises readings reached its 10 second limit and will continue at the next pass | Normal while catching up. If continuous for hours, the host is too small for the fleet |
ping.unavailable |
WARN | The ping command cannot run on the collector’s host, because it is not installed or is not allowed to send ping packets. Every ping target reads as unreachable until fixed | Install the ping tool, or grant it the CAP_NET_RAW capability, or use tcp:// targets in NF_PING_TARGETS |
ping.defaults_icmp |
INFO | At startup, the collector is using its built-in ping targets | On a network that blocks ping, set tcp:// targets in NF_PING_TARGETS |
connection.target_invalid |
WARN | An NF_CONNECTION_TARGETS entry is not host:port and was ignored |
Fix the entry |
connection.targets_override |
INFO | At startup, NF_CONNECTION_TARGETS replaced the built-in targets |
None |
connection.targets_defaulted |
WARN | NF_CONNECTION_TARGETS was set but had no usable entry, and the built-in targets apply |
Fix the list. Every entry must be host:port |
connection.public_ip_unreachable |
WARN | The lookup of the site’s public address failed. The last known result is kept | If the network blocks the built-in lookup services, set NF_CONNECTION_IP_URLS and NF_CONNECTION_ASN_URLS |
connection.uplink_unrecognised |
WARN | A VPN holds the default route, so the collector cannot tell which interface is the internet connection | Set NF_CONNECTION_UPLINK_INTERFACE to the interface |
failover.route_probe_unavailable |
WARN | The collector cannot read the host’s routes, so it cannot detect a change of connection | Install iproute2. Run a Docker collector with --network host |
failover.route_probe_unsupported |
INFO | Reading the host’s routes is not supported on this platform | None. Expected on Windows today |
Live view
| Event | Level | Fires when | Action |
|---|---|---|---|
live.subscribed |
INFO | A user opened a terminal’s Live view | None |
live.unsubscribed |
INFO | A Live view was closed | None |
live.session_ended |
INFO | A Live view was closed because the user’s session ended | None |
live.rejected_over_cap |
WARN | The limit of open Live views was reached, and a new one was refused | If sustained, many users are watching one organisation. A refused view falls back to a refresh every 5 seconds |
live.dropped_lagging |
WARN | A Live view fell behind and caught up from the current state | None |
live.dropped_slow_client |
WARN | A Live view’s browser could not keep up and one update was skipped | None. Readings are not affected |
live.extra_rejected |
WARN | A collector sent a live item, such as a sky map, larger than the fleet server accepts | Investigate the collector’s host |
operator.tool_requested |
INFO | A user pressed Speed test or Reboot | None |
geocode.upstream_failed |
WARN | The place search on the Location setting got no answer from the map service | Check the fleet server’s internet access. Choosing a point on the map still works |
collector.speed_test_started |
INFO | The collector started a speed test | None. The test uses the connection’s full bandwidth for about a minute |
collector.speed_test_finished |
INFO | The speed test ended | None. The result is in the detail field and on the terminal’s page |
collector.reboot_requested |
INFO | The collector received a reboot instruction | The terminal drops and returns within minutes. The collector declines the reboot when the site’s network would go down with the terminal, and the detail field says so |
collector.ping_targets_applied |
INFO | The collector applied the connectivity check targets set on the dashboard | None. persisted=false means the data directory is not writable |
ping.targets_set |
INFO | A user saved the connectivity check targets | None |
collector.live_backfill |
INFO | A user opened a terminal’s Live view, and the collector sent its recent readings to fill the charts | None |
collector.live_backfill_device |
INFO | The terminal’s own recent readings were sent as well | None |
collector.snapshot_shipped |
INFO | The collector sent the terminal’s daily sky map | None. If no map arrives for a day, the collector cannot fetch it from the terminal |