Troubleshooting

The common faults in Nexus Telemetry Fleet, listed by what the dashboard or a log shows, with the steps for each. Each section ends with the page that covers the subject. A collector is the software at a site that reports its terminal to the fleet server. See Add a collector.

A collector that never appears

  1. Read the collector’s log. Enrolment failed: token expired or exhausted means the token was already used, or has expired. Enrolment failed: invalid or revoked token means the token was revoked. Enrolment refused by the fleet server's licence means the fleet server has no valid licence key.
  2. Open Tokens in the dashboard and read the token’s Status: used or expired confirms the log.
  3. Generate a new token with Add telemetry collector and run the new command on the collector’s host.

See Tokens.

A collector that cannot reach the terminal

The collector runs and the terminal is on the dashboard, but its readings are empty and the collector’s log repeats collector.domain_poll_failed for the terminal.

  1. Read collector.dish_addr at the top of the collector’s log. It records the address the collector reads, http://192.168.100.1:9200 by default.
  2. From the collector’s host, check that 192.168.100.1 answers. The terminal answers at that address whether or not it is in Bypass mode.
  3. Where the host sits behind another router, add a static route on that router for 192.168.100.1/32 through the interface the terminal is connected to. A site network managed by a service provider requires the provider to add it. No port is opened and nothing is exposed to the internet.
  4. Where the terminal is at another address, set NF_STARLINK_ADDR. See Collector configuration.

The log records collector.domain_recovered once the terminal is read again.

Unknown source name

  1. Read the log for Unknown NF_SOURCES entry '...' (known: host, starlink, connection, ping).
  2. Compare NF_SOURCES with those 4 names. A misspelt name stops the collector at startup.
  3. Correct or remove the name and restart the collector. The log then lists the sources it is reading.

See Collector configuration.

A terminal in Outage

  1. Read the cause on the terminal’s card. unreachable means the collector stopped reporting without notice. stopped at the terminal means the collector reported that it was being shut down.
  2. Check power and network at the site, and whether anyone stopped the collector.
  3. Once readings resume, the card shows Stabilising for 5 minutes, then the state returns to Healthy. Nothing is done on the dashboard.

See States.

A terminal reading stale

The Live view of a terminal’s page shows a freshness word beside the state, which says how recent the last reading is. stale means the collector is connected and nothing has arrived for more than 3 times its own reporting interval. The Stale threshold setting, 1 day by default, is the separate point at which the terminal is marked offline.

  1. On the terminal’s Live view, read last seen for when the last reading arrived.
  2. Check power and network at the site, and whether anyone stopped the collector.

While the terminal is past the threshold, the Status view reads Outage, and the List view and the terminal’s own page read Offline.

See States.

A version reading behind

  1. On the List view, the Version column marks a collector that is behind its release channel with an amber behind chip.
  2. On the terminal’s Admin view, Software offers Update to <version>. Press it, then press Update in the panel.
  3. The update can wait. The collector keeps reporting on its current version until it is updated.

See Release channels.

A licence chip, or an enrolment refused with 402

A self-hosted fleet server runs on a licence key. As the key nears expiry and after it, the dashboard shows a licence chip in the top bar.

  1. Read the chip: Licence: 1 day, Licence expired or Licence: read-only.
  2. With the licence read-only, a change on the dashboard is refused with The licence has lapsed and the dashboard is read-only. Apply a renewed licence key on the Licence page to restore changes., and a collector’s enrolment is refused with HTTP status 402.
  3. Open Settings, Licence, paste the new key under Licence key, and press Apply key.

See Licensing.

A certificate that cannot renew

  1. On the Certificates page, find the collector whose Status shows the amber chip offline · cannot self-renew.
  2. Compare its Expires date with today.
  3. Get the collector back online before that date. The fleet server sends the queued renewal when the collector connects.

See Certificates page.

A lost token

  1. The fleet server does not store a token, only a hash of it, so a token cannot be shown again.
  2. Generate another with Add telemetry collector.
  3. Revoke the lost token on Tokens, or leave it to expire.

See Add a collector.

A collector revoked by mistake

  1. Check that the collector is still yours and still running. Recovery is done on its host.
  2. Generate a token with Add telemetry collector.
  3. On the host, enrol again with --force.
bash
sudo nf-collector enrol --server <your-address> --token <token> --force

The collector attaches to its existing record, and its terminal returns to the dashboard with its name and history. Without --force the command refuses, because the collector already has an identity.

See Retire and revoke.

A CA rotation that does not complete

A CA rotation replaces the CA certificate, and each collector moves to the new one when it next connects. The banner counts the collectors that have not moved yet.

  1. On the Certificates page, compare the number in the CA rotation in progress banner with the number of collectors that are offline. If they match, the rotation is waiting for those collectors.
  2. Wait. Each collector moves to the new CA certificate when it connects, and the number drops.
  3. Press Force-complete only for collectors that will never return. It revokes every collector that has not migrated.

See Certificates page.

Gaps in a terminal’s readings

  1. Note where the gap is in the terminal’s charts.
  2. A recent gap on a terminal whose collector was offline is filled in when the collector reconnects. It sends every reading it stored while it was cut off, in blocks, until the gap is filled.
  3. Raw readings older than the Raw readings retention period, 7 days by default, are deleted. The 5-minute, hourly and daily summaries have their own longer periods and remain.

See Timings.

A dashboard unreachable from another host

The fleet server answers only on its own host until NF_LISTEN is set to another address. Before opening it:

  1. Create the first user, so that the dashboard requires a sign-in.
  2. Put TLS or a reverse proxy in front of the dashboard.
  3. Set NF_LISTEN and restart the fleet server.

See Hardening.

← Security FAQ Support →