Backup and restore

A self-hosted Nexus Telemetry Fleet server keeps its certificate authority, every collector’s identity and every reading in one data directory. None of it can be rebuilt if the directory is lost, so it is backed up. Managed fleet servers are backed up by the platform.

Backup contents

A backup is a folder named nexus-fleet- and the UTC time it was taken. It contains 4 items.

Item Contains
fleet.db The registry: organisations, users, the CA certificates, tokens, the deny list and the settings
dek.key The key that decrypts the registry and the private keys
ca/ The certificate authority: its signing keys and certificates, including the certificate chain every collector checks the fleet server against
orgs/ One database per organisation, with its collectors and every reading, and beside each database the key that decrypts it

Everything on disk is encrypted. Without the keys, the databases and the private keys cannot be read.

Warning: a copy of fleet.db on its own restores nothing. Keep the 4 items together, and protect the backup as the data directory is protected. It contains the fleet’s keys.

Backup command

The backup command runs as the fleet server’s own account, with the same data directory, and runs while the fleet server is live. It takes a snapshot of each database, then opens the snapshot and checks that it is intact. Only then does it record the backup as complete.

bash
sudo -u nexus-fleet NF_DATA_DIR=/var/lib/nexus-fleet nf-server backup

A snapshot that fails its check stops the whole backup. The backup is written under backups/ inside the data directory, or under the directory given with --out. Old backups in that folder are not deleted.

Off-host copies

A backup on the same disk as the data is lost with the data. The nf-server package installs a scheduled backup that takes a backup and sends it off the host. It is off until it is configured.

Installed Where
The script /usr/bin/backup-stream-to-r2.sh
The example configuration /etc/nexus-fleet/backup.example.env
The backup timer nf-backup.timer, nightly at 03:00, with up to 15 minutes of random delay
The check timer nf-backup-check.timer, at 09:00 and 21:00. It fails, and Server health reports Problem, when the newest successful copy is older than NF_BACKUP_MAX_AGE_HOURS

Copy the example configuration and make it readable by root only.

bash
sudo cp /etc/nexus-fleet/backup.example.env /etc/nexus-fleet/backup.env && sudo chmod 0600 /etc/nexus-fleet/backup.env
Setting Effect
NF_BACKUP_PASSPHRASE Required. Encrypts every copy. Keep a second copy of the passphrase away from this fleet server
NF_BACKUP_TARGET_DIR A directory on other hardware to copy to. Old copies there are never deleted
CLOUDFLARE_ACCOUNT_ID, NF_BACKUP_R2_BUCKET_NAME, NF_BACKUP_R2_ACCESS_KEY_ID, NF_BACKUP_R2_SECRET_ACCESS_KEY A Cloudflare R2 bucket to copy to. 7 daily, 4 weekly and 3 monthly copies are kept when rclone is installed. Set either the directory or the bucket, not both
NF_BACKUP_MAX_AGE_HOURS How old the newest successful copy may be before the check timer fails. The default is 26 hours, which allows one missed night. Set it in server.env as well, or Server health keeps the 26 hour default

Each run copies one backup as several files, one for the registry and one per organisation. It then fetches each file back and decrypts it, to prove the copy can be restored. Fill in the configuration, then enable both timers.

bash
sudo systemctl enable --now nf-backup.timer nf-backup-check.timer

The Backups check on Server health reads the same record: Not set up until the first run, Good while copies succeed, and Problem once the newest success is older than the limit.

Restore

There is no restore command. The 4 items are put back and the fleet server is started.

bash
sudo systemctl stop nf-server

Copy fleet.db, dek.key, ca/ and orgs/ from the backup into the data directory, replacing what is there. Or point NF_DATA_DIR at the backup folder. Then start the fleet server.

bash
sudo systemctl start nf-server

The fleet server opens the restored registry, loads the certificate authority and resumes. Collectors reconnect on their own. Nothing is re-enrolled. Check that the database opened.

bash
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/ready

200 means it opened. 503 means the database did not open, or the fleet server is still starting. Read the journal before signing in.

Older backups

Readings between the backup and the restore are lost. Collectors continue from the present and do not send the missing readings. After a restore the fleet server allows an hour for collectors to reconnect before it reports one as an outage.

A collector enrolled after the backup is unknown to the restored registry. It enrols again as a new record with the same collector id, and is not counted against the licence again.

A certificate revoked after the backup is not revoked in the restored registry, so revoke it again. Collectors guard against this: the deny list has a version number that only goes up, each collector remembers the highest version it has seen, and a collector refuses a deny list older than that and keeps its own. The fleet server’s version passes the collector’s once the revocation is repeated.

Root key

A backup contains the organisation’s root key only if the key was on the host when the backup was taken. A root key kept offline is restored from its own sealed export. See Root key.

← Hardening Server updates →