Backing Up and Restoring Servers

Maintaining regular backups helps guard against data loss and hardware failure. Having a recent backup will allow you to redeploy SecureDrop without changing onion URLs, recreating journalist accounts, or losing previous submissions from sources.

Note

Only the Application Server is backed up and restored, including historical submissions and both Source Interface and Journalist Interface URLs. The Monitor Server needs to be configured from scratch in the event of a hardware migration.

Minimizing Disk Use

Since the backup and restore operations both involve transferring all of your SecureDrop’s stored submissions over Tor, the process can take a long time.

Encouraging journalists to regularly delete older, unneeded submissions from the Journalist Interface will save time and improve reliability when doing backups.

Tip

Although it varies, the average throughput of an onion service is about 3 Mbps, or roughly 90 minutes for 2GB. Plan your backup and restore accordingly.

On the Application Server, open a Terminal via Apps ▸ System Tools ▸ Console on the Admin Workstation and run

ssh app sudo du -sh /var/lib/securedrop/store

Compare the output of this command (which approximates the size of a backup archive) to the amount of free space on your Tails persistent volume via Tails’ Disks utility to ensure you have sufficient space to perform a backup.

If you find you cannot perform a backup or restore due to this constraint, and have already deleted old submissions from the Journalist Interface, contact us through the SecureDrop Support Portal.

Note

Submissions are deleted asynchronously and one at a time, so if you delete a lot of submissions through the Journalist Interface, it may take a while for all of the submissions to actually be deleted. SecureDrop uses shred to securely erase files, which takes significantly more time than normal file deletion. You can monitor the progress of queued deletion jobs by logging in to the Application Server over SSH and running:

sudo journalctl -u securedrop_rqworker

Backing Up

Check Connectivity

Open a Terminal via Apps ▸ System Tools ▸ Console on your Admin Workstation and verify it is able to run Ansible and connect to the SecureDrop servers.

ssh app uptime

If this command fails, see Troubleshooting.

Create the Backup

When you are ready to begin the backup, run

securedrop-admin backup

The backup command will display updates on its progress as the backup is created. Run time will vary depending on connectivity and the number of submissions saved on the Application Server.

When the backup action is complete, the backup will be stored as a compressed archive in ~/.config/securedrop-admin. The filename will begin sd-backup, followed by a timestamp of when the backup was initiated, and ending with .tar.gz. You can find the full path to the backup archive in the output of the backup command.

Warning

The backup file contains sensitive information! It should only be stored on the Admin Workstation, or on a dedicated encrypted backup USB.

Note

When dealing with larger backups, the securedrop-admin backup command may fail with a MemoryError at this stage of the operation: “Fetch the backup tarball back to the Admin Workstation”.

If this happens, a backup was successfully generated, but it is still on the server. Run this command from your ~/.config/securedrop-admin directory to copy the backup your Admin Workstation:

rsync -av --progress --partial app:$(ssh app ls -1rt /tmp/sd-backup* | tail -1) ~/.config/securedrop-admin/

If the transfer fails or is interrupted, you can simply run this command again to resume it.

Note that this method will only work if you have first run the securedrop-admin backup command, and the backup has successfully progressed at least until the “Fetch the backup tarball” stage.

Restoring from a Backup

Prerequisites

To perform a restore, boot into the Admin Workstation and ensure that your .tar.gz backup archive has been copied to ~/.config/securedrop-admin. (If you are using the same Admin Workstation as you did when you took the backup, the archive will already be in place).

If you are restoring data onto an existing instance (for example, for data recovery purposes), see Restoring a Backup on an Existing Instance.

If you are reinstalling SecureDrop and then restoring from a backup (for example, for hardware migration, operating system upgrade, or disaster recovery purposes), see Migrating Using a Backup.

For other data recovery scenarios, see Additional Information or contact Support.

Restoring a Backup on an Existing Instance

To restore an existing instance to a previous state, run the command:

securedrop-admin restore sd-backup-2020-07-22--01-06-25.tar.gz

Make sure to replace sd-backup-2020-07-22--01-06-25.tar.gz with the filename for your backup archive.

This command attempts to restore submissions, source and journalist accounts, and configuration details for the onion services used by the web interfaces and SSH (if configured).

Migrating Using a Backup

Moving a SecureDrop instance to new hardware involves:

Backing up the old instance and preserving configuration and credentials from the Admin Workstation;

Installing SecureDrop on new hardware;

Restoring the backup to the new instance and repairing credentials.

Note

If you need to restore from a backup from an instance configured to use SSH-over-LAN onto an SSH-over-Tor instance, you must either first update the target instance to use SSH-over-LAN or perform a data-only backup. See Data-only Restores for more information.

Note

The instructions below assume that you are using the same Admin Workstation that was used to manage your old instance. If you are using a new Admin Workstation you will need to first install the securedrop-admin package and prerequisites on it. Then you may copy the config directory ~/.config/securedrop-admin and backup archive from the old Admin Workstation to the new workstation (using an encrypted Transfer Device), and proceed with the instructions below.

If you have not already done so, back up the existing installation. The instructions below assume that the backup has been created and renamed sd-backup-old.tar.gz.
Move the existing Admin Workstation SSH configuration out of the way via the Terminal via Apps ▸ System Tools ▸ Console, using the commands:
```
ssh-add -D
find ~/.ssh/ -type f -exec mv {} {}.bak \;
```
Note

You will be generating fresh SSH credentials for the servers, and any other Admin Workstation USBs will have to be provisioned with updated credentials.
Ensure your Admin Workstation is connected to a LAN port on your network firewall, and configure the Admin Workstation’s IP address.
Install Ubuntu 24.04 on the Application and Monitor Servers, following the server setup instructions to install with the correct settings, test connectivity, and set up SSH keys to allow for Admin Workstation access.

Note

You may need to wait approximately 10-15 minutes after installing Ubuntu 24.04 for the servers to become reachable via SSH.
Reinstall SecureDrop on the servers, following the installation instructions. During the configuration stage (securedrop-admin sdconfig), the values will be prepopulated based on the old instance’s configuration, which is still stored in ~/.config/securedrop-admin. Press Enter to accept each value.

Proceed through the installation by running securedrop-admin install then securedrop-admin localconfig. If SSH-over-Tor is configured, run ssh app uptime and ssh mon uptime in the Terminal to verify SSH connectivity.
Restore from the old instance’s backup (e.g. sd-backup-old.tar.gz) using the Terminal command:
```
securedrop-admin restore sd-backup-old.tar.gz
```
The restore task will proceed for some time, removing v2 services if a v2+v3 backup was used.
Synchronize the server and Admin Workstation’s web interface config and authentication keys using the Terminal commands:
```
securedrop-admin install
securedrop-admin localconfig
```
Test the new instance to verify that the web interfaces are available and the servers can be reached via SSH.
If you have migrated to new hardware, ensure your old servers have been decommissioned and/or destroyed by following the relevant sections of our decommissioning documentation.

Repair Additional Admin Workstations

If you have additional Admin Workstation USBs, they will no longer have valid SSH credentials and will need to be repaired. In these steps, the “primary Admin Workstation” is the one which you used to complete the above migration process.

Prepare a fresh LUKS-encrypted USB. You may record the passphrase in your primary Admin Workstation KeePassXC password manager.
Copy the following files from your primary Admin Workstation onto the LUKS-encrypted USB:
- ~/.config/securedrop-admin/tor_v3_keys.json
- ~/.config/securedrop-admin/mon-ssh.auth_private
- ~/.ssh/id_rsa.pub
- ~/.ssh/id_rsa
Note

Alternatively, if you wish to use different SSH credentials for each Admin Workstation, you may do so. In this case, copy only the first two files above to your additional Admin Workstations.

Generate per-machine SSH keys and use a clean LUKS-encrypted USB drive to transfer the public portions of those keys to your primary Admin Workstation, where you will then add them to the servers’ authorized_keys files, as described here. You may also contact Support for assistance.
Boot into each additional Admin Workstation. Set an administration password and unlock the persistent volume on the Tails welcome screen. Once logged in, attach the LUKS-encrypted USB and unlock it.
Ensure that this Admin Workstation is using an up-to-date version of Tails and is running the latest SecureDrop application code, 2.13.0.
As you did with the primary Admin Workstation, archive the existing SSH configuration:
```
ssh-add -D
find ~/.ssh/ -type f -exec mv {} {}.bak \;
```
From the LUKS-encrypted USB, copy ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub to the ~/.ssh/ directory.
From the LUKS-encrypted USB, copy tor_v3_keys.json and mon-ssh.auth_private to the ~/.config/securedrop-admin directory.
In the Terminal, type the following commands:
```
securedrop-admin localconfig
```
Test connectivity to each server by running ssh app uptime and ssh mon uptime.
Once all Admin Workstations have been updated, securely wipe the files on the LUKS-encrypted USB, by right-clicking them in the file manager and selecting Wipe. Then, reformat the device using the Disks utility.

Additional Information

Data-Only Restores

The restore command normally restores both the data and the Tor configuration of an instance, including the .onion URLs for your instance.

You may, however, restore data, such as submissions and journalist and source accounts, without altering an instance’s Tor configuration, with the following command:

securedrop-admin restore --preserve-tor-config sd-backup-2020-07-22--01-06-25.tar.gz

This is a suitable option if you have a backup archive taken from an instance with v2 onion services, and wish to restore it to an instance that is now using v3 onion services.

If you require any assistance with migration or data recovery, please contact Support.