Backing Up and Restoring Servers

Maintaining regular backups helps guard against data loss and hardware failure. Having a recent backup will allow you to redeploy SecureDrop without changing onion URLs, recreating journalist accounts, or losing previous submissions from sources.

Note

Only the Application Server is backed up and restored, including historical submissions and both Source Interface and Journalist Interface URLs. The Monitor Server needs to be configured from scratch in the event of a hardware migration.

Minimizing Disk Use

Since the backup and restore operations both involve transferring all of your SecureDrop’s stored submissions over Tor, the process can take a long time.

Encouraging journalists to regularly delete older, unneeded submissions from the Journalist Interface will save time and improve reliability when doing backups.

Tip

Although it varies, the average throughput of an onion service is about 3 Mbps, or roughly 90 minutes for 2GB. Plan your backup and restore accordingly.

On the Application Server, open a Terminal on the Admin Workstation and run

ssh app sudo du -sh /var/lib/securedrop/store

Compare the output of this command (which approximates the size of a backup archive) to the amount of free space on your Tails persistent volume via Tails’ Disks utility to ensure you have sufficient space to perform a backup.

If you find you cannot perform a backup or restore due to this constraint, and have already deleted old submissions from the Journalist Interface, contact us through the SecureDrop Support Portal.

Note

Submissions are deleted asynchronously and one at a time, so if you delete a lot of submissions through the Journalist Interface, it may take a while for all of the submissions to actually be deleted. SecureDrop uses shred to securely erase files, which takes significantly more time than normal file deletion. You can monitor the progress of queued deletion jobs by logging in to the Application Server over SSH and running:

sudo journalctl -u securedrop_rqworker

Backing Up

Open a Terminal on the Admin Workstation and cd to ~/Persistent/securedrop.

Run git describe --exact-match to verify that your workstation is running the latest version of SecureDrop, 2.8.0. If not, you should update it before proceeding.

Check Connectivity

Verify that your Admin Workstation is able to run Ansible and connect to the SecureDrop servers.

ssh app uptime

If this command fails, see Troubleshooting.

Create the Backup

When you are ready to begin the backup, run

./securedrop-admin backup

The backup command will display updates on its progress as the backup is created. Run time will vary depending on connectivity and the number of submissions saved on the Application Server.

When the backup action is complete, the backup will be stored as a compressed archive in ~/Persistent/securedrop/install_files/ansible-base. The filename will begin sd-backup, followed by a timestamp of when the backup was initiated, and ending with .tar.gz. You can find the full path to the backup archive in the output of the backup command.

Warning

The backup file contains sensitive information! It should only be stored on the Admin Workstation, or on a dedicated encrypted backup USB.

Note

When dealing with larger backups, the securedrop-admin backup command may fail with a MemoryError at this stage of the operation: “Fetch the backup tarball back to the Admin Workstation”.

If this happens, a backup was successfully generated, but it is still on the server. Run this command from your ~/Persistent/securedrop directory to copy the backup your Admin Workstation:

rsync -av --progress --partial app:$(ssh app ls -1rt /tmp/sd-backup* | tail -1) install_files/ansible-base/

If the transfer fails or is interrupted, you can simply run this command again to resume it.

Note that this method will only work if you have first run the securedrop-admin backup command, and the backup has successfully progressed at least until the “Fetch the backup tarball” stage.

Restoring from a Backup

Prerequisites

To perform a restore, boot into the Admin Workstation and ensure that your .tar.gz backup archive has been copied to ~/Persistent/securedrop/install_files/ansible-base. (If you are using the same Admin Workstation as you did when you took the backup, the archive will already be in place).

If you are restoring data onto an existing instance (for example, for data recovery purposes), see Restoring a Backup on an Existing Instance.

If you are reinstalling SecureDrop and then restoring from a backup (for example, for hardware migration, operating system upgrade, or disaster recovery purposes), see Migrating Using a Backup.

For other data recovery scenarios, see Additional Information or contact Support.

Restoring a Backup on an Existing Instance

To restore an existing instance to a previous state, run the command:

./securedrop-admin restore sd-backup-2020-07-22--01-06-25.tar.gz

Make sure to replace sd-backup-2020-07-22--01-06-25.tar.gz with the filename for your backup archive.

This command attempts to restore submissions, source and journalist accounts, and configuration details for the onion services used by the web interfaces and SSH (if configured).

Note

You cannot restore v2 onion service configurations to a v3-only server (this includes any SecureDrop installation based on Ubuntu 20.04). If a restore operation would leave you without a valid onion service configuration, the restore script will fail with an error. In this case, you can still perform a data-only restore. See Data-only Restores for more information.

Migrating Using a Backup

Moving a SecureDrop instance to new hardware involves:

Backing up the old instance and preserving configuration and credentials from the Admin Workstation;

Installing SecureDrop on new hardware;

Restoring the backup to the new instance and repairing credentials.

SecureDrop now supports only v3 onion services, so the final configuration will only include v3 onion services regardless of the backup state.

Note

If you need to restore from a backup from an instance configured to use SSH-over-LAN onto an SSH-over-Tor instance, you must either first update the target instance to use SSH-over-LAN or perform a data-only backup. See Data-only Restores for more information.

The restore process differs based on the onion services that were configured on the old instance and preserved in the backup:

Migrating using a v2+v3 or v3-only backup

Migrating using a v2-only backup

Migrating Using a V2+V3 or V3-Only Backup

Note

The instructions below assume that you are using the same Admin Workstation that was used to manage your old instance. If you are using a new Admin Workstation you will need to copy the directory ~amnesia/Persistent/securedrop from the old workstation to the new workstation (using a Transfer Device) before proceeding.

If you have not already done so, back up the existing installation. The instructions below assume that the backup has been created and renamed sd-backup-old.tar.gz.
Move the existing Admin Workstation SecureDrop code out of the way, by opening a Terminal via Applications ▸ System Tools ▸ Terminal and running the command:
```
mv ~/Persistent/securedrop ~/Persistent/sd.bak
```
Move the existing Admin Workstation SSH configuration out of the way via the Terminal, using the commands:
```
ssh-add -D
find ~/.ssh/ -type f -exec mv {} {}.bak \;
```
Note

You will be generating fresh SSH credentials for the servers, and any other Admin Workstation USBs will have to be provisioned with updated credentials.
Re-clone the SecureDrop repository to the Admin Workstation using the following Terminal commands:
```
cd ~/Persistent
git clone https://github.com/freedomofpress/securedrop
```
Verify that the current release tag was signed with the release signing key:
```
cd ~/Persistent/securedrop/
git fetch --tags
git tag -v 2.8.0
```
The output should include the following two lines:
```
gpg:                using RSA key 2359E6538C0613E652955E6C188EDD3B7B22E6A3
gpg: Good signature from "SecureDrop Release Signing Key <securedrop-release-key-2021@freedom.press>" [unknown]
```
Important

If you do not see the message above, signature verification has failed and you should not proceed with the installation. If this happens, please contact us at securedrop@freedom.press.

Verify that each character of the fingerprint matches what is on the screen of your workstation. If it does, you can check out the new release:
```
git checkout 2.8.0
```
Important

If you see the warning refname '2.8.0' is ambiguous in the output, we recommend that you contact us immediately at securedrop@freedom.press (GPG encrypted).

Copy the old instance’s configuration files and backup from ~/Persistent/sd.bak into ~/Persistent/securedrop using the following Terminal commands:

cd ~/Persistent/securedrop
export SD_OLD=~/Persistent/sd.bak/install_files/ansible-base
export SD_NEW=~/Persistent/securedrop/install_files/ansible-base
cp $SD_OLD/group_vars/all/site-specific $SD_NEW/group_vars/all/
cp $SD_OLD/tor_v3_keys.json $SD_NEW/
cp $SD_OLD/sd-backup-old.tar.gz $SD_NEW/

You will also need to copy the old instance’s Submission Public Key, OSSEC Alert Public Key, and, if configured, the Journalist Alert Public Key. Assuming that these are named SecureDrop.asc, ossec.asc, and journalist.asc respectively, run the following commands:

cp $SD_OLD/SecureDrop.asc $SD_NEW/
cp $SD_OLD/ossec.asc $SD_NEW/
cp $SD_OLD/journalist.asc $SD_NEW/

(Optional): If your old instance was configured to provide the Source Interface via HTTPS, you should also copy across the certificate, certificate key, and chain file. Assuming that these are named sd.crt, sd.key, and ca.crt respectively, run the following commands:
```
cp $SD_OLD/sd.{crt,key} $SD_NEW/
cp $SD_OLD/ca.crt $SD_NEW/
```
Ensure your Admin Workstation is connected to a LAN port on your network firewall, and configure the Admin Workstation’s IP address.
Install Ubuntu 20.04 on the Application and Monitor Servers, following the server setup instructions to install with the correct settings, test connectivity, and set up SSH keys to allow for Admin Workstation access.

Note

You may need to wait approximately 10-15 minutes after installing Ubuntu 20.04 for the servers to become reachable via SSH.
Reinstall SecureDrop on the servers, following the installation instructions. During the configuration stage (./securedrop-admin sdconfig), the values will be prepopulated based on the old instance’s configuration. Press Enter to accept each value.

Proceed through the installation by running ./securedrop-admin install then ./securedrop-admin tailsconfig. If SSH-over-Tor is configured, run ssh app uptime and ssh mon uptime in the Terminal to verify SSH connectivity.
Restore from the old instance’s backup (e.g. sd-backup-old.tar.gz) using the Terminal command:
```
./securedrop-admin restore sd-backup-old.tar.gz
```
The restore task will proceed for some time, removing v2 services if a v2+v3 backup was used.
Synchronize the server and Admin Workstation’s web interface config and authentication keys using the Terminal commands:
```
./securedrop-admin install
./securedrop-admin tailsconfig
```
Test the new instance to verify that the web interfaces are available and the servers can be reached via SSH.
If you have migrated to new hardware, ensure your old servers have been decommissioned and/or destroyed by following the relevant sections of our decommissioning documentation.

Repair Additional Admin Workstations

If you have additional Admin Workstation USBs, they will no longer have valid SSH credentials and will need to be repaired. In these steps, the “primary Admin Workstation” is the one which you used to complete the above migration process.

Prepare a fresh LUKS-encrypted USB. You may record the passphrase in your primary Admin Workstation KeePassXC password manager.
Copy the following files from your primary Admin Workstation onto the LUKS-encrypted USB:
- ~/Persistent/securedrop/install_files/ansible-base/tor_v3_keys.json
- ~/Persistent/securedrop/install_files/ansible-base/mon-ssh.auth_private
- ~/.ssh/id_rsa.pub
- ~/.ssh/id_rsa
Note

Alternatively, if you wish to use different SSH credentials for each Admin Workstation, you may do so. In this case, copy only the first two files above to your additional Admin Workstations.

Generate per-machine SSH keys and use a clean LUKS-encrypted USB drive to transfer the public portions of those keys to your primary Admin Workstation, where you will then add them to the servers’ authorized_keys files, as described here. You may also contact Support for assistance.
Boot into each additional Admin Workstation. Set an administration password and unlock the persistent volume on the Tails welcome screen. Once logged in, attach the LUKS-encrypted USB and unlock it.
Ensure that this Admin Workstation is using an up-to-date version of Tails and is running the latest SecureDrop application code, 2.8.0.
As you did with the primary Admin Workstation, archive the existing SSH configuration:
```
ssh-add -D
find ~/.ssh/ -type f -exec mv {} {}.bak \;
```
From the LUKS-encrypted USB, copy ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub to the ~/.ssh/ directory.
From the LUKS-encrypted USB, copy tor_v3_keys.json and mon-ssh.auth_private to the ~/Persistent/securedrop/install_files/ansible-base directory.

In the Terminal, type the following commands:

cd ~/Persistent/securedrop
./securedrop-admin tailsconfig

Test connectivity to each server by running ssh app uptime and ssh mon uptime.
Once all Admin Workstations have been updated, securely wipe the files on the LUKS-encrypted USB, by right-clicking them in the file manager and selecting Wipe. Then, reformat the device using the Disks utility.

Migrating Using a V2-Only Backup

V2 onion services are no longer supported by SecureDrop, so v2 Source and Journalist Interface addresses will be replaced by v3 addresses when you perform a migration using a v2-only backup. However, it is possible to migrate submissions, source accounts, and journalist accounts. To do so, follow the steps below:

Note

The instructions below assume that you are using the same Admin Workstation that was used to manage your old instance. If you are using a new Admin Workstation you will need to copy the directory ~amnesia/Persistent/securedrop from the old workstation to the new workstation (using a Transfer Device) before proceeding.

If you have not already done so, back up the existing installation. The instructions below assume that the backup has been created and renamed sd-backup-old.tar.gz.
Move the existing Admin Workstation SecureDrop code out of the way, by opening a Terminal via Applications ▸ System Tools ▸ Terminal and running the command:
```
mv ~/Persistent/securedrop ~/Persistent/sd.bak
```
Move the existing Admin Workstation SSH configuration out of the way via the Terminal, using the commands:
```
ssh-add -D
find ~/.ssh/ -type f -exec mv {} {}.bak \;
```
Reinstall SecureDrop on the Admin Workstation using the following Terminal commands:
```
cd ~/Persistent
git clone https://github.com/freedomofpress/securedrop
```
Verify that the current release tag was signed with the release signing key:
```
cd ~/Persistent/securedrop/
git fetch --tags
git tag -v 2.8.0
```
The output should include the following two lines:
```
gpg:                using RSA key 2359E6538C0613E652955E6C188EDD3B7B22E6A3
gpg: Good signature from "SecureDrop Release Signing Key <securedrop-release-key-2021@freedom.press>" [unknown]
```
Important

If you do not see the message above, signature verification has failed and you should not proceed with the installation. If this happens, please contact us at securedrop@freedom.press.

Verify that each character of the fingerprint matches what is on the screen of your workstation. If it does, you can check out the new release:
```
git checkout 2.8.0
```
Important

If you see the warning refname '2.8.0' is ambiguous in the output, we recommend that you contact us immediately at securedrop@freedom.press (GPG encrypted).
Copy the old instance’s configuration files and backup from ~/Persistent/sd.bak into ~/Persistent/securedrop using the following Terminal commands:
```
cd ~/Persistent/securedrop
export SD_OLD=~/Persistent/sd.bak/install_files/ansible-base
export SD_NEW=~/Persistent/securedrop/install_files/ansible-base
cp $SD_OLD/group_vars/all/site-specific $SD_NEW/group_vars/all/
cp $SD_OLD/sd-backup-old.tar.gz $SD_NEW/
```
You will also need to copy the old instance’s Submission Public Key, OSSEC Alert Public Key, and, if configured, the Journalist Alert Public Key. Assuming that these are named SecureDrop.asc, ossec.asc, and journalist.asc respectively, run the following commands:
```
cp $SD_OLD/SecureDrop.asc $SD_NEW/
cp $SD_OLD/ossec.asc $SD_NEW/
```
Ensure your Admin Workstation is connected to a LAN port on your network firewall, and configure the Admin Workstation’s IP address.
Install Ubuntu 20.04 on the Application and Monitor Servers, following the server setup instructions to install with the correct settings, test connectivity, and set up SSH keys to allow for Admin Workstation access.

Note

You may need to wait approximately 10-15 minutes after installing Ubuntu 20.04 for the servers to become reachable via SSH.
Reinstall SecureDrop on the servers, following the installation instructions. During the configuration stage (./securedrop-admin sdconfig), the values will be prepopulated based on the old instance’s configuration. Press Enter to accept each value.

Note

If your old instance served the Source Interface over HTTPS, you will need to set up your new instance using HTTP instead, and update it to use HTTPS after the initial migration. The web interface addresses change as part of the process, and so your certificate is no longer valid.

Proceed through the installation by running ./securedrop-admin install then ./securedrop-admin tailsconfig. If SSH-over-Tor is configured, run ssh app uptime and ssh mon uptime in the Terminal to verify SSH connectivity and add the new onion URLs to your known_hosts file.
Restore from the old instance’s backup (e.g. sd-backup-old.tar.gz) using the Terminal command:
```
./securedrop-admin restore --preserve-tor-config sd-backup-old.tar.gz
```
The new instance’s onion service addresses will be unchanged, but the old instance’s data and accounts will now be available.
As part of this process, your .onion URLs have changed, and any old Journalist and Admin Workstations will no longer work. To create additional Journalist Workstations, see our onboarding documentation.
If you have migrated to new hardware, ensure your old servers have been decommissioned and/or destroyed by following the relevant sections of our decommissioning documentation.

Additional Information

Data-Only Restores

The restore command normally restores both the data and the Tor configuration of an instance, including the .onion URLs for your instance.

You may, however, restore data, such as submissions and journalist and source accounts, without altering an instance’s Tor configuration, with the following command:

./securedrop-admin restore --preserve-tor-config sd-backup-2020-07-22--01-06-25.tar.gz

This is a suitable option if you have a backup archive taken from an instance with v2 onion services, and wish to restore it to an instance that is now using v3 onion services.

If you require any assistance with migration or data recovery, please contact Support.