Upgrade from 0.5.x to 0.6.x

Updating the Tails Workstations

All Tails drives should be updated to Tails 3.6, released concurrently with SecureDrop 0.6. For the Secure Viewing Station, Admin Workstation and Journalist Workstation, you need to manually upgrade as explained in the Tails documentation.

For the Journalist Workstations and the Admin Workstation, you will also need to update the SecureDrop code using the following manual method:

cd ~/Persistent/securedrop
git fetch --tags
git checkout 0.6
gpg --recv-key "2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77"
git tag -v 0.6 # Output should include "Good signature"
./securedrop-admin setup

From this point forward, you will be able to use the command securedrop-admin check_for_updates to check whether updates are available, and securedrop-admin update to check for updates and to apply them in a single step.

Troubleshooting Linux Kernel Issues

The latest Linux kernel (version 4.4.115) will be automatically installed on your SecureDrop servers within 24 hours of the 0.6 release (13th of March at 22:00 UTC). If you are using hardware other than our official hardware recommendations, you face moderate risk of a SecureDrop outage.

If you are experiencing an outage as part of the 0.6 release (e.g. you cannot reach the source interface, you cannot SSH into the servers etc.), you will have to perform the steps outlined below.

First, you need to physically access each server. Power down the server (safely if possible), attach required peripherals (keyboard, monitor), and power the server back up. If you did not configure Google Authenticator for console access, you will need to use single user mode in order to login to each server.

Boot into Single-User Mode

To access single user mode, you will have to edit the boot options for the new kernel. You can do so using the GRUB bootloader, pictured below:

GRUB in default state

Press any key quickly just once. You will only have about 2 to 3 seconds before Ubuntu starts booting. If you miss that window, just log in normally and reboot safely, provided you can log in. Do not unplug or forcibly shut down the server.

Once you hit a key, you will be able to interact with the menu with the up (⬆) and down (⬇) keys. Select “Ubuntu” as shown above, and press “e” to edit the boot options. In the line that begins “linux”, replace the word “quiet” with “single”. Note that the word “quiet” may be wrapped, as in the screenshot below:

GRUB in edit mode

Press the “F10” key to boot.

Test the New Kernel

Observe the boot process. It is possible that the system will fail to boot completely; if so, the log information will help us to understand what is happening.

Provided that you can log in, check if you have network access. Try a command such as sudo host freedom.press. If you don’t have network access, it is most likely due to the upgraded kernel missing a network driver for your hardware.

If everything appears to be operating normally, the outage may not be kernel-related. In that case, please still follow the steps at the end of this document to send us log information along with an issue report, and we will help you investigate.

If you are experiencing network issues or other kernel problems, we recommend that you roll back to an older kernel, and that you report the issue to us immediately.

Compare the Behavior of the Old Kernel

Reboot the server in a safe way with sudo reboot. After the BIOS screen, you can select a new kernel from the GRUB boot menu by selecting Advanced Options for Ubuntu, pictured below.

GRUB with advanced options selected

Under Advanced Options, choose the option with kernel version 3.14.x-grsec. As before, you may need to edit the kernel options to enter single user mode. The boot process should proceed normally. Wait until you get a login prompt and log in.

Once you are logged in, check to see if you have network access. If you do, then your instance is having an issue with the 4.x kernel. In that case, we need to temporarily set an older kernel as the default.

Roll Back to the Old Kernel

Important

The older kernel has reached end of life, so it is of critical importance for the near term security of your instance that we work together to resolve any compatibility issues. Rolling back to an older version is only a stopgap measure to avoid a prolonged outage of your SecureDrop instance.

Inspect the file /boot/grub/grub.cfg. You should find a menuentry line with the same text that you selected during boot, e.g.:

submenu 'Advanced options for Ubuntu'…

  menuentry 'Ubuntu, with Linux 3.14.xx-grsec…

Take note of its position among the other submenu entries (it will most likely be third). Then edit the GRUB configuration:

sudo vim /etc/default/grub

Make a backup of the file or take a note of the current value of GRUB_DEFAULT somewhere, so you can restore the previous behavior easily at a later point.

Once you have done so, set the GRUB_DEFAULT variable to point to the index of the menu and submenu. Note that the index starts at 0, so for a typical setup, the line in /etc/default/grub would look like this:

GRUB_DEFAULT=”1>2”

The “1” means the second entry of the main menu (“Advanced options”), the “2” means the third entry of the submenu. Again, update these numbers consistent with your configuration.

Caution

Ensure that you have chosen the right index for the main menu and the submenu, and double-check that you are beginning the count at 0, not 1; otherwise, you may boot into the wrong kernel.

This change still has to be applied to take effect on the next boot:

sudo update-grub2

Now you can reboot into the old, working kernel.

sudo reboot

The server should come up automatically. From here on, you should be able to perform all administrative tasks via SSH as usual. If you want additional confirmation of the kernel version, the command uname -r should display 3.14.79-grsec.

Please notify us of the compatibility issue so we can help you resolve it ASAP.

Report Compatibility Issues

If you have encountered issues with the kernel upgrade, it is important that you report them to us so that we may incorporate any necessary changes to our updated kernel, and so that we can work with you to switch back to the new kernel as soon as possible.

Run the following commands via SSH from the Admin Workstation:

cd ~/Persistent/securedrop/
source .venv/bin/activate
cd install_files/ansible-base
ansible all -b -m setup > server-facts.log

Please also send us a copy of /var/log/syslog and /var/log/dmesg for analysis.

You can share server-facts.log, syslog and dmesg with us as follows:

  • If you are a member of our Support Portal, please create a new issue and attach the files to it.
  • Alternatively, email us at securedrop@freedom.press (GPG encrypted) with the subject “SecureDrop kernel facts” and the files attached.

Once we get your information, we can try to provide assistance to resolve compatibility issues.

Getting Support

Should you require further support with your SecureDrop installation or upgrade, we are happy to help!