Troubleshooting Kernel Updates

Kernel updates address known bugs and security vulnerabilities in the Linux kernel. They may be installed automatically on your Application and Monitor Servers as part of a SecureDrop release. All kernel updates are tested extensively against recommended hardware. If things do go wrong (e.g., the server does not boot after a kernel update), the following instructions will help you to roll back to the previous, working kernel. You can then report compatibility issues to us so we can work together to resolve them as quickly as possible.

First, you need to physically access each server. Power down the server (safely if possible), attach required peripherals (keyboard, monitor), and power the server back up.

If you have access to the password for your admin user, you can use it to log into each server without the use of two-factor authentication, which was disabled for keyboard logins in SecureDrop 0.8.0. You may have saved the password in the KeePassX database on your Admin Workstation. If you do not have the password, you can boot into single user mode instead.

Boot into Single User Mode

To access single user mode, you will have to edit the boot options for the new kernel. You can do so using the GRUB bootloader, pictured below:

GRUB in default state

Press any key quickly just once. You will only have about 2 to 3 seconds before Ubuntu starts booting. If you miss that window, just log in normally and reboot safely, provided you can log in. Do not unplug or forcibly shut down the server.

Once you hit a key, you will be able to interact with the menu with the up (⬆) and down (⬇) keys. Select “Ubuntu” as shown above, and press “e” to edit the boot options. In the line that begins “linux”, replace the word “quiet” with “single”. Note that the word “quiet” may be wrapped. Once you have performed the replacement, the output on your console should look similar to the screenshot below.

GRUB in edit mode

Press the “F10” key to boot.

Test the New Kernel

Observe the boot process. It is possible that the system will fail to boot completely; if so, the log information will help us to understand what is happening.

Provided that you can log in, check if you have network access. Try a command such as sudo host freedom.press. If you don’t have network access, it is most likely due to the upgraded kernel missing a network driver for your hardware.

If everything appears to be operating normally, the outage may not be kernel-related. In that case, you may still wish to follow the steps at the end of this document to send us log information along with an issue report, and we will help you investigate.

If you are experiencing network issues or other kernel problems, we recommend that you roll back to an older kernel, and that you report the issue to us immediately.

Compare the Behavior of the Old Kernel

Reboot the server in a safe way with sudo reboot. After the BIOS screen, you can select a different kernel from the GRUB boot menu by selecting Advanced Options for Ubuntu, pictured below.

GRUB with advanced options selected

The next menu should give you a list of kernels, similar to the one pictured below:

Selecting a specific kernel in GRUB

Choose the option with the previous kernel version. If unsure, please consult the release notes for the most recent release of SecureDrop, which will include details about kernel version changes.

As before, you may need to edit the kernel options to enter single user mode. The boot process should proceed normally. Wait until you get a login prompt and log in.

Once you are logged in, check to see if you have network access. If you do, then your instance is having an issue with the newer kernel. In that case, we need to temporarily set an older kernel as the default.

Roll Back to the Old Kernel

Important

It is of critical importance for the security of your instance that we work together to resolve any compatibility issues. Rolling back to an older version is only a stopgap measure to avoid a prolonged outage of your SecureDrop instance.

Inspect the file /boot/grub/grub.cfg. You should find a menuentry line with the same text that you selected during boot, e.g.:

submenu 'Advanced options for Ubuntu'…

  menuentry 'Ubuntu, with Linux 4.xxx.xx-grsec…

Take note of its position among the other submenu entries (it will most likely be third). Then edit the GRUB configuration:

sudo vim /etc/default/grub

Make a backup of the file or take a note of the current value of GRUB_DEFAULT somewhere, so you can restore the previous behavior easily at a later point.

Once you have done so, set the GRUB_DEFAULT variable to point to the index of the menu and submenu. Note that the index starts at 0, so for a typical setup, the line in /etc/default/grub would look like this:

GRUB_DEFAULT=”1>2”

The “1” means the second entry of the main menu (“Advanced options”), the “2” means the third entry of the submenu. Again, update these numbers consistent with your configuration.

Caution

Ensure that you have chosen the right index for the main menu and the submenu, and double-check that you are beginning the count at 0, not 1; otherwise, you may boot into the wrong kernel.

This change still has to be applied to take effect on the next boot:

sudo update-grub2

Now you can reboot into the old, working kernel.

sudo reboot

The server should come up automatically. From here on, you should be able to perform all administrative tasks via SSH as usual. If you want additional confirmation of the kernel version, the command uname -r should display the expected kernel version number.

Please notify us of the compatibility issue so we can help you resolve it ASAP.

Report Compatibility Issues

If you have encountered issues with a kernel update, it is important that you report them to us so that we may incorporate any necessary changes to our updated kernel, and so that we can work with you to switch back to the new kernel as soon as possible.

Run the following commands via SSH from the Admin Workstation:

cd ~/Persistent/securedrop/
source .venv/bin/activate
cd install_files/ansible-base
ansible all -b -m setup > server-facts.log

Please also send us a copy of /var/log/syslog and /var/log/dmesg for analysis.

You can share server-facts.log, syslog and dmesg with us as follows:

  • If you are a member of our Support Portal, please create a new issue and attach the files to it.
  • Alternatively, email us at securedrop@freedom.press (GPG encrypted) with the subject “SecureDrop kernel facts” and the files attached.

Once we get your information, we can try to provide assistance to resolve compatibility issues.

If you are not a member of our Support Portal, we also encourage you to request help in the SecureDrop Community Forums. Choose carefully what information to disclose publicly. For example, raw logs may contain sensitive information useful to potential attackers.