Setting Up the Development Environment

Note

SecureDrop maintains two versions of documentation: stable and latest. stable is the default used by our Read the Docs site, and is built from our latest signed git tag. latest is built from the head of the main git branch of the securedrop-docs repository. In almost all cases involving development work, you’ll want to make sure you have the latest version selected by using the menu in the bottom left corner of the Read the Docs site.

Overview

SecureDrop is a multi-machine design. To make development and testing easy, we provide a set of virtual environments, each tailored for a specific type of development task. We use Vagrant, VirtualBox, and Docker and our Ansible playbooks can provision these environments on either virtual machines or physical hardware.

Note

SecureDrop is written in Python 3 only.

Quick Start

The Docker based environment is suitable for developing the web application and updating the documentation.

Follow the instructions below to install the requirements for the Docker-based environment for your operating system.

Ubuntu or Debian GNU/Linux

Run the following commands to update the package index and to install Git and make:

sudo apt-get update
sudo apt-get install -y make git

We recommend using the stable version of Docker CE (Community Edition) which can be installed via the official documentation links:

Make sure to follow the post-installation steps for Linux.

Fedora Linux

Note

To install Docker Engine, you need the 64-bit version of Fedora 30 or higher.

Run the following command to update the package index and to install Git and make:

sudo dnf install -y make git

We recommend using the stable version of Docker CE (Community Edition) which can be installed via the official documentation link:

Make sure to follow the post-installation steps for Linux.

macOS

Install Docker.

Qubes

Create a StandaloneVM based on Debian 10, called sd-dev. You can use the Q menu to configure a new VM, or run the following in dom0:

qvm-clone --class StandaloneVM debian-10 sd-dev
qvm-start sd-dev
qvm-sync-appmenus sd-dev

The commands above create a new StandaloneVM, boot it, and then update the Qubes menus with applications within that VM. Open a terminal in sd-dev, and proceed with installing Docker CE for Debian.

Tip

If you experience an error with the aufs-dkms dependency when installing Docker CE, you can safely skip that package using the --no-install-recommends argument for apt.

Fork & Clone the Repository

Now you are ready to get your own copy of the source code. Visit our repository, fork it, and clone it on your local machine.

git clone git@github.com:<your_github_username>/securedrop.git

Note

Pull requests should be opened against the develop branch of our repository, which is the primary branch used for development.

Using the Docker Environment

The Docker based helpers are intended for rapid development on the SecureDrop web application and documentation. They use Docker images that contain all the dependencies required to run the tests, a demo server etc.

Tip

When run for the first time, building Docker images will take a few minutes, even one hour when your Internet connection is not fast. If you are unsure about what happens, you can get a more verbose output by setting the environment variable export DOCKER_BUILD_VERBOSE=true.

The SecureDrop repository is bind mounted into the container and files modified in the container are also modified in the repository. This container has no security hardening or monitoring.

To get started, you can try the following:

cd securedrop
make dev                                               # run development servers
make test                                              # run tests
securedrop/bin/dev-shell bin/run-test tests/functional # functional tests only
securedrop/bin/dev-shell bash                          # shell inside the container

Tip

The interactive shell in the container does not run redis, Xvfb etc. However you can import shell helper functions with source bin/dev-deps and call run_xvfb, maybe_create_config_py etc.

SecureDrop consists of two separate web applications (the Source Interface and the Journalist Interface) that run concurrently. In the development environment they are configured to detect code changes and automatically reload whenever a file is saved. They are made available on your host machine by forwarding the following ports:

You should use Tor Browser to test web application changes, see here for instructions.

A test administrator (journalist) and non-admin user (dellsberg) are created by default when running make dev. In addition, sources and submissions are present. The test users have the following credentials. Note that the password and TOTP secret are the same for both accounts for convenience during development.

  • Username: journalist or dellsberg
  • Password: correct horse battery staple profanity oil chewy
  • TOTP secret: JHCO GO7V CER3 EJ4L

If you need to generate the six digit two-factor code, use the TOTP secret in combination with an authenticator application that implements RFC 6238, such as FreeOTP (Android and iOS) or oathtool (command line tool, multiple platforms). Instead of typing the TOTP code, you can simply scan the following QR code:

../_images/test-users-totp-qrcode.png

You can also generate the two-factor code using the Python interpreter:

>>> import pyotp
>>> pyotp.TOTP('JHCOGO7VCER3EJ4L').now()
u'422038'

Setting Up a Multi-Machine Environment

Note

You do not need this step if you only plan to work on the web application or the documentation.

To get started, you will need to install Vagrant, VirtualBox, Docker, and Ansible on your development workstation.

Ubuntu or Debian GNU/Linux

Note

Tested on: Ubuntu 16.04 and Debian GNU/Linux stretch

sudo apt-get update
sudo apt-get install -y build-essential libssl-dev libffi-dev python3-dev \
    dpkg-dev git linux-headers-$(uname -r) virtualbox

We recommend using the latest stable version of Vagrant, 1.8.5 at the time of this writing, which might be newer than what is in your distro’s package repositories. Older versions of Vagrant has been known to cause problems (GitHub #932, GitHub #1381). If apt-cache policy vagrant says your candidate version is not at least 1.8.5, you should download the current version from the Vagrant Downloads page and then install it.

# If your OS vagrant is recent enough
sudo apt-get install vagrant
# OR this, if you downloaded the deb package.
sudo dpkg -i vagrant.deb

Warning

We do not recommend installing vagrant-cachier. It destroys apt’s state unless the VMs are always shut down/rebooted with Vagrant, which conflicts with the tasks in the Ansible playbooks. The instructions in Vagrantfile that would enable vagrant-cachier are currently commented out.

VirtualBox should be at least version 5.x. See GitHub #1381 for documentation of incompatibility with the older VirtualBox 4.x release series.

Finally, install Ansible so it can be used with Vagrant to automatically provision VMs. We recommend installing Ansible from PyPi with pip to ensure you have the latest stable version.

sudo apt-get install python3-pip

The version of Ansible recommended to provision SecureDrop VMs may not be the same as the version in your distro’s repos, or may at some point flux out of sync. For this reason, and also just as a good general development practice, we recommend using a Python virtual environment to install Ansible and other development-related tooling. Using virtualenvwrapper:

sudo apt-get install virtualenvwrapper
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
mkvirtualenv -p /usr/bin/python3 securedrop

Note

You’ll want to add the command to source virtualenvwrapper.sh to your ~/.bashrc (or whatever your default shell configuration file is) so that the command-line utilities virtualenvwrapper provides are automatically available in the future.

macOS

Install the dependencies for the development environment:

  1. Vagrant
  2. VirtualBox
  3. Ansible
  4. rsync >= 3.1.0

If you use homebrew-cask to manage macOS apps, you can install Vagrant and VirtualBox that way. As for Ansible, we strongly recommend installing it in a virtual environment using virtualenvwrapper and pip, so as not to install the older version we use system-wide. You should create your virtualenv using the Python 3 install on your system. If you are using a different version, the path to virtualenvwrapper.sh will differ. Running pip show virtualenvwrapper should help you find it.

sudo easy_install pip # if you don't already have pip
sudo -H pip install -U virtualenvwrapper --ignore-installed six
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv -p python3 securedrop

Note

You’ll want to add the command to source virtualenvwrapper.sh to your ~/.bashrc (or whatever your default shell configuration file is) so that the command-line utilities virtualenvwrapper provides are automatically available in the future.

The version of rsync installed by default on macOS is extremely out-of-date, as is Apple’s custom. We recommend using Homebrew to install a modern version (3.1.0 or greater): brew install rsync.

Fork & Clone the Repository

Now you are ready to get your own copy of the source code. Visit our repository fork it and clone it on you local machine:

git clone git@github.com:<your_github_username>/securedrop.git

Install Python Requirements

SecureDrop uses many third-party open source packages from the Python community. Ensure your virtualenv is activated and install the packages.

pip install --no-deps --require-hashes -r securedrop/requirements/python3/develop-requirements.txt

Note

You will need to run this everytime new packages are added.

Qubes

To configure a multi-machine evironment in Qubes, follow the Quick Start instructions above to create a standalone VM named sd-dev, then follow the Linux instructions above to install the required packages, omitting Virtualbox.

Then, complete the steps described in Deploying SecureDrop staging instance on Qubes.