SecureDrop Workstation Release Management

SecureDrop Workstation code spans across two repositories:

https://github.com/freedomofpress/securedrop-client (Debian packages)
https://github.com/freedomofpress/securedrop-workstation (RPM package)

The components in the Debian packages are all released together, while the workstation RPM package is released independently.

Release a Debian package

Releasing a release candidate (RC) package is the first step before you begin QA or any signing ceremonies. Even when you are releasing a hotfix, RC packages are still recommended for QA purposes.

Production releases will require at least two maintainers, one of which will need access to the SecureDrop release key.

Step 0: Tracking issue

Before beginning the release proces, create a tracking issue titled Release <package name> <version>. It should contain estimated timelines and assignees for release management, QA, and stakeholder communications. Pin the issue for ease of access and visibility.

Step 1: Create a release candidate (RC) tag

Create a release branch named release/<major>.<minor>.<patch>.
Push a commit adding a new changelog entry and incrementing the version.
Push an RC tag in the format <major>.<minor>.<patch>~rcN on your new commit. We will be building from this tag in the next step.

Step 2: Build and deploy the package to `apt-test`

Clone securedrop-client and securedrop-builder.

git clone git@github.com:freedomofpress/securedrop-client.git
git clone git@github.com:freedomofpress/securedrop-builder.git

Check out the newly pushed tag and then build the packages.

cd securedrop-client
git checkout <major>.<minor>.<patch>~rcN
make build-debs

Save and publish build metadata.
Open a PR to https://github.com/freedomofpress/securedrop-apt-test with the packages you want to deploy. Once merged, the packages will be deployed to https://apt-test.freedom.press.

Step 3: Begin QA

You can now start the QA process! If a bug is found, a fix should be developed, merged into the main branch and cherry-picked into the release branch. If desired, release another RC set of packages for further testing.

Once QA testers are satisfied with the packages, you are ready to move on to the next step.

Step 4: Create a release tag

Update the changelog and version. Remove any references to the RC versions from the changelogs.
Generate a release tag named <major>.<minor>.<patch> (same as the previous tags, without the ~rcN part).
Sign the tag with the SecureDrop release key or ask another maintainer to do this and push the signed tag

Step 5: Build and deploy the packages to `apt-qa`

Clone securedrop-client and securedrop-builder.

git clone git@github.com:freedomofpress/securedrop-client.git
git clone git@github.com:freedomofpress/securedrop-builder.git

Check out the newly pushed tag and then build the packages.

cd securedrop-client
git tag -v <major>.<minor>.<patch> # Signed by SecureDrop Release Key
git checkout <major>.<minor>.<patch>
make build-debs

Save and publish build metadata.
Add your packages to a new branch called release in https://github.com/freedomofpress/securedrop-apt-prod.
Update the apt repo distribution files by running ./tools/publish and push those changes to the release branch as well.
Regenerate and sign the apt release file or ask another maintainer to do this. The packages will now be installable from https://apt-qa.freedom.press.
Open a PR to merge the release branch into main.
Another maintainer should also build the packages (following the same steps as earlier) and verify their newly built packages are bit-for-bit identical to those pushed to apt-qa.

Step 6: Perform the `apt-qa` preflight check

First, provision a production workstation from the most recently-released securedrop-workstation-dom0-config production package. Ensure your machine has been updated (either via Qubes native updater or SDW GUI updater).

At minimum, perform the full test. Additional QAers may perform smoketest to save time if there is already full test coverage.

Full test (includes updater)

As root, edit /srv/salt/sd-default-config.yml so that the prod apt_repo_url points to https://apt-qa.freedom.press.
Run the SDW GUI updater. To force an updater run, invoke the updater via /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0.
Start the Client application, and observe the updated version string, indicating the required packages were installed. Perform testing according to the test plan.

Smoketest (no updater run)

Start the Template VMs.
In each template VM, edit /etc/apt/sources.list.d/securedrop_workstation.list file to point to https://apt-qa.freedom.press.
Update the package system and install the new packages via apt update && apt upgrade -y.
Verify that the updated packages were installed in the templates. Shut down template VMs and all VMs associated with SecureDrop Workstation.
Start the Client application and perform testing according to test plan.

Step 7: Deploy the package to `apt-prod`

Merge the release branch into main to deploy your package to https://apt.freedom.press.
Once you see the package land on https://apt.freedom.press, run the updater to install it in a production environment and ensure that it works as expected.

Release an RPM package

Release `securedrop-workstation-dom0-config`

Verify the tag of the project you wish to build: git tag -v VERSION and ensure the tag is signed with the official release key.
git checkout VERSION
Now you are ready to build. Build RPMs following the documentation in an environment sufficient for building production artifacts. For securedrop-workstation you run make dom0-rpm to build the RPM.
sha256sum the built template (and store hash in the build logs/commit message).
Commit the (unsigned) version of this RPM to a branch in the securedrop-yum-prod repository.
Copy the RPM to the signing environment.
Verify integrity of RPM prior to signing (use sha256sums to compare). Note for reviewers: Using rpm --delsign on a signed artifact (for example, a release candidate) in order to verify the checksum of the unsigned .rpm file must be done in the same type of build environment (Linux distribution and rpm version) as the .rpm was built in, or the checksums may not match.
Sign RPM in place (see Signing section below).
Move the signed RPM back to the environment for committing to the lfs repository.
Save and publish build metadata.
Commit the RPM in a second commit on the branch you began above in securedrop-yum-prod. Make a PR.
Upon merge to master, ensure that changes deploy to yum.securedrop.org without issue.

Release `qubes-template-securedrop-workstation`

The SecureDrop workstation template is RPM packaged, and is first deployed to yum-test.securedrop.org before being promoted to production (yum.securedrop.org) using the following procedure:

Verify the tag in the qubes-template-securedrop-workstation repository: git tag -v VERSION and ensure the tag is signed with the official release key.
git checkout VERSION
Rebuild template following documentation in qubes-template-securedrop-workstation.
sha256sum the built template (and store hash in the build logs/commit message).
Commit unsigned template for historical purposes.
Sign template RPM with test key (rpm --resign) (see Signing section below).
Commit signed template.
Push those two commits to a PR in securedrop-yum-test. Make the PR.
Save and publish build metadata.
Upon merge of the PR into securedrop-yum-test, the template will be deployed to yum-test.securedrop.org.
Install the template in dom0 and test it. Provided you’ve run the Salt configurations, find the template via: sudo qubes-dom0-update --action=search qubes-template-securedrop-workstation.
Once template is sufficiently tested, remove test sig: rpm --delsign <file>.
Verify unsigned template sha256sum from build logs/commit message.
Sign template with prod key: rpm --resign <file>
Push commit to a branch in the securedrop-yum-prod repository. Make a PR.
Upon merge to master, ensure that changes deploy to yum.securedrop.org without issue.

Signing procedures

Sign the tag with the SecureDrop release key

If the tag does not already exist, create a new annotated and unsigned tag: git tag -a VERSION.
Output the tag to a file: git cat-file tag VERSION > VERSION.tag.
Copy the tag file into your signing environment and then verify the tag commit hash.
Sign the tag with the SecureDrop release key: gpg --armor --detach-sign VERSION.tag.
Append ASCII-armored signature to tag file (ensure there are no blank lines): cat VERSION.tag.sig >> VERSION.tag.
Move tag file with signature appended back to the release environment.
Delete old unsigned tag: git tag -d VERSION.
Create new signed tag: git mktag < VERSION.tag > .git/refs/tags/VERSION.
Verify the tag’s signature: git tag -v VERSION.
Push the tag to the shared remote: git push origin VERSION.

Regenerate and sign the apt release file

From the release branch containing the new package, update the apt repository distribution files.

git clone https://github.com/freedomofpress/securedrop-apt-prod
cd securedrop-apt-prod
git checkout -b release
./tools/publish

Copy the regenerated file called Release into your signing environment and then verify the hash to ensure the file transfer was successful.
Sign the Release file with the SecureDrop release key.

gpg --armor --detach-sign Release

Copy the Release.gpg file into your release environment and move it to repo/public/dists/<debian-codename>/ on your release branch.
Verify that the release file was signed with the production key.

gpg --verify ./repo/public/dists/<debian-codename>/Release{.gpg,}

Sign the RPM package

The entire RPM must be signed. This process also requires a Fedora machine/VM on which the GPG signing key (either in GPG keyring or in qubes-split-gpg) is setup. You will need to add the public key to RPM for verification (see below).

rpm -Kv indicates if digests and sigs are OK. Before signature it should not return signature, and rpm -qi <file>.rpm will indicate an empty Signature field. Set up your environment (for prod you can use the ~/.rpmmacros example file at the bottom of this section):

sudo dnf install rpm-build rpm-sign  # install required packages
echo "vault" | sudo tee /rw/config/gpg-split-domain  # edit 'vault' as required
cat << EOF > ~/.rpmmacros
%_signature gpg
%_gpg_name <gpg_key_id>
%__gpg /usr/bin/qubes-gpg-client-wrapper
%__gpg_sign_cmd %{__gpg} --no-verbose -u %{_gpg_name} --detach-sign %{__plaintext_filename} --output %{__signature_filename}
EOF

Now we’ll sign the RPM:

rpm --resign <rpm>.rpm  # --addsign would allow us to apply multiple signatures to the RPM
rpm -qi<file.rpm>  # should now show that the file is signed
rpm -Kv  # should contain NOKEY errors in the lines containing Signature
# This is because the (public) key of the RPM signing key is not present,
# and must be added to the RPM client config to verify the signature:
sudo rpm --import <publicKey>.asc
rpm -Kv  # Signature lines will now contain OK instead of NOKEY

You can then proceed with distributing the package, via the “test” or “prod” repo, as appropriate.