Internationalization (i18n)

SecureDrop is translated into a number of languages. We use a web-based collaborative translation platform called Weblate to make it easier. Under the hood, all translation is done using GNU gettext.

With gettext, text to be translated is specially marked in source code. A Python example:

if not (msg or fh):
    flash(gettext("You must enter a message or choose a file to submit."), "error")
    return redirect(url_for('main.lookup'))

In this code, the string You must enter a message or choose a file to submit. can be automatically extracted for translation. The gettext function to which it is passed is used as a marker by pybabel or similar tools to collect the strings to be translated and store them into a .pot file at securedrop/translations/messages.pot. For instance:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr ""

The .pot file serves as a template for all the language-specific .po files, which are where Weblate stores the contributed translations. For each language to be translated, a directory is created, such as securedrop/translations/fr_FR, and populated with a .po file derived from the template. For instance, securedrop/translations/fr_FR/LC_MESSAGES/messages.po is almost identical to securedrop/translations/messages.pot except for the msgstr fields, which will contain the French translations, e.g.:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr "Vous devez saisir un message ou sélectionner un fichier à envoyer."

There’s one last type of file in the gettext system, a machine-readable version of the .po translations called a .mo file. Applications use these to get translations at runtime. The .po files are compiled to .mo files when the SecureDrop package is installed.

The Weblate web application is used to translate strings and relies on gettext behind the scenes. It owns the .pot and .po files. When preparing a SecureDrop release, a pull request is created to pull in all the translations that have been updated in Weblate.

The desktop icons installed on SecureDrop workstations are also translated. The icon templates are in the install_files/ansible-base/roles/tails-config/templates directory. Their labels are collected in the desktop.pot file and translated in the corresponding .po files in the same directory (fr.po, de.po etc.). All translations are merged from the *.j2.in files into the corresponding *.j2 file and committed to the SecureDrop repository. They are then installed when configuring Tails with the tasks/create_desktop_shortcuts.yml tasks.

i18n_tool.py

Most of the work in managing translations within the SecureDrop code base is supported by securedrop/i18n_tool.py. It provides convenient wrappers around pybabel and gettext , and is used to update strings to be translated; pull translations from Weblate; to compile translations before running tests and while packaging SecureDrop.

Development tasks

Add a new language

A user with weblate admin rights must visit the Weblate translation creation page and the Weblate desktop translation creation page to add a new language.

SecureDrop only supports a subset of all the languages being worked on in Weblate: some of them are partially translated or not fully reviewed. The list of fully supported languages is hard-coded in the i18n_tool.py file, in the SUPPORTED_LANGUAGES variable. When a new language is completely translated and reviewed, the i18n_tool.py file must be manually edited to add this new language to the SUPPORTED_LANGUAGES variable.

Update strings to be translated

After strings are modified in the SecureDrop source code, templates or desktop labels, the securedrop/translations/messages.pot files must also be updated. Individual developers should not do this whenever changing strings in the code; the translations are updated in bulk when it’s time to update the Weblate fork.

Translations can be updated with the following command:

$ make translate

This wraps i18n_tool.py translate-messages and i18n_tool.py translate-desktop. These commands update the .pot files for the SecureDrop server code and the desktop icons, as well as the .po files for each language.

Note

The new source strings will only be visible to translators in Weblate after the develop branch is merged into the Weblate fork.

Merge develop into the Weblate fork

Weblate works on a long standing fork of the SecureDrop git repository and is exclusively responsible for the content of the *.pot and *.po files. The content of the develop branch must be merged into the i18n branch to make updated source strings available to Weblate.

Translation must be suspended in Weblate, and any uncommitted changes committed and pushed, to avoid conflicts:

Weblate commit Lock

  • Click Commit.
  • Click Push.
  • And finally, click Lock.

Weblate commit Locked

The develop branch can now be merged into i18n:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add i18n git@github.com:freedomofpress/securedrop-i18n.git
$ git fetch i18n
$ git checkout -b i18n i18n/i18n
$ git merge origin/develop
$ make -C securedrop translate

The translate Makefile target uses the i18n_tool.py command to keep the *.pot and *.po files in sync with the SecureDrop source code. After running make -C securedrop translate, carefully review the output of git diff. Check messages.pot first for updated strings, looking for formatting problems. Then review the messages.po of one existing translation, with a focus on fuzzy translations. There is no need to review other translations because they are processed in the same way. When you are satisfied with the result, it can be merged with:

$ git commit -a -m 'l10n: sync with upstream origin/develop'
$ git push i18n i18n
  • Go to the Weblate commit page for SecureDrop and verify the commit hash matches the last commit of the i18n branch. This must happen instantly after the branch is pushed because Weblate is notified via a webhook. If it is different, ask for help.
  • Click Unlock.

Weblate commit Unlock

Weblate pushes the translations done via the web interface to the develop branch in a fork of the SecureDrop git repository. These commits must be manually cherry-picked and proposed as pull requests for the SecureDrop git repository.

Weblate commit Unlocked

Merge translations back to develop

Weblate automatically pushes the translations done via the web interface as a series of commits to the i18n branch in the Weblate SecureDrop branch, which is a fork of the develop branch of the SecureDrop git repository. These translations need to be submitted back to the develop branch via pull requests. When you create a branch for this, begin its name with i18n-, as that prefix triggers special CI tests for translations.

To fetch the latest translations from the i18n branch into your working copy of the SecureDrop repository, run these commands in your repo root:

$ git checkout -b i18n-merge origin/develop
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-from-weblate
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-docs

You now have the latest translations on your i18n-merge branch.

Note

It is very important to check that each translated string looks like a plausible translation, with no markup. Even if the reviewer does not understand the language, if a translated string looks strange, someone other than the reviewer must be consulted to verify it means something. It is extremely unlikely that a contributor will manipulate a translated string to introduce a vulnerability in SecureDrop, but any suspicious translation should be investigated.

To check the new translations, you’ll need to compile them and verify them by running our automated tests and, ideally, by checking them in the SecureDrop source and journalist interfaces.

Compile translations

At runtime, gettext needs a compiled file for each language (the .mo files). Before you can check the translations in the SecureDrop web interfaces, these need to be created:

$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-messages --compile

For the desktop icons of the source and journalist interfaces, compilation updates their template files with all the translations collected from the .po files.

This can be done by running the following command:

$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-desktop --compile

Verify translations

SecureDrop web interfaces

After a translation is compiled, the web page in which it appears can be verified visually by starting the SecureDrop development servers and navigating via http://localhost:8080 for the source interface or http://localhost:8081 for the journalist interface. You can start the development servers with:

$ make -C securedrop dev

The translations can be checked automatically by running the SecureDrop page layout tests:

$ export PAGE_LAYOUT_LOCALES="en_US,fr_FR"  # may be set to any supported languages
$ make -C securedrop test TESTFILES=tests/pageslayout
[...]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_account_edit_hotp_secret[en_US] PASSED
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_account_edit_hotp_secret[fr_FR] PASSED
[...]

Note

if unset, PAGE_LAYOUT_LOCALES defaults to en_US (US English) and ar (Arabic).

After running the tests, screenshots for each locale are available in securedrop/tests/pageslayout/screenshots/<locale>, e.g. securedrop/tests/pageslayout/screenshots/fr_FR. Screenshot filenames can be found in the tests that created them, in securedrop/tests/pageslayout/test_journalist.py or securedrop/tests/pageslayout/test_source.py.

Desktop icons

The translated templates for the desktop icons are:

  • install_files/ansible-base/roles/tails-config/templates/desktop-journalist-icon.j2
  • install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2

Check that each of them contains a Name line for each of SecureDrop’s supported locales.

Push your branch and create a pull request

After you’ve checked the translations, you’re ready to push your i18n-merge branch and create a pull request to get the translations merged to the SecureDrop develop branch:

$ git commit -m "l10n: compile desktop icons' translations" # if needed
$ git push i18n-merge

Note

The CI job translation-tests will automatically run the above page layout tests in all supported languages on branches named with the prefix i18n-. If you’ve followed that naming convention, the translation tests should soon be run on your pull request.

If you have an abundance of time, you can run all the translation tests locally with:

$ make translation-test

And at long last, you’re done. Go to https://github.com/freedomofpress/securedrop and propose a pull request.

Note

Unlike the SecureDrop application translations, the desktop icon translations are compiled and merged into the repository. They need to be available in their translated form when securedrop-admin tailsconfig is run, because the development environment is not available.

Release Management

Two weeks before the release: string freeze

When features for a new SecureDrop release are frozen, the localization manager for the release will:

Release day

Translator credits

Correct acknowledgment of translators’ contributions is important, so i18n_tool.py makes it easy to list the translators who have helped since the last merge of Weblate translations, with i18n_tool.py list-translators. A list of everyone who has ever contributed translations to SecureDrop can be obtained with i18n_tool.py list-translators --all. There are Makefile targets for these, list-translators and list-all-translators, e.g:

$ make list-all-translators
ar:
  A. Nonymous
  Ahmad Gharbeia
  Ahmed Essam
  Ali Boshanab
[...]

Weblate administration

Note

The privilege escalation workflow is different for code maintainers and translation maintainers.

A translation admin has special permissions on Weblate and the repositories. When someone is willing to become an admin, a thread is started in the translation section of the forum. If there is consensus after a week, the permissions of the new admin are elevated. If there is not yet consensus, a public vote is organized among the current admins.

The privileges of an admin who has not been active for six months or more are revoked, but they can apply again at any time.

The community of SecureDrop translators works very closely with the SecureDrop developers and some of them participate in both groups. However, the translator community has a different set of rules and permissions, and therefore independent policies from SecureDrop itself.

Admin permissions

The full set of admin permissions can be granted at:

Granting reviewer privileges in Weblate

  • Visit https://weblate.securedrop.org/admin/weblate_auth/user/.
  • Click on the user name.
  • In the Groups block:
    • Select Localizationlab in the Available groups list and click on the right arrow to move it to the Chosen groups list.
    • Select Users in the Chosen groups list and click on the left arrow to remove it.

Update the Weblate full text index

Weblate’s full-text index can occasionally get out of sync. When this happens, Weblate’s search may fail to find a word that you know exists in the source strings. You can rebuild the index with:

$ ssh debian@weblate.securedrop.org
$ cd /app/weblate
$ sudo docker-compose run weblate rebuild_index --all --clean

Note that the new index may not be used right away. Some workers may still have the old index open. If the index is holding up translators with a release looming, the server can be rebooted.