Internationalization (i18n)

The code, templates and JavaScript user visible strings must all be wrapped with gettext functions to be substituted with the equivalent localized string. For instance:

if not (msg or fh):
  flash(gettext("You must enter a message or choose a file to submit."), "error")
  return redirect(url_for('main.lookup'))

The gettext function reads .mo files to find the translation for the string given in argument at runtime. It is used as a marker by pybabel or similar tools to collect the strings to be translated and store them into a .pot file at securedrop/translations/messages.pot. For instance:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr ""

For each language to be translated, a directory is created such as securedrop/translations/fr_FR and populated with a .po file derived from securedrop/translations/messages.pot, for translators to work with. For instance securedrop/translations/fr_FR/LC_MESSAGES/messages.po is almost identical to securedrop/translations/messages.pot except for the msgstr field which contains the translation:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr "Vous devez saisir un message ou sélectionner un fichier à envoyer."

This file is compiled into an optimized binary form, the .mo file used by the gettext function at runtime.

The Weblate web application is used to translate strings and relies on the gettext format behind the scene. It owns the securedrop/translations/messages.pot file and all other translation related files. The SecureDrop code is modified by sending a pull request but the translations are exclusively modified via Weblate.

The desktop icon are in the install_files/ansible-base/roles/tails-config/templates directory. Their labels are collected in the desktop.pot file and translated in the corresponding .po files in the same directory (fr.po, de.po etc.). All translations are merged from the *.j2.in files into the corresponding *.j2 file and committed to the repository. They are then installed when configuring Tails with the tasks/create_desktop_shortcuts.yml tasks.

The i18n_tool.py Translation Helpers

The pybabel and gettext command line is wrapped into the i18n_tool.py translate-messages and i18n_tool.py translate-desktop helpers for convenience. It is designed to be used by developers, to run tests with fixtures and for packaging.

Creating New Translations

A user with weblate admin rights must visit the Weblate translation creation page and the Weblate desktop translation creation page to add the desired languages.

Updating Strings to be Translated

After strings are modified in the code, templates, JavaScript or desktop labels, the securedrop/translations/messages.pot files must also be updated. Individual developers should NOT do this when changing strings in the code; the translations will be updated in bulk later on.

Translations can be updated with the following command:

make translate

This wraps i18n_tool.py translate-messages and i18n_tool.py translate-desktop. These commands will update securedrop/translations/messages.pot, install_files/ansible-base/roles/tails-config/templates/desktop.pot, and the messages.po files for each language.

Note

The changes will only be visible in the Weblate web interface used by translators after Merging Develop into the Weblate Fork.

Compiling Translations

gettext needs a compiled file for each language (the *.mo files). This can be done by running the following command:

securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-messages --compile

For desktop files the compilation phases creates a modified version of the original file which includes all the translations collected from the .po files.

This can be done by running the following command:

securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-desktop --compile

Verifying Translations

After a translation is compiled, the web page in which it shows can be verified visually by navigating to the corresponding state from http://localhost:8080 for the source interface or http://localhost:8081 for the journalist interface after running the following:

make -C securedrop dev

An easier way is to generate screenshots for each desired language with:

$ securedrop/bin/dev-shell bash
$ export PAGE_LAYOUT_LOCALES=en_US,fr_FR
$ ./i18n_tool.py --verbose translate-messages --compile
$ pytest -v --page-layout tests/pages-layout
...
...TestJournalistLayout::test_col_no_documents[en_US] PASSED
...TestJournalistLayout::test_col_no_documents[fr_FR] PASSED
...

Note

if unset, PAGE_LAYOUT_LOCALES defaults to en_US

The screenshots for fr_FR are available in securedrop/tests/pages-layout/screenshots/fr_FR and the name of the file can be found in the function that created it in securedrop/tests/pages-layout/test_journalist.py or securedrop/tests/pages-layout/test_source.py.

Merging Translations Back to Develop

Weblate automatically pushes the translations done via the web interface as a series of commit to the i18n branch in the Weblate SecureDrop branch which is a fork of the develop branch of the SecureDrop git repository. These translations need to be submitted to the develop branch via pull requests for merge on a regular basis.

SecureDrop only supports a subset of all the languages being worked on in Weblate: some of them are partially translated or not fully reviewed. The list of supported languages is hard-coded in the i18n_tool.py file, in the SUPPORTED_LANGUAGES variable. When a new language is fully translated and reviewed, the i18n_tool.py file must be manually edited to add this new language to the SUPPORTED_LANGUAGES variable.

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git checkout -b wip-i18n origin/develop
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-from-weblate
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-desktop --compile
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-docs
$ git commit -m 'l10n: compile desktop files' translations # if needed
$ git push wip-i18n # and make a pull request from the branch

Warning

It is very important to carefully check each translated string does not look strange. Even if the reviewer does not understand the language, if a translated string looks strange, someone other than the reviewer must be consulted to verify it means something. It is extremely unlikely that a reviewer will manipulate a translated string to introduce a vulnerability in SecureDrop. But it is easy to check visually and significantly reduce the risk.

List contributors for each supported language:

$ for l in $sm ; do echo -n "$l " ; git log --format=%aN lab/i18n -- install_files/ansible-base/roles/tails-config/templates/$l.po securedrop/translations/$l/LC_MESSAGES/messages.po | sort -u | tr '\n' ',' | sed -e 's/,/, /g' ; echo ; done
nl Anne M, kwadronaut, Yarno Ritzen,
fr Alain-Olivier,
...

Verify the translations are not broken:

$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-messages --compile
$ PAGE_LAYOUT_LOCALES=$(echo $sm | tr ' ' ',') \
    pytest -v --page-layout tests/pages-layout

Go to https://github.com/freedomofpress/securedrop and propose a pull request.

Note

contrary to the applications translations, the desktop translations are compiled and merged into the repository. They need to be available in their translated form when securedrop-admin tailsconfig is run because the development environment is not available.

Merging Develop into the Weblate Fork

Weblate works on a long standing fork of the SecureDrop git repository and is exclusively responsible for the content of the *.pot and *.po files. The content of the develop branch must be merged into the i18n branch to extract new strings to translate or existing strings that were updated.

The translations must be suspended in Weblate to avoid conflicts.

Weblate commit Lock

  • Click Lock

Weblate commit Locked

The develop branch can now be merged into i18n as follows:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add lab http://lab.securedrop.club/bot/securedrop/tree/i18n
$ git fetch lab
$ git checkout -b i18n lab/i18n
$ git merge origin/develop
$ make -C securedrop translate

The translate Makefile target relies on the i18n_tool.py command to examine all the source files, looking for strings that need to be translated (i.e. gettext('translate me') etc.) and update the *.pot and *.po files, removing, updating and inserting strings to keep them in sync with the sources. Carefully review the output of git diff. Check messages.pot first for updated strings, looking for formatting problems. Then review the messages.po of one existing translation, with a focus on fuzzy translations. There is no need to review other translations because they are processed in the same way. When you are satisfied with the result, it can be merged with:

$ git commit -a -m 'l10n: sync with upstream origin/develop'
$ git push lab i18n
  • Go to the Weblate commit page for SecureDrop and verify the commit hash matches the last commit of the i18n branch. This must happen instantly after the branch is pushed because Weblate is notified via a webhook. If it is different, ask for help.
  • Click Unlock

Weblate commit Unlock

Weblate pushes the translations done via the web interface to the develop branch in a fork of the SecureDrop git repository. These commits must be manually cherry-picked and proposed as pull requests for the SecureDrop git repository.

Weblate commit Unlocked

Updating the Full Text Index

The full text index can occasionally not be up to date. The symptom may be that the search function fails to find a word that you know exists in the source strings. If that happens you can rebuild the index from scratch with:

$ ssh debian@weblate.securedrop.club
$ cd /app/weblate
$ sudo docker-compose run weblate rebuild_index --all --clean

Note that the new index will not be used right away, some workers may still have the old index open. Rebooting the machine is an option, waiting for a few hours is another option.

Release Management

Two Weeks Before the Release: Update

The new and updated strings are uploaded to Weblate. This is done late in the SecureDrop release cycle so translators get less notifications. It would be inconvenient if there were hundreds of strings needing attention. But SecureDrop is small and it is ok to postpone notifications.

  • Merge develop into the Weblate fork
  • Post an announcement to the translation section of the forum (see an example)
  • Add a prominent Weblate whiteboard announcement that reads The X.Y.Z deadline is MM DD, YY midnight. String freeze will be in effect MM DD, YY midnight.
  • Create a pull request for every source string suggestion coming from translators
  • Backport every commit changing a source string to the release branch
  • Update the i18n timeline and Weblate whiteboard

One Week Before the Release: String Freeze

  • Verify develop and the release branch have the same source strings
  • Merge develop into the Weblate fork
  • Post an announcement to the translation section of the forum (see an example)
  • Remind all developers about the string freeze, in the chat room
  • Add a prominent Weblate whiteboard announcement that reads The X.Y.Z deadline is Month day, year midnight. String freeze is in effect: no source string are modified before the release.
  • Update the i18n timeline and Weblate whiteboard

The Day of the Release

Translator Credits

Verify the names and emails look ok, otherwise add to .mailmap until it does:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add lab http://lab.securedrop.club/bot/securedrop/tree/i18n
$ git fetch lab
$ previous_version=0.4.4
$ git log --pretty='%aN <%aE>' $previous_version..lab/i18n -- \
   securedrop/translations install_files/ansible-base/roles/tails-config/templates | sort -u

We do not want to publish the translator emails so we strip them:

git log --pretty='%aN' $previous_version..lab/i18n -- \
 securedrop/translations install_files/ansible-base/roles/tails-config/templates | sort -u

Translations Admins

Note

The privilege escalation workflow is different for code maintainers and translation maintainers.

A translation admin is a person who is actively performing administrative duties. They have special permissions on the repositories and the translation platform. When someone is willing to become an admin, a thread is started in the translation section of the forum. If there is a consensus, the permissions of the new admin are elevated after a week or more. If there is no consensus, a public vote is organized among the current admins.

All admins are listed in the forum introduction page

The privileges of an admin who has not been active for six months or more are revoked. They can apply again at any time.

The community of SecureDrop translators works very closely with the SecureDrop developers and some of them participate in both groups. However, the translators community has a different set of rules and permissions, reason why it makes sense to have an independent policy.

Admin Permissions

An admin may not need or want all permissions but they are entitled to have all of them.

Granting Reviewer Privileges in Weblate

  • visit https://weblate.securedrop.club/admin/auth/user/
  • click on the user name
  • in the Groups block
    • select Localizationlab in the Available groups list and click on the right arrow to move it to the Chosen groups list
    • select Users in the Chosen groups list and click on the left arrow to remove it