Internationalization (i18n)

The code, templates and JavaScript user visible strings must all be wrapped with gettext functions to be substituted with the equivalent localized string. For instance:

if not (msg or fh):
  flash(gettext("You must enter a message or choose a file to submit."), "error")
  return redirect(url_for('main.lookup'))

The gettext function reads .mo files to find the translation for the string given in argument at runtime. It is used as a marker by pybabel or similar tools to collect the strings to be translated and store them into a .pot file at securedrop/translations/messages.pot. For instance:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr ""

For each language to be translated, a directory is created such as securedrop/translations/fr_FR and populated with a .po file derived from securedrop/translations/messages.pot, for translators to work with. For instance securedrop/translations/fr_FR/LC_MESSAGES/messages.po is almost identical to securedrop/translations/messages.pot except for the msgstr field which contains the translation:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr "Vous devez saisir un message ou sélectionner un fichier à envoyer."

This file is compiled into an optimized binary form, the .mo file used by the gettext function at runtime.

The Weblate web application is used to translate strings and relies on the gettext format behind the scene. It owns the securedrop/translations/messages.pot file and all other translation related files. The SecureDrop code is modified by sending a pull request but the translations are exclusively modified via Weblate.

The desktop icon are in the install_files/ansible-base/roles/tails-config/templates directory. Their labels are collected in the desktop.pot file and translated in the corresponding .po files in the same directory (fr.po, de.po etc.). All translations are merged from the *.j2.in files into the corresponding *.j2 file and committed to the repository. They are then installed when configuring Tails with the tasks/create_desktop_shortcuts.yml tasks.

The manage.py translate helpers

The pybabel and gettext command line is wrapped into the manage.py translate-messages and manage.py translate-desktop helpers for convenience. It is designed to be used by developers, to run tests with fixtures and for packaging.

Creating new translations

A user with weblate admin rights must visit the Weblate translation creation page and the Weblate desktop translation creation page to add the desired languages.

Updating strings to be translated

After modifying a string in the code, templates, JavaScript or desktop labels, the securedrop/translations/messages.pot files must be updated by running the following command in /vagrant/securedrop, in the development virtual machine:

make translate

which wraps manage.py translate-messages and manage.py translate-desktop. The updated securedrop/translations/messages.pot and install_files/ansible-base/roles/tails-config/templates/desktop.pot should then be reviewed and committed.

Note

The changes will only be visible in the Weblate web interface used by translators after Merging develop into the weblate fork.

Compiling translations

gettext needs a compiled file for each language (the *.mo files). This can be done by running the following command in /vagrant/securedrop, in the development virtual machine:

./manage.py --verbose translate-messages --compile

For desktop files the compilation phases creates a modified version of the original file which includes all the translations collected from the .po files.

This can be done by running the following command in /vagrant/securedrop, in the development virtual machine:

./manage.py --verbose translate-desktop --compile

Verifying translations

After a translation is compiled, the web page in which it shows can be verified visually by navigating to the corresponding state from http://localhost:8080 for the source interface or http://localhost:8081 for the journalist interface after running the following:

./manage.py run

An easier way is to generate screenshots for each desired language with:

$ export PAGE_LAYOUT_LOCALES=en_US,fr_FR
$ ./manage.py --verbose translate-messages --compile
$ pytest -v --page-layout tests/pages-layout
...
...TestJournalistLayout::test_col_no_documents[en_US] PASSED
...TestJournalistLayout::test_col_no_documents[fr_FR] PASSED
...

Note

if unset, PAGE_LAYOUT_LOCALES defaults to en_US

The screenshots for fr_FR are available in securedrop/tests/pages-layout/screenshots/fr_FR and the name of the file can be found in the function that created it in securedrop/tests/pages-layout/test_journalist.py or securedrop/tests/pages-layout/test_source.py.

Merging translations back to develop

Weblate automatically pushes the translations done via the web interface as a series of commit to the i18n branch in the Weblate SecureDrop branch which is a fork of the develop branch of the SecureDrop git repository. These translations need to be submitted to the develop branch via pull requests for merge on a regular basis.

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add lab http://lab.securedrop.club/bot/securedrop/tree/i18n
$ git fetch lab
$ git checkout -b wip-i18n origin/develop
$ git checkout lab/i18n -- securedrop/translations
$ sm="nl fr_FR de_DE nb_NO pt_BR es_ES zh_Hant tr it_IT ar"
$ sd="nl fr de_DE nb_NO pt_BR es_ES zh_Hant tr it ar"
$ for l in $sm ; do \
     git add securedrop/translations/$l/LC_MESSAGES/messages.po ; \
  done
$ for l in $sd ; do \
     git checkout lab/i18n -- \
         install_files/ansible-base/roles/tails-config/templates/$l.po ; \
     git add install_files/ansible-base/roles/tails-config/templates/$l.po ; \
  done
$ vagrant ssh development
$ cd /vagrant/securedrop ; ./manage.py --verbose translate-desktop --compile
$ git commit -m 'sync with weblate' translations
$ git push wip-i18n

Warning

It is very important to carefully check each translated string does not look strange. Even if the reviewer does not understand the language, if a translated string looks strange, someone other than the reviewer must be consulted to verify it means something. It is extremely unlikely that a reviewer will manipulate a translated string to introduce a vulnerability in SecureDrop. But it is easy to check visually and significantly reduce the risk.

List contributors for each supported language:

$ for l in $sm ; do echo -n "$l " ; git log --format=%aN lab/i18n -- install_files/ansible-base/roles/tails-config/templates/$l.po securedrop/translations/$l/LC_MESSAGES/messages.po | sort -u | tr '\n' ',' | sed -e 's/,/, /g' ; echo ; done
nl Anne M, kwadronaut, Yarno Ritzen,
fr Alain-Olivier,
...

Verify the translations are not broken:

$ vagrant ssh development
$ cd /vagrant/securedrop
$ ./manage.py --verbose translate-messages --compile
$ PAGE_LAYOUT_LOCALES=$(echo $sm | tr ' ' ',') \
    pytest -v --page-layout tests/pages-layout

Go to https://github.com/freedomofpress/securedrop and propose a pull request.

Note

contrary to the applications translations, the desktop translations are compiled and merged into the repository. They need to be available in their translated form when securedrop-admin tailsconfig is run because the development environment is not available.

Merging develop into the weblate fork

Weblate works on a long standing fork of the SecureDrop git repository and is exclusively responsible for the content of the *.pot and *.po files. The content of the develop branch must be merged into the i18n branch to extract new strings to translate or existing strings that were updated.

The translations must be suspended in Weblate to avoid conflicts.

Weblate commit Lock

  • Click Lock

Weblate commit Locked

The develop branch can now be merged into i18n as follows:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add lab http://lab.securedrop.club/bot/securedrop/tree/i18n
$ git fetch lab
$ git checkout -b i18n lab/i18n
$ git merge origin/develop
$ make translate

The manage.py command examines all the source files, looking for strings that need to be translated (i.e. gettext(‘translate me’) etc.) and update the *.pot and *.po files, removing, updating and inserting strings to keep them in sync withe the sources. Carefully review the output of git diff. Check messages.pot first for updated strings, looking for formatting problems. Then review the messages.po of one existing translation, with a focus on fuzzy translations. There is no need to review other translations because they are processed in the same way. When you are satisfied with the result, it can be merged with:

$ git commit -a -m 'l10n: sync with upstream origin/develop'
$ git push lab i18n
  • Go to the Weblate commit page for SecureDrop and verify the commit hash matches the last commit of the i18n branch. This must happen instantly after the branch is pushed because Weblate is notified via a webhook. If it is different, ask for help.

Weblate commit Unlock

Weblate pushes the translations done via the web interface to the develop branch in a fork of the SecureDrop git repository. These commits must be manually cherry-picked and proposed as pull requests for the SecureDrop git repository.

Weblate commit Unlocked

Updating the full text index

The full text index can occasionally not be up to date. The symptom may be that the search function fails to find a word that you know exists in the source strings. If that happens you can rebuild the index from scratch with:

$ ssh [email protected]
$ cd /app/weblate
$ sudo docker-compose run weblate rebuild_index --all --clean

Note that the new index will not be used right away, some workers may still have the old index open. Rebooting the machine is an option, waiting for a few hours is another option.

Translator credits

Verify the names and emails look ok, otherwise add to .mailmap until it does:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add lab http://lab.securedrop.club/bot/securedrop/tree/i18n
$ git fetch lab
$ previous_version=0.4.4
$ git log --pretty='%aN <%aE>' $previous_version..lab/i18n -- \
   securedrop/translations install_files/ansible-base/roles/tails-config/templates | sort -u

We do not want to publish the translator emails so we strip them:

git log --pretty='%aN' $previous_version..lab/i18n -- \
 securedrop/translations install_files/ansible-base/roles/tails-config/templates | sort -u

Translations admins

Note

The privilege escalation workflow is different for code maintainers and translation maintainers.

A translation admin is a person who is actively performing administrative duties. They have special permissions on the repositories and the translation platform. When someone is willing to become an admin, a thread is started in the translation section of the forum. If there is a consensus, the permissions of the new admin are elevated after a week or more. If there is no consensus, a public vote is organized among the current admins.

All admins are listed in the forum introduction page

The privileges of an admin who has not been active for six months or more are revoked. They can apply again at any time.

The community of SecureDrop translators works very closely with the SecureDrop developers and some of them participate in both groups. However, the translators community has a different set of rules and permissions, reason why it makes sense to have an independent policy.

Admin permissions

An admin may not need or want all permissions but they are entitled to have all of them.

Granting reviewer privileges in Weblate

  • visit https://weblate.securedrop.club/admin/auth/user/
  • click on the user name
  • in the Groups block
    • select Localizationlab in the Available groups list and click on the right arrow to move it to the Chosen groups list
    • select Users in the Chosen groups list and click on the left arrow to remove it