Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale

by   Marta R. Costa-Jussà, et al.

We introduce a multilingual extension of the HOLISTICBIAS dataset, the largest English template-based taxonomy of textual people references: MULTILINGUALHOLISTICBIAS. This extension consists of 20,459 sentences in 50 languages distributed across all 13 demographic axes. Source sentences are built from combinations of 118 demographic descriptors and three patterns, excluding nonsensical combinations. Multilingual translations include alternatives for gendered languages that cover gendered translations when there is ambiguity in English. Our benchmark is intended to uncover demographic imbalances and be the tool to quantify mitigations towards them. Our initial findings show that translation quality for EN-to-XX translations is an average of 8 spBLEU better when evaluating with the masculine human reference compared to feminine. In the opposite direction, XX-to-EN, we compare the robustness of the model when the source input only differs in gender (masculine or feminine) and masculine translations are an average of almost 4 spBLEU better than feminine. When embedding sentences to a joint multilingual sentence representations space, we find that for most languages masculine translations are significantly closer to the English neutral sentences when embedded.


page 5

page 7

page 8

page 9

page 10

page 11


Automatic Discrimination of Human and Neural Machine Translation in Multilingual Scenarios

We tackle the task of automatically discriminating between human and mac...

Toxicity in Multilingual Machine Translation at Scale

Machine Translation systems can produce different types of errors, some ...

Multilingual Representation Distillation with Contrastive Learning

Multilingual sentence representations from large models can encode seman...

How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points

In this paper, as a case study, we present a systematic study of gender ...

Fairness in Language Models Beyond English: Gaps and Challenges

With language models becoming increasingly ubiquitous, it has become ess...

Translational Grounding: Using Paraphrase Recognition and Generation to Demonstrate Semantic Abstraction Abilities of MultiLingual NMT

In this paper, we investigate whether multilingual neural translation mo...

Using Deep Neural Networks to Translate Multi-lingual Threat Intelligence

The multilingual nature of the Internet increases complications in the c...

Please sign up or login with your details

Forgot password? Click here to reset