FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE

This paper presents the FBK contribution to the IWSLT-2024 {`}Test suites{'} shared subtask, part of the Offline Speech Translation Task. Our contribution consists of the MuST-SHE-IWSLT24 benchmark evaluation, designed to assess gender bias in speech translation. By focusing on the en-de language pair, we rely on a newly created test suite to investigate systems{'} ability to correctly translate feminine and masculine gender. Our results indicate that {--} under realistic conditions {--} current ST systems achieve reasonable and comparable performance in correctly translating both feminine and masculine forms when contextual gender information is available. For ambiguous references to the speaker, however, we attest a consistent preference towards masculine gender, thus calling for future endeavours on the topic. Towards this goal we make MuST-SHE-IWSLT24 freely available at: https://mt.fbk.eu/must-she/