This paper investigates the feasibility of using crowd-sourcing services for the hu- man assessment of machine translation quality of translations into non-English tar- get languages. Non-expert graders are hired through the CrowdFlower interface to Amazon’s Mechanical Turk in order to carry out a ranking-based MT evaluation of utterances taken from the travel conver- sation domain for 10 Indo-European and Asian languages. The collected human as- sessments are analyzed for their worker characteristics, evaluation costs, and qual- ity of the evaluations in terms of the agree- ment between non-expert graders and ex- pert/oracle judgments. Moreover, data quality control mechanisms including “lo- cale qualification” “qualificatio testing”, and “on-the-fl verification are investi- gated in order to increase the reliability of the crowd-based evaluation results.

Crowd-based MT Evaluation for non-English Target Languages

Bentivogli, Luisa;Federico, Marcello
2012-01-01

Abstract

This paper investigates the feasibility of using crowd-sourcing services for the hu- man assessment of machine translation quality of translations into non-English tar- get languages. Non-expert graders are hired through the CrowdFlower interface to Amazon’s Mechanical Turk in order to carry out a ranking-based MT evaluation of utterances taken from the travel conver- sation domain for 10 Indo-European and Asian languages. The collected human as- sessments are analyzed for their worker characteristics, evaluation costs, and qual- ity of the evaluations in terms of the agree- ment between non-expert graders and ex- pert/oracle judgments. Moreover, data quality control mechanisms including “lo- cale qualification” “qualificatio testing”, and “on-the-fl verification are investi- gated in order to increase the reliability of the crowd-based evaluation results.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11582/106204
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
social impact