Evaluating Machine Translation in a Usage Scenario

Rosa Gaudio, Aljoscha Burchardt, António Branco

In: Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis (Hrsg.). Proceedings of the Tenth International Conference on Language Resources and Evaluation. International Conference on Language Resources and Evaluation (LREC-2016) May 23-28 Portoroz Slovenia ISBN 978-2-9517408-9-1 European Language Resources Association (ELRA) Paris, France 5/2016.


In this document we report on a user-scenario-based evaluation aiming at assessing the performance of machine translation (MT) systems in a real context of use. We describe a sequel of experiments that has been performed to estimate the usefulness of MT and to test if improvements of MT technology lead to better performance in the usage scenario. One goal is to find the best methodology for evaluating the eventual benefit of a machine translation system in an application. The evaluation is based on the QTLeap corpus, a novel multilingual language resource that was collected through a real-life support service via chat. It is composed of naturally occurring utterances produced by users while interacting with a human technician providing answers. The corpus is available in eight different languages: Basque, Bulgarian, Czech, Dutch, English, German, Portuguese and Spanish.


58_Paper.pdf (pdf, 638 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence