Many applications within natural language processing involve performing text-to-text transformations, i.e., given a text in natural language as input, systems are required to produce a version of this text (e.g., a translation), also in natural language, as output. Automatically evaluating the output of such systems is an important component in developing text-to-text applications. Two approaches have been proposed for this problem: (i) to compare the system outputs against one or more reference outputs using string matching based evaluation metrics and (ii) to build models based on human feedback to predict the quality of system outputs without reference texts. Despite their popularity, reference-based evaluation metrics are faced with the challenge that multiple good (and bad) quality outputs can be produced by text-to-text approaches for the same input. This variation is very hard to capture, even with multiple reference texts. In addition, reference-based metrics cannot be used in production (e.g., online machine translation systems), when systems are expected to produce outputs for any unseen input. In this book, we focus on the second set of metrics, so-called Quality Estimation (QE) metrics, where the goal is to provide an estimate on how good or reliable the texts produced by an application are without access to gold-standard outputs. QE enables different types of evaluation that can target different types of users and applications. Machine learning techniques are used to build QE models with various types of quality labels and explicit features or learnt representations, which can then predict the quality of unseen system outputs. This book describes the topic of QE for text-to-text applications, covering quality labels, features, algorithms, evaluation, uses, and state-of-the-art approaches. It focuses on machine translation as the application, since this represents most of the QE work done to date. It also briefly describes QE for several other applications, including text simplification, text summarization, grammatical error correction, and natural language generation.
Table of Contents
Acknowledgments
Introduction
Quality Estimation for MT at Subsentence Level
Quality Estimation for MT at Sentence Level
Quality Estimation for MT at Document Level
Quality Estimation for other Applications
Final Remarks
Bibliography
Authors’ Biographies
About the Author(s)
Lucia Specia, University of Sheffield
Lucia Specia is a Professor of Language Engineering in the Department of Computer Science at the University of Sheffield. Her research focuses on various aspects of data-driven approaches to multilingual language processing with a particular interest in multimodal context models for language grounding that has applications in machine translation, quality estimation, and text adaptation. She is the recipient of an ERC Starting Grant on multimodal machine translation (2016-2021) and has been involved in a number of other research projects on machine translation (EC FP7 QTLaunch-Pad and EXPERT, EC H2020 QT21 21, CRACKER) and EC H2020 text adaptation (SIMPATICO). Before joining the University of Sheffield in 2012, she was Senior Lecturer at the University of Wolverhampton (2010-2011) and research engineer at the Xerox Research Centre, France (2008–2009). She received a Ph.D. in Computer Science from the University of Sao Paulo, Brazil, in 2008.
Carolina Scarton, University of Sheffield
Carolina Scarton is a Research Associate at the University of Sheffield and a member of the Natural Language Processing Group. She holds a Ph.D. on Quality Estimation of Machine Translation from the University of Sheffield (2017) and her research interests are quality estimation, text simplification, evaluation of NLP task outputs at the document level, and readability assessment. Currently, Dr. Scarton works for the EC H2020 SIMPATICO project, where she develops approaches for sentence simplification. Previously, she was a Marie Skłodowska-Curie Early Stage Researcher working for the EXPERT project on the topic of machine translation at the University of Sheffield (2013-2016).
Gustavo Henrique Paetzold, Federal University of Technology – Brazil
Gustavo Henrique Paetzold is an Adjunct Professor at the Federal University of Technology – Parana. He holds a Ph.D. from the University of Sheffield (2016). His main research interests are text simplification, psycholinguistics, quality estimation, and machine learning applied to natural language processing. Prior to joining the Federal University of Technology, Dr. Paetzold worked as a Research Associate at the University of Sheffield on the EC H2020 SIMPATICO project, where he developed approaches for lexical simplification (2016-2018).
