The Recognizing Textual Entailment Challenges: Datasets and Methodologies

While semantic inference has always been a major focus in Computational Linguistics, the topic has benefited of new attention in the field thanks to the Recognizing Textual Entailment (RTE) framework, first launched in 2004, which has provided an operational definition of entailment based on human judgements over portions of text. On top of such definition, a task has been designed, which includes both guidelines for dataset annotation and evaluation metrics for assessing systems' performance. This chapter presents the successful experience of creating Textual Entailment datasets. We show how, during the years, RTE datasets have been developed in several variants, not only to address complex phenomena underlying entailment, but also to demonstrate the potential application of entailment inference into concrete scenarios, including summarization, knowledge base population, answer validation for question answering, and student answer assessment.