In this paper we present a method to evaluate Question Answering systems in restricted domain applications, which present peculiar features with respect to open domain Question Answering. While the answers provided by the system are still evaluated manually as in QA evaluation based on Information Retrieval, we propose to assess the overall quality of answers rather than concentrating merely on answers’ correctness. We propose to evaluate each answer from different perspectives, i.e. not only from the point of view of the exactness of the answer, but of the data provided as well, while at the same time taking into consideration aspects connected to comprehensibility and answer presentation.
Assessing Answer Quality in Task-oriented Question Answering
Speranza, Manuela;Gretter, Roberto
2009-01-01
Abstract
In this paper we present a method to evaluate Question Answering systems in restricted domain applications, which present peculiar features with respect to open domain Question Answering. While the answers provided by the system are still evaluated manually as in QA evaluation based on Information Retrieval, we propose to assess the overall quality of answers rather than concentrating merely on answers’ correctness. We propose to evaluate each answer from different perspectives, i.e. not only from the point of view of the exactness of the answer, but of the data provided as well, while at the same time taking into consideration aspects connected to comprehensibility and answer presentation.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
