The paper discusses some significant issues in the empirical evaluation of interactive recommender: the role of experiments, the contingent and constructive nature of users’ interaction strategies, and the generalizability of the results. We propose to adopt as the main evaluation goal the construction of a situation-specific account of the user-system behavior, and we suggest applying the context matching approach to cope with the contingencies of the user behavior. To make clear the limitations of high-level single step experimental evaluations, we present a critical analysis of a case study, a pilot evaluation of a travel recommender system (ITR). The examination of this study shows the danger of overlooking the detailed aspects of user-system interaction, and underlines the need to iteratively refine the evaluation hypotheses and design when no detailed model of the user-system behavior is initially available

Understanding Recommender Systems: Experimental Evaluation Challenges

Ricci, Francesco
2003

Abstract

The paper discusses some significant issues in the empirical evaluation of interactive recommender: the role of experiments, the contingent and constructive nature of users’ interaction strategies, and the generalizability of the results. We propose to adopt as the main evaluation goal the construction of a situation-specific account of the user-system behavior, and we suggest applying the context matching approach to cope with the contingencies of the user behavior. To make clear the limitations of high-level single step experimental evaluations, we present a critical analysis of a case study, a pilot evaluation of a travel recommender system (ITR). The examination of this study shows the danger of overlooking the detailed aspects of user-system interaction, and underlines the need to iteratively refine the evaluation hypotheses and design when no detailed model of the user-system behavior is initially available
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11582/949
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
social impact