Towards a comprehensive framework for automated testing of user experience

User Experience (UX) has a great impact in today’s market, making good UX evaluations crucial for the development process. However, UX evaluations pose many challenges and significant costs. A possible solution to reduce costs and empower conducting UX evaluations is to automate a significant part of the process. Automated UX evaluations offer a solution around some important issues, such as the diversity of the audience that tests the system and the possibility of performing continuous testing of different versions. It also opens new possibilities, such as simulating long-term or multiparty interactions. We argue that to move to automated testing in a sound and comprehensive way, we need good definitions and good UX models. In this article, we present a comprehensive framework to assist the definition of UX tests as a first step towards automated UX testing. We believe this framework will help the formalization and systematization of the automated tests, and its modular structure makes it comprehensive enough to cover different aspects of UX and types of systems. We discuss previous UX definitions and models that inspired our framework, and we also discuss concepts from functional testing, such as coverage, applied to UX testing.