This work discusses the results of two user studies aiming to evaluate the NESPOLE! speech-to-speech translation system, which provides for multilingual and multimodal communication in the tourism and in the medical domain, allowing users to interact through the Internet by sharing maps, web-pages and pen-based gestures. The purpose is to investigate the overall effectiveness of the combination of multimodality and multilinguality in respect to a speech-only system. Specifically, we will examine the improvement in usability offered by the multimodal system, which is gauged by evaluating the interaction between novice users and the system at several stages of development. Additionally, we will discuss how data collection contributed to system improvement