Design of multimodal interaction with mobile devices. Challenges for visually impaired and elderly users

This paper presents two early studies aimed at investigating issues concerning the design of multimodal interaction - based on voice commands and mid-air gestures - with mobile technology specifically designed for visually impaired and elderly users. These studies have been carried out on a new device allowing enhanced speech recognition (interpreting lip movements) and mid-air gesture interaction on Android devices (smartphone and tablet PC). The initial findings and challenges raised by these novel interaction modalities are discussed. These mainly centre on issues of feedback and feedforward, the avoidance of false positives and point of reference or orientation issues regarding the device and the mid-air gestures.