Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.

Seamless bare-hand interaction in mixed reality

Messelodi, Stefano;Poiesi, Fabio
2018-01-01

Abstract

Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.
File in questo prodotto:
File Dimensione Formato  
seamless-bare-hand.pdf

solo utenti autorizzati

Descrizione: Articolo sottomesso
Tipologia: Documento in Pre-print
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 3.75 MB
Formato Adobe PDF
3.75 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11582/316188
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
social impact