BeHere: a VR/SAR remote collaboration system based on virtual replicas sharing gesture and avatar in a procedural task

In this paper, we focus on the significance of remote collaboration using virtual replicas, avatar, and gesture on a procedural task in industry; thus, we present a Virtual Reality (VR)/Spatial Augmented Reality (SAR) remote collaboration system, BeHere, based on 3D virtual replicas and sharing gestures and avatar. BeHere enables a remote expert in VR to guide a local worker in real-time to complete a procedural task in the real-world. For the remote VR site, we construct a 3D virtual environment using virtual replicas, and the user can manipulate them by using gestures in an intuitive interaction and see their partners’ 3D virtual avatar. For the local site, we use SAR to enable the local worker to see instructions projected onto the real-world based on the shared virtual replicas and gestures. We conducted a formal user study to evaluate the prototype system in terms of performance, social presence, workload, and ranking and user preference. We found that the combination of visual cues of gestures, avatar, and virtual replicas plays a positive role in improving user experience, especially for remote VR users. More significantly, our study provides useful information and important design implications for further research on the use of gesture-, gaze- and avatar-based cues as well as virtual replicas in VR/AR remote collaboration on a procedural task in industry. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s10055-023-00748-5.