IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation

Florent P Audonnet 1 Ixchel G. Ramirez-Alpizar 2 Gerardo Aragon-Camarasa 1
1University of Glasgow , 2National Institute of Advanced Industrial Science and Technology
Paper arXiv Code

IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation

Abstract

We present IMMERTWIN, a mixed reality framework for enhance robotic arm teleoperation using a digital twin as a bridge for interaction between the user and the robotics devices.

We evaluated IMMERTWIN by performing a medium scale user survey with 26 participants on two different robots. Users were asked to teleoperate both robots inside the virtual environment to pick and place 3 cubes in a tower and to repeat this task as many times as possible in 10 minutes, with only 5 minutes of training beforehand.

Our experimental results show that most users were able to succeed by building at least a tower of 3 cubes regardless of the robot used. In addition user's preferred to use the IMMERTWIN over our previous work TELESIM, as it caused them less mental workload.

Our Framework

Overview of our Framework

Our framework, IMMERWTIN, shown in the green dotted line, accepts the pose of any 3D VR controller (shown in the black dotted line) to update the position of the user's virtual hand in Unreal Engine. The user can then grab the robotic gripper and move the robot where they want. The new robotic goal is then transmitted to TELESIM (shown in the blue dotted line) to perform motion planning and collision avoidance. The state of the virtual robot is then transmitted to the real robot, shown in the red dotted line, to update its position. Finally, the state of the real robot is transmitted back into Unreal Engine to create a closed-loop digital twin.

Experimental Setup

Overview of the experimental setup

Our experimental setup comprises the following components: (1) The view from the user inside Unreal Engine, with their virtual hands (4) and the virtual gripper (5). (2) The room containing both the Baxter robot (8) and the UR3 (7), along with 4 ZED 2I cameras (9). (3) The user wearing the VR headset, with the black security tape (6).

Results

Population percentage for each tower completed for both robots for TELESIM and IMMERTWIN.

Participants of IMMERTWIN successfully constructed a maximum of 10 towers with both robots. Notably, users operating the UR3 with IMMERTWIN consistently completed more towers than participants using any other robot across both TELESIM and IMMERTWIN.

NASA Mental Workload

NASA mental workload.

NASA Physical Workload

NASA physical workload.

IMMERTWIN was favoured by most participants who had previously used TELESIM. This preference underscores the potential benefits of immersive VR environments in enhancing user experience, even if quantifiable performance gains are not immediately evident. The mental effort required by IMMERTWIN was significantly lower than that of TELESIM, suggesting that the immersive nature of the VR environment may alleviate cognitive load. However, physical effort remained at the same level, particularly with the UR3 in TELESIM, which was noted to be more exhausting than other setups. This highlights the need for further refinement in balancing cognitive and physical demands in VR-based teleoperation systems.

Acknowledgements

We thank the University of Glasgow for the use of the Baxter robot and the UR3 robot. We also thank the EPSCR for funding this research. We thank AIST in Japan for their help in developing the ideas for IMMERTWIN. Finally we thank all the participants who took part in our experiments.

BibTeX

@article{audonnet2023telesim,
        title={TELESIM: A Modular and Plug-and-Play Framework for Robotic Arm Teleoperation using a Digital Twin}, 
        author={Florent P Audonnet and Jonathan Grizou and Andrew Hamilton and Gerardo Aragon-Camarasa},
        year={2023},
        eprint={2309.10579},
        archivePrefix={arXiv},
        primaryClass={cs.RO}
  }