Interacting with Maps in Virtual and Augmented Reality

Abstract. Augmented reality (AR) and virtual reality (VR) technology are increasingly used for the analysis and visualisation of geospatial data. It has become simple to create an immersive three-dimensional AR or VR map with a combination of game engines (e.g., Unity), software development kits for streaming and rendering geospatial data (e.g., Mapbox), and affordable hardware (e.g., HTC Vive). However, it is not clear how to best interact with geospatial visualisations in AR and VR. For example, there are no established standards to efficiently zoom and pan, select map features, or place markers on AR and VR maps. In this paper, we explore interaction with AR and VR maps using gestures and handheld controllers.

As for gesture-controlled interaction, we present the results of recent research projects exploring how body gestures can control basic AR and VR map operations. We use motion-tracking controllers (e.g., Leap Motion) to capture and interpret gestures. We conducted a set of user studies to identify, explore and compare various gestures for controlling map-related operations. This includes, for example, mid-air hand gestures for zooming and panning (Satriadi et al. 2019), selecting points of interest, adjusting the orientation of maps, or placing markers on maps. Additionally, we present novel VR interfaces and interaction methods for controlling the content of maps with gestures.

As for handheld controllers, we discuss interaction with exocentric globes, egocentric globes (where the user stands inside a large virtual globe), flat maps, and curved maps in VR. We demonstrate controller-based interaction for adjusting the centre of world maps displayed on these four types of projection surfaces (Yang et al. 2018), and illustrate the utility of interactively movable VR maps by the example of three-dimensional origin-destination flow maps (Yang et al. 2019).