Three-Dimensional Mapping of the Rotation of Interactive Virtual Objects with Eye-Tracking Data

Summary

We have developed a simple, customizable, and efficient method for recording quantitative processual data from interactive spatial tasks and mapping these rotation data with eye-tracking data.

Abstract

We present a method for real-time recording of human interaction with three-dimensional (3D) virtual objects. The approach consists of associating rotation data of the manipulated object with behavioral measures, such as eye tracking, to make better inferences about the underlying cognitive processes.

The task consists of displaying two identical models of the same 3D object (a molecule), presented on a computer screen: a rotating, interactive object (iObj) and a static, target object (tObj). Participants must rotate iObj using the mouse until they consider its orientation to be identical to that of tObj. The computer tracks all interaction data in real time. The participant's gaze data are also recorded using an eye tracker. The measurement frequency is 10 Hz on the computer and 60 Hz on the eye tracker.

The orientation data of iObj with respect to tObj are recorded in rotation quaternions. The gaze data are synchronized to the orientation of iObj and referenced using this same system. This method enables us to obtain the following visualizations of the human interaction process with iObj and tObj: (1) angular disparity synchronized with other time-dependent data; (2) 3D rotation trajectory inside what we decided to call a "ball of rotations"; (3) 3D fixation heatmap. All steps of the protocol have used free software, such as GNU Octave and Jmol, and all scripts are available as supplementary material.

With this approach, we can conduct detailed quantitative studies of the task-solving process involving mental or physical rotations, rather than only the outcome reached. It is possible to measure precisely how important each part of the 3D models is for the participant in solving tasks, and thus relate the models to relevant variables such as the characteristics of the objects, cognitive abilities of the individuals, and the characteristics of human-machine interface.

Introduction

Mental rotation (MR) is a cognitive ability that enables individuals to mentally manipulate and rotate objects, facilitating a better understanding of their features and spatial relationships. It is one of the visuospatial abilities, a fundamental cognitive group that was studied as early as 1890¹. Visuospatial abilities are an important component of an individual's cognitive repertoire that is influenced by both inherited and environmental factors^2,3,4,5. Interest in visuospatial abilities has grown throughout th....

Protocol

1. Preparation of data collection tools

1. Set up the online data collection (optional).
   NOTE: This step describes how to set up a customizable clone of the project code and working webpage (refer to Supplemental File 1). This step was adapted from the tutorials available at https://pages.github.com/ and https://github.com/jamiewilson/form-to-google-sheets. If users are only interested in the data processing method and not in the data recording, they can use the webpage h.......

Discussion

As previously stated, this paper aims to present a detailed procedure of real-time mapping of fixation and saccade motion data on interactive 3D objects, which is easily customizable and only uses software available for free, providing step-by-step instructions to make everything work.

While this experimental setup involved a highly interactive task, such as moving a 3D object to match another object's orientation with PR in two of the three possible axes, we ensured thorough documentation.......

Materials

Name	Company	Catalog Number	Comments
Firefox	Mozilla Foundation (Open Source)		Any updated modern browser that is compatible with WebGL (https://caniuse.com/webgl), and in turn with Jmol, can be used
GNU Octave	Open Source		https://octave.org/
Google Apps Script	Google LLC		script.google.com
Google Sheets	Google LLC		https://www.google.com/sheets/about/
Laptop			Any computer that can run the eye tracking system software.
Mangold Software Suite	Mangold		Software interface used for the Eye tracking device. Any Software that outputs the data with system time values can be used.
Mouse			Any mouse capable of clicking and dragging with simple movements should be compatible. Human interfaces analogous to a mouse with the same capabilities, such as a touchscreen or pointer, should be compatible, but may behave differently.
Vt3mini	EyeTech Digital Systems		60 Hz. Any working Eye Tracking device should be compatible.