Virtual Reality Experiments with Physiological Measures

Summary

Virtual reality (VR) experiments can be difficult to implement and require meticulous planning. This protocol describes a method for the design and implementation of VR experiments that collect physiological data from human participants. The Experiments in Virtual Environments (EVE) framework is employed to accelerate this process.

Abstract

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and laboratory experiments, respectively. VR is especially useful for geographic visualizations and investigations of spatial behavior. In spatial behavior research, VR provides a platform for studying the relationship between navigation and physiological measures (e.g., skin conductance, heart rate, blood pressure). Specifically, physiological measures allow researchers to address novel questions and constrain previous theories of spatial abilities, strategies, and performance. For example, individual differences in navigation performance may be explained by the extent to which changes in arousal mediate the effects of task difficulty. However, the complexities in the design and implementation of VR experiments can distract experimenters from their primary research goals and introduce irregularities in data collection and analysis. To address these challenges, the Experiments in Virtual Environments (EVE) framework includes standardized modules such as participant training with the control interface, data collection using questionnaires, the synchronization of physiological measurements, and data storage. EVE also provides the necessary infrastructure for data management, visualization, and evaluation. The present paper describes a protocol that employs the EVE framework to conduct navigation experiments in VR with physiological sensors. The protocol lists the steps necessary for recruiting participants, attaching the physiological sensors, administering the experiment using EVE, and assessing the collected data with EVE evaluation tools. Overall, this protocol will facilitate future research by streamlining the design and implementation of VR experiments with physiological sensors.

Introduction

Understanding how individuals navigate has important implications for several fields, including cognitive science^1,2,3, neuroscience^4,5, and computer science^6,7. Navigation has been investigated in both real and virtual environments. One advantage of real-world experiments is that navigation does not require the mediation of a control interface and thus may produce more realistic spatial behavior. In contrast, virtual reality (VR) experiments allow for more precise measurement of behavioral (e.g., walking trajectories) and physiological (e.g., heart rate) data, as well as more experimental control (i.e., internal validity). In turn, this approach can result in simpler interpretations of the data and thus more robust theories of navigation. In addition, neuroscience can benefit from VR because researchers can investigate the neural correlates of navigation while participants are engaged in the virtual environment but cannot physically move. For computer scientists, navigation in VR requires unique developments in processing power, memory, and computer graphics in order to ensure an immersive experience. Findings from VR experiments can also be applied in architecture and cartography by informing the design of the building layouts⁸ and map features⁹ to facilitate real-world navigation. Recently, advances in VR technology combined with a dramatic decrease in its cost have led to an increase in the number of laboratories employing VR for their experimental designs. Because of this growing popularity, researchers need to consider how to streamline the implementation of VR applications and standardize the experiment workflow. This approach will help shift resources from implementation to the development of theory and extend the existing capabilities of VR.

VR setups can range from more to less realistic in terms of displays and controls. More realistic VR setups tend to require additional infrastructure such as large tracking spaces and high-resolution displays¹⁰. These systems often employ redirected walking algorithms in order to inject imperceptible rotations and translations into the visual feedback provided to users and effectively enlarge the virtual environment through which participants can move^11,12. These algorithms can be generalized in that they do not require the knowledge of environmental structure¹³ or predictive in that they assume particular paths for the user¹⁴. Although most research on redirected walking has used head-mounted displays (HMDs), some researchers employ a version of this technique with walking-in-place as part of a large projection system (e.g., CAVEs)¹⁵. While HMDs can be carried on the head of the participant, CAVE displays tend to provide a wider horizontal field of view^16,17. However, less infrastructure is needed for VR systems using desktop displays^18,19. Neuroscientific research has also employed VR systems in combination with functional magnetic resonance imaging (fMRI) during scanning²⁰, in combination with fMRI after scanning^21,22, and in combination with electroencephalography (EEG) during recording^23,24. Software frameworks are needed in order to coordinate the variety of displays and controls that are used for navigation research.

Research that incorporates VR and physiological data poses additional challenges such as data acquisition and synchronization. However, physiological data allows for the investigations of implicit processes that may mediate the relationship between navigation potential and spatial behavior. Indeed, the relationship between stress and navigation has been studied using desktop VR and a combination of different physiological sensors (i.e., heart rate, blood pressure, skin conductance, salivary cortisol, and alpha-amylase)^25,26,27,28. For example, van Gerven and colleagues²⁹ investigated the impact of stress on navigation strategy and performance using a virtual reality version of a Morris water maze task and several physiological measures (e.g., skin conductance, heart rate, blood pressure). Their results revealed that stress predicted navigation strategy in terms of landmark use (i.e., egocentric versus allocentric) but was not related to navigation performance. In general, findings from previous studies are somewhat inconsistent regarding the effect of stress on navigation performance and spatial memory. This pattern may be attributable to the separation of the stressor (e.g., the cold pressor procedure²⁶, the Star Mirror Tracing Task²⁵) from the actual navigation task, the use of simple maze-like virtual environments (e.g., virtual Morris water maze²⁶, virtual radial arm maze²⁸), and differences in methodological details (e.g., type of stressor, type of physiological data). Differences in the format of collected physiological data can also be problematic for the implementation and analysis of such studies.

The Experiments in Virtual Experiments (EVE) framework facilitates the design, implementation, and analysis of VR experiments, especially those with additional peripheral devices (e.g., eye trackers, physiological devices)³⁰. The EVE framework is freely available as an open-source project on GitHub (https://cog-ethz.github.io/EVE/). This framework is based on the popular Unity 3D game engine (https://unity3d.com/) and the MySQL database management system (https://www.mysql.com/). Researchers can use the EVE framework in order to prepare the various stages of a VR experiment, including pre- and post-study questionnaires, baseline measurements for any physiological data, training with the control interface, the main navigation task, and tests for spatial memory of the navigated environment (e.g., judgments of relative direction). Experimenters can also control the synchronization of data from different sources and at different levels of aggregation (e.g., across trials, blocks, or sessions). Data sources may be physical (i.e., connected to the user; see Table of Materials) or virtual (i.e., dependent on interactions between the participant's avatar and the virtual environment). For example, an experiment may require recording heart rate and position/orientation from the participant when that participant's avatar moves through a particular area of the virtual environment. All of this data is automatically stored in a MySQL database and evaluated with replay functions and the R package evertools (https://github.com/cog-ethz/evertools/). Evertools provides exporting functions, basic descriptive statistics, and diagnostic tools for distributions of data.

The EVE framework may be deployed with a variety of physical infrastructures and VR systems. In the present protocol, we describe one particular implementation at the NeuroLab at ETH Zürich (Figure 1). The NeuroLab is a 12 m by 6 m room containing an isolated chamber for conducting EEG experiments, a cubicle containing the VR system (2.6 m x 2.0 m), and a curtained area for attaching physiological sensors. The VR system includes a 55" ultra-high definition television display, a high-end gaming computer, a joystick control interface, and several physiological sensors (see Table of Materials). In the following sections, we describe the protocol for conducting a navigation experiment in the NeuroLab using the EVE framework and physiological sensors, present representative results from one study on stress and navigation, and discuss the opportunities and challenges associated with this system.

Protocol

The following protocol was conducted in accordance with guidelines approved by the Ethics Commission of ETH Zürich as part of the proposal EK 2013-N-73.

1. Recruit and Prepare Participants

1. Select participants with particular demographics (e.g., age, gender, educational background) using a participant recruitment system or mailing list (e.g., UAST; http://www.uast.uzh.ch/).
2. Contact selected participants by e-mail. In this e-mail, remind the participants of the session time and requirements. Let the participants know that they must wear a loose-fitting top (for blood pressure monitoring), refrain from alcohol for 12 h before the experiment, and refrain from several other activities (i.e., caffeine, smoking, eating, and exercise) for 3 h before the experiment.

2. Prepare the Experiment and Physiological Devices Using EVE

1. Before each experimental session, start the computer, the experimenter monitor, and the testing monitor.
2. Ensure that the room fan, the thermometer and the humidity monitor are on.
3. Switch on the machine measuring the electrodermal activity (EDA) and the electrocardiography (ECG; e.g., PowerLab from ADInstruments). See Table of Materials.
4. Open the EDA/ECG software (EVE currently supports Labchart from ADInstruments) and create a new settings file. Select a sampling rate of 1,000 Hz and the appropriate number of channels (e.g., one for EDA and one for ECG). Save this settings file and re-save a version with a different name for each experimental session.
5. For the EDA electrodes, perform an open-circuit zero (i.e., without the electrodes attached to anything) to obtain a baseline measure of system conductivity.
6. Ensure that the control interface (e.g., joystick) is connected to the computer.
7. On the experimenter monitor, open the executable Unity file for the experiment.
   1. Open the "Experiment Settings" menu in EVE, and enter the experiment parameters (e.g., participant ID number, physiological measurement file, experimental condition, room temperature and humidity).
   2. Click "Start Experiment".

3. Experimental Procedure

1. Introduction and consent procedure
   1. Pick up the participant at the agreed meeting location and guide him/her to the laboratory.
   2. Indicate that the session will take approximately 90 min and ask the participant to store his or her watch and/or mobile phone.
   3. Ask the participant to sit in the experimental chair and explain the experimental procedure according to the prepared verbal script.
   4. Ask the participant to read the information sheet and sign the informed consent form.
2. Connection of EDA and ECG sensors
   1. Clean the index finger and the ring finger of the non-dominant hand with a wet tissue without soap. Ensure that they are dry and connect the two EDA electrodes to the medial phalanges.
   2. Clean the skin on the chest where the ECG electrodes will be placed with a wet cloth.
   3. Place the white, black, and red electrodes on the participant's body between the ribs according to Figure 2. Place the white electrode on the upper right abdomen (UR), and the black electrode on the upper left abdomen (UL). Place the red electrode on the lower left abdomen (LL). Ensure that the three electrodes are not directly over a rib.
   4. Connect the three color-coded ECG wires to the corresponding electrodes attached to the participant's body.
3. Pre-experiment questionnaires
   1. Provide the participant with a keyboard and a mouse that will be used to answer the questionnaires (e.g., demographic questions, the first part of the Short Stress State Questionnaire, the Santa Barbara Sense of Direction Scale), and inform him or her that they will be asked a series of questions on the computer.
   2. Inform the participants that they can ask the experimenter questions regarding the questionnaires at any time.
   3. Close the two side walls of the cubicle while the participant is completing the questionnaires.
4. Preparations for physiological measurement. These steps can be conducted while the participant is completing the questionnaires.
   1. Inform the participant that the experimenter will now prepare the physiological devices.
   2. Check that the electrodes are attached to the correct locations.
   3. Attach the blood pressure cuff to the non-dominant arm.
   4. Provide instructions to the participant regarding the accurate measurement of blood pressure. Tell the participant to minimize arm and body movements, keep the blood pressure cuff at heart level, and maintain an upright posture with his or her feet flat on the floor.
   5. Connect the two EDA wires to the electrodes on the fingers.
   6. Turn off the light above the monitor and dim all other overhead lights to the lowest setting.
   7. Hand the joystick to the participant and ensure that the mouse is off the screen of the testing monitor.
   8. Zero the EDA channel in order to obtain a measure of an individual's starting level of skin conductance.
   9. In the EDA/ECG software, open the "Bio Amp" dialog box. Choose the signal range in which the heart beat signal covers around one third of the preview window (5 mV in most cases).
   10. Start recording with the EDA/ECG software, and check whether a signal is visible in the EDA/ECG software window on the experimenter monitor.
   11. Start blood pressure recording by pressing the appropriate button in the blood pressure machine.
   12. Switch to the open Unity program, and press "Start Measurements". A fixation cross should appear.
5. Joystick training and baseline video
   1. Ask the participant to watch and follow the training video that instructs him or her how to use the joystick.
   2. Ask the participants to complete the training maze in order to practice using the joystick. In this training maze, participants are instructed to follow the arrows that indicate a route and collect floating gems.
   3. If the experiment includes sound, place the headphones on the participant's head.
   4. Ask the participant to watch the baseline nature video without moving. This video is used to account for a baseline measurement of the participant's physiological data during the subsequent analysis.
6. Navigation task
   1. Ensure that the participants have read the instructions regarding the to-be-completed navigation task. Inquire as to whether the participant has any questions before the navigation task begins. Tell the participant that they should not ask questions during the navigation task.
   2. Ask the participant to press the trigger on the joystick when he or she is ready to begin the navigation task.
7. Final physiological measures and detachment of physiological sensors
   1. Wait until the system has completed the final blood pressure measurement.
   2. Stop recording EDA and ECG by pressing the stop button in the EDA/ECG software.
   3. Remove the blood pressure cuff.
   4. Remove the EDA electrodes from the participant.
   5. Ask the participants not to remove the ECG electrodes until the end of the experiment.
   6. Remove the joystick and headphones.
8. Post-experiment questionnaires
   1. Provide the participant with a keyboard and a mouse for the post-experiment questionnaires (e.g., the second part of the Short Stress State Questionnaire, the Self-Assessment Manikin, the Simulator Sickness Questionnaire).
   2. Inform the participants that they will be asked another series of questions on the computer and that he or she can ask questions if necessary.
9. End of the experimental session
   1. Inform the participant that the experimental part is now finished. Thank her or him for participating in the experiment.
   2. Tell the participant that he/she can now remove the ECG electrodes.
   3. Pay out the participants and ask them to sign the printed receipt.
   4. Ask if the participant has any questions regarding the purpose of the experiment, and escort him or her outside of the experimental room.

4. After Each Experimental Session

1. Open the "Evaluation" menu in EVE in order to conduct experiment diagnostics (e.g., replay trajectories), and save the physiological measurement files in the EDA/ECG software.
2. In the "Evaluation" menu in EVE, press the "Add Event Marker" button to mark events in the physiological measurement files. This step is critical for the analysis of the physiological data in terms of particular experimental phases.
3. Save the EDA/ECG file in the physiological measurement file in the EDA/ECG software.
4. Export the experimental data for backup using the evertools package.
5. Switch off the EDA/ECG machine and clean the EDA electrodes with alcohol pads.
6. Mark that the participant showed up in the participant recruitment system.

Results

From each participant in the NeuroLab, we typically collect physiological data (e.g., ECG), questionnaire data (e.g., the Santa Barbara Sense of Direction Scale or SBSOD³¹), and navigation data (e.g., paths through the virtual environment). For example, changes in heart rate (derived from ECG data) have been associated with changes in stress states in combination with other physiological³² and self-report measures<...

Discussion

In the present paper, we described a protocol for conducting experiments in VR with physiological devices using the EVE framework. These types of experiments are unique because of additional hardware considerations (e.g., physiological devices and other peripherals), the preparatory steps for collecting physiological data using VR, and data management requirements. The present protocol provides the necessary steps for experimenters that intend to collect data from multiple peripherals simultaneously. For example...

Materials

Name	Company	Catalog Number	Comments
Alienware Area 51 Base	Dell	210-ADHC	Computation
138 cm 4K Ultra-HD LED-TV	Samsung	UE55JU6470U	Display
SureSigns VS2+	Philips Healthcare	863278	Blood Pressure
PowerLab 8/35	AD Instruments	PL3508	Skin Conductance
PowerLab 26T (LTS)	AD Instruments	ML4856	Heart Rate
Extreme 3D Pro Joystick	Logitech	963290-0403	HID