The authors are incredibly grateful to Mr. Li Yan for his help in recruiting pilot participants and acknowledge Ms. Bu Lingyun for her work on drawing pictures. The research was supported by the National Natural Science Foundation of China (grant number T2192931, 72071185), the National Brain Project (grant number&#160;STI2030-Major Projects2022ZD0208500), the National Key Laboratory Project of Human Factors Engineering (grant number SYFD062003), the National Key Laboratory Project of Human Factors Engineering (grant number 6142222210201), and year 2022 Major Projects of Military Logistic Research Grant and Key Project of Air Force Equipment Comprehensive Research&#160;(grant number KJ2022A000415).

All methods described here have been approved by the Institutional Review Board (IRB) of Tsinghua University, and informed consent was obtained from all participants. After completion, all participants were paid $12 (or a gift of equal value).
1. Participant selection
<ol>
	<li>Recruit participants according to a prior study of power analysis using G*Power software35 (see Table of Materials) to ensure the participant number meets the expected sample size given by G*Power, which equals 21. 
	NOTE: The expected sample size given by G*Power refers to the estimated number of participants needed in a study to achieve a desired level of statistical power based on the specified effect size, significance level, and statistical test used. The expected sample size is an important consideration in research planning. It helps researchers determine the feasibility and cost-effectiveness of their study design and ensures that the study has sufficient statistical power to detect meaningful effects.</li>
	<li>Ensure none participants have a history of epilepsy, heart or brain disease, recent endocrine or psychiatric medication, or severe skin allergies.</li>
	<li>Use the Snellen eye chart (use the metric system 6/6)36 to confirm that participants&#39; vision is normal or corrected-to-normal and that they don&#39;t have any visual impairments like color blindness or color weakness.</li>
	<li>Ensure that none of the participants have consumed alcohol or drugs within the previous 24 h that may affect their ability to fly.</li>
	<li>Ensure the participants have had no less than 6 h of sleep and are in good mental condition before the experiment.</li>
</ol>
2. Flight simulator hardware
<ol>
	<li>Check that all of the flight simulator&#39;s hardware is complete. According to its function, this hardware is organized into three modules (Table 1) (see Table of Materials). 
	NOTE: Researchers need to touch a metal rod before touching the equipment to avoid the risk posed by electrostatic induction.
	<ol>
		<li>Check the components of the VR, HMD (head-mounted display), and eye-tracking module with the help of Table 2.</li>
		<li>Ensure that all the flight simulator PC module components meet the following minimum requirements: 3.6G Hz processor, 4G internal memory, 64-bit operating system, and graphics card. The system will support all flight controllers.</li>
		<li>Check the components of the flight control module and ensure the device&#39;s parameter configuration is consistent with Table 3.</li>
	</ol>
	</li>
	<li>Install the hardware of the flight simulator according to the layout in Figure 1. Figure 2 shows how the hardware is connected.
	<ol>
		<li>Join the throttle and control panel physically and treat them as a unit.</li>
		<li>Connect the throttle, joystick, and pedal to the flight simulator PC module via USB.</li>
		<li>Connect the HMD to the flight simulator PC module via the link box.</li>
		<li>Connect the base stations and VR controllers to the HMD via the VR software on the PC.</li>
	</ol>
	</li>
</ol>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td rowspan="2">VR head-mounted display(HMD) and eye tracking module</td>
			<td>1. Base Station</td>
		</tr>
		<tr>
			<td>2. VR HMD</td>
		</tr>
		<tr>
			<td>Flight Simulator PC module</td>
			<td>3. Flight Simulator PC</td>
		</tr>
		<tr>
			<td rowspan="3">Flight control module</td>
			<td>4. Flight Throttle</td>
		</tr>
		<tr>
			<td>5. Flight Joystick</td>
		</tr>
		<tr>
			<td>6. Flight Pedal</td>
		</tr>
	</tbody>
</table>
Table 1: Components of the three modules of flight simulator hardware.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Main component</td>
			<td>Accessories</td>
		</tr>
		<tr>
			<td rowspan="4">VR HMD</td>
			<td>Headset cable (attached)</td>
		</tr>
		<tr>
			<td>Face cushion (attached)</td>
		</tr>
		<tr>
			<td>Cleaning cloth</td>
		</tr>
		<tr>
			<td>Earphone hole cap &#215; 2&#160;</td>
		</tr>
		<tr>
			<td rowspan="4">Link box</td>
			<td>&#160;Power adapter</td>
		</tr>
		<tr>
			<td>DisplayPort cable</td>
		</tr>
		<tr>
			<td>USB 3.0 cable</td>
		</tr>
		<tr>
			<td>Mounting pad</td>
		</tr>
		<tr>
			<td rowspan="3">Controllers (2018) &#215; 2</td>
			<td>Power adapters &#215; 2&#160;</td>
		</tr>
		<tr>
			<td>Lanyards &#215; 2&#160;</td>
		</tr>
		<tr>
			<td>Micro-USB cables &#215; 2&#160;</td>
		</tr>
		<tr>
			<td rowspan="2">Base Station 2.0 &#215; 2</td>
			<td>Power adapters &#215; 2&#160;</td>
		</tr>
		<tr>
			<td>Mounting Kit (2 mounts, 4 screws, and 4 wall anchors)</td>
		</tr>
	</tbody>
</table>
Table 2: List of components of the VR HMD and eye-tracking module.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Device</td>
			<td>Parameter configuration</td>
		</tr>
		<tr>
			<td rowspan="5">Flight Joystick</td>
			<td>Nineteen action buttons</td>
		</tr>
		<tr>
			<td>One 8-way &#34;point of view&#34; hat</td>
		</tr>
		<tr>
			<td>Several 3D magnetic sensors</td>
		</tr>
		<tr>
			<td>One 5-coil spring system</td>
		</tr>
		<tr>
			<td>One 16-bit resolution (65536 x 65536 values).</td>
		</tr>
		<tr>
			<td rowspan="3">Flight Control Panel</td>
			<td>Fifteen action buttons</td>
		</tr>
		<tr>
			<td>One TRIM wheel</td>
		</tr>
		<tr>
			<td>Five programmable LEDs</td>
		</tr>
		<tr>
			<td rowspan="5">Flight Throttle</td>
			<td>Seventeen action buttons</td>
		</tr>
		<tr>
			<td>One mouse hat with a push button</td>
		</tr>
		<tr>
			<td>One 8-way &#34;point of view&#34; hat</td>
		</tr>
		<tr>
			<td>Several 3D magnetic sensors</td>
		</tr>
		<tr>
			<td>Two 14-bit resolution</td>
		</tr>
		<tr>
			<td rowspan="2">Flight Pedal</td>
			<td>Tension between 2.5 kg and 5 kg</td>
		</tr>
		<tr>
			<td>Angle between 35&#176; and 75&#176;</td>
		</tr>
	</tbody>
</table>
Table 3: The parameter configuration of the devices of the flight control module.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/65170/65170fig01.jpg" /> 
Figure 1: The layout of the VR flight simulator hardware. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/65170/65170fig02.jpg" /> 
Figure 2: The connection of flight simulator hardware. (A) Flight control module. The throttle and control panel are physically joined and treated as a unit. If the term &#34;throttle&#34; is used in this study, it refers to both the throttle and the control panel. (B) Flight simulator PC module. A computer that meets the requirements outlined in step 2.2. (C) HMD and eye-tracking module. The software development kits (SDKs) for eye tracking and the 3D engine are kept in sync when installed on the same computer. Therefore, the eye-tracking functions and the operating system interact and work together. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
3. Flight simulator software
<ol>
	<li>Ensure all the software (see Table of Materials) has been installed before the experiment begins. Information about all software used in the experiment is shown in Table 4.</li>
</ol>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Name</td>
			<td colspan="4">Description</td>
		</tr>
		<tr>
			<td>VR software</td>
			<td colspan="4">A widely used tool for experiencing VR content on the hardware.</td>
		</tr>
		<tr>
			<td>VR app store</td>
			<td colspan="4">The app store for virtual reality where customers can explore, create, connect, and experience the content they love and need.</td>
		</tr>
		<tr>
			<td>Eye-tracking software</td>
			<td colspan="4">Eye-tracking software developed by the research team via Eye-tracking and 3D engine SDKs.</td>
		</tr>
		<tr>
			<td>FlySimulator</td>
			<td colspan="4">The main program of the flight simulator software, developed by the research team.</td>
		</tr>
		<tr>
			<td>Screen recording software</td>
			<td colspan="4">A free and open source software for video recording and live streaming.</td>
		</tr>
	</tbody>
</table>
Table 4: Information about all software used in the experiment.
4. Preparation before launching the flight simulator
NOTE: If this is one&#39;s first time running the eye-tracking program, perform the additional steps according to Figure 3. The eye-tracking program will activate automatically after the initial run.
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/65170/65170fig03.jpg" /> 
Figure 3: The additional steps when running the eye-tracking program for the first time. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<ol>
	<li>Ensure that the HMD is turned on and connected to the computer.</li>
	<li>Place the base station diagonally to ensure the HMD can always be inside the base station&#39;s monitoring range. Keep the base station fixed to keep a stable VR environment.</li>
	<li>Set up a standing-only play area according to the prompts given by the VR software on the computer. 
	NOTE: The prompt allows a room-scale area or sets up a standing-only play area. Standing-only mode is generally used to experience VR scenes that do not require walking and can be selected when the user has limited space to move around. Therefore, in this experiment, set up a standing-only play area.</li>
	<li>Set the eye-tracking calibration. The eye-tracking system must be recalibrated every time the participant is switched.
	<ol>
		<li>Confirm that the participants are not using contact lenses, which causes eye tracking to malfunction.</li>
		<li>Open the eye-tracking calibration program using the VR controllers (see Table of Materials).</li>
		<li>Adjust the device height, interpupillary distance (IDP), and gaze point as the system directs.</li>
		<li>Get the participant&#39;s eyes to light up each spot for 2 s in a clockwise direction to verify the effectiveness of the eye-tracking calibration.</li>
	</ol>
	</li>
	<li>Connect the tuned flight control module and large screen display (i.e., at least a 27 inch monitor) to the same computer as the VR HMD. The screen allows the experimenter to view what is happening in the VR HMD simultaneously.</li>
</ol>
5. Experimental procedure
NOTE: The experiment is divided into four steps: &#34;collect information,&#34; &#34;introduce the task and operation,&#34; &#34;practice before the experiment,&#34; and &#34;conduct a formal experiment.&#34; The experimental process is summarized in Figure 4.
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/65170/65170fig04.jpg" /> 
Figure 4: The flowchart of the experiment. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<ol>
	<li>Collect information from participants (15 min).
	<ol>
		<li>Fill out the payment details form with the payment information of the participants.</li>
		<li>Inform the participants to read the informed consent document and sign it.</li>
	</ol>
	</li>
	<li>Introduce the task and simulator operation to the participants (5 min).
	<ol>
		<li>Explain to the participants the traffic pattern and map (Figure 5).</li>
		<li>Instruct participants on how to use the VR flight simulator.</li>
		<li>Inform the participants that they can leave the experiment if they experience discomfort while simulating a flight.</li>
	</ol>
	</li>
	<li>Ask the participants to practice using the VR flight simulator (15 min).
	<ol>
		<li>Fit the VR HMD to the participants and calibrate the eye movements. Adjust the device height, interpupillary distance (IDP), and gaze point as instructed by the system.</li>
		<li>Open the screen recording software.</li>
		<li>Launch the FlySimulator program to assist participants with flight training.</li>
		<li>Instruct the participants to use the flight pedal, joystick, and throttle in concert to steer the aircraft as closely as possible to the reference line of flight (Figure 5).</li>
		<li>Reset the flight throttle and engine switch buttons after exiting the FlySimulator program, and stop recording the screen after the subject has completed the practice.</li>
	</ol>
	</li>
	<li>Conduct formal flight experiments (20 min).
	<ol>
		<li>Put on the VR HMD for the participants, and run OBS Studio to start recording the screen.</li>
		<li>Start the FlySimulator program, choose a fit perspective, start the engine, let go of the parking brake, and put the flight throttle to full.</li>
		<li>Exit the FlySimulator program, then reset the flight throttle and engine switch buttons. Stop recording the screen.</li>
	</ol>
	</li>
</ol>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/65170/65170fig05.jpg" /> 
Figure 5: The traffic pattern for the VR flight simulator. The pitch angle is the rotation angle around the left and right axes of the aircraft, and the reference line (or the center reference line) is right in the middle of the red and green lines. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig05large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
6. Data analysis
<ol>
	<li>Analyze eye movement data.
	<ol>
		<li>Divide five area of interest (AOI) zones in this study; for more information about the calculation of AOIs, see Figure 6, based on the actual panel of the flight cockpit corresponding to the flight cockpit instrument (Figure 7).</li>
		<li>Calculate the AOI difference test using the percentage of dwell time in the AOI zone37, given the equipment feature of the VR oculomotor used in this study. 
		NOTE: The average percentage of dwell time is the cumulative percentage spent looking within an AOI divided by the total fixation time and averaged across participants.</li>
	</ol>
	</li>
	<li>Analyze flight performance data with custom-developed programming scripts using Python 3.10. The core algorithm for performance metrics was adapted from the QAR analysis method, inspired by the 35 BAE-146 aircraft QAR data provided by the National Aeronautics and Space Administration (NASA)38. 
	NOTE: Flight performance indicators for this study: the total flight time (the total length of time from takeoff to landing for each participant, in seconds), pitch angle 1 s before landing (pitch angle of the aircraft 1 s before landing, obtained from raw data, in degrees), mean distance to reference line (mean error of the spatial distance between the aircraft and the reference line during flight, in meters), and standard deviation of distance to reference line (standard deviation of the spatial distance between the plane and the reference line during flight, in meters).</li>
	<li>Conduct the statistical analyses.
	<ol>
		<li>Use the Shapiro-Wilk test39 to confirm the data&#39;s normality.</li>
		<li>Use statistical software (see Table of Materials) for descriptive statistics, Mann-Whitney U test, and Student&#39;s t-test.</li>
		<li>Depict the violin plots to indicate better the shape of the data distribution40. 
		NOTE: The explanation for the effect size measured using Cohen&#39;s d is as follows: 0.1 is very small, 0.2 is small, 0.5 is medium, 0.8 is large, 1.2 is very large, and 2.0 is huge41,42. The significance level was set at p &#60; 0.05.</li>
	</ol>
	</li>
</ol>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/65170/65170fig06.jpg" /> 
Figure 6: AOI preprocessing and calculating flow process. Sections 1 to 4 describe how the present study processed the pilots&#39; eye movement data up to the independent-sample t-test. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig06large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/65170/65170fig07.jpg" /> 
Figure 7: Schematic diagram of AOI division of flight instrument. The function of instruments: (A) The airspeed indicator indicates &#65279;the aircraft&#39;s speed relative to the air. (B) The altitude indicator shows the aircraft&#39;s pitch and roll altitude. (C) The vertical speed indicator indicates the aircraft&#39;s ascent or descent speed. (D) The altitude indicator indicates the aircraft&#39;s barometric altitude. (E) The engine speed indicator indicates the speed of the aircraft engine. <a href="https://www.jove.com/files/ftp_upload/65170/65170fig07large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

&#65279;The authors declared no financial disclosure or conflicts of interest.

&#65279;The current study assessed if participants with flight experience had better flight performance than those without flight experience in a VR-based flight simulator. More importantly, it evaluated whether a more optimized eye movement pattern could be found in these participants with better flight performance. The results have significant differences between participants with and without flight experience in three key flying QAR indicators: pitch angle 1 s before landing, the mean distance to the reference line, a...

The European Aviation Safety Agency (2012) categorizes flight simulators as training facilities, flight and navigation program trainers, flight training equipment, and complete flight simulators1. To date, a range of flight simulators is available for training, from low-level tabletop systems to highly complicated motion-based full flight simulators2. The traditional simulator includes a flight dynamics model, a system simulation, a hardware cockpit, an external visualization, and an optional motion simulation3.
These traditional flight simulators have some advantages as effective flight training equipment. However, their cost is high and environmentally unfriendly, as the drive of each system requires substantial electrical energy, especially a full flight simulator, which requires high temperature and high-pressure fluid or air pressure, consumes much power and generates a lot of noise4.
However, a simple desktop simulator system is flexible and low-cost, with lower immersion and fewer interactions than a full flight simulator2. Therefore, it is essential to develop new flight simulators that combine the advantages of desktop systems and full flight simulators (in other words, the flexibility of a tabletop simulation and the immersion and interaction level close to a full flight simulator).
With the development of computer technology, especially virtual reality (VR) technology, a new type of flight simulator based on emerging VR technology is becoming a reality. The VR-based flight simulator is flexible, portable, low-cost, and has fewer space requirements than conventional flight simulators5. Researchers have created flight simulators based on VR technology over the past 20 years6,7,8,9,10,11; however, these VR flight simulators are mainly for flight training, and there are few for pilot selection. Still, with cost reduction and technology enhancement, VR-based simulators are changing and becoming feasible for personal selection. Some studies have used VR-based simulators for personal selection in different domains: Schijven et al.12 selected surgical trainees using a virtual reality simulator. Huang et al.13 developed a psychology selection instrument based on virtual reality technology for air force pilot recruitment. Wojciechowski and Wojtowicz14 assessed a candidate&#39;s capabilities as an unmanned aerial vehicle (UAV) pilot based on VR technology. Given that pilot selection is critical for the aviation industry, it is pressing to develop a new VR-based flight simulator focusing on pilot selection, as large-scale pilot selection is susceptible to the cost of the simulator and demands in the portability simulator system.
Eye movements provide cues for a pilot&#39;s performance. Different studies have found that the eye-scanning mode distinguishes the performance between expert and novice pilots. By comparing the scanning pattern between experts and novices, experts&#39; efficient and structural eye movement behavior and the inadequate scanning methods of beginners could be differentiated. Several aviation studies have found that pilots&#39; eye-scanning strategy highly relates to the level of expertise15,16,17,18,19,20,21,22,23,24. According to Bellenkes et al.25, the duration of experts&#39; fixations is shorter, and the frequency of their fixations on instruments is higher than that of novices. Almost the same conclusion was drawn by Kasarskis et al.26, who discovered that expert pilots have more fixations combined with shorter durations than novices, suggested that expert pilots have a better visual mode than novices. In another study, Lorenz et al.27 found that experts spend more time looking outside the cockpit than novices. These results have great practical value in the selection of newcomers.
Flight performance assessment is another critical factor for pilot selection. However, the following problems exist in pilot flight performance evaluation: conflicting expert opinions, more selection norms, and a unified selection theory. In the driving field, Horrey et al.28 compared the absolute value of lane departure from the centerline for different experimental conditions to assess driving performance. Back to the aviation domain, the flight quick access recorder (QAR) records all sorts of pilot manipulation parameters, aircraft parameters, environments, and warning information during flight29. More specifically, as the QAR indicators, the pitch angle is the rotation angle around the left and right axes of the aircraft30, and the reference line (or the center reference line) is right in the middle of the red and green lines28; these two flight parameters are used to evaluate the flight performance of participants with or without experience in the current study. These QAR data can be used to evaluate flight performance, yet to our best knowledge, they have seldom been used for personal training and selection in scientific research31,32.
Measurements of eye movement patterns can be used to assess and predict flight performance and guide pilot training and selection. Gerathewohl33 stated that the eye is the most important sensory organ of the pilot, processing 80% of the flight information. Pilots must acquire visual information from instruments in the cockpit and integrate it into a coherent image to manage the flight22. Further, optimal scanning behavior is essential to accomplish better flight performance15. However, no affordable flight simulator currently integrates an eye tracker to facilitate quantitative studies of the relationship between eye movements and flight performance.
The current study developed a new VR flight simulator to assess if participants with flight experience had better flight performance than those without flight experience. The VR flight simulator integrates eye tracking and a flight dynamics system allowing eye-movement pattern analysis and flight performance evaluation. In particular, it is worth mentioning that the VR flight simulator uses a VR eye tracker34, not a glass-like or desktop eye tracker, to analyze the area of interest (AOI)-based eye movement without time-consuming frame counting.
Finally, the present work can lead to an omnibus measurement for pilot selection in the future, from eye scanning path to objective flying performance data. With the help of the virtual flight simulator, the cost of flight selection will be significantly reduced, and the norm of pilots can be formed based on extensive data gathering. The work fills a gap between conventional and desktop simulators for flight selection needs.

<table><tbody><tr><td>3D engine SDK</td><td>Epic Games</td><td>Unreal Engine 4&lt;br/&gt; GameAnalytics Unreal SDK</td><td>This SDK is a powerful yet flexible free analytics tool designed for games.</td></tr><tr><td>CPU</td><td>Intel</td><td>IntelCore i9</td><td>One of the most powerful CPU on the mainstream market.</td></tr><tr><td>Eye tracking SDK</td><td>Tobii</td><td>Tobii XR SDK</td><td>This SDK provide device agnostic access to eye tracking data to allow development for headsets from many different hardware vendors and is not limited to devices using Tobii Eye Tracking hardware.</td></tr><tr><td>Eye tracking software</td><td>Developed by the research team</td><td></td><td>A program that tracks the movement of a person&#039;s eyes while they are using a virtual reality HMD.</td></tr><tr><td>FlySimulator program</td><td>Developed by the research team</td><td></td><td>A software that simulates flying experiences in a virtual environment, using VR HMD and hand-held controllers.</td></tr><tr><td>Graphics card</td><td>NVIDIA</td><td>GeForce RTX 3090&lt;br/&gt; 10496 NVIDIA CUDA Cores&lt;br/&gt; 1.70 GHz Boost Clock&amp;nbsp;&amp;nbsp;&lt;br/&gt; 24 GB Memory Size&lt;br/&gt; GDDR6X Memory Type</td><td>One of the most powerful graphics card on the mainstream market.</td></tr><tr><td>Operating system (OS)</td><td>Microsoft</td><td>Windows XP</td><td>An operating system (OS) developed and exclusively distributed by Microsoft Corporation</td></tr><tr><td>Replica control panel</td><td>THRUSTMASTER</td><td>2960720 2971004 2962072 2960748 2960769</td><td>U.S. Air Force A-10C attack aircraft HOTAS</td></tr><tr><td>Replica joystick</td><td>THRUSTMASTER</td><td>2960720</td><td>U.S. Air Force A-10C attack aircraft HOTAS</td></tr><tr><td>Replica pedal</td><td>THRUSTMASTER</td><td></td><td>TPR pendular rudder</td></tr><tr><td>Replica throttle</td><td>THRUSTMASTER</td><td></td><td>U.S. Air Force A-10C attack aircraft HOTAS</td></tr><tr><td>Screen connected to PC</td><td>Redmi</td><td>RMMNT27NF, 27-inch, 1920 X 1080 resolution ratio</td><td>Screen allows the experimenter to simultaneously view what is happening in the VR HMD</td></tr><tr><td>Screen recording software</td><td>OBS Project</td><td>OBS Studio Version 28.0</td><td>A free and open source software for video recording and live streaming</td></tr><tr><td>Statistical power analysis software</td><td>Open-Source</td><td>G*power Version 3.1.9.6</td><td>A free and user-friendly tool for estimating statistical power and sample size.</td></tr><tr><td>Statistical software</td><td>IBM</td><td>SPSS Version 24.0</td><td>A powerful statistical software platform</td></tr><tr><td>Versatile statistics tool</td><td>GraphPad Software</td><td>GraphPad Prism Version 9.4.0</td><td>A versatile statistics tool purpose-built for scientists-not statisticians</td></tr><tr><td>VR app store</td><td>HTC Corporation</td><td>VIVE Software 2.0.17.6 / 2.1.17.6</td><td>An app store for virtual reality where customers can explore, create, connect, and experience the content they love and need.</td></tr><tr><td>VR head-mounted display (HMD)</td><td>HTC Corporation</td><td>VIVE Pro Eye</td><td>A VR headset with precision eye tracking</td></tr><tr><td>VR software</td><td>Steam</td><td>Steam VR Version 1.23</td><td>A tool for experiencing VR content on the hardware</td></tr></tbody></table>

evaluating flight performance and eye movement patterns using virtual reality flight simulator

Efficient and economical performance evaluation of pilots has become critical to the aviation industry. With the development of virtual reality (VR) and the combination of eye-tracking technology, solutions to meet these needs are becoming a reality. Previous studies have explored VR-based flight simulators, focusing mainly on technology validation and flight training. The current study developed a new VR flight simulator to evaluate pilots' flight performance based on eye movement and flight indicators in a 3D immersive scene. During the experiment, 46 participants were recruited: 23 professional pilots and 23 college students without flight experience. The experiment results showed significant differences in flight performance between participants with and without flight experience, the former being higher than the latter. In contrast, those with flight experience showed more structured and efficient eye-movement patterns. These results of the differentiation of flight performance demonstrate the validity of the current VR flight simulator as a flight performance assessment method. The different eye-movement patterns with flight experience provide the basis for future flight selection. However, this VR-based flight simulator has shortcomings like motion feedback compared to traditional flight simulators. This flight simulator platform is highly flexible except for the apparent low cost. It can meet the diverse needs of researchers (e.g., measuring situation awareness, VR sickness, and workload by adding relevant scales).

For the current experiment, 23 experts with flight experience and 23 novices without flight experience were chosen. The participants were between 25 and 58 years of age (experts: M = 32.52 years, SD = 7.28 years; novices: M = 29.57 years, SD = 5.74 years). The gender of all participants was male. All the novices were recruited from Tsinghua University (students or faculty), and all the experts were from China Eastern Airlines.
Eye movement<b...

Watch this Scientific Journal Video about Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator at JoVE.com

Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator

A new virtual reality flight simulator was built, which enables efficient and low-cost evaluation of flight performance and eye movement patterns. It also provides a high-potential research tool for ergonomics and other research.

Efficient and economical performance evaluation of pilots has become critical to the aviation industry. With the development of virtual reality (VR) and ...

evaluating-flight-performance-eye-movement-patterns-using-virtual

Research

JoVE Journal

Behavior

868 Views.  Tsinghua University. A new virtual reality flight simulator was built, which enables efficient and low-cost evaluation of flight performance and eye movement patterns. It also provides a high-potential research tool for ergonomics and other research.

Video: Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator

Development of a Virtual Reality Assessment of Everyday Living Skills

Cognitive impairments affect the majority of patients with schizophrenia and these impairments predict poor long term psychosocial outcomes.&#160; Treatment studies aimed at cognitive impairment in patients with schizophrenia not only require demonstration of improvements on cognitive tests, but also evidence that any cognitive changes lead to clinically meaningful improvements.&#160; Measures of &#8220;functional capacity&#8221; index the extent to which individuals have the potential to perform skills required for real world functioning. &#160;Current data do not support the recommendation of any single instrument for measurement of functional capacity.&#160; The Virtual Reality Functional Capacity Assessment Tool (VRFCAT) is a novel, interactive gaming based measure of functional capacity that uses a realistic simulated environment to recreate routine activities of daily living. Studies are currently underway to evaluate and establish the VRFCAT&#8217;s sensitivity, reliability, validity, and practicality.&#160;This new measure of functional capacity is practical, relevant, easy to use, and has several features that improve validity and sensitivity of measurement of function in clinical trials of patients with CNS disorders.

A challenge for proving treatment efficacy for cognitive impairments in schizophrenia is finding the optimizing measurement of skills related to everyday functioning.&#160; The Virtual Reality Functional Capacity Assessment Tool (VRFCAT) is an interactive gaming based computerized measure aimed at skills associated with everyday functioning, including baseline impairments and treatment related changes.

Cognitive impairments affect the majority of patients with schizophrenia and these impairments predict poor long term psychosocial outcomes.&#160; ...

Driving Simulation in the Clinic: Testing Visual Exploratory Behavior in Daily Life Activities in Patients with Visual Field Defects

Patients suffering from homonymous hemianopia after infarction of the posterior cerebral artery (PCA) report different degrees of constraint in daily life, despite similar visual deficits. We assume this could be due to variable development of compensatory strategies such as altered visual scanning behavior. Scanning compensatory therapy (SCT) is studied as part of the visual training after infarction next to vision restoration therapy. SCT consists of learning to make larger eye movements into the blind field enlarging the visual field of search, which has been proven to be the most useful strategy1, not only in natural search tasks but also in mastering daily life activities2. Nevertheless, in clinical routine it is difficult to identify individual levels and training effects of compensatory behavior, since it requires measurement of eye movements in a head unrestrained condition. Studies demonstrated that unrestrained head movements alter the visual exploratory behavior compared to a head-restrained laboratory condition3. Martin et al.4 and Hayhoe et al.5 showed that behavior demonstrated in a laboratory setting cannot be assigned easily to a natural condition. Hence, our goal was to develop a study set-up which uncovers different compensatory oculomotor strategies quickly in a realistic testing situation: Patients are tested in the clinical environment in a driving simulator. SILAB software (Wuerzburg Institute for Traffic Sciences GmbH (WIVW)) was used to program driving scenarios of varying complexity and recording the driver's performance. The software was combined with a head mounted infrared video pupil tracker, recording head- and eye-movements (EyeSeeCam, University of Munich Hospital, Clinical Neurosciences).
The positioning of the patient in the driving simulator and the positioning, adjustment and calibration of the camera is demonstrated. Typical performances of a patient with and without compensatory strategy and a healthy control are illustrated in this pilot study. Different oculomotor behaviors (frequency and amplitude of eye- and head-movements) are evaluated very quickly during the drive itself by dynamic overlay pictures indicating where the subjects gaze is located on the screen, and by analyzing the data. Compensatory gaze behavior in a patient leads to a driving performance comparable to a healthy control, while the performance of a patient without compensatory behavior is significantly worse. The data of eye- and head-movement-behavior as well as driving performance are discussed with respect to different oculomotor strategies and in a broader context with respect to possible training effects throughout the testing session and implications on rehabilitation potential.

Patients with visual deficits after stroke report about different constraints in daily life most likely due to variable compensatory strategies, which are difficult to differentiate in clinical routine. We present a clinical set-up which allows measurement of different compensatory head- and eye-movement-strategies and evaluating their effects on driving performance.

Patients suffering from homonymous hemianopia after infarction of the posterior cerebral artery (PCA) report different degrees of constraint in daily ...

Medicine

Measuring the Kinematics of Daily Living Movements with Motion Capture Systems in Virtual Reality

The inability to complete instrumental activities of daily living (IADL) is a precursor to various neuropsychological diseases. Questionnaire-based assessments of IADL are easy to use but prone to subjective bias. Here, we describe a novel virtual reality (VR) test to assess two complex IADL tasks: handling financial transactions and using public transportation. While a participant performs the tasks in a VR setting, a motion capture system traces the position and orientation of the dominant hand and head in a three-dimensional Cartesian coordinate system. Kinematic raw data are collected and converted into &#39;kinematic performance measures,&#39; i.e., motion trajectory, moving distance, and time to completion. Motion trajectory is the path of a particular body part (e.g., dominant hand or head) in space. Moving distance refers to the total distance of the trajectory, and time to completion is how long it took to complete an IADL task. These kinematic measures could discriminate patients with cognitive impairment from healthy controls. The development of this kinematic measuring protocol allows detection of early IADL-related cognitive impairments.

We designed a virtual reality test to assess instrumental activities of daily living (IADL) with a motion capture system. We propose a detailed kinematic analysis to interpret the participant&#39;s various movements, including trajectory, moving distance, and time to completion to evaluate IADL capabilities.

The inability to complete instrumental activities of daily living (IADL) is a precursor to various neuropsychological diseases. Questionnaire-based ...

A Networked Desktop Virtual Reality Setup for Decision Science and Navigation Experiments with Multiple Participants

Investigating the interactions among multiple participants is a challenge for researchers from various disciplines, including the decision sciences and spatial cognition. With a local area network and dedicated software platform, experimenters can efficiently monitor the behavior of the participants that are simultaneously immersed in a desktop virtual environment and digitalize the collected data. These capabilities allow for experimental designs in spatial cognition and navigation research that would be difficult (if not impossible) to conduct in the real world. Possible experimental variations include stress during an evacuation, cooperative and competitive search tasks, and other contextual factors that may influence emergent crowd behavior. However, such a laboratory requires maintenance and strict protocols for data collection in a controlled setting. While the external validity of laboratory studies with human participants is sometimes questioned, a number of recent papers suggest that the correspondence between real and virtual environments may be sufficient for studying social behavior in terms of trajectories, hesitations, and spatial decisions. In this article, we describe a method for conducting experiments on decision-making and navigation with up to 36 participants in a networked desktop virtual reality setup (i.e., the Decision Science Laboratory or DeSciL). This experiment protocol can be adapted and applied by other researchers in order to set up a networked desktop virtual reality laboratory.

This paper describes a method for conducting multi-user experiments on decision-making and navigation using a networked computer laboratory.

Investigating the interactions among multiple participants is a challenge for researchers from various disciplines, including the decision sciences and ...

Virtual Reality Experiments with Physiological Measures

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and laboratory experiments, respectively. VR is especially useful for geographic visualizations and investigations of spatial behavior. In spatial behavior research, VR provides a platform for studying the relationship between navigation and physiological measures (e.g., skin conductance, heart rate, blood pressure). Specifically, physiological measures allow researchers to address novel questions and constrain previous theories of spatial abilities, strategies, and performance. For example, individual differences in navigation performance may be explained by the extent to which changes in arousal mediate the effects of task difficulty. However, the complexities in the design and implementation of VR experiments can distract experimenters from their primary research goals and introduce irregularities in data collection and analysis. To address these challenges, the Experiments in Virtual Environments (EVE) framework includes standardized modules such as participant training with the control interface, data collection using questionnaires, the synchronization of physiological measurements, and data storage. EVE also provides the necessary infrastructure for data management, visualization, and evaluation. The present paper describes a protocol that employs the EVE framework to conduct navigation experiments in VR with physiological sensors. The protocol lists the steps necessary for recruiting participants, attaching the physiological sensors, administering the experiment using EVE, and assessing the collected data with EVE evaluation tools. Overall, this protocol will facilitate future research by streamlining the design and implementation of VR experiments with physiological sensors.

Virtual reality (VR) experiments can be difficult to implement and require meticulous planning. This protocol describes a method for the design and implementation of VR experiments that collect physiological data from human participants. The Experiments in Virtual Environments (EVE) framework is employed to accelerate this process.

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and ...

Using a Virtual Reality Walking Simulator to Investigate Pedestrian Behavior

To cross a road successfully, individuals must coordinate their movements with moving vehicles. This paper describes use of a walking simulator in which people walk on a treadmill to intercept gaps between two moving vehicles in an immersive virtual environment. Virtual reality allows for a safe and ecologically varied investigation of gap crossing behavior. Manipulating the initial starting distance can further the understanding of a participant&#8217;s speed regulation while approaching a gap. The speed profile may be assessed for various gap crossing variables, such as initial distance, vehicle size, and gap size. Each walking simulation results in a position/time series that can inform how velocity is adjusted differently depending on the gap characteristics. This methodology can be used by researchers investigating pedestrian behavior and behavioral dynamics while employing human participants in a safe and realistic setting.

This protocol describes use of a walking simulator that serves as a safe and ecologically valid method to study pedestrian behavior in the presence of moving traffic.

To cross a road successfully, individuals must coordinate their movements with moving vehicles. This paper describes use of a walking simulator in which ...

Controlled Rotation of Human Observers in a Virtual Reality Environment

The low cost and availability of Virtual Reality (VR) systems have supported a recent acceleration of research into perception and behavior under more naturalistic, multisensory, and immersive conditions. One area of research that has particularly benefited from the use of VR systems is multisensory integration, for example, the integration of visual and vestibular cues to give rise to a sense of self-motion. For this reason, an accessible method for the controlled physical rotation of an observer in a virtual environment represents a useful innovation. This paper presents a method for automating the rotation of an office swivel chair along with a method for integrating that motion into a VR experience. Using an example experiment, it is demonstrated that the physical motion, thus produced, is integrated with the visual experience of an observer in a way consistent with expectations; high integration when the motion is congruent with the visual stimulus and low integration when the motion is incongruent.

The controlled physical rotation of a human observer is desirable for certain experimental, recreational, and educational applications. This paper outlines a method for converting an office swivel chair into a medium for controlled physical rotation in a virtual reality environment.

The low cost and availability of Virtual Reality (VR) systems have supported a recent acceleration of research into perception and behavior under more ...

Eye Tracking During A Complex Aviation Task For Insights Into Information Processing

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill performance. Previous work has shown that traditional and advanced gaze metrics reliably demonstrate robust differences in pilot expertise, cognitive load, fatigue, and even situation awareness (SA).
This study describes the methodology for using a wearable eye tracker and gaze mapping algorithm that captures naturalistic head and eye movements (i.e., gaze) in a high-fidelity flight motionless simulator. The method outlined in this paper describes the area of interest (AOI)-based gaze analyses, which provides more context related to where participants are looking, and dwell time duration, which indicates how efficiently they are processing the fixated information. The protocol illustrates the utility of a wearable eye tracker and computer vision algorithm to assess changes in gaze behavior in response to an unexpected in-flight emergency.
Representative results demonstrated that gaze was significantly impacted when the emergency event was introduced. Specifically, attention allocation, gaze dispersion, and gaze sequence complexity significantly decreased and became highly concentrated on looking outside the front window and at the airspeed gauge during the emergency scenario (all p values &#60; 0.05). The utility and limitations of employing a wearable eye tracker in a high-fidelity motionless flight simulation environment to understand the spatiotemporal characteristics of gaze behavior and its relation to information processing in the aviation domain are discussed.

Eye tracking is a non-invasive method to probe information processing. This article describes how eye tracking can be used to study gaze behavior during a flight simulation emergency task in low-time pilots (i.e., &lt;350 flight hours).

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill ...

Motor Imagery Using VR Digital Avatars in a Brain-Computer Interface System

Motor Imagery Performance Through Embodied Digital Twins in a Virtual Reality-Enabled Brain-Computer Interface Environment

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) technologies with the customization of three-dimensional (3D) avatars. Traditional approaches to rehabilitation often fail to fully engage patients, primarily due to their inability to provide a deeply immersive and interactive experience. This research endeavors to fill this gap by utilizing motor imagery (MI) techniques, where participants visualize physical movements without actual execution. This method capitalizes on the brain's neural mechanisms, activating areas involved in movement execution when imagining movements, thereby facilitating the recovery process. The integration of VR's immersive capabilities with the precision of electroencephalography (EEG) to capture and interpret brain activity associated with imagined movements forms the core of this system. Digital Twins in the form of personalized 3D avatars are employed to significantly enhance the sense of immersion within the virtual environment. This heightened sense of embodiment is crucial for effective rehabilitation, aiming to bolster the connection between the patient and their virtual counterpart. By doing so, the system not only aims to improve motor imagery performance but also seeks to provide a more engaging and efficacious rehabilitation experience. Through the real-time application of BCI, the system allows for the direct translation of imagined movements into virtual actions performed by the 3D avatar, offering immediate feedback to the user. This feedback loop is essential for reinforcing the neural pathways involved in motor control and recovery. The ultimate goal of the developed system is to significantly enhance the effectiveness of motor imagery exercises by making them more interactive and responsive to the user's cognitive processes, thereby paving a new path in the field of neurological rehabilitation.

Author Spotlight: Enhancing Neurorehabilitation Through EEG, Motor Imagery, and Virtual Reality

Motor imagery in a virtual reality environment has wide applications in brain-computer interface systems. This manuscript outlines the use of personalized digital avatars that resemble the participants performing movements imagined by the participant in a virtual reality environment to enhance immersion and a sense of body ownership.

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) ...

Bioengineering

Virtual Prism Adaptation Therapy: Protocol for Validation in Healthy Adults

Hemispatial neglect is a common impairment after stroke. It is associated with poor functional and social outcomes. Therefore, an adequate intervention is imperative for the successful management of hemispatial neglect. However, the clinical use of various interventions is limited in real clinical practice. Prism adaptation therapy is one of the most evidence-based rehabilitation modalities to treat hemispatial neglect. To overcome any possible shortcoming that may occur with prism therapy, we developed a new system using immersive virtual reality and depth-sensing camera to create a virtual prism adaptation therapy (VPAT). To validate the VPAT system, we designed an experimental protocol investigating the behavioral errors and changes in cortical activation via the VPAT system. Cortical activation was measured by functional near infrared spectroscopy (fNIRS). The experiment consisted of four phases. All four included clicking, pointing or rest applied to right-handed healthy people. Clicking versus pointing was used for investigating the cortical region related with the gross motor task, and pointing with VPAT versus pointing without VPAT was used for investigating the cortical region associated with visuospatial perception. The preliminary results from four healthy participants showed that pointing errors by the VPAT system was similar to the conventional prism adaptation therapy. Further analysis with more participants and fNIRS data, as well as a study in patients with stroke may be required.

This experimental protocol demonstrates the use of virtual prism adaptation therapy (VPAT) in healthy adults and the association between VPAT and functional near infrared spectroscopy to determine the effect of VPAT on cortical activation. Results suggest that VPAT may be feasible and could induce similar behavioral adaptation as conventional prism adaptation therapy.

Hemispatial neglect is a common impairment after stroke. It is associated with poor functional and social outcomes. Therefore, an adequate intervention is ...