This study was supported by the Philosophy and Social Science Research Project of Jiangsu Provincial Department of Education (2018SJA1089), Jiangsu Government Scholarship for Overseas Studies (JS-2018-262), the Natural Science Foundation of Zhejiang Province (LY19G020018) and the National Natural Science Foundation of China (NSFC) (72001096).

This study was conducted in accordance with the principles of the Helsinki Declaration. All participants were informed of the purpose and safety of the experiment and signed the informed consent form before participation. This study was approved by the institutional review board of Jiangsu University of Science and Technology.
1. Experiment procedure
<ol>
	<li>Preparation for the experiment
	<ol>
		<li>Explain informed consent to the participants and ask them to sign the consent form.</li>
		<li>Perform a color vision test on the participants to confirm that they have normal color discrimination.</li>
		<li>Present 30 mineral water brands to participants and ask them to pick the brands that they are familiar with, to ensure that they are unfamiliar with the brands of mineral water used in the experiment.</li>
		<li>Introduce the procedure of the experiment to the participants.</li>
		<li>Conduct a pre-experiment on the participants using mineral water brands other than the ones used in the study, and ensure that they are familiar with the AR and website operations.</li>
		<li>Direct each participant from Group A to perform the AR experiment first, and then to perform the website experiment. Direct each participant from Group B to perform the website experiment first, and then to perform the AR experiment.</li>
	</ol>
	</li>
	<li>Wearing instruments
	<ol>
		<li>fNIRS probes
		<ol>
			<li>Clean the forehead skin of the participants with skin preparation gel (see Table of Materials).</li>
			<li>Wrap the probes (see Table of Materials) in a plastic wrap for sweaty circumstances. Fix the probes on the FP1 and FP2 position according to the international 10-20 system with a black headband45. Use a black bandana to cover the probes to protect them against ambient light and improve signal quality.</li>
			<li>Clean the transmitter and receiver of the probes using a 70% isopropyl alcohol pad after finishing the experiment.</li>
		</ol>
		</li>
		<li>Eye tracker: Capture eye movement in real-world environments using eye tracking glasses (see Table of Materials). Fit prescription lenses on the head unit of the eye tracking glasses magnetically (if required) and ensure that the participants can walk around freely with a corrected sight. 
		&#8203;NOTE: Because the eye tracking glasses are not designed to work in conjunction with standard eye glasses, participants who wear eye glasses may still be included in the study by using optional prescription lenses for correction of either short-sightedness or long-sightedness. Participants can also wear standard contact lenses because although they may slightly increase noise, they do not normally introduce errors in the data. Participants cannot use colored or other lenses that change the appearance of the pupil or iris.</li>
	</ol>
	</li>
</ol>
2. Measures
<ol>
	<li>fNIRS
	<ol>
		<li>Open the recording software (see Table of Materials). Connect the probes to a laptop via a Bluetooth adapter, and then record the changes of concentration of oxygenated hemoglobin (O2Hb) in the participants&#39; prefrontal cortex through the laptop, with a sampling frequency of 10 Hz.</li>
		<li>Check the received light intensity and tissue saturation index (TSI) quality to control data quality. Ensure that the received signal is located between 1% and 95%.</li>
		<li>Ensure that the participants sit in a comfortable position on a chair and maintain the resting state for 2 min to collect baseline data before the real experiment.</li>
		<li>Click on the Start button on the software interface to record the fNIRS data.</li>
	</ol>
	</li>
	<li>Eye tracking
	<ol>
		<li>Set up the hardware for the eye tracking glasses. Connect the USB plug of the eye tracking glasses to a USB port on the laptop. Open the recording software (see Table of Materials) and set the sampling frequency to 120 Hz.</li>
		<li>Calibration: Perform a one-point-calibration. Ask the participant to focus on a clearly identifiable object in their field of view at 0.6 m. Move the crosshair cursor to the object, where the participant is focused on in the scene video and click on the object.</li>
		<li>Press the Record button on the software interface to start recording.</li>
	</ol>
	</li>
	<li>Questionnaire and scale: Present the usability questionnaire and NASA-TLX scale to the participants after they complete the tasks of the AR/website.</li>
</ol>
3. Data analysis
<ol>
	<li>fNIRS data processing
	<ol>
		<li>Convert the optical density values obtained from the fNIRS recording software into concentrations (&#956;mol) according to the modified Beer-Lambert Law46.</li>
		<li>Filter the raw data at low-pass 0.5 Hz to remove systematic noises such as heartbeats and respiration.</li>
		<li>Check and correct the data for motion artifacts by removing the data segments that exceeded three standard deviations above the entire time-series47.</li>
		<li>Export the mean and maximum fNIRS data in the AR and website conditions, and then subtract them from the baseline data.</li>
	</ol>
	</li>
	<li>Eye tracking data processing
	<ol>
		<li>Export the fixation frequency (count/s), total fixation time (ms), average fixation time (ms), saccade frequency (count/s), average saccade time (ms), and average scan path length (px/s) of the participants.</li>
	</ol>
	</li>
	<li>Statistical analysis
	<ol>
		<li>Perform a two-tailed test at a significance level of 0.05. Check the data normality using the Shapiro-Wilk test and perform a difference test. Perform multiple comparison corrections for p-values using the false discovery rate (FDR) method. 
		NOTE: While performing the difference test, the data that followed a normal distribution were analyzed using a paired samples t-test, and the data that did not follow a normal distribution were analyzed using Wilcoxon signed rank test.</li>
	</ol>
	</li>
</ol>

The authors have nothing to disclose.

Critical steps within the protocol 
During the experiment, several steps were considered to ensure reliability of the results. First, participants who are familiar with the brands of mineral water used in the experiment were excluded, because these participants would have performed the task based on their knowledge of the brand. Second, the participants completed a pre-experiment using other brands of mineral water, which was employed to ensure that the participants were familiar with AR and website...

Six frontier technologies that interact with consumers, typically represented by augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data, are reshaping many theoretical models of consumer behavior1. Augmented Reality (AR) is a new technology that could enhance consumer experience and improve consumer satisfaction. It superimposes textual information, images, videos, and other virtual items onto real scenarios to fuse virtuality and reality, thus enhancing information in the real world through explanation, guidance, evaluation, and prediction2. AR provides a new kind of human-computer interaction, creating an immersive shopping experience for consumers, and has led to the development of many applications3,4. However, the consumer acceptance of AR services is still minimal, and many companies are thus cautious about adopting AR technology5,6. The technology acceptance model (TAM) has been widely used to explain and predict the adoption behavior of new information technologies7,8. According to the TAM, the adoption intention for a new technology largely depends on its usability9. Therefore, a possible explanation for the slow consumer acceptance of AR services from the TAM perspective may relate to the usability of the new techniques, which highlights the need to evaluate the usability of AR while shopping10,11.
Usability is defined as the effectiveness, efficiency, and satisfaction of achieving specified goals in a specified context by specified users12. Currently, there are two main methods for evaluating usability: subjective and objective evaluations13. Subjective evaluations rely mainly on self-report methods using questionnaires and scales. Following this line of research, the questionnaire used in this study included five features associated with the information search mode to achieve a goal: (1) efficiency, (2) ease of use, (3) memorability (easy to remember), (4) satisfaction (the information search mode is comfortable and pleasant), and (5) generalizability to other objects14,15,16. In addition, cognitive load, representing the load while performing a particular task on the cognitive system of a learner17, is another core indicator of usability18,19. Thus, this study additionally used the NASA Task Load Index (NASA-TLX)13,20 as a subjective metric to measure the cognitive load while shopping using AR versus shopping using website services. It is noteworthy that self-report methods rely on the ability and willingness of individuals to accurately report their attitudes and/or prior behaviors21, leaving open the possibility for mis-reporting, under-reporting, or bias. Thus, objective measures could be a valuable complement to traditional subjective methods22.
Neuro-Information-Systems (NeuroIS) methods are used for objective evaluation of AR usability. NeuroIS, coined by Dimoka et al. at the 2007 ICIS conference, is attracting increasing attention in the field of information systems (IS)23. NeuroIS uses theories and tools of cognitive neuroscience to better understand the development, adoption, and impact of IS technologies24,25. To date, cognitive neuroscience tools, such as functional magnetic resonance imaging (fMRI), electroencephalogram (EEG), positron emission computed tomography, magnetoencephalography (MEG), and functional near-infrared spectroscopy (fNIRS), are commonly used in NeuroIS studies26,27. For instance, Dimoka and Davis used fMRI to measure the subjects&#39; activations when they interacted with the website, and revealed that perceived ease of use influenced activation in the prefrontal cortex (PFC)28. Similarly, using EEG, Moridis et al. found that frontal asymmetry was closely associated with usefulness29. These results indicate that the PFC may play a key role in usability.
Although achievements have been made in previous NeuroIS studies, the paradigms used in these studies had limited body movements of subjects with low ecological validity, limiting their theoretical and practical contributions. Interacting with technologies such as AR while shopping requires free body movements, and subject constraints largely impair consumer experience as discussed in He et al.22. Thus, brain-imaging tools with high ecological validity are needed for a usability test of information systems. In this regard, fNIRS has unique technical advantages: during fNIRS experiments, subjects can move freely30 to some extent. For instance, previous studies have measured subjects&#39; brain activations during several outdoor activities such as cycling using portable fNIRS31. In addition, fNIRS is low-cost and enables the measurement of brain activations for long periods of time32. In this study, fNIRS was used to objectively measure subjects&#39; level of cognitive load while using the shopping services of AR versus a website.
Eye tracking has been a valuable psychophysiological technique for detecting users&#39; visual attention during a usability test in recent years33 and has also been widely used in NeuroIS studies34. The technique relies on the eye-mind hypothesis, which assumes that the observer&#39;s focus goes where the attention is directed, that visual attention represents the mental process, and that patterns of visual attention reflect human cognitive strategies35,36,37. In the area of AR research, Yang et al. used eye tracking to find that AR advertisement improved consumers&#39; attitudes toward the advertisement by increasing their curiosity and attention38. In the current study, eye tracking was used to measure subjects&#39; attention, including parameters such as total fixation duration, average fixation duration, fixation frequency, saccade frequency, average saccade duration, and average scan path length.
In summary, this study proposes a usability test method that combines subjective and objective evaluations with AR applications as an example. A usability questionnaire and a NASA-TLX scale were used for subjective evaluation, and multimodal measures combining fNIRS and eye tracking were used for objective evaluation39,40.
Experimental design 
Experimental materials: To simulate a real-life shopping context, a product shelf was built in a laboratory, and two different brands of mineral water were placed on the shelf as experimental materials. As essential goods, mineral water was selected because participants would not have bias in subjective evaluations on the basis of their occupational background, gender, and purchasing ability. The price, capacity, and familiarity of the brands were controlled (see Table of Materials) to eliminate the interference of irrelevant variables.
The usability test included two conditions: a smartphone-based AR application (Supplemental Figure 1)&#160;and a website (Supplemental Figure 2). The AR application was programmed based on an AR engine. The website was developed using Python, based on Bootstrap for the front-end and Flask for the back-end. The AR application and website were run and browsed on a smartphone. Among the two different brands of mineral water, one was used as the experimental material in the AR condition, and the other was used in the website condition.
Experimental tasks: Participants were asked to perform four information search tasks that derived from IoT application contexts: the quality of water, the storage temperature, the matching diet, and the price per liter. These four information items are what consumers normally pay attention to when they buy mineral water. There was no time constraint for participants to complete the tasks.
Quality of water: The quality of mineral water commonly includes two indicators: the total dissolved solids (TDS) and the pH value. The TDS reflects the mineral content, and the pH value describes the acidity/alkalinity of the water. These two indicators are related to trace elements contained in the mineral water and influence taste. For instance, Bruvold and Ongerth divided the sensory quality of water into five grades according to its TDS content41. Marcussen et al. found that water has good sensory qualities in the range of 100-400 mg/L TDS42. The TDS and pH value of the two brands of mineral water used in this study were measured using TDS and pH meters, respectively, and then marked on the AR application and the website. While performing the task, participants were required to report the TDS and pH values of the mineral water and confirm whether these values were within the nominal range. In the AR condition, participants could acquire this information by scanning the bottle of water. In the website condition, participants were required to perform four steps: (1) finding a numeric code on the back of the bottle of mineral water, (2) entering the numeric code into a query box to obtain the TDS and pH values for mineral water, (3) searching the nominal range for mineral water on the website, and (4) verbally reporting whether TDS and pH value are within the nominal range for the product.
Storage temperature: The quality of mineral water may decrease during transportation and storage owing to changes in temperature. Experiments have shown that the appropriate temperature for mineral water is between 5 &#176;C and 25 &#176;C during transport and storage. In this temperature range, water does not have a bad odor43. In the present experiment, the storage temperature of the two types of mineral water in different places was marked on the AR application and the website. While performing the task, participants were required to report the storage location and corresponding temperature of the water. In the AR condition, participants could acquire this information by scanning the bottle of water. In the website condition, participants could acquire this information by entering the numeric code into a query box.
Matching diet: Different brands of mineral water are suitable for different menus due to their unique mineral composition and bubble content44. In the present experiment, dietary recommendations for the two mineral waters were marked on the AR application and website. While performing the task, participants were required to report how the mineral water matches the food in the menu. In the AR condition, participants could acquire this information by scanning the water bottle. In the website condition, participants could search for this information on the website.
Price per liter: Currently, the labels on the mineral water bottles in China do not display the price per liter information. This makes it difficult for consumers to distinguish the difference in unit prices of different types of mineral water. Therefore, the present experiment required participants to report the price per liter. In the AR application, participants could acquire the price per liter directly by scanning the bottle of water. In the website condition, the information could be calculated from the unit price and volume on the label.
This study used a within-participant design, with participant inclusion and exclusion criteria as described in Table 1. A total of 40 participants completed the experiment (20 males and 20 females, mean age = 21.31 &#177; 1.16 years). All participants were undergraduates of Jiangsu University of Science and Technology and were randomly arranged into two groups (A and B). In order to avoid the order effect, the experimental order was counterbalanced across the two groups (A/B). Specifically, one group performed the AR condition first and then the website condition, while the other group performed the website first and then the AR condition. The participants were required to complete preparation for the experiment, wear the instruments, and perform the experimental tasks. The inter-experiment interval was set to 10 s to allow cortical activation to return to the baseline level, avoiding cross-influence in the subsequent task. At the end of the AR/website experiment, participants were required to complete the usability questionnaire and the NASA-TLX scale. The experimental flowchart is shown in Figure 1. A photograph of the experimental setup is presented in Figure 2.
Table 1: Inclusion and exclusion criteria for the study. <a href="https://www.jove.com/files/ftp_upload/64667/Table 1 Inclusion and exclusion criteria.xlsx" target="_blank">Please click here to download this Table.</a>
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/64667/64667fig1v2.jpg" /> 
Figure 1: Experimental flowchart. Each experiment lasted ~45 min, with a rest period of 10 s between the tasks. <a href="https://www.jove.com/files/ftp_upload/64667/64667fig1v2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64667/64667fig02.jpg" /> 
Figure 2: Example setup of the experimental scene. The experimental materials, the participant, and the equipment are shown. <a href="https://www.jove.com/files/ftp_upload/64667/64667fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table><tbody><tr><td>AR Engine</td><td>Unity Technologies</td><td>2020.3.1</td><td>AR development platform</td></tr><tr><td>AR SDK</td><td>PTC</td><td>Vuforia Engine 9.8.5</td><td>AR development kit</td></tr><tr><td>Eye Tracker (eye tracking glasses)</td><td>SMI, Germany</td><td>SMI ETG</td><td>Head-mounted eye tracking&lt;br/&gt; system</td></tr><tr><td>Eye Tracker Recording software</td><td>SMI, Germany</td><td>iViewETG Software</td><td>Eye Tracker Recording software</td></tr><tr><td>fNIRS probes</td><td>Artinis Medical Systems BV, Netherlands</td><td>Artinis PortaLite</td><td>Light source: Light emitting diodes&lt;br/&gt; Wavelengths: Standard nominal 760 and 850 nm</td></tr><tr><td>fNIRS software</td><td>Artinis Medical Systems BV, Netherlands</td><td>OxySoft 3.2.70</td><td>fNIRS data recording and analysis software</td></tr><tr><td>Mineral Water</td><td>Groupe Danone</td><td>Badoit&amp;nbsp;</td><td>Experimental material in the AR condition&amp;nbsp;&amp;nbsp; Capacity: 330ml&lt;br/&gt; Price: &lt;img alt=&quot;Equation 1&quot; src=&quot;/files/ftp_upload/64667/64667eq01.jpg&quot; style=&quot;margin: auto;&quot;/&gt;6</td></tr><tr><td>Mineral Water</td><td>Nestl&amp;eacute;</td><td>Acqua Panna</td><td>Experimental material in the website condition Capacity: 250ml&lt;br/&gt; Price: &lt;img alt=&quot;Equation 1&quot; src=&quot;/files/ftp_upload/64667/64667eq01.jpg&quot; style=&quot;margin: auto;&quot;/&gt;5.4</td></tr><tr><td>Skin Preparation Gel</td><td>Weaver and Company</td><td>Nuprep</td><td>Clean the forehead skin of the participants</td></tr><tr><td>Smartphone</td><td>Xiaomi</td><td>Redmi K30 Ultra</td><td>Smartphone-based AR application and website</td></tr></tbody></table>

usability evaluation of augmented reality: a neuro-information-systems study

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test included both subjective and objective evaluations. For the subjective evaluation, a usability questionnaire and a NASA-TLX scale were adopted. For the objective evaluation, methods of Neuro-Information-Systems (NeuroIS) were used. From a NeuroIS perspective, this study used mobile fNIRS and eye tracking glasses for multimodal measurements, which solved the problem of ecological validity of cognitive neuroscience tools used in real-world behavior experiments. This study used Augmented Reality (AR) integrated into the Internet of Things (IoT) as an experimental object. Comparing the differences in the neuroimaging data, the physiological data, the usability questionnaire, and the NASA-TLX scale data between the two information-search modes (AR versus a website), information search with AR had a higher efficiency and a lower cognitive load compared with information search with a website during the process of consumption decision-making. The usability experiment results demonstrate that AR, as an emerging technology in retail, can effectively enhance consumer experiences and increase their purchase intention. The experimental paradigm, combining both subjective and objective evaluations in this study, could be applied to a usability test for emerging technologies, such as augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data. It provides a practical experimental solution for the user experience in human-computer-interactions with the adoption of emerging technologies.

The representative results of this study include the usability questionnaire results, eye tracking data analysis, NASA-TLX scale data, fNIRS data analysis, and dynamic cognitive load changes. For the usability questionnaire results, eye tracking data analysis, NASA-TLX scale data and fNIRS data analysis, normality tests, and differences tests were conducted. For dynamic cognitive load changes, this study selected fNIRS and eye tracking data from a single participant to demonstrate the validity of the multimodal measureme...

Watch this Scientific Journal Video about Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study at JoVE.com

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

This study presents an experimental paradigm for&#160;a&#160;usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test ...

usability-evaluation-augmented-reality-neuro-information-systems

Research

JoVE Journal

Behavior

2.2K Views.  Jiangsu University of Science and Technology. Deze studie introduceert een experimenteel paradigma voor een bruikbaarheidstest van opkomende technologieën in managementinformatiesystemen. De bruikbaarheidstest omvatte zowel subjectieve als objectieve evaluaties. Mobiele fNIRS en eye tracking-brillen werden gebruikt voor multimodale metingen die het ecologische validiteitsprobleem van cognitieve neurowetenschappelijke hulpmiddelen die worden gebruikt in gedragsexperimenten in de echte wereld oplossen.De procedure wordt gedemonstre...

Video: Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

A Networked Desktop Virtual Reality Setup for Decision Science and Navigation Experiments with Multiple Participants

Investigating the interactions among multiple participants is a challenge for researchers from various disciplines, including the decision sciences and spatial cognition. With a local area network and dedicated software platform, experimenters can efficiently monitor the behavior of the participants that are simultaneously immersed in a desktop virtual environment and digitalize the collected data. These capabilities allow for experimental designs in spatial cognition and navigation research that would be difficult (if not impossible) to conduct in the real world. Possible experimental variations include stress during an evacuation, cooperative and competitive search tasks, and other contextual factors that may influence emergent crowd behavior. However, such a laboratory requires maintenance and strict protocols for data collection in a controlled setting. While the external validity of laboratory studies with human participants is sometimes questioned, a number of recent papers suggest that the correspondence between real and virtual environments may be sufficient for studying social behavior in terms of trajectories, hesitations, and spatial decisions. In this article, we describe a method for conducting experiments on decision-making and navigation with up to 36 participants in a networked desktop virtual reality setup (i.e., the Decision Science Laboratory or DeSciL). This experiment protocol can be adapted and applied by other researchers in order to set up a networked desktop virtual reality laboratory.

This paper describes a method for conducting multi-user experiments on decision-making and navigation using a networked computer laboratory.

Investigating the interactions among multiple participants is a challenge for researchers from various disciplines, including the decision sciences and ...

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to use. Therefore, new approaches like immersive analytics are welcome. Immersive analytics promise to experience high-dimensional data sets in a convenient manner for various user groups and data sets. Technically, virtual-reality devices are used to enable immersive analytics. In Industry 4.0, for example, scenarios like the identification of outliers or anomalies in high-dimensional data sets are pursued goals of immersive analytics. In this context, two important questions should be addressed for any developed technical solution on immersive analytics: First, is the technical solutions being helpful or not? Second, is the bodily experience of the technical solution positive or negative? The first question aims at the general feasibility of a technical solution, while the second one aims at the wearing comfort. Extant studies and protocols, which systematically address these questions are still rare. In this work, a study protocol is presented, which mainly investigates the usability for immersive analytics in Industry 4.0 scenarios. Specifically, the protocol is based on four pillars. First, it categorizes users based on previous experiences. Second, tasks are presented, which can be used to evaluate the feasibility of the technical solution. Third, measures are presented, which quantify the learning effect of a user. Fourth, a questionnaire&#160;evaluates the stress level when performing tasks. Based on these pillars, a technical setting was implemented that uses mixed reality smartglasses to apply the study protocol. The results of the conducted study show the applicability of the protocol on the one hand and the feasibility of immersive analytics in Industry 4.0 scenarios on the other. The presented protocol includes a discussion of discovered limitations.

This protocol delineates the technical setting of a developed mixed reality application that is used for immersive analytics. Based on this, measures are presented, which were used in a study to gain insights into usability aspects of the developed technical solution.

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to ...

Engineering

The Immersive Cleveland Clinic Virtual Reality Shopping Platform for the Assessment of Instrumental Activities of Daily Living

A decline in the performance of instrumental activities of daily living (IADLs) has been proposed as a prodromal marker of neurological disease. Existing clinical and performance-based IADL assessments are not feasible for integration into clinical medicine. Virtual reality (VR) is a powerful yet underutilized tool that could advance the diagnosis and treatment of neurological disease. An impediment to the adoption and scaling of VR in clinical neurology is VR-related sickness resulting from sensory inconsistencies between the visual and vestibular systems (i.e., locomotion problem).
The Cleveland Clinic Virtual Reality Shopping (CC-VRS) platform attempts to solve the locomotion problem by coupling an omnidirectional treadmill with high-resolution VR content, enabling the user to physically navigate a virtual grocery store to simulate shopping. The CC-VRS consists of Basic and Complex shopping experiences; both require walking 150 m and retrieving five items. The Complex experience has additional scenarios that increase the cognitive and motor demands of the task to better represent the continuum of activities associated with real-world shopping. The CC-VRS platform provides objective and quantitative biomechanical and cognitive outcomes related to the user&#39;s IADL performance. Initial data indicate that the CC-VRS results in minimal VR-sickness and is feasible and tolerable for older adults and patients with Parkinson&#39;s disease (PD). The considerations underlying the development, design, and hardware and software technology are reviewed, and initial models of integration into primary care and neurology are provided.

Virtual reality (VR) is a powerful yet underutilized approach to advance the diagnosis and treatment of neurological disease. The Cleveland Clinic Virtual Reality Shopping platform combines state-of-the-art VR content with an omnidirectional treadmill to quantify instrumental activities of daily living-a proposed prodromal marker of neurological disease.

A decline in the performance of instrumental activities of daily living (IADLs) has been proposed as a prodromal marker of neurological disease. Existing ...

A Single-Channel and Non-Invasive Wearable Brain-Computer Interface for Industry and Healthcare

The present work focuses on how to build a wearable brain-computer interface (BCI). BCIs are a novel means of human-computer interaction that relies on direct measurements of brain signals to assist both people with disabilities and those who are able-bodied. Application examples include robotic control, industrial inspection, and neurorehabilitation. Notably, recent studies have shown that steady-state visually evoked potentials (SSVEPs) are particularly suited for communication and control applications, and efforts are currently being made to bring BCI technology into daily life. To achieve this aim, the final system must rely on wearable, portable, and low-cost instrumentation. In exploiting SSVEPs, a flickering visual stimulus with fixed frequencies is required. Thus, in considering daily-life constraints, the possibility to provide visual stimuli by means of smart glasses was explored in this study. Moreover, to detect the elicited potentials, a commercial device for electroencephalography (EEG) was considered. This consists of a single differential channel with dry electrodes (no conductive gel), thus achieving the utmost wearability and portability. In such a BCI, the user can interact with the smart glasses by merely staring at icons appearing on the display. Upon this simple principle, a user-friendly, low-cost BCI was built by integrating extended reality (XR) glasses with a commercially available EEG device. The functionality of this wearable XR-BCI was examined with an experimental campaign involving 20 subjects. The classification accuracy was between 80%-95% on average depending on the stimulation time. Given these results, the system can be used as a human-machine interface for industrial inspection but also for rehabilitation in ADHD and autism.

This paper discusses how to build a brain-computer interface by relying on consumer-grade equipment and steady-state visually evoked potentials. For this, a single-channel electroencephalograph exploiting dry electrodes was integrated with augmented reality glasses for stimuli presentation and output data visualization. The final system was non-invasive, wearable, and portable.

The present work focuses on how to build a wearable brain-computer interface (BCI). BCIs are a novel means of human-computer interaction that relies on ...

Motor Imagery Using VR Digital Avatars in a Brain-Computer Interface System

Motor Imagery Performance Through Embodied Digital Twins in a Virtual Reality-Enabled Brain-Computer Interface Environment

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) technologies with the customization of three-dimensional (3D) avatars. Traditional approaches to rehabilitation often fail to fully engage patients, primarily due to their inability to provide a deeply immersive and interactive experience. This research endeavors to fill this gap by utilizing motor imagery (MI) techniques, where participants visualize physical movements without actual execution. This method capitalizes on the brain's neural mechanisms, activating areas involved in movement execution when imagining movements, thereby facilitating the recovery process. The integration of VR's immersive capabilities with the precision of electroencephalography (EEG) to capture and interpret brain activity associated with imagined movements forms the core of this system. Digital Twins in the form of personalized 3D avatars are employed to significantly enhance the sense of immersion within the virtual environment. This heightened sense of embodiment is crucial for effective rehabilitation, aiming to bolster the connection between the patient and their virtual counterpart. By doing so, the system not only aims to improve motor imagery performance but also seeks to provide a more engaging and efficacious rehabilitation experience. Through the real-time application of BCI, the system allows for the direct translation of imagined movements into virtual actions performed by the 3D avatar, offering immediate feedback to the user. This feedback loop is essential for reinforcing the neural pathways involved in motor control and recovery. The ultimate goal of the developed system is to significantly enhance the effectiveness of motor imagery exercises by making them more interactive and responsive to the user's cognitive processes, thereby paving a new path in the field of neurological rehabilitation.

Author Spotlight: Enhancing Neurorehabilitation Through EEG, Motor Imagery, and Virtual Reality

Motor imagery in a virtual reality environment has wide applications in brain-computer interface systems. This manuscript outlines the use of personalized digital avatars that resemble the participants performing movements imagined by the participant in a virtual reality environment to enhance immersion and a sense of body ownership.

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) ...

Bioengineering

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

Medicine

Virtual Reality Tools for Assessing Unilateral Spatial Neglect: A Novel Opportunity for Data Collection

Unilateral spatial neglect (USN) is a syndrome characterized by inattention to or inaction in one side of space and affects between 23-46% of acute stroke survivors. The diagnosis and characterization of these symptoms in individual patients can be challenging and often requires skilled clinical staff. Virtual reality (VR) presents an opportunity to develop novel assessment tools for patients with USN.
We aimed to design and build a VR tool to detect and characterize subtle USN symptoms, and to test the tool on subjects treated with inhibitory repetitive transcranial magnetic stimulation (TMS) of cortical regions associated with USN.
We created three experimental conditions by applying TMS to two distinct regions of cortex associated with visuospatial processing- the superior temporal gyrus (STG) and the supramarginal gyrus (SMG) - and applied sham TMS as a control. We then placed subjects in a virtual reality environment in which they were asked to identify the flowers with lateral asymmetries of flowers distributed across bushes in both hemispaces, with dynamic difficulty adjustment based on each subject's performance. 
We found significant differences in average head yaw between subjects stimulated at the STG and those stimulated at the SMG and marginally significant effects in the average visual axis.
VR technology is becoming more accessible, affordable, and robust, presenting an exciting opportunity to create useful and novel game-like tools. In conjunction with TMS, these tools could be used to study specific, isolated, artificial neurological deficits in healthy subjects, informing the creation of VR-based diagnostic tools for patients with deficits due to acquired brain injury. This study is the first to our knowledge in which artificially generated USN symptoms have been evaluated with a VR task.

The goal was to design, build, and pilot a novel virtual reality task to detect and characterize unilateral spatial neglect, a syndrome affecting 23-46% of acute stroke survivors, expanding the role of virtual reality in the study and management of neurologic disease.

Unilateral spatial neglect (USN) is a syndrome characterized by inattention to or inaction in one side of space and affects between 23-46% of acute stroke ...

Enhancing Stroke Rehabilitation with Virtual Reality-Based Occupational Training

Cognitive Function and Upper Limb Rehabilitation Training Post-Stroke Using a Digital Occupational Training System

Stroke rehabilitation often requires frequent and intensive therapy to improve functional recovery. Virtual reality (VR) technology has shown the potential to meet these demands by providing engaging and motivating therapy options. The digital occupational training system is a VR application that utilizes cutting-edge technologies, including multi-touch screens, virtual reality, and human-computer interaction, to offer diverse training techniques for advanced cognitive capacity and hand-eye coordination abilities. The objective of this study was to assess the effectiveness of this program in enhancing cognitive function and upper extremity rehabilitation in stroke patients. The training and assessment consist of five cognitive modules covering perception, attention, memory, logical reasoning, and calculation, along with hand-eye coordination training. This research indicates that after eight weeks of training, the digital occupational training system can significantly improve cognitive function, daily living skills, attention, and self-care abilities in stroke patients. This software can be employed as a time-saving and clinically effective rehabilitation aid to complement traditional one-on-one occupational therapy sessions. In summary, the digital occupational training system shows promise and offers potential financial benefits as a tool to support the functional recovery of stroke patients.

Author Spotlight: Rehabilitation of Stroke Patients With a Digital Occupational Training System

The current protocol outlines how the VR-based digital occupational training system enhances the rehabilitation of patients with cognitive impairment and upper limb dysfunction following a stroke.

Stroke rehabilitation often requires frequent and intensive therapy to improve functional recovery. Virtual reality (VR) technology has shown the ...

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) for minimally invasive pedicle screw placement. The cadaveric porcine specimens were placed on a surgical table and draped with sterile covers. The levels of interest were identified using fluoroscopy, and a dynamic reference frame was attached to the spinous process of a vertebra in the region of interest. Cone beam computerized tomography (CBCT) was performed, and a 3D rendering was automatically generated, which was used for the subsequent planning of the pedicle screw placements. Each surgeon was fitted with an HMD that was individually eye-calibrated and connected to the spinal navigation system.
Navigated instruments, tracked by the navigation system and displayed in 2D and 3D in the HMD, were used for 33 pedicle cannulations, each with a diameter of 4.5 mm. Postprocedural CBCT scans were assessed by an independent reviewer to measure the technical (deviation from the planned path) and clinical (Gertzbein grade) accuracy of each cannulation. The navigation time for each cannulation was measured. The technical accuracy was 1.0 mm &#177; 0.5 mm at the entry point and 0.8 mm &#177; 0.1 mm at the target. The angular deviation was 1.5&#176; &#177; 0.6&#176;, and the mean insertion time per cannulation was 141 s &#177; 71 s. The clinical accuracy was 100% according to the Gertzbein grading scale (32 grade 0; 1 grade 1). When used for minimally invasive pedicle cannulations in a porcine model, submillimeter technical accuracy and 100% clinical accuracy could be achieved with this protocol.

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) ...

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

Samenvatting

Meer video's verkennen

Hoofdstukken in deze video