Name: usability evaluation of augmented reality: a neuro-information-systems study
Uploaded: 2022-11-30T14:40:00
Description: Watch this Scientific Journal Video about Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study at JoVE.com

This study was supported by the Philosophy and Social Science Research Project of Jiangsu Provincial Department of Education (2018SJA1089), Jiangsu Government Scholarship for Overseas Studies (JS-2018-262), the Natural Science Foundation of Zhejiang Province (LY19G020018) and the National Natural Science Foundation of China (NSFC) (72001096).

This study was conducted in accordance with the principles of the Helsinki Declaration. All participants were informed of the purpose and safety of the experiment and signed the informed consent form before participation. This study was approved by the institutional review board of Jiangsu University of Science and Technology.
1. Experiment procedure
<ol>
	<li>Preparation for the experiment
	<ol>
		<li>Explain informed consent to the participants and ask them to sign the con...

Critical steps within the protocol 
During the experiment, several steps were considered to ensure reliability of the results. First, participants who are familiar with the brands of mineral water used in the experiment were excluded, because these participants would have performed the task based on their knowledge of the brand. Second, the participants completed a pre-experiment using other brands of mineral water, which was employed to ensure that the participants were familiar with AR and website...

Six frontier technologies that interact with consumers, typically represented by augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data, are reshaping many theoretical models of consumer behavior1. Augmented Reality (AR) is a new technology that could enhance consumer experience and improve consumer satisfaction. It superimposes textual information, images, videos, and other virtual items onto real scenarios to fuse virtuality and reality, thus enhancing information in the real world through explanation, guidance, evaluation, and prediction2. AR provides a new kind of human-computer interaction, creating an immersive shopping experience for consumers, and has led to the development of many applications3,4. However, the consumer acceptance of AR services is still minimal, and many companies are thus cautious about adopting AR technology5,6. The technology acceptance model (TAM) has been widely used to explain and predict the adoption behavior of new information technologies7,8. According to the TAM, the adoption intention for a new technology largely depends on its usability9. Therefore, a possible explanation for the slow consumer acceptance of AR services from the TAM perspective may relate to the usability of the new techniques, which highlights the need to evaluate the usability of AR while shopping10,11.
Usability is defined as the effectiveness, efficiency, and satisfaction of achieving specified goals in a specified context by specified users12. Currently, there are two main methods for evaluating usability: subjective and objective evaluations13. Subjective evaluations rely mainly on self-report methods using questionnaires and scales. Following this line of research, the questionnaire used in this study included five features associated with the information search mode to achieve a goal: (1) efficiency, (2) ease of use, (3) memorability (easy to remember), (4) satisfaction (the information search mode is comfortable and pleasant), and (5) generalizability to other objects14,15,16. In addition, cognitive load, representing the load while performing a particular task on the cognitive system of a learner17, is another core indicator of usability18,19. Thus, this study additionally used the NASA Task Load Index (NASA-TLX)13,20 as a subjective metric to measure the cognitive load while shopping using AR versus shopping using website services. It is noteworthy that self-report methods rely on the ability and willingness of individuals to accurately report their attitudes and/or prior behaviors21, leaving open the possibility for mis-reporting, under-reporting, or bias. Thus, objective measures could be a valuable complement to traditional subjective methods22.
Neuro-Information-Systems (NeuroIS) methods are used for objective evaluation of AR usability. NeuroIS, coined by Dimoka et al. at the 2007 ICIS conference, is attracting increasing attention in the field of information systems (IS)23. NeuroIS uses theories and tools of cognitive neuroscience to better understand the development, adoption, and impact of IS technologies24,25. To date, cognitive neuroscience tools, such as functional magnetic resonance imaging (fMRI), electroencephalogram (EEG), positron emission computed tomography, magnetoencephalography (MEG), and functional near-infrared spectroscopy (fNIRS), are commonly used in NeuroIS studies26,27. For instance, Dimoka and Davis used fMRI to measure the subjects&#39; activations when they interacted with the website, and revealed that perceived ease of use influenced activation in the prefrontal cortex (PFC)28. Similarly, using EEG, Moridis et al. found that frontal asymmetry was closely associated with usefulness29. These results indicate that the PFC may play a key role in usability.
Although achievements have been made in previous NeuroIS studies, the paradigms used in these studies had limited body movements of subjects with low ecological validity, limiting their theoretical and practical contributions. Interacting with technologies such as AR while shopping requires free body movements, and subject constraints largely impair consumer experience as discussed in He et al.22. Thus, brain-imaging tools with high ecological validity are needed for a usability test of information systems. In this regard, fNIRS has unique technical advantages: during fNIRS experiments, subjects can move freely30 to some extent. For instance, previous studies have measured subjects&#39; brain activations during several outdoor activities such as cycling using portable fNIRS31. In addition, fNIRS is low-cost and enables the measurement of brain activations for long periods of time32. In this study, fNIRS was used to objectively measure subjects&#39; level of cognitive load while using the shopping services of AR versus a website.
Eye tracking has been a valuable psychophysiological technique for detecting users&#39; visual attention during a usability test in recent years33 and has also been widely used in NeuroIS studies34. The technique relies on the eye-mind hypothesis, which assumes that the observer&#39;s focus goes where the attention is directed, that visual attention represents the mental process, and that patterns of visual attention reflect human cognitive strategies35,36,37. In the area of AR research, Yang et al. used eye tracking to find that AR advertisement improved consumers&#39; attitudes toward the advertisement by increasing their curiosity and attention38. In the current study, eye tracking was used to measure subjects&#39; attention, including parameters such as total fixation duration, average fixation duration, fixation frequency, saccade frequency, average saccade duration, and average scan path length.
In summary, this study proposes a usability test method that combines subjective and objective evaluations with AR applications as an example. A usability questionnaire and a NASA-TLX scale were used for subjective evaluation, and multimodal measures combining fNIRS and eye tracking were used for objective evaluation39,40.
Experimental design 
Experimental materials: To simulate a real-life shopping context, a product shelf was built in a laboratory, and two different brands of mineral water were placed on the shelf as experimental materials. As essential goods, mineral water was selected because participants would not have bias in subjective evaluations on the basis of their occupational background, gender, and purchasing ability. The price, capacity, and familiarity of the brands were controlled (see Table of Materials) to eliminate the interference of irrelevant variables.
The usability test included two conditions: a smartphone-based AR application (Supplemental Figure 1)&#160;and a website (Supplemental Figure 2). The AR application was programmed based on an AR engine. The website was developed using Python, based on Bootstrap for the front-end and Flask for the back-end. The AR application and website were run and browsed on a smartphone. Among the two different brands of mineral water, one was used as the experimental material in the AR condition, and the other was used in the website condition.
Experimental tasks: Participants were asked to perform four information search tasks that derived from IoT application contexts: the quality of water, the storage temperature, the matching diet, and the price per liter. These four information items are what consumers normally pay attention to when they buy mineral water. There was no time constraint for participants to complete the tasks.
Quality of water: The quality of mineral water commonly includes two indicators: the total dissolved solids (TDS) and the pH value. The TDS reflects the mineral content, and the pH value describes the acidity/alkalinity of the water. These two indicators are related to trace elements contained in the mineral water and influence taste. For instance, Bruvold and Ongerth divided the sensory quality of water into five grades according to its TDS content41. Marcussen et al. found that water has good sensory qualities in the range of 100-400 mg/L TDS42. The TDS and pH value of the two brands of mineral water used in this study were measured using TDS and pH meters, respectively, and then marked on the AR application and the website. While performing the task, participants were required to report the TDS and pH values of the mineral water and confirm whether these values were within the nominal range. In the AR condition, participants could acquire this information by scanning the bottle of water. In the website condition, participants were required to perform four steps: (1) finding a numeric code on the back of the bottle of mineral water, (2) entering the numeric code into a query box to obtain the TDS and pH values for mineral water, (3) searching the nominal range for mineral water on the website, and (4) verbally reporting whether TDS and pH value are within the nominal range for the product.
Storage temperature: The quality of mineral water may decrease during transportation and storage owing to changes in temperature. Experiments have shown that the appropriate temperature for mineral water is between 5 &#176;C and 25 &#176;C during transport and storage. In this temperature range, water does not have a bad odor43. In the present experiment, the storage temperature of the two types of mineral water in different places was marked on the AR application and the website. While performing the task, participants were required to report the storage location and corresponding temperature of the water. In the AR condition, participants could acquire this information by scanning the bottle of water. In the website condition, participants could acquire this information by entering the numeric code into a query box.
Matching diet: Different brands of mineral water are suitable for different menus due to their unique mineral composition and bubble content44. In the present experiment, dietary recommendations for the two mineral waters were marked on the AR application and website. While performing the task, participants were required to report how the mineral water matches the food in the menu. In the AR condition, participants could acquire this information by scanning the water bottle. In the website condition, participants could search for this information on the website.
Price per liter: Currently, the labels on the mineral water bottles in China do not display the price per liter information. This makes it difficult for consumers to distinguish the difference in unit prices of different types of mineral water. Therefore, the present experiment required participants to report the price per liter. In the AR application, participants could acquire the price per liter directly by scanning the bottle of water. In the website condition, the information could be calculated from the unit price and volume on the label.
This study used a within-participant design, with participant inclusion and exclusion criteria as described in Table 1. A total of 40 participants completed the experiment (20 males and 20 females, mean age = 21.31 &#177; 1.16 years). All participants were undergraduates of Jiangsu University of Science and Technology and were randomly arranged into two groups (A and B). In order to avoid the order effect, the experimental order was counterbalanced across the two groups (A/B). Specifically, one group performed the AR condition first and then the website condition, while the other group performed the website first and then the AR condition. The participants were required to complete preparation for the experiment, wear the instruments, and perform the experimental tasks. The inter-experiment interval was set to 10 s to allow cortical activation to return to the baseline level, avoiding cross-influence in the subsequent task. At the end of the AR/website experiment, participants were required to complete the usability questionnaire and the NASA-TLX scale. The experimental flowchart is shown in Figure 1. A photograph of the experimental setup is presented in Figure 2.
Table 1: Inclusion and exclusion criteria for the study. <a href="https://www.jove.com/files/ftp_upload/64667/Table 1 Inclusion and exclusion criteria.xlsx" target="_blank">Please click here to download this Table.</a>
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/64667/64667fig1v2.jpg" /> 
Figure 1: Experimental flowchart. Each experiment lasted ~45 min, with a rest period of 10 s between the tasks. <a href="https://www.jove.com/files/ftp_upload/64667/64667fig1v2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64667/64667fig02.jpg" /> 
Figure 2: Example setup of the experimental scene. The experimental materials, the participant, and the equipment are shown. <a href="https://www.jove.com/files/ftp_upload/64667/64667fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>AR Engine</td>
			<td>Unity Technologies</td>
			<td>2020.3.1</td>
			<td>AR development platform</td>
		</tr>
		<tr>
			<td>AR SDK</td>
			<td>PTC</td>
			<td>Vuforia Engine 9.8.5</td>
			<td>AR development kit</td>
		</tr>
		<tr>
			<td>Eye Tracker (eye tracking glasses)</td>
			<td>SMI, Germany</td>
			<td>SMI ETG</td>
			<td>Head-mounted eye tracking 
			system</td>
		</tr>
		<tr>
			<td>Eye Tracker Recording software</td>
			<td>SMI, Germany</td>
			<td>iViewETG Software</td>
			<td>Eye Tracker Recording software</td>
		</tr>
		<tr>
			<td>fNIRS probes</td>
			<td>Artinis Medical Systems BV, Netherlands</td>
			<td>Artinis PortaLite</td>
			<td>Light source: Light emitting diodes 
			Wavelengths: Standard nominal 760 and 850 nm</td>
		</tr>
		<tr>
			<td>fNIRS software</td>
			<td>Artinis Medical Systems BV, Netherlands</td>
			<td>OxySoft 3.2.70</td>
			<td>fNIRS data recording and analysis software</td>
		</tr>
		<tr>
			<td>Mineral Water</td>
			<td>Groupe Danone</td>
			<td>Badoit&#160;</td>
			<td>Experimental material in the AR condition&#160;&#160; Capacity: 330ml 
			Price: <img alt="Equation 1" src="/files/ftp_upload/64667/64667eq01.jpg" style="margin: auto;" />6</td>
		</tr>
		<tr>
			<td>Mineral Water</td>
			<td>Nestl&#233;</td>
			<td>Acqua Panna</td>
			<td>Experimental material in the website condition Capacity: 250ml 
			Price: <img alt="Equation 1" src="/files/ftp_upload/64667/64667eq01.jpg" style="margin: auto;" />5.4</td>
		</tr>
		<tr>
			<td>Skin Preparation Gel</td>
			<td>Weaver and Company</td>
			<td>Nuprep</td>
			<td>Clean the forehead skin of the participants</td>
		</tr>
		<tr>
			<td>Smartphone</td>
			<td>Xiaomi</td>
			<td>Redmi K30 Ultra</td>
			<td>Smartphone-based AR application and website</td>
		</tr>
	</tbody>
</table>

usability evaluation of augmented reality: a neuro-information-systems study

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test included both subjective and objective evaluations. For the subjective evaluation, a usability questionnaire and a NASA-TLX scale were adopted. For the objective evaluation, methods of Neuro-Information-Systems (NeuroIS) were used. From a NeuroIS perspective, this study used mobile fNIRS and eye tracking glasses for multimodal measurements, which solved the problem of ecological validity of cognitive neuroscience tools used in real-world behavior experiments. This study used Augmented Reality (AR) integrated into the Internet of Things (IoT) as an experimental object. Comparing the differences in the neuroimaging data, the physiological data, the usability questionnaire, and the NASA-TLX scale data between the two information-search modes (AR versus a website), information search with AR had a higher efficiency and a lower cognitive load compared with information search with a website during the process of consumption decision-making. The usability experiment results demonstrate that AR, as an emerging technology in retail, can effectively enhance consumer experiences and increase their purchase intention. The experimental paradigm, combining both subjective and objective evaluations in this study, could be applied to a usability test for emerging technologies, such as augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data. It provides a practical experimental solution for the user experience in human-computer-interactions with the adoption of emerging technologies.

The representative results of this study include the usability questionnaire results, eye tracking data analysis, NASA-TLX scale data, fNIRS data analysis, and dynamic cognitive load changes. For the usability questionnaire results, eye tracking data analysis, NASA-TLX scale data and fNIRS data analysis, normality tests, and differences tests were conducted. For dynamic cognitive load changes, this study selected fNIRS and eye tracking data from a single participant to demonstrate the validity of the multimodal measureme...

Watch this Scientific Journal Video about Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study at JoVE.com

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

This study presents an experimental paradigm for&#160;a&#160;usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test ...

This study introduces an experimental paradigm for a usability test of emerging technologies in management information systems. The usability test included both subjective and objective evaluations. Mobile fNIRS and eye tracking glasses were used for multimodal measurements solving the ecological validity problem of cognitive neuroscience tools used in real world behavior experiments.Demonstrating the procedure will be Di Zhang, a master student from my laboratory. Yi Wang, a master student, will act as a participant. Begin by explaining the informed consent to the participants, and ask them to sign the consent form.Next, perform a color vision test on the participants to confirm that they have normal color discrimination. Then present 30 mineral water brands to participants and instruct them to pick their familiar brands to ensure that they were unfamiliar with the brands of mineral water used in the experiment. Next, introduce the experimental procedure to the participants.Conduct a pre experiment on the participants using mineral water brands unused in the study to make them familiar with augmented reality or AR and website operations. To wear the fNIRS probes, clean the forehead skin of the participants with skin preparation gel. Wrap the probes in plastic wrap to avoid sweaty circumstances, and fix them on the FP one and FP two position according to the international 10 to 20 system with a black headband.Then cover the probes with a black bandana. Use eye tracking glasses to capture eye movement in real world environments. Fit prescription lenses on the head unit of the eye tracking glasses magnetically if required and ensure that the participants can walk around freely with a corrected sight.Once all the participants are familiar with the experiment direct each participant from group A to perform the AR experiment first and then the website experiment. Similarly, ask the participants from group B to perform the website experiment first and then the AR experiment. Ensure the participants sit comfortably on a chair and rest for two minutes to collect baseline data before the experiment.To measure the fNIRS data, open the recording software and connect the probes to a laptop via Bluetooth adapter. Then record the concentration changes of oxygenated hemoglobin in the participant's prefrontal cortex with a sampling frequency of 10 hertz. Check the received light intensity and tissue saturation index quality to control data quality.Ensure that the received signal is located between one and 95%To measure the eye tracking, set up the hardware for the eye tracking glasses. Connect the plug of the eye tracking glasses to a USB port on the laptop. Then open the recording software and set the sampling frequency to 120 hertz.Perform a one point calibration by asking the participant to focus on an identifiable object in their field of view At 0.6 meters. Move the cross hair cursor to the focused object in the scene video and click on the object. Then click the record button on the software interface to start recording.Present the usability questionnaire and NASA TLX scale to the participants after they complete the tasks of the AR and website. After completing the experiment clean the transmitter and receiver of the probes using a 70%isopropyl alcohol pad. The median difference scores of the usability questionnaire showed that the participants had a better user experience in the AR condition than in the website condition.The eye tracking data revealed that the participants had higher information search efficiency when using AR than the website. The median difference scores of the NASA TLX scale indicated that the AR technique led to a lower cognitive load than the website technique. The fNIRS data analyzed for the differences in the mean oxygenated, hemoglobin and website conditions showed dynamic cognitive load changes.Participants familiar with the brands of mineral water used in the experiment were excluded because they would have performed the task based on their knowledge of the brand. This study proposes a experimental paradigm combining subjective and objective evaluations in the mix which can effectively evaluate the usability of emerging technologies.

Research

JoVE Journal

Behavior

Assessment and Evaluation of the High Risk Neonate: The NICU Network Neurobehavioral Scale

There has been a long-standing interest in the assessment of the neurobehavioral integrity of the newborn infant.  The NICU Network Neurobehavioral Scale ...

Two-photon Calcium Imaging in Mice Navigating a Virtual Reality Environment

In recent years, two-photon imaging has become an invaluable tool in neuroscience, as it allows for chronic measurement of the activity of genetically ...

Development of a Virtual Reality Assessment of Everyday Living Skills

Cognitive impairments affect the majority of patients with schizophrenia and these impairments predict poor long term psychosocial outcomes.&#160; ...

Evaluation of a Smartphone-based Human Activity Recognition System in a Daily Living Environment

An evaluation method that includes continuous activities in a daily-living environment was developed for Wearable Mobility Monitoring Systems (WMMS) that ...

Force and Position Control in Humans - The Role of Augmented Feedback

During motor behaviour, humans interact with the environment by for example manipulating objects and this is only possible because sensory feedback is ...

Mobile Game-based Virtual Reality Program for Upper Extremity Stroke Rehabilitation

Stroke rehabilitation requires repetitive, intensive, goal-oriented therapy. Virtual reality (VR) has the potential to satisfy these requirements. ...

Tickling, a Technique for Inducing Positive Affect When Handling Rats

Handling small animals such as rats can lead to several adverse effects. These include the fear of humans, resistance to handling, increased injury risk ...

Measuring the Kinematics of Daily Living Movements with Motion Capture Systems in Virtual Reality

The inability to complete instrumental activities of daily living (IADL) is a precursor to various neuropsychological diseases. Questionnaire-based ...

A Networked Desktop Virtual Reality Setup for Decision Science and Navigation Experiments with Multiple Participants

Investigating the interactions among multiple participants is a challenge for researchers from various disciplines, including the decision sciences and ...