Name: a methodology for capturing joint visual attention using mobile eye-trackers
Uploaded: 2020-01-18T14:00:00
Description: Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

The development of this methodology was supported by the National Science Foundation (NSF #0835854), the Leading House Technologies for Vocation Education, funded by the Swiss State Secretariat for Education, Research and Innovation, and the Harvard School of Education&#39;s Dean Venture Fund.

1. Participant Screening
<ol>
	<li>Ensure that participants with normal or corrected-to-normal vision are recruited. Because participants will be asked to wear a mobile eye-tracker, they can wear contact lenses but not regular eyeglasses.</li>
</ol>
2. Preparation for the Experiment
<ol>
	<li>Eye-tracking devices
	<ol>
		<li>Use any mobile eye-tracker capable of capturing eye movement in real world environments. 
		NOTE: The mobile eye-trackers used here were two Tobii Pro Glasses 2 (see ...

<ol>
	<li>Tomasello, M., Moore, C., Dunham, P. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Joint+attention+as+social+cognition.">Joint attention as social cognition.</a> Joint attention: Its origins and role in development. , 103-130 (1995).</li><li>Mundy, P., Sigman, M., Kasari, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+longitudinal+study+of+joint+attention+and+language+development+in+autistic+children.">A longitudinal study of joint attention and language development in autistic children.</a> Journal of Autism and Developmental Disorders. 20, 115-128 (1990).</li><li>Clark, H. H., Brennan, S. E., Resnick, L. B., Levine, J. M., Teasley, S. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Grounding+in+communication.">Grounding in communication.</a> Perspectives on socially shared cognition. , 127-149 (1991).</li><li>Siposova, B., Carpenter, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+look+at+joint+attention+and+common+knowledge.">A new look at joint attention and common knowledge.</a> Cognition. 189, 260-274 (2019).</li><li>Gergle, D., Clark, A. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=See+What+I'm+Saying?:+Using+Dyadic+Mobile+Eye+Tracking+to+Study+Collaborative+Reference.">See What I'm Saying?: Using Dyadic Mobile Eye Tracking to Study Collaborative Reference.</a> Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work. , 435-444 (2011).</li><li>Renner, P., Pfeiffer, T., Wachsmuth, I., Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Spatial+References+with+Gaze+and+Pointing+in+Shared+Space+of+Humans+and+Robots.">Spatial References with Gaze and Pointing in Shared Space of Humans and Robots.</a> Spatial Cognition IX. , 121-136 (2014).</li><li>Shvarts, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+detection+of+gaze+convergence+in+multimodal+collaboration:+a+dual+eye-tracking+technology.">Automatic detection of gaze convergence in multimodal collaboration: a dual eye-tracking technology.</a> The Russian Journal of Cognitive Science. 5, 4 (2018).</li><li>. Chilitags: Robust Fiducial Markers for Augmented Reality [software] Available from: <a target="_blank" href="https://github.com/chili-epfl/qml-chilitags">https://github.com/chili-epfl/qml-chilitags</a> (2013)</li><li>Jermann, P., Mullins, D., N&#252;ssli, M. -. A., Dillenbourg, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Collaborative+Gaze+Footprints:+Correlates+of+Interaction+Quality.+Connecting+Computer-Supported+Collaborative+Learning+to+Policy+and+Practice.">Collaborative Gaze Footprints: Correlates of Interaction Quality. Connecting Computer-Supported Collaborative Learning to Policy and Practice.</a> CSCL2011 Conference Proceedings., Volume I - Long Papers. , 184-191 (2011).</li><li>Richardson, D. C., Dale, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Looking+To+Understand:+The+Coupling+Between+Speakers'+and+Listeners'+Eye+Movements+and+Its+Relationship+to+Discourse+Comprehension.">Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension.</a> Trends in Cognitive Sciences. 29, 1045-1060 (2005).</li><li>Richardson, D. C., Dale, R., Kirkham, N. Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Art+of+Conversation+Is+Coordination+Common+Ground+and+the+Coupling+of+Eye+Movements+During+Dialogue.">The Art of Conversation Is Coordination Common Ground and the Coupling of Eye Movements During Dialogue.</a> Psychological Science. 18, 407-413 (2007).</li><li>Schneider, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+Mobile+Eye-Trackers+to+Unpack+the+Perceptual+Benefits+of+a+Tangible+User+Interface+for+Collaborative+Learning.">Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning.</a> ACM Transactions on Computer-Human Interaction. 23, 1-23 (2016).</li><li>Meier, A., Spada, H., Rummel, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+rating+scheme+for+assessing+the+quality+of+computer-supported+collaboration+processes.">A rating scheme for assessing the quality of computer-supported collaboration processes.</a> Int. J. Comput.-Support. Collab. Learn. 2, 63-86 (2007).</li><li>Schneider, B., Pea, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-time+mutual+gaze+perception+enhances+collaborative+learning+and+collaboration+quality.">Real-time mutual gaze perception enhances collaborative learning and collaboration quality.</a> Journal of Computer-Supported Collaborative Learning. 8, 375-397 (2013).</li></ol>

The methodology described in this paper provides a rigorous way to capture JVA in colocated dyads. With the emergence of affordable sensing technology and improved computer vision algorithms, it is now possible to study collaborative interactions with an accuracy that was previously unavailable. This methodology leverages fiducial markers disseminated in the environment and uses homographies as a way to remap participants&#39; gazes onto a common plane. This allows researchers to rigorously study JVA in colocated groups....

JVA has been extensively studied over the last century, especially by developmental psychologists studying language acquisition. It was quickly established that joint attention is more than just a way to learn words but rather a precursor to children&#39;s theories of mind1. Thus, it plays a significant role in many social processes, such as communicating with others, collaborating, and developing empathy. Autistic children, for instance, lack the ability to coordinate their visual attention with their caregivers, which is associated with significant social impairments2. Humans need joint attention to become functional members of society, to coordinate their actions, and to learn from others. From children acquiring their first words, teenagers learning from schoolteachers, students collaborating on projects, and to groups of adults working toward common goals, joint attention is a fundamental mechanism to establish common ground between individuals3. In this paper, I focus on the study of JVA in educational research. Understanding how joint attention unfolds over time is of primary importance for the study of collaborative learning processes. As such, it plays a predominant role in socioconstructivist settings.
The exact definition of joint attention is still debated4. This paper is concerned with a subconstruct of joint attention (JA), namely JVA. JVA happens when two subjects are looking at the same place at the same time. It should be noted that JVA does not provide any information about other important constructs of interest in the study of JA, such as monitoring common, mutual, and shared attention, or more generally, awareness of the cognition of another group member. This paper operationalizes and simplifies JVA by combining the eye-tracking data from two participants and analyzing the frequency in which they align their gazes. For a more comprehensive discussion, the interested reader can learn more about the study of the JA construct in Siposovaet al.4.
Over the past decade, technological advances have radically transformed research on JVA. The main paradigm shift was to use multiple eye-trackers to obtain quantitative measures of attentional alignments, as opposed to qualitatively analyzing video recordings in a laboratory or ecological setting. This development has allowed researchers to collect precise, detailed information about dyads&#39; visual coordination. Additionally, eye-trackers are becoming more affordable: until recently, their use was reserved to academic settings or large corporations. It is now possible to purchase inexpensive eye-trackers that generate reliable datasets. Finally, the progressive inclusion of gaze-tracking capabilities into existing devices like high-end laptops and virtual and augmented reality headsets suggests that eye-tracking will soon become ubiquitous.
Because of the popularization of eye-tracking devices, it is important to understand what they can and cannot tell us about social interactions. The methodology presented in this paper marks a first step in this direction. I address two challenges in capturing JVA from multiple eye-trackers: synchronizing the data on 1) the temporal scale, and 2) on the spatial scale. More specifically, this protocol makes use of fiducial markers placed in real-world environments to inform computer vision algorithms where participants are orienting their gaze. This new kind of methodology paves the way to rigorous analysis of human behavior in small groups.
This research protocol complies with the guidelines of Harvard University&#39;s human research ethics committee.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Tobii Glasses 2</td>
			<td>Tobii</td>
			<td>N/A</td>
			<td><a href="https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/" target="_blank">https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/</a></td>
		</tr>
		<tr>
			<td>Fiducial markers</td>
			<td>Chili lab &#8211; EPFL, Switzerland</td>
			<td>N/A</td>
			<td><a href="https://github.com/chili-epfl/chilitags" target="_blank">https://github.com/chili-epfl/chilitags</a></td>
		</tr>
	</tbody>
</table>

a methodology for capturing joint visual attention using mobile eye-trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

The methodology presented above was used to study students who were following a vocational training program in logistics (n = 54)12. In this experiment, pairs of students interacted with a Tangible User Interface (TUI) that simulated a small-scale warehouse. The fiducial markers placed on the TUI allowed the research team to remap students&#39; gazes onto a common plane and compute levels of JVA. Findings indicated that groups who had higher levels of JVA tended to do better at the task given to t...

Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Research

JoVE Journal

Behavior

Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have ...

The 5-Choice Serial Reaction Time Task: A Task of Attention and Impulse Control for Rodents

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. ...

Topographical Estimation of Visual Population Receptive Fields by fMRI

Visual cortex is retinotopically organized so that neighboring populations of cells map to neighboring parts of the visual field. Functional magnetic ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

A Method to Quantify Visual Information Processing in Children Using Eye Tracking

Visual problems that occur early in life can have major impact on a child's development. Without verbal communication and only based on observational ...

Visualizing Visual Adaptation

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical ...

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

Central and Divided Visual Field Presentation of Emotional Images to Measure Hemispheric Differences in Motivated Attention

Two dominant theories on lateralized processing of emotional information exist in the literature. One theory posits that unpleasant emotions are processed ...

Methodology for Establishing a Community-Wide Life Laboratory for Capturing Unobtrusive and Continuous Remote Activity and Health Data

An end-to-end suite of technologies has been established for the unobtrusive and continuous monitoring of health and activity changes occurring in the ...

Visual Classical Conditioning in Wood Ants

Several species of insects have become model systems for studying learning and memory formation. Although many studies focus on freely moving animals, ...