Name: a methodology for capturing joint visual attention using mobile eye-trackers
Uploaded: 2020-01-18T14:00:00
Description: Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

The development of this methodology was supported by the National Science Foundation (NSF #0835854), the Leading House Technologies for Vocation Education, funded by the Swiss State Secretariat for Education, Research and Innovation, and the Harvard School of Education&#39;s Dean Venture Fund.

1. Participant Screening
<ol>
	<li>Ensure that participants with normal or corrected-to-normal vision are recruited. Because participants will be asked to wear a mobile eye-tracker, they can wear contact lenses but not regular eyeglasses.</li>
</ol>
2. Preparation for the Experiment
<ol>
	<li>Eye-tracking devices
	<ol>
		<li>Use any mobile eye-tracker capable of capturing eye movement in real world environments. 
		NOTE: The mobile eye-trackers used here were two Tobii Pro Glasses 2 (see ...

<ol>
	<li>Tomasello, M., Moore, C., Dunham, P. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Joint+attention+as+social+cognition.">Joint attention as social cognition.</a> Joint attention: Its origins and role in development. , 103-130 (1995).</li><li>Mundy, P., Sigman, M., Kasari, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+longitudinal+study+of+joint+attention+and+language+development+in+autistic+children.">A longitudinal study of joint attention and language development in autistic children.</a> Journal of Autism and Developmental Disorders. 20, 115-128 (1990).</li><li>Clark, H. H., Brennan, S. E., Resnick, L. B., Levine, J. M., Teasley, S. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Grounding+in+communication.">Grounding in communication.</a> Perspectives on socially shared cognition. , 127-149 (1991).</li><li>Siposova, B., Carpenter, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+look+at+joint+attention+and+common+knowledge.">A new look at joint attention and common knowledge.</a> Cognition. 189, 260-274 (2019).</li><li>Gergle, D., Clark, A. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=See+What+I'm+Saying?:+Using+Dyadic+Mobile+Eye+Tracking+to+Study+Collaborative+Reference.">See What I'm Saying?: Using Dyadic Mobile Eye Tracking to Study Collaborative Reference.</a> Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work. , 435-444 (2011).</li><li>Renner, P., Pfeiffer, T., Wachsmuth, I., Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Spatial+References+with+Gaze+and+Pointing+in+Shared+Space+of+Humans+and+Robots.">Spatial References with Gaze and Pointing in Shared Space of Humans and Robots.</a> Spatial Cognition IX. , 121-136 (2014).</li><li>Shvarts, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+detection+of+gaze+convergence+in+multimodal+collaboration:+a+dual+eye-tracking+technology.">Automatic detection of gaze convergence in multimodal collaboration: a dual eye-tracking technology.</a> The Russian Journal of Cognitive Science. 5, 4 (2018).</li><li>. Chilitags: Robust Fiducial Markers for Augmented Reality [software] Available from: <a target="_blank" href="https://github.com/chili-epfl/qml-chilitags">https://github.com/chili-epfl/qml-chilitags</a> (2013)</li><li>Jermann, P., Mullins, D., N&#252;ssli, M. -. A., Dillenbourg, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Collaborative+Gaze+Footprints:+Correlates+of+Interaction+Quality.+Connecting+Computer-Supported+Collaborative+Learning+to+Policy+and+Practice.">Collaborative Gaze Footprints: Correlates of Interaction Quality. Connecting Computer-Supported Collaborative Learning to Policy and Practice.</a> CSCL2011 Conference Proceedings., Volume I - Long Papers. , 184-191 (2011).</li><li>Richardson, D. C., Dale, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Looking+To+Understand:+The+Coupling+Between+Speakers'+and+Listeners'+Eye+Movements+and+Its+Relationship+to+Discourse+Comprehension.">Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension.</a> Trends in Cognitive Sciences. 29, 1045-1060 (2005).</li><li>Richardson, D. C., Dale, R., Kirkham, N. Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Art+of+Conversation+Is+Coordination+Common+Ground+and+the+Coupling+of+Eye+Movements+During+Dialogue.">The Art of Conversation Is Coordination Common Ground and the Coupling of Eye Movements During Dialogue.</a> Psychological Science. 18, 407-413 (2007).</li><li>Schneider, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+Mobile+Eye-Trackers+to+Unpack+the+Perceptual+Benefits+of+a+Tangible+User+Interface+for+Collaborative+Learning.">Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning.</a> ACM Transactions on Computer-Human Interaction. 23, 1-23 (2016).</li><li>Meier, A., Spada, H., Rummel, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+rating+scheme+for+assessing+the+quality+of+computer-supported+collaboration+processes.">A rating scheme for assessing the quality of computer-supported collaboration processes.</a> Int. J. Comput.-Support. Collab. Learn. 2, 63-86 (2007).</li><li>Schneider, B., Pea, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-time+mutual+gaze+perception+enhances+collaborative+learning+and+collaboration+quality.">Real-time mutual gaze perception enhances collaborative learning and collaboration quality.</a> Journal of Computer-Supported Collaborative Learning. 8, 375-397 (2013).</li></ol>

The methodology described in this paper provides a rigorous way to capture JVA in colocated dyads. With the emergence of affordable sensing technology and improved computer vision algorithms, it is now possible to study collaborative interactions with an accuracy that was previously unavailable. This methodology leverages fiducial markers disseminated in the environment and uses homographies as a way to remap participants&#39; gazes onto a common plane. This allows researchers to rigorously study JVA in colocated groups....

JVA has been extensively studied over the last century, especially by developmental psychologists studying language acquisition. It was quickly established that joint attention is more than just a way to learn words but rather a precursor to children&#39;s theories of mind1. Thus, it plays a significant role in many social processes, such as communicating with others, collaborating, and developing empathy. Autistic children, for instance, lack the ability to coordinate their visual attention with their caregivers, which is associated with significant social impairments2. Humans need joint attention to become functional members of society, to coordinate their actions, and to learn from others. From children acquiring their first words, teenagers learning from schoolteachers, students collaborating on projects, and to groups of adults working toward common goals, joint attention is a fundamental mechanism to establish common ground between individuals3. In this paper, I focus on the study of JVA in educational research. Understanding how joint attention unfolds over time is of primary importance for the study of collaborative learning processes. As such, it plays a predominant role in socioconstructivist settings.
The exact definition of joint attention is still debated4. This paper is concerned with a subconstruct of joint attention (JA), namely JVA. JVA happens when two subjects are looking at the same place at the same time. It should be noted that JVA does not provide any information about other important constructs of interest in the study of JA, such as monitoring common, mutual, and shared attention, or more generally, awareness of the cognition of another group member. This paper operationalizes and simplifies JVA by combining the eye-tracking data from two participants and analyzing the frequency in which they align their gazes. For a more comprehensive discussion, the interested reader can learn more about the study of the JA construct in Siposovaet al.4.
Over the past decade, technological advances have radically transformed research on JVA. The main paradigm shift was to use multiple eye-trackers to obtain quantitative measures of attentional alignments, as opposed to qualitatively analyzing video recordings in a laboratory or ecological setting. This development has allowed researchers to collect precise, detailed information about dyads&#39; visual coordination. Additionally, eye-trackers are becoming more affordable: until recently, their use was reserved to academic settings or large corporations. It is now possible to purchase inexpensive eye-trackers that generate reliable datasets. Finally, the progressive inclusion of gaze-tracking capabilities into existing devices like high-end laptops and virtual and augmented reality headsets suggests that eye-tracking will soon become ubiquitous.
Because of the popularization of eye-tracking devices, it is important to understand what they can and cannot tell us about social interactions. The methodology presented in this paper marks a first step in this direction. I address two challenges in capturing JVA from multiple eye-trackers: synchronizing the data on 1) the temporal scale, and 2) on the spatial scale. More specifically, this protocol makes use of fiducial markers placed in real-world environments to inform computer vision algorithms where participants are orienting their gaze. This new kind of methodology paves the way to rigorous analysis of human behavior in small groups.
This research protocol complies with the guidelines of Harvard University&#39;s human research ethics committee.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Tobii Glasses 2</td>
			<td>Tobii</td>
			<td>N/A</td>
			<td><a href="https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/" target="_blank">https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/</a></td>
		</tr>
		<tr>
			<td>Fiducial markers</td>
			<td>Chili lab &#8211; EPFL, Switzerland</td>
			<td>N/A</td>
			<td><a href="https://github.com/chili-epfl/chilitags" target="_blank">https://github.com/chili-epfl/chilitags</a></td>
		</tr>
	</tbody>
</table>

a methodology for capturing joint visual attention using mobile eye-trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

The methodology presented above was used to study students who were following a vocational training program in logistics (n = 54)12. In this experiment, pairs of students interacted with a Tangible User Interface (TUI) that simulated a small-scale warehouse. The fiducial markers placed on the TUI allowed the research team to remap students&#39; gazes onto a common plane and compute levels of JVA. Findings indicated that groups who had higher levels of JVA tended to do better at the task given to t...

Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Hi, my name is Bertrand Schneider and I'm an assistant professor at the Harvard Graduate School of Education. In this video I'm going to show you how we can use mobile eye-trackers to capture a central construct into social science, joint visual attention. Joint visual attention has been extensively studied by psychologists and has been found to be closely correlated to the quality of the interactions between group members.It turns out that when people build a common ground and create a shared understanding of a task, they tend to frequently look at the same place at the same time. Traditionally, researchers have studied joint visual attention qualitatively by manually coding videos. I'm going to show you how we can use mobile eye-trackers to get a quantitative measure of this construct in co-located settings.In this video we will be using the Tobii Pro Glasses Two. These glasses are wearable eye-trackers that can capture eye movement in real-world environments. In addition to specialized cameras on the frame to track eye movements, the device is also equipped with a full-HD scene camera and a microphone, so the gaze behavior can be visualized within the context of the wearer's visual field.For these glasses gaze is captured 50 times per second and a live video feed from the glasses can be streamed to a computer either wirelessly, or through an ethernet cable. The glasses do have one limitation though, as they will not work over regular eyeglasses. The procedure to set up the eye-tracker is relatively straightforward.First, the participants will be asked to put on the eye-tracking glasses as they would a normal pair of glasses. Based on the participants'distinct facial features, nose pieces with different heights may need to be used to preserve data quality. After turning on eye-tracker, the participants should clip the data collection unit to their persons to allow for unbridled body movements.The Tobii Pro Glasses controller should be opened and the participants should be instructed to look at the center of the calibration marker provided by Tobii, while the calibration function of the software is enabled. Once calibration is complete, recording can be started from within the software. After the recording session is complete, terminate the recording from the Tobii software before instructing the participant to remove the eye-tracking glasses and the data collection unit.And then, turn off the unit. Data can be extracted through another software, Tobii Pro Lab, by removing the SD card from the data collection unit and inserting the card to the computer. Tobii Pro Lab is able to import all of the recording sessions stored in the SD card at the same time.Files can then be processed within Tobii Pro Lab to generate videos, different visualizations or be outputted as tab-separated values or TSV files for further analysis. Here you can see the raw eye-tracking data from a study we recently conducted, where two participants were learning to program a robot. On each side you can see the video stream generated by each eye-tracker with the location of the participant's gaze.As you can see, it is impossible to tell if they're looking at the same place at the same time, because the point of view of each participant is different. Additionally, data recording may start at different times. This means that the data needs to be synchronized temporally and spatially.I'm going to show you how to address these two issues in this video. First, I'm going to describe a procedure to synchronize the data temporarily. For the first participant you have a certain number of video frames.Some of them are before or after the actual experimental task. Like the first frame, where the experimenter is calibrating the eye-tracker. Similarly, for the other participants you have the same kind of data.It is not shown here but each frame of the video is also associated with an x and y coordinate that represents the gaze of each participant. To synchronize the data, we briefly show a fiducial marker on the computer screen before and after the experimental task. By using a computer vision algorithm, we can detect when this marker is presented to each participant, which allows us to trim and align the data.So, this is one way of dealing with data synchronization issues. In the next parts, we are going to look at the second issue:How to synchronize the data spatially. As mentioned earlier, the data is coming from each eye-tracker in the form of a video feed associated with the location of each participant's gaze, here in blue and green.While the x and y-coordinates might be the same for both participants, it doesn't mean that they're looking at the same place because they're looking at the screen from two different perspectives. One way of solving this issue is to build an image of the experimental setup that will serve as a reference, and where we're going to remap the location of each participant's case. This allows us to detect for each frame of the eye-tracking video, if the participants are looking at the same place at the same time.But how do we remap these coordinates into the image on the left? We are going to use the same cooperative vision algorithm that allowed us to synchronize the data earlier. By applying it on each frame of the video recordings, we can now detect the location of the fiducial markers from the perspective of the participants.This allows us to connect the same markers on the reference image on the left. By knowing the coordinates of this shared set of points, we can infer the location of each person's gaze using a mathematical operation known as a homography. By applying this procedure on each frame, we can generate a video to make sure that the homography worked.On the right side you can see the video recording of each participant with the location of their gaze in blue and green. The same fiducial markers are connected with a white line between the image on the left and the participant's point of view on the right hand side. The remapped gazes are shown on the left and they turn red when there is some joint visual attention.Generating this video is an important step toward making sure that the data is clean and that the homography was performed correctly. Additionally, there are two other visualizations that can be produced to sanity check the data. The first visualization is a heat map.For each participant we can plot each gaze point on the image of the experimental setup. This ensures that the homography worked correctly and allows us to categorize these case points into different areas of interests. Here, for example, we can see that most of the time was spent looking at the computer screen and very little time was spent looking at the cheat sheets.The second visualization is called a cross recurrence graph. Cross recurrence graphs allows us to visualize eye-tracking data for pair of participants. Time for the first participant is displayed on the x-axis, time for the second participant is displayed on the y-axis.Black squares mean that both participants are looking at the same place, white square represent missing data and gray square represent when participants are looking at different locations. Black squares along the diagonal mean that they're looking at the same place at the same time. Black squares off the diagonal mean that participants are looking at the same place but at different times.On the left, you can see a dyad with high levels of joint visual attention. In the middle, a dyad with low levels of joint visual attention. On the right side, a group with a lot of missing data.By performing this sanity checks, you can make sure that you have correctly synchronized and remapped your data into a common image of the experimental setup. These steps are critical and need to be performed before any analysis takes place. Lastly, there are two parameters that need to be picked before computing a measure of joint visual attention.The first parameter is the time window in which to participants can look at the same location. Previous work by Richardson and Dale had established that it can take up to two seconds for participants to disengage from what they're doing to pay attention to what their partner is doing. Thus, we determined that there is joint visual attention if two participants have looked at the same place within a plus and minus two second window.The second parameter is the minimum distance between two gaze points for them to qualify as joint visual attention. This distance is context-dependent and needs to be defined by researchers depending on the task administered and their research questions. For some tasks, the distance might be short.Here, for example, we used 100 pixels. For other tasks this distance might be larger. Next, I'm going to present some results found using this methodology.After you get an estimate for the amount of joint visual attention in the group, you can correlate this measure with other variables of interest. For example, in the work we have correlated this measure with the rating scheme developed into learning sciences that captures a groups quality of collaboration. For each group we assigned a score on the nines of dimensions presented here.For example, how well people sustained mutual understanding or how easily they reached a consensus. These scores have to receive acceptable inter reliability with another quota. Finally, we can also aggregate those scores into one general metric that approximates collaboration quality for each group.A result that has been found in our work as well as other studies is that joint visual attention is significantly correlated with collaboration quality, as measured by the rating scheme presented earlier. Groups that are rated highly using this coding scheme tend to have more joint visual attention than the groups who received low scores. This shows that productive interactions are oftentimes associated with more joint visual attention.On the next slide I'm going to show you another result that builds upon this finding. So, one advantage of having fine eye-tracking data is that we can extract other measures of joint visual attention. For example, we can compute who initiated and responded to an offer of joint visual attention.In particular, on the x-axis of this graph the score of zero means equal distribution of these behaviors and a score of one means that one person was always responding or initiating moments of join visual attention. In this study we found an inverse correlation with learning gains shown on the y-axis as measured by pre and post-tests. Groups where the same person was consistently initiating or responding to moments of joint visual attention were less likely to learn and groups where this responsibility was equally shared were more likely to score higher on the post-tests when controlling for scores on the pre-test.In this video I've presented the methodology that helps researchers synchronize mobile eye-tracking data both temporally and spatially. Findings suggest that dual eye-tracking data can provide indicators of collaboration by computing measures of joint visual attention. Additionally, I've presented results showing that we can go beyond simple measures of joint attention, for example, by looking at who initiated or responded to an episode of joint visual attention.We found that this measure was related to other outcome measures, such as learning gains. Computing this kind of measure would not be possible without eye-tracking data. In summary, we found that the methodology presented in this video can help researchers gain new insights into collaborative processes.Thank you so much for watching this video and feel free to refer to the paper for more information.

Research

JoVE Journal

Behavior

Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have ...

The 5-Choice Serial Reaction Time Task: A Task of Attention and Impulse Control for Rodents

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. ...

Topographical Estimation of Visual Population Receptive Fields by fMRI

Visual cortex is retinotopically organized so that neighboring populations of cells map to neighboring parts of the visual field. Functional magnetic ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

A Method to Quantify Visual Information Processing in Children Using Eye Tracking

Visual problems that occur early in life can have major impact on a child's development. Without verbal communication and only based on observational ...

Visualizing Visual Adaptation

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical ...

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

Central and Divided Visual Field Presentation of Emotional Images to Measure Hemispheric Differences in Motivated Attention

Two dominant theories on lateralized processing of emotional information exist in the literature. One theory posits that unpleasant emotions are processed ...

Methodology for Establishing a Community-Wide Life Laboratory for Capturing Unobtrusive and Continuous Remote Activity and Health Data

An end-to-end suite of technologies has been established for the unobtrusive and continuous monitoring of health and activity changes occurring in the ...

Visual Classical Conditioning in Wood Ants

Several species of insects have become model systems for studying learning and memory formation. Although many studies focus on freely moving animals, ...