The development of this methodology was supported by the National Science Foundation (NSF #0835854), the Leading House Technologies for Vocation Education, funded by the Swiss State Secretariat for Education, Research and Innovation, and the Harvard School of Education&#39;s Dean Venture Fund.

1. Participant Screening
<ol>
	<li>Ensure that participants with normal or corrected-to-normal vision are recruited. Because participants will be asked to wear a mobile eye-tracker, they can wear contact lenses but not regular eyeglasses.</li>
</ol>
2. Preparation for the Experiment
<ol>
	<li>Eye-tracking devices
	<ol>
		<li>Use any mobile eye-tracker capable of capturing eye movement in real world environments. 
		NOTE: The mobile eye-trackers used here were two Tobii Pro Glasses 2 (see Table of Materials). In addition to specialized cameras that can track eye movements, the glasses are also equipped with an HD scene camera and a microphone so that the gaze can be visualized in the context of the user&#39;s visual field. These glasses capture gaze data 50 times per second. Other researchers have used ASL Mobile Eye5, SMI6, or Pupil-labs7, all of which provide video streams from the scene camera and eye-tracking coordinates at varying sampling rates (30&#8211;120 Hz). The procedure below might vary slightly with other eye-tracking devices.</li>
	</ol>
	</li>
	<li>Fiducial Markers
	<ol>
		<li>The two steps below (i.e., temporal and spatial alignments) require the use of fiducial markers. There are several computer vision libraries that provide researchers with these markers and algorithms to detect them on an image or video feed. The protocol described uses the Chilitag library8.</li>
	</ol>
	</li>
	<li>Temporal alignment
	<ol>
		<li>Because the eye-tracking data are recorded on two separate units, ensure that the data are properly synchronized (Figure 1). Two main methods can be used. This manuscript only covers the first method, because server synchronization works differently with each brand of mobile eye-tracker.
		<ol>
			<li>Briefly show a fiducial marker on a computer screen to mark the beginning and the end of a session. This is similar to a visual &#34;hand clap&#34; (Figure 2).</li>
			<li>Alternatively, use a server to synchronize the clocks of the two data collection units. This method is slightly more accurate and recommended if a higher temporal accuracy is required.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Spatial alignment
	<ol>
		<li>To find if two participants are looking at the same place at the same time, map their gazes to a common plane. This plane can be a picture of the experimental setting (see the left side of Figure 3). Carefully design this image before the experiment.</li>
		<li>Size of the fiducial markers: The general size of the fiducial markers depends on the algorithm used to detect them from the eye-tracking video. Surfaces close to the participants can have smaller fiducial markers, while surfaces further away from them need to be larger, so that they look similar from the participants&#39; perspective. Try different sizes beforehand to make sure that they can be detected from the eye-tracking video.</li>
		<li>Number of fiducial markers: To make the process of mapping gaze points into a common plane successful, make sure to have several fiducial markers visible from the participants&#39; point of view at any given time.</li>
		<li>Location of the fiducial markers: Frame relevant areas of interest with strips of fiducial markers (e.g., see the laptop screen on Figure 3).</li>
	</ol>
	</li>
	<li>Finally, run pilots to test the synchronization procedure and determine the optimal location, size, and number of fiducial markers. Eye-tracking videos can be processed through a computer vision algorithm to see if the fiducial markers are reliably detected.</li>
</ol>
3. Running the experiment
<ol>
	<li>Instructions
	<ol>
		<li>Instruct participants to put on the eye-tracking glasses as they would a normal pair of glasses. Based on the participants&#39; distinct facial features, nose pieces of different heights may need to be used to preserve data quality.</li>
		<li>After turning on the eye-tracker, have the participants clip the recording unit to themselves to allow for natural body movement.</li>
	</ol>
	</li>
	<li>Calibration
	<ol>
		<li>Instruct the participants to look at the center of the calibration marker provided by Tobii while the calibration function of the software is enabled. Once the calibration is complete, recording can be started from within the software.</li>
		<li>Instruct participants to not move the mobile eye-trackers after calibration. If they do, the data are likely to be inaccurate and the calibration procedure will need to be performed again.</li>
	</ol>
	</li>
	<li>Data monitoring
	<ol>
		<li>Monitor the data collection process during the study and ensure that the eye-tracking data are being collected properly. Most mobile eye-trackers can provide a live stream on a separate device (e.g., a tablet) for this purpose.</li>
	</ol>
	</li>
	<li>Data export
	<ol>
		<li>After the recording session is complete, instruct the participant to remove the eye-tracking glasses and the data collection unit. Turn off the unit.</li>
		<li>Extract data using another software, Tobii Pro Lab, by removing the SD card from the data collection unit importing the session data. Tobii Pro Lab can be used to replay the video, create visualizations, and export the eye-tracking data as comma-separated (.csv) or tab-separated (.tsv) files.</li>
	</ol>
	</li>
</ol>
4. Preprocessing the dual eye-tracking data
<ol>
	<li>Sanity checking eye-tracking data
	<ol>
		<li>Check the eye-tracking data visually after data collection. It is not uncommon for some participants to have missing data. For example, some particular eye physiology can pose problems to eye-tracking algorithms, the glasses might shift during the experiment, the data collection software might crash, etc.</li>
		<li>Use descriptive statistics to check how much data were lost during each session and exclude sessions that have significant amounts of missing or noisy data.</li>
	</ol>
	</li>
	<li>Temporal alignment
	<ol>
		<li>Trim the data from each mobile eye-tracker to only include interactions between the participants. This can be achieved by using the method described above (i.e., presenting two special fiducial markers to participants at the start and the end of the session). These fiducial markers can then be detected from the eye-tracking video to trim the datasets.</li>
	</ol>
	</li>
	<li>Spatial alignment 
	NOTE: To detect whether two participants are looking at the same place at the same time, it is necessary to remap the participants&#39; gaze onto a common plane (i.e., an image of the experimental setting). A computational method for achieving this goal is a homography (i.e., a perspective transformation of a plane). From a technical perspective, two images of the same planar surface in space are related by a homography matrix. Based on a common set of points, this matrix can be used to infer the location of additional points between two planes. In Figure 3, for example, if a computer vision algorithm knows where the fiducial markers are on the handout, it can remap the gaze of the participant onto the common plane on the left side. The white lines connect the two sets of points shared by the video feed of each participant and the scene, which are then used for building the homography to remap the green and blue dots on the left side.
	<ol>
		<li>Use the Python version of OpenCV, e.g., to compute the homography matrix from the fiducial markers and then to remap the eye-tracking data to the scene of the experimental setting (or any other suitable library in your language of choice). OpenCV provides two useful functions: findHomography() to get the homography matrix, and perspectiveTransform() to transform the point from one perspective to the other.</li>
		<li>To use findHomography(), run with two arguments: the X,Y coordinates of the source points (i.e., the fiducial markers detected from the participants&#39; scene video, shown on the right in Figure 3) and the corresponding destination points (i.e., the same fiducials markers detected on the scene image, shown on the left in Figure 3).</li>
		<li>Feed the resulting homography matrix into the perspectiveTransform() function, along with a new point that needs to be mapped from the source image to the destination image (e.g., the eye-tracking data shown as a blue/green dot on the right side of Figure 3). The perspectiveTransform function returns the new coordinate of the same point on the scene image (i.e., the blue/green dots shown on the left side of Figure 3). 
		NOTE: For more information, the OpenCV official documentation provides sample code and examples to implement the homography: docs.opencv.org/master/d1/de0/tutorial_py_feature_homography.html.</li>
	</ol>
	</li>
	<li>Sanity checking the homography
	<ol>
		<li>Complete section 4.3 for the entire session, and perform a homography on each frame of the mobile eye-tracking video to check the quality of the homography. While there are no automated ways to estimate the accuracy of the resulting eye-tracking data, videos like the one shown in Figure 4 should be used to manually sanity check each session.</li>
		<li>If the quality is lower than expected, consider additional parameters to improve the results of the homography:
		<ol>
			<li>Number of fiducial markers detected: Only perform the homography if enough fiducial markers can be detected from the video stream. This number can be determined by examining the video produced above.</li>
			<li>Location of the fiducial markers: If different markers are at different depths and orientations, the quality of the homography usually increases when the markers closest to the gaze coordinates are selected, given that there are enough markers to build a robust homography.</li>
			<li>Orientation of the fiducial markers: Combining fiducial markers that have different orientations (e.g., horizontal and vertical) will produce inaccurate homographies. It is recommended to first detect which plane or areas of interests (AOIs) the participant is looking at (e.g., the computer screen, the cheat sheet, the table, see Figure 3) and then use the fiducial markers on this plane for the homography.</li>
			<li>Quality of the video stream: Sudden head movements can blur video frames and make the data unusable, because fiducial markers cannot be reliably detected (Figure 4). The methodology of this paper is not appropriate for experiments that involve a lot of sudden head movements.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
5. Analyzing the dual eye-tracking data
<ol>
	<li>Missing data
	<ol>
		<li>In order to make sure that the data were properly remapped onto the reference image, produce visualization graphs (e.g., Figure 5, Figure 6) and descriptive statistics to check how much data are missing.</li>
	</ol>
	</li>
	<li>Cross-recurrence graphs
	<ol>
		<li>Use cross-recurrence graphs9 to represent visual synchronization between two participants (Figure 6), where the X-axis represents time for the first participant, and the Y-axis represents time for the second participant. Black squares indicate that participants are looking at the same area, a black diagonal line describes two subjects looking at the same thing at exactly the same time, and black squares off the diagonal line describes when two subjects looking at the same thing with a time lag. Finally, differentiating between missing data (white square) and existing data with no JVA (gray squares) helps identify problematic sessions. This provides researchers with a visual sanity check.</li>
	</ol>
	</li>
	<li>Computing JVA
	<ol>
		<li>After filtering for missing data, compute a metric for JVA by counting the number of times that participants&#39; gazes are in the same radius in the scene (defined below) in a -2/+2 s time window. Divide this number by the number of valid data points that can be used to compute JVA. The result of the division represents the percentage of time that two subjects were jointly looking at the same place. This last step is necessary to avoid inflating the scores of groups with more data after the homography. 
		NOTE: Two parameters need to be set before JVA can be computed, the minimal distance between two gaze points, and the time window between them (Figure 7): 1) Time window: An early foundational study10 used a single eye-tracker to measure JVA between a listener and a speaker. The researchers asked a first set of participants (&#34;speakers&#34;) to talk about a television show whose characters were displayed in front of them. A second set of participants (&#34;listeners&#34;) then watched the same show while listening to audio recording of the speakers. The eye movements of the speakers and listeners were compared, and it was found that a listener&#39;s eye movements closely matched a speaker&#39;s eye movement with a delay of 2 s. In subsequent work11 researchers analyzed live dialogues and found that a delay of 3 s best captured moments of JVA. Because each task is unique and might exhibit different time lags, it is also suggested to explore how different time lags affect the results of a given experiment. Overall, it is common to look for JVA in a &#177; 2/3 s time window depending on the experimental task and then explore how different time lags might change the results. 2) Distance between gazes: there is no empirically defined distance between two gazes for them to count as JVA. This distance is dependent on the research questions defined by the researchers. The research questions should inform the size of the targets of interest. In the example seen in Figure 7, a radius of 100 pixels on the scene image (blue/green circles) was chosen for the analysis because it is sufficient to capture when participants are looking at the robot in the maze, as well as at similar user interface elements on the computer screen, which are the two main areas of interest for this experimental task.</li>
	</ol>
	</li>
</ol>

<ol><li>Tomasello, M. Joint attention as social cognition. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aTomasello%2C+M+%22Joint+attention%3A+Its+origins+and+role+in+development%22">Joint attention: Its origins and role in development</a>. Moore, C., Dunham, P. J. , Lawrence Erlbaum Associates, Inc. Hillsdale, NJ, England. 103-130 (1995).</li>
<li>Mundy, P., Sigman, M., Kasari, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+longitudinal+study+of+joint+attention+and+language+development+in+autistic+children%5BTitle%5D)+AND+%22Journal+of+Autism+and+Developmental+Disorders%22%5BJournal%5D)">A longitudinal study of joint attention and language development in autistic children.</a> Journal of Autism and Developmental Disorders. 20, 115-128 (1990).</li>
<li>Clark, H. H., Brennan, S. E. Grounding in communication. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aClark%2C+HH+%22Perspectives+on+socially+shared+cognition%22">Perspectives on socially shared cognition</a>. Resnick, L. B., Levine, J. M., Teasley, S. D. , American Psychological Association. Washington, DC, US. 127-149 (1991).</li>
<li>Siposova, B., Carpenter, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+new+look+at+joint+attention+and+common+knowledge%5BTitle%5D)+AND+%22Cognition%22%5BJournal%5D)">A new look at joint attention and common knowledge.</a> Cognition. 189, 260-274 (2019).</li>
<li>Gergle, D., Clark, A. T. See What I'm Saying?: Using Dyadic Mobile Eye Tracking to Study Collaborative Reference. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aGergle%2C+D+%22Proceedings+of+the+ACM+2011+Conference+on+Computer+Supported+Cooperative+Work%22">Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work</a>. , ACM. New York, NY, USA. 435-444 (2011).</li>
<li>Renner, P., Pfeiffer, T., Wachsmuth, I. Spatial References with Gaze and Pointing in Shared Space of Humans and Robots. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aRenner%2C+P+%22Spatial+Cognition+IX%22">Spatial Cognition IX</a>. Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. , Springer International Publishing. 121-136 (2014).</li>
<li>Shvarts, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Automatic+detection+of+gaze+convergence+in+multimodal+collaboration%3A+a+dual+eye-tracking+technology%5BTitle%5D)+AND+%22The+Russian+Journal+of+Cognitive+Science%22%5BJournal%5D)">Automatic detection of gaze convergence in multimodal collaboration: a dual eye-tracking technology.</a> The Russian Journal of Cognitive Science. 5, 4(2018).</li>
<li>Bonnard, Q., et al. Chilitags: Robust Fiducial Markers for Augmented Reality [software]. , Available from: 
 <a target="_blank" href="https://github.com/chili-epfl/qml-chilitags">https://github.com/chili-epfl/qml-chilitags</a> (2013).</li>
<li>Jermann, P., Mullins, D., Nüssli, M. -A., Dillenbourg, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Collaborative+Gaze+Footprints%3A+Correlates+of+Interaction+Quality.+Connecting+Computer-Supported+Collaborative+Learning+to+Policy+and+Practice%5BTitle%5D)+AND+%22CSCL2011+Conference+Proceedings.%2C+Volume+I+-+Long+Papers%22%5BJournal%5D)">Collaborative Gaze Footprints: Correlates of Interaction Quality. Connecting Computer-Supported Collaborative Learning to Policy and Practice.</a> CSCL2011 Conference Proceedings., Volume I - Long Papers. , 184-191 (2011).</li>
<li>Richardson, D. C., Dale, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Looking+To+Understand%3A+The+Coupling+Between+Speakers%27+and+Listeners%27+Eye+Movements+and+Its+Relationship+to+Discourse+Comprehension%5BTitle%5D)+AND+%22Trends+in+Cognitive+Sciences%22%5BJournal%5D)">Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension.</a> Trends in Cognitive Sciences. 29, 1045-1060 (2005).</li>
<li>Richardson, D. C., Dale, R., Kirkham, N. Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+Art+of+Conversation+Is+Coordination+Common+Ground+and+the+Coupling+of+Eye+Movements+During+Dialogue%5BTitle%5D)+AND+%22Psychological+Science%22%5BJournal%5D)">The Art of Conversation Is Coordination Common Ground and the Coupling of Eye Movements During Dialogue.</a> Psychological Science. 18, 407-413 (2007).</li>
<li>Schneider, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Using+Mobile+Eye-Trackers+to+Unpack+the+Perceptual+Benefits+of+a+Tangible+User+Interface+for+Collaborative+Learning%5BTitle%5D)+AND+%22ACM+Transactions+on+Computer-Human+Interaction%22%5BJournal%5D)">Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning.</a> ACM Transactions on Computer-Human Interaction. 23, 1-23 (2016).</li>
<li>Meier, A., Spada, H., Rummel, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+rating+scheme+for+assessing+the+quality+of+computer-supported+collaboration+processes%5BTitle%5D)+AND+%22Int.+J.+Comput.-Support.+Collab.+Learn%22%5BJournal%5D)">A rating scheme for assessing the quality of computer-supported collaboration processes.</a> Int. J. Comput.-Support. Collab. Learn. 2, 63-86 (2007).</li>
<li>Schneider, B., Pea, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Real-time+mutual+gaze+perception+enhances+collaborative+learning+and+collaboration+quality%5BTitle%5D)+AND+%22Journal+of+Computer-Supported+Collaborative+Learning%22%5BJournal%5D)">Real-time mutual gaze perception enhances collaborative learning and collaboration quality.</a> Journal of Computer-Supported Collaborative Learning. 8, 375-397 (2013).</li>
</ol>

The authors declare that they have no competing financial interests.

The methodology described in this paper provides a rigorous way to capture JVA in colocated dyads. With the emergence of affordable sensing technology and improved computer vision algorithms, it is now possible to study collaborative interactions with an accuracy that was previously unavailable. This methodology leverages fiducial markers disseminated in the environment and uses homographies as a way to remap participants&#39; gazes onto a common plane. This allows researchers to rigorously study JVA in colocated groups.
This method includes multiple sanity checks that need to be performed at various point of the experiment. Because this is a complex procedure, researchers need to make sure that the resulting datasets are complete and valid. Finally, it is recommended to conduct pilot studies before the actual experiment, and to reconstruct participants&#39; interactions though a video after data collection is completed (Figure 3, Figure 4, Figure 5, Figure 6).
There are several limitations associated with this method:
Number of participants. While this methodology works well for two participants, analysis become more complicated with larger groups. Fiducial markers can still be used to remap gazes onto a ground truth but knowing how to identify JVA becomes a more nuanced process. Should JVA be defined as the times when everyone is looking at the same place at the same time, or when two participants are gazing at the same place? Additionally, visualizations like the cross-recurrence graph become impractical with more than 2&#8211;3 people.
Settings. The method described in this paper is appropriate for small, controlled settings (e.g., laboratory studies). Open-ended settings, such as outdoors or large spaces, are usually too complicated to instrument with fiducial markers and thus can limit the usefulness of the eye-tracking data. Additionally, the fiducial markers can be distracting and clutter the environment. In the future, better computer vision algorithms will be able to automatically extract common features between two perspectives. There are already algorithms that exist for this purpose, but we found that the level of accuracy was not yet acceptable for the type of experiment described above.
AOIs. Related to the point above, computing homography and the cross-recurrence graph work well with a stable number of areas of interest, but corrections have to be made when comparing different tasks with different numbers of areas of interest.
Use of equipment. Mobile eye-trackers can be obtrusive, affecting participants&#39; behavior or failing to work with particular eye physiology.
In conclusion, the methodology described in this paper is a promising way to study colocated interactions. It allows researchers to capture a precise metric for JVA, which is a critical construct in the social sciences1. Additionally, it is possible to detect more fine-grained indicators of collaborative learning through this methodology12 compared to traditional qualitative analyses. In short, it is a more efficient and accurate way to study social interactions.
Potential application of this method includes designing interventions to support collaboration through real-time eye-tracking data. Some pioneering work has produced shared gaze visualizations using remote eye-trackers, which has been shown to benefit collaborative learning from a distance14. Dyads who could see the gaze of their partner in real time exhibited more JVA, collaborated better and achieved higher learning gains compared to a control group. Future work will examine whether this kind of intervention can support collaborative processes in colocated settings (e.g., through virtual or augmented reality headsets).

JVA has been extensively studied over the last century, especially by developmental psychologists studying language acquisition. It was quickly established that joint attention is more than just a way to learn words but rather a precursor to children&#39;s theories of mind1. Thus, it plays a significant role in many social processes, such as communicating with others, collaborating, and developing empathy. Autistic children, for instance, lack the ability to coordinate their visual attention with their caregivers, which is associated with significant social impairments2. Humans need joint attention to become functional members of society, to coordinate their actions, and to learn from others. From children acquiring their first words, teenagers learning from schoolteachers, students collaborating on projects, and to groups of adults working toward common goals, joint attention is a fundamental mechanism to establish common ground between individuals3. In this paper, I focus on the study of JVA in educational research. Understanding how joint attention unfolds over time is of primary importance for the study of collaborative learning processes. As such, it plays a predominant role in socioconstructivist settings.
The exact definition of joint attention is still debated4. This paper is concerned with a subconstruct of joint attention (JA), namely JVA. JVA happens when two subjects are looking at the same place at the same time. It should be noted that JVA does not provide any information about other important constructs of interest in the study of JA, such as monitoring common, mutual, and shared attention, or more generally, awareness of the cognition of another group member. This paper operationalizes and simplifies JVA by combining the eye-tracking data from two participants and analyzing the frequency in which they align their gazes. For a more comprehensive discussion, the interested reader can learn more about the study of the JA construct in Siposovaet al.4.
Over the past decade, technological advances have radically transformed research on JVA. The main paradigm shift was to use multiple eye-trackers to obtain quantitative measures of attentional alignments, as opposed to qualitatively analyzing video recordings in a laboratory or ecological setting. This development has allowed researchers to collect precise, detailed information about dyads&#39; visual coordination. Additionally, eye-trackers are becoming more affordable: until recently, their use was reserved to academic settings or large corporations. It is now possible to purchase inexpensive eye-trackers that generate reliable datasets. Finally, the progressive inclusion of gaze-tracking capabilities into existing devices like high-end laptops and virtual and augmented reality headsets suggests that eye-tracking will soon become ubiquitous.
Because of the popularization of eye-tracking devices, it is important to understand what they can and cannot tell us about social interactions. The methodology presented in this paper marks a first step in this direction. I address two challenges in capturing JVA from multiple eye-trackers: synchronizing the data on 1) the temporal scale, and 2) on the spatial scale. More specifically, this protocol makes use of fiducial markers placed in real-world environments to inform computer vision algorithms where participants are orienting their gaze. This new kind of methodology paves the way to rigorous analysis of human behavior in small groups.
This research protocol complies with the guidelines of Harvard University&#39;s human research ethics committee.

<table><tbody><tr><td>Tobii Glasses 2</td><td>Tobii</td><td>N/A</td><td>&lt;a href=&quot;https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/&quot; target=&quot;_blank&quot;&gt;https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/&lt;/a&gt;</td></tr><tr><td>Fiducial markers</td><td>Chili lab &amp;ndash; EPFL, Switzerland</td><td>N/A</td><td>&lt;a href=&quot;https://github.com/chili-epfl/chilitags&quot; target=&quot;_blank&quot;&gt;https://github.com/chili-epfl/chilitags&lt;/a&gt;</td></tr></tbody></table>

a methodology for capturing joint visual attention using mobile eye-trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

The methodology presented above was used to study students who were following a vocational training program in logistics (n = 54)12. In this experiment, pairs of students interacted with a Tangible User Interface (TUI) that simulated a small-scale warehouse. The fiducial markers placed on the TUI allowed the research team to remap students&#39; gazes onto a common plane and compute levels of JVA. Findings indicated that groups who had higher levels of JVA tended to do better at the task given to them, learned more, and had a better quality of collaboration13 (Figure 8, left side). Dual eye-tracking datasets also allowed us to capture particular group dynamics like the free-rider effect. We estimated this effect by identifying who was likely to have initiated each moment of JVA (i.e., whose gaze was there first) and who responded to it (i.e., whose gaze was there second). We found a significant correlation between learning gains and the students&#39; tendency to equally share the responsibility of initiating and responding to offers of JVA. In other words, groups in which the same person always initiated moments of JVA were less likely to learn (Figure 8, right side) and groups where this responsibility was equally shared were more likely to learn. This finding shows that we can go beyond merely quantifying JV, and actually identify group dynamics and productivity using dual eye-tracking data.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/60670/60670fig1.jpg" /> 
Figure 1: Each participant generates two video feeds with the X,Y coordinates of their gaze on each video frame. This methodology addresses synchronizing the data temporally and spatially between the participants. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig1alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/60670/60670fig2.jpg" /> 
Figure 2: A methodology for synchronizing the two datasets. Briefly showing unique fiducial marker on a computer screen to tag the start and the end of the activity. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig2alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/60670/60670fig3.jpg" /> 
Figure 3: Using fiducial markers disseminated in the environment to remap participants&#39; gazes onto a common plan (left side). White lines indicate fiducial markers that have been detected in both images. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig3alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/60670/60670fig4.jpg" /> 
Figure 4: Examples of poor data quality. Left: A blurred frame from the eye-tracking video caused by a sudden head movement. Fiducial markers could not be detected in this image. Right: A failed homography where the fiducial marker data were not properly synchronized with the video feed. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig4alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/60670/60670fig5.jpg" /> 
Figure 5: Heatmaps. Left: A heatmap of the eye-tracking data remapped onto the experimental scene. This visualization was used as a sanity check for the homography. Right: A group that had too much missing data and had to be discarded. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig5alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/60670/60670fig6.jpg" /> 
Figure 6: Cross recurrence graph generated from three dyads to visualize JVA. P1 represents time for the first participant, P2 represents time for the second participant. Black squares show JVA; gray squares show moments where participants are looking at different places; white squares show missing data. Square along the main diagonal indicates moments where participants looked at the same place at the same time. This visualization was used as a sanity check for measures of JVA from the combined eye-tracking data. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig6alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/60670/60670fig7.jpg" /> 
Figure 7: A video frame where JVA was detected between two participants (red dots). Richardson et al.11 recommend looking at a time window of +/-2 s. when computing JVA. Additionally, researchers need to define the minimal distance between two gaze points to count as JVA. A radius of 100 pixel was chosen on the middle image above. <a href="https://www.jove.com/files/ftp_upload/60670/60670fig7large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 8" class="xfigimg" src="/files/ftp_upload/60670/60670fig8.jpg" /> 
Figure 8: Examples of results. Data from Schneider et al.12 where the percentage of time looking at the same place at the same time was correlated with participants&#39; quality of collaboration: r(24) = 0.460, P = 0.018 (left side) and imbalances in initiating/responding to offers of JVA was correlated with their learning gains: r(24) = &#8722;0.47, P = 0.02 (right side). <a href="https://www.jove.com/files/ftp_upload/60670/60670fig8alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

a-methodology-for-capturing-joint-visual-attention-using-mobile-eye

Nanyang Technological University

Research

JoVE Journal

Behavior

7.6K Views.  Harvard University. Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

Video: A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

How to Build a Dichoptic Presentation System That Includes an Eye Tracker

The presentation of different stimuli to the two eyes, dichoptic presentation, is essential for studies involving 3D vision and interocular suppression. There is a growing literature on the unique experimental value of pupillary and oculomotor measures, especially for research on interocular suppression. Although obtaining eye-tracking measures would thus benefit studies that use dichoptic presentation, the hardware essential for dichoptic presentation (e.g. mirrors) often interferes with high-quality eye tracking, especially when using a video-based eye tracker. We recently described an experimental setup that combines a standard dichoptic presentation system with an infrared eye tracker by using infrared-transparent mirrors1. The setup is compatible with standard monitors and eye trackers, easy to implement, and affordable (on the order of US$1,000). Relative to existing methods it has the benefits of not requiring special equipment and posing few limits on the nature and quality of the visual stimulus. Here we provide a visual guide to the construction and use of our setup.

We recently proposed a method that allows dichoptic visual stimulus presentation and binocular eye tracking simultaneously1. The key is the combination of an infrared eye tracker and the corresponding infrared-transparent mirrors. This manuscript provides an in depth protocol for initial setup and everyday operation.

The presentation of different stimuli to the two eyes, dichoptic presentation, is essential for studies involving 3D vision and interocular suppression. ...

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. Eye-tracking informatics are known to connect to a range of feelings, health conditions, mental states and actions. But because vision is the result of constant eye-movements, teasing out what is important from what is noise is complex and data intensive. Furthermore, a significant challenge is controlling for what people look at compared to what is presented to them.
The following presents a methodology for combining and analyzing eye-tracking on a video of a natural and complex scene with a machine learning technique for analyzing the content of the video. In the protocol we focus on analyzing data from filmed videos, how a video can be best used to record participants&#39; eye-tracking data, and importantly how the content of the video can be analyzed and combined with the eye-tracking data. We present a brief summary of the results and a discussion of the potential of the method for further studies in complex environments.

The objective of the protocol is to detail how to collect video data for use in the laboratory; how to record eye-tracking data of participants looking at the data and how to efficiently analyze the content of the videos that they were looking at using a machine learning technique.

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. ...

Environment

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test&#160;theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to&#160;comprehension informed by&#160;comprehenders&#39; experience. Among the examined aspects of the visual context are actions, events, a speaker&#39;s gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eye-tracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension.

The present article reviews an eye-tracking methodology for studies on language comprehension. To obtain reliable data, key steps of the protocol must be followed. Among these are the correct set-up of the eye tracker (e.g., ensuring good quality of the eye and head images) and accurate calibration.

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing ...

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

Gaze in Action: Head-mounted Eye Tracking of Children's Dynamic Visual Attention During Naturalistic Behavior

Young children's visual environments are dynamic, changing moment-by-moment as children physically and visually explore spaces and objects and interact with people around them. Head-mounted eye tracking offers a unique opportunity to capture children's dynamic egocentric views and how they allocate visual attention within those views. This protocol provides guiding principles and practical recommendations for researchers using head-mounted eye trackers in both laboratory and more naturalistic settings. Head-mounted eye tracking complements other experimental methods by enhancing opportunities for data collection in more ecologically valid contexts through increased portability and freedom of head and body movements compared to screen-based eye tracking. This protocol can also be integrated with other technologies, such as motion tracking and heart-rate monitoring, to provide a high-density multimodal dataset for examining natural behavior, learning, and development than previously possible. This paper illustrates the types of data generated from head-mounted eye tracking in a study designed to investigate visual attention in one natural context for toddlers: free-flowing toy play with a parent. Successful use of this protocol will allow researchers to collect data that can be used to answer questions not only about visual attention, but also about a broad range of other perceptual, cognitive, and social skills and their development.

Young children do not passively observe the world, but rather actively explore and engage with their environment. This protocol provides guiding principles and practical recommendations for using head-mounted eye trackers to record infants' and toddlers' dynamic visual environments and visual attention in the context of natural behavior.

Young children's visual environments are dynamic, changing moment-by-moment as children physically and visually explore spaces and objects and interact ...

Investigating the Deployment of Visual Attention Before Accurate and Averaging Saccades via Eye Tracking and Assessment of Visual Sensitivity

This experimental protocol was designed to investigate whether visual attention is obligatorily deployed at the endpoint of saccades. To this end, we recorded the eye position of human participants engaged in a saccade task via eye tracking and assessed visual orientation discrimination performance at various locations during saccade preparation. Importantly, instead of using a single saccade target paradigm for which the saccade endpoint typically coincides roughly with the target, this protocol comprised the presentation of two nearby saccade targets, leading to a distinct spatial dissociation between target locations and saccade endpoint on a substantial number of trials. The paradigm allowed us to compare presaccadic visual discrimination performance at the endpoint of accurate saccades (landing at one of the saccade targets) and of averaging saccades (landing at an intermediate location in between the two targets). We observed a selective enhancement of visual sensitivity at the endpoint of accurate saccades but not at the endpoint of averaging saccades. Rather, before the execution of averaging saccades, visual sensitivity was equally enhanced at both targets, suggesting that saccade averaging follows from unresolved attentional selection among the saccade targets. These results argue against a mandatory coupling between visual attention and saccade programming based on a direct measure of presaccadic visual sensitivity rather than saccadic reaction times, which have been used in other protocols to draw similar conclusions. While our protocol provides a useful framework to investigate the relationship between visual attention and saccadic eye movements at the behavioral level, it can also be combined with electrophysiological measures to extend insights at the neuronal level.

This experimental protocol combined eye tracking and the assessment of presaccadic visual sensitivity in a dual task paradigm, consisting of a free choice saccade task and a visual discrimination task, to investigate the deployment of visual spatial attention before both, accurate and averaging saccades.

This experimental protocol was designed to investigate whether visual attention is obligatorily deployed at the endpoint of saccades. To this end, we ...

Assessing Binocular Central Visual Field and Binocular Eye Movements in a Dichoptic Viewing Condition

Macular degeneration typically results in heterogeneous binocular central visual defects. Currently available approaches to assess central visual field, like the microperimetry, can test only one eye at a time. Therefore, they cannot explain how the defects in each eye affect the binocular interaction and real-world function. Dichoptic stimulus presentation with a gaze-controlled system could provide a reliable measure of monocular/binocular visual fields. However, dichoptic stimulus presentation and simultaneous eye-tracking are challenging because optical devices of instruments that present stimulus dichoptically (e.g., haploscope) always interfere with eye-trackers (e.g., infrared video-based eye-trackers). Therefore, the goals were 1) to develop a method for dichoptic stimulus presentation with simultaneous eye-tracking, using 3D-shutter glasses and 3D-ready monitors, that is not affected by interference and 2) to use this method to develop a protocol for assessing central visual field in subjects with central vision loss. The results showed that this setup provides a practical solution for reliably measuring eye-movements in dichoptic viewing condition. In addition, it was also demonstrated that this method can assess gaze-controlled binocular central visual field in subjects with central vision loss.

Presented here is a protocol for assessing binocular eye movements and gaze-controlled central visual field screening in participants with central vision loss.

Macular degeneration typically results in heterogeneous binocular central visual defects. Currently available approaches to assess central visual field, ...

Eye Tracking During A Complex Aviation Task For Insights Into Information Processing

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill performance. Previous work has shown that traditional and advanced gaze metrics reliably demonstrate robust differences in pilot expertise, cognitive load, fatigue, and even situation awareness (SA).
This study describes the methodology for using a wearable eye tracker and gaze mapping algorithm that captures naturalistic head and eye movements (i.e., gaze) in a high-fidelity flight motionless simulator. The method outlined in this paper describes the area of interest (AOI)-based gaze analyses, which provides more context related to where participants are looking, and dwell time duration, which indicates how efficiently they are processing the fixated information. The protocol illustrates the utility of a wearable eye tracker and computer vision algorithm to assess changes in gaze behavior in response to an unexpected in-flight emergency.
Representative results demonstrated that gaze was significantly impacted when the emergency event was introduced. Specifically, attention allocation, gaze dispersion, and gaze sequence complexity significantly decreased and became highly concentrated on looking outside the front window and at the airspeed gauge during the emergency scenario (all p values &#60; 0.05). The utility and limitations of employing a wearable eye tracker in a high-fidelity motionless flight simulation environment to understand the spatiotemporal characteristics of gaze behavior and its relation to information processing in the aviation domain are discussed.

Eye tracking is a non-invasive method to probe information processing. This article describes how eye tracking can be used to study gaze behavior during a flight simulation emergency task in low-time pilots (i.e., &lt;350 flight hours).

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill ...

Multiple Object Tracking

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University

In a staggeringly complex and engaging world, it is crucial to selectively process some stimuli at the expense of others. Experimental psychologists call this ability attention. Specifically, visual attention refers to the ability to selectively process aspects of a visual scene.
Many paradigms used to study visual attention involve brief, punctuated, and repeated trials. However, everyday situations often place sustained demands on attention, as opposed to requiring only brief focus. For example, compare driving through busy city streets, which demands sustained attention, with crossing a busy street, which demands just a few moments of caution. To investigate sustained visual attention, experimental psychologists typically rely on a paradigm called multiple object tracking.
This video demonstrates standard procedures for investigating sustained visual attention through multiple object tracking.

Multiple Object Tracking: Studying Sustained Visual Attention

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University

In a staggeringly complex and engaging world, it is crucial to selectively process ...