Name: combining eye-tracking data with an analysis of video content from free-viewing a video of a walk in an urban park environment
Uploaded: 2019-05-07T12:30:00
Description: Watch this Scientific Journal Video about Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment at JoVE.com

This work was financially supported by the City of Melbourne and partially by ARC DP 150103135. We would like to thank Eamonn Fennessy for his advice and collaborative approach. With special thanks to researcher assistants Isabelle Janecki and Ethan Chen whom also helped collect and analyze this data. All errors remain the authors.

Ethical approval for this project was given by the Australian Catholic University ethics committee - approval number #201500036E. This ensured that informed consent was gained from all participants and all participants participated voluntarily, and that participants data remained anonymous and confidential. In addition the approval was given due the method and equipment meeting Australian standards safety regulations.
1. Filming Urban Scenes that Can Be Used in an Eye-Tracking Study
<ol>
	<l...

<ol>
	<li>Patrik, P., Stigsdotter, U. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+relation+between+perceived+sensory+dimensions+of+urban+green+space+and+stress+restoration.">The relation between perceived sensory dimensions of urban green space and stress restoration.</a> Landscape and Urban Planning. 94 (3-4), 264-275 (2010).</li><li>Bj&#248;rn, G., Patil, G. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Biophilia:+does+visual+contact+with+nature+impact+on+health+and+well-being?.">Biophilia: does visual contact with nature impact on health and well-being?.</a> International Journal of Environmental Research and Public Health. 6 (9), 2332-2343 (2009).</li><li>Velarde, M. a. D., Fry, G., Tveit, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Health+effects+of+viewing+landscapes-Landscape+types+in+environmental+psychology.">Health effects of viewing landscapes-Landscape types in environmental psychology.</a> Urban Forestry &amp; Urban Greening. 6 (4), 199-212 (2007).</li><li>Polat, A. T., Ahmet, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Relationships+between+the+visual+preferences+of+urban+recreation+area+users+and+various+landscape+design+elements.">Relationships between the visual preferences of urban recreation area users and various landscape design elements.</a> Urban Forestry &amp; Urban Greening. 14 (3), 573-582 (2015).</li><li>Peter, P., Giannopoulos, I., Raubal, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Where+am+I?+Investigating+map+matching+during+self-localization+with+mobile+eye+tracking+in+an+urban+environment.">Where am I? Investigating map matching during self-localization with mobile eye tracking in an urban environment.</a> Transactions in GIS. 18 (5), 660-686 (2014).</li><li>Berto, R., Massaccesi, S., Pasini, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+Eye+Movements+Measured+across+High+and+Low+Fascination+Photographs+Differ?+Addressing+Kaplan's+Fascination+Hypothesis.">Do Eye Movements Measured across High and Low Fascination Photographs Differ? Addressing Kaplan's Fascination Hypothesis.</a> Journal of Environmental Psychology. 28 (2), 185-191 (2008).</li><li>Kaplan, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+restorative+benefits+of+nature:+Towards+an+integrative+framework.">The restorative benefits of nature: Towards an integrative framework.</a> Journal of Environmental Psychology. 15, 169-182 (1995).</li><li>Duchowski, A. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Eye Tracking Methodology: Theory and Practice. , (2017).</li><li>Amati, M., Ghanbari Parmehr, E., McCarthy, C., Sita, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+eye-catching+are+natural+features+when+walking+through+a+park?+Eye-+tracking+responses+to+videos+of+walks?.">How eye-catching are natural features when walking through a park? Eye- tracking responses to videos of walks?.</a> Urban Forestry and Urban Greening. 31, 67-78 (2018).</li><li>Gould, S. D. A. R. W. I. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Framework+for+Machine+Learning+and+Computer+Vision+Research+and+Development.">A Framework for Machine Learning and Computer Vision Research and Development.</a> Journal of Machine Learning Research. (Dec), 3533-3537 (2012).</li><li>Richardson, D., Matlock, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+integration+of+figurative+language+and+static+depictions:+an+eye+movement+study+of+fictive+motion.">The integration of figurative language and static depictions: an eye movement study of fictive motion.</a> Cognition. 102 (1), 129-138 (2007).</li><li>Bojko, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Eye Tracking the User Experience: A Practical Guide to Research. , (2013).</li></ol>

Generally, in standard software packages for analyzing eye-tracking data, a vector AOI is used. Even for a single still image, the size of the vector cannot be easily measured. Furthermore, including all AOIs in an image and calculating the relative amounts of AOIs is laborious. It is almost impossible to do this manually on a video without a machine learning technique such as the one described. This was a relatively simple statement that infers a free viewing situation. A much more precise scenario can be used and diffe...

Our daily lived experiences of urban environments greatly impact on our health and well-being. Our well-being can depend on the amount of green spaces that we view and experience1,2,3, and these views can be quantified using eye-tracking equipment to guide decision making about park design. However, an issue arises with the volume of eye tracking data that is generated and making sense of this data. As the equipment for recording gaze data in a lab or natural setting becomes easier to use and more powerful, researchers need to consider how we can collect and analyze data validly to help with decision-making questions.
So far, a great deal of eye tracking research has used photographs in a survey or laboratory setting4. While this methodology allows for a great deal of reproducibility and control over results, it is unable to take advantage of the latest advances in eye-tracking technology which include the use of video and wearable mobile eye-trackers. Furthermore, we would argue that the act of walking and relaxing is necessarily dynamic particularly when oriented towards a task such as wayfinding5. Therefore a fully scientific understanding of these settings should take place outside of the laboratory. However, at the moment, eye-tracking in a real-life naturalistic setting makes comparing the experience between subjects very difficult. For example, if we wanted to compare whether one respondent looks at trees more than another, how could we control for the fact their point of view would be constantly changing compared to others or that their head might have turned. Detailed analysis in these conditions is near impossible with current analysis techniques. We would argue that it is important to control the viewing areas available to the individual being studied and in the analysis to be able to account for the total scene being viewed at any one point in time.
There are a set of theories linking stress levels and perceptions of safety to landscape views and well-evolved measures of stress6,7. There has also been a rapid increase in the sophistication of eye-tracking equipment to measure gaze8. Eye-tracking is important because involuntary eye-movements may be more reliably connected to preference, stress and other traditional measures than surveys and intrusive, physiological tests such as salivary cortisol levels. The objective of this research is to develop tools that enable a more precise measurement of eye-tracking data applied to more naturalistic settings, so as to provide further evidence for or refute long-standing landscape theories that have informed park design for decades.
The aim of this project is to develop and test a novel analysis technique that can generate relevant eye-tracking data for different videos of park walk simulations. Our work reported here and elsewhere9 represents a half-way point between the naturalistic setting of a fully mobile eye-tracking system and the lab-based photo studies referred to above. In particular, we concentrate on using videos as the stimulus material, exploring how this material can be used to test the amount of fascination that different parks generate in the City of Melbourne. Our work is based on the assumption that detailed analysis of videos is a necessary step to breach before undertaking a fuller, more naturalistic assessment of the potential of parks to provide restoration from stress.
In this study, we employed a desktop eye-tracker with videos of walks through urban parks and asked participants to imagine that they were taking a relaxing walk through a park. We describe a method to allow the amount of time that participants spent looking at different objects to be comparable between parks. Desktop studies are generally easier to control compared to mobile ET studies and allow comparative analysis of each subject.
Standard eye-tracking software uses a manual area of interest tool in which an operator can manually draw boundaries around objects of interest in each scene. This enables the amount of time that participants spent looking at different objects to be automatically counted. For video data, this process is labour intensive and subject to operator subjectivity and error. In later versions of eye-tracking analysis software, AOIs can automatically track objects across frames when they are the same size in the video. This is an improvement, however, this is only intended to be used for a small number of stimuli in each image and each image must be checked and confirmed.
Manual labelling of objects in an image is common and supported by image editing software such as GNU Image Manipulation Program (GIMP). Given that 1 s produces 30 frames or images, the manual labelling of videos is impractical. In addition, AOI labelling by drawing vector polygons around the edge of complex objects such as tree canopies is very time consuming. Finally, while it is conceivably possible to calculate the size of objects in a field of view using vector labelling, this feature is not currently available.
The method that we report on below deals with these limitations. This study employed automatic labelling of objects. This is possible using an image processing technique known as semantic labelling, in which every pixel in every frame of the video is assigned a label indicating an object class. Machine learning is used to derive pixel classifiers for each object class of interest. These classifiers provide a probabilistic label for each pixel (known as unary potentials), which are then refined in a subsequent optimization process to achieve the final labelled output. These classifiers learn statistical decision boundaries between each object classes in the space of features extracted from the image, including texture, histogram of edge orientations, RGB color values, and the normalized image coordinates. An appropriate technique for this is implemented in the DARWIN machine learning toolbox10 and is described below.

<table border="0" cellpadding="0" cellspacing="0" style="width:572px;" width="573">
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">12 mm lens</td>
			<td style="width:71px;">Olympus</td>
			<td style="width:119px;"></td>
			<td style="width:167px;">Lens</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">Panasonic GH4&#160;</td>
			<td>Panasonic</td>
			<td style="width:119px;"></td>
			<td>Video Camera</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">Tobii Studio version (2.1.14)&#160;&#160;</td>
			<td>Tobii</td>
			<td style="width:119px;"></td>
			<td>Software</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">Tobii x120 desktop eye-tracker</td>
			<td>Tobii</td>
			<td style="width:119px;"></td>
			<td>Eye-tracker</td>
		</tr>
	</tbody>
</table>

combining eye-tracking data with an analysis of video content from free-viewing a video of a walk in an urban park environment

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. Eye-tracking informatics are known to connect to a range of feelings, health conditions, mental states and actions. But because vision is the result of constant eye-movements, teasing out what is important from what is noise is complex and data intensive. Furthermore, a significant challenge is controlling for what people look at compared to what is presented to them.
The following presents a methodology for combining and analyzing eye-tracking on a video of a natural and complex scene with a machine learning technique for analyzing the content of the video. In the protocol we focus on analyzing data from filmed videos, how a video can be best used to record participants&#39; eye-tracking data, and importantly how the content of the video can be analyzed and combined with the eye-tracking data. We present a brief summary of the results and a discussion of the potential of the method for further studies in complex environments.

Figure 1 and Figure 2 show the result of taking all eye-tracking data for the whole video across all participants and producing a heat map; this is the standard approach available in eye-tracking software packages. By comparing Figure 1 and Figure 2 it is possible to identify that on average participants scanned left and right on the x coordinate of the video in ...

Watch this Scientific Journal Video about Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment at JoVE.com

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

The objective of the protocol is to detail how to collect video data for use in the laboratory; how to record eye-tracking data of participants looking at the data and how to efficiently analyze the content of the videos that they were looking at using a machine learning technique.

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. ...

Research

JoVE Journal

Environment

Analysis of Fatty Acid Content and Composition in Microalgae

A method to determine the content and composition of total fatty acids present in microalgae is described. Fatty acids are a major constituent of ...

Design and Construction of an Urban Runoff Research Facility

As the urban population increases, so does the area of irrigated urban landscape. Summer water use in urban areas can be 2-3x winter base line water use ...

An Aquatic Microbial Metaproteomics Workflow: From Cells to Tryptic Peptides Suitable for Tandem Mass Spectrometry-based Analysis

Meta-omic technologies such as metagenomics, metatranscriptomics and metaproteomics can aid in the understanding of microbial community structure and ...

The Use of an Automated System (GreenFeed) to Monitor Enteric Methane and Carbon Dioxide Emissions from Ruminant Animals

The Use of an Automated System GreenFeed to Monitor Enteric Methane and Carbon Dioxide Emissions from Ruminant Animals

Ruminant animals (domesticated or wild) emit methane (CH4) through enteric fermentation in their digestive tract and from decomposition of manure during ...

Extracting Metrics for Three-dimensional Root Systems: Volume and Surface Analysis from In-soil X-ray Computed Tomography Data

Plant roots play a critical role in plant-soil-microbe interactions that occur in the rhizosphere, as well as processes with important implications to ...

Spotting Cheetahs: Identifying Individuals by Their Footprints

The cheetah (Acinonyx jubatus) is Africa's most endangered large felid and listed as Vulnerable with a declining population trend by the IUCN1. It ranges ...

A Video Surveillance System to Monitor Breeding Colonies of Common Terns (Sterna Hirundo)

A Video Surveillance System to Monitor Breeding Colonies of Common Terns Sterna Hirundo

Many waterbird populations have faced declines over the last century, including the common tern (Sterna hirundo), a waterbird species with a widespread ...

Protocol for Producing Three-Dimensional Infrared Video of Freezing in Plants

Freezing in plants can be monitored using infrared (IR) thermography, because when water freezes, it gives off heat. However, problems with color contrast ...

Quantifying Plant Soluble Protein and Digestible Carbohydrate Content, Using Corn (Zea mays) As an Exemplar

Quantifying Plant Soluble Protein and Digestible Carbohydrate Content, Using Corn Zea mays As an Exemplar

Elemental data are commonly used to infer plant quality as a resource to herbivores. However, the ubiquity of carbon in biomolecules, the presence of ...

Long-term Video Tracking of Cohoused Aquatic Animals: A Case Study of the Daily Locomotor Activity of the Norway Lobster (Nephrops norvegicus)