This work was financially supported by the City of Melbourne and partially by ARC DP 150103135. We would like to thank Eamonn Fennessy for his advice and collaborative approach. With special thanks to researcher assistants Isabelle Janecki and Ethan Chen whom also helped collect and analyze this data. All errors remain the authors.

Ethical approval for this project was given by the Australian Catholic University ethics committee - approval number #201500036E. This ensured that informed consent was gained from all participants and all participants participated voluntarily, and that participants data remained anonymous and confidential. In addition the approval was given due the method and equipment meeting Australian standards safety regulations.
1. Filming Urban Scenes that Can Be Used in an Eye-Tracking Study
<ol>
	<li>Use a high-quality digital video camera attached to a gimbal to create a simulated &#8216;walking&#8217; or &#8216;sitting&#8217; video.</li>
	<li>Record the film with smooth fluid movements in 4k resolution at 25 fps and save them at 1920 x 1080 HD.</li>
	<li>Use a professional filming company to produce digital films if unfamiliar with this or lacking access to equipment.</li>
	<li>Shoot film in one take, to a single length across all videos with precise predefined routes and characteristics. 
	NOTE: Example video details: Each simulated walk film was 3 min 50 s long and each simulated sitting film was 1 minute long. For both videos in this study, the videos started with a short stair climb leading to a path. The path was followed for exactly 1 min 30 s and then the camera stopped and panned left for 45-60 degrees and stopped for 15 seconds. The camera then continued along the walking path until the 3 min 25 s mark, panned left 40-70 degrees for 15 seconds, continued along the path until the 3 min 50 s mark, then faded to black.</li>
	<li>Ensure that the films are comparable, in other words they do not introduce novel gaze data from different walk trajectories or camera movement. One film was taken of each walk, two in total.</li>
	<li>As eye movements are affected by sound, record a file of generic urban park sounds from one of the walks for the same length of the video without interruptions from sudden loud noises outside of the park (e.g., car alarms). This can be played instead of the existing soundtrack from both videos so that the visual impact of the video can be isolated. 
	NOTE: Excessive movement of vehicles and other objects can affect eye-movements and skew the results. If some of this intrusion does occur, the film can be played to the participants but the frames may need to be excluded from the analysis.</li>
	<li>If interested in different activities in a walk, such as walking and then sitting on a bench, save or edit the sitting and walking scenes as separate files so as to test them independently.</li>
</ol>
2. Setup and Desktop Calibration of the Eye-tracking Equipment
<ol>
	<li>Show the filmed sequences to individuals in a room where natural light can be excluded to avoid reflections on the screen. Show it to them on as large a screen as possible to occupy as much of the visual field, thereby avoiding distraction from outside the field of view. A widescreen (16:9) PC monitor (22 inch) with participants approximately 60 cm away from the screen is an acceptable standard.</li>
	<li>Perform a calibration to the screen for each participant using the inbuilt calibration tool on the eye-tracking software. Require participants to look at a red ball moving around the screen as part of this and stopping at 5 key calibration points.</li>
	<li>During the calibration observe the participants and ensure that they are not moving their head as well. If they are too close to the screen and the screen is very large this can be a problem. Adjust the distance from the screen accordingly.</li>
	<li>Edit each film to have a white cross situated in the top left corner of the early frames of the video. This can be done using a video editing software. Show this screen for 3 seconds, then begin to play the walk. This is to check the eye-tracking calibration and to ensure that the eye-tracking data collection times could be matched with individual frame numbers.</li>
</ol>
3. Recruitment and Ethics
<ol>
	<li>Use a professional research recruitment company to ensure a spread of different genders, ages and occupations among the sample or recruit on site from among the student and staff body.</li>
	<li>Screen participants for known ocular or neurological conditions and/or injuries. They must be asked whether they are taking medications that are known to affect eye movements (e.g. benzodiazepines).</li>
	<li>Conduct near vision reading tests and a pen torch eye movement excursion test to demonstrate that they have full eye movement excursions.</li>
	<li>Survey participants about their age, gender and frequency of park use.</li>
</ol>
4. Participant Experimental Setup
<ol>
	<li>Ask participants to imagine themselves in need of restoration. Use a sentence that allows them to imagine the context of the eye-tracking video used such as: &#8216;Imagine that it is midday and you are walking alone in Melbourne. You are mentally tired from intense concentration at work and are looking for somewhere to go for a walk, sit down and rest for a little while, before going back to work&#8217;.</li>
	<li>Sit participants comfortably in front of the computer screen at a distance of 60&#8211;65 cm to view the films.</li>
	<li>Using a spreadsheet program order the films randomly before playing in front of participants.</li>
	<li>Use speakers to play the same audio for all films.</li>
	<li>Record the gaze trajectory of the participants using a desktop eye-tracking system. Mount the desktop eye-tracking device as per the manufacturer&#8217;s instructions. This could be just below the screen on a table, or it could clip to the edge of the screen, on top for example.</li>
	<li>Use an eye-tracking unit that has a sampling rate of 120 Hz and an accuracy of 0.50&#176;, which allows large freedom of head movements enabling recording during natural head movements. A lower frequency eye-tracker is also acceptable. At 57.3 cm, 1&#176; of visual angle equates to ~1 cm on the screen.</li>
	<li>Allow participants to watch the footage on a monitor in high definition.</li>
	<li>Record eye-movements using the eye-tracking software</li>
	<li>Employ a fixation filter to convert the raw eye sample data for analysis. From the main menu, click on Tools &#62; Settings. Select the Fixation filters tab. Select the fixation filter according to the manufacturer&#8217;s specifications on the best type of filter to aggregate the raw eye-tracking data into fixations.</li>
	<li>If recording eye-tracking data using multiple films, give breaks between recording sessions whenever participants request it.</li>
</ol>
5. Connecting the Viewed Parks to Impressions of the Videos
<ol>
	<li>Compare the extent to which people viewed objects and their opinions of the videos by asking participants to rate each of the parks on a scale of 1&#8211;10, firstly for whether they felt they would be able to rest and recover in that environment (1, not very much, to 10, very much) and secondly how much they liked the park (1, not very much, to 10, very much).</li>
	<li>Ask the participants whether they would use the park to relax or relieve stress (Y/N) and whether they recognized the park they were looking at (Y/N) to control for the potential impact of a prior association with the park.</li>
	<li>Record short answer responses from the participants to explain their impressions using a voice recorder and then transcribe these.</li>
</ol>
6. Automatic Video Analysis for Area of Interest Extraction
<ol>
	<li>Selection of AOIs 
	<ol>
		<li>Choose items that are of interest to park designers, urban designers, planners or architects, such as trees, shrubs, signposts, buildings, turf, paths, steps, etc.</li>
		<li>For optimal performance and minimal training requirements (discussed further below), use elements that are easily visually distinguishable from each other to the naked eye, and/or consistently occupy different regions of each video frame. In general, sufficient training examples depicting visually distinguishing differences of each AOI should be sufficient for robust performance.</li>
	</ol>
	</li>
	<li>Training classifiers for AOI extraction
	<ol>
		<li>Selection of the training frames to be used, number and rationale
		<ol>
			<li>Choose an appropriate number of training frames (henceforth referred to as the Training Set). There is no fixed number that is appropriate. 
			NOTE: The frames must provide sufficient coverage of the range of visual appearance of each object class (i.e., those objects to be labelled) throughout the video. For example, 40 frames out of a 15,000 frame video sequence depicting a single parkland scene were found to achieve sufficient accuracy. If a sequence contains significant variation throughout then more frames may be required.</li>
			<li>Consider the frame content when choosing training frames. These include: lighting, shapes with respect to what is being classified (e.g., not just one type of tree but a range of tree types, their position in the image, the way they are lit, etc.), texture, and color.</li>
			<li>Consider also the number of frames to include. Specifically, the length and variation of visual conditions exhibited across the video being analyzed, as well as the number of object classes to identify, and the frequency of their appearance.</li>
		</ol>
		</li>
		<li>Manual pixel labelling for training frames 
		NOTE: Manual labelling the training frames will associate pixels with object classes of interest.
		<ol>
			<li>Pixel labelling of training frames
			<ol>
				<li>In turn, open each training frame from the video in the image editing software.</li>
				<li>For each training frame from the video, overlay a transparent image layer on the loaded image for labelling and create a color palette, providing one color for each given object class of interest (i.e., AOI).</li>
				<li>Ensure that the color palette and the mapping of color to object classes is the same throughout the labelling process.</li>
				<li>Select the color for the sample AOI.</li>
				<li>Color regions of sample AOIs by selecting, with a mouse click and drag, pixels within the area to &#8220;color in&#8221; using the appropriate palette choice. 
				NOTE: It is possible to color large objects imprecisely. However, for narrow/small objects of only a few pixels width, take greater care to ensure that manual labelling accurately captures the object&#8217;s visual appearance.</li>
				<li>Once labelling of a frame is complete, export the overlaid layer as a separate image file. Ensure the base file name matches the original frame base file name, but with a &#8220;c&#8221; appended to the end. For example, if the original frame filename was &#8220;1234.png&#8221;, then the labeled layer file name should be &#8220;1234c.png&#8221;.</li>
				<li>Ensure all labeled images are stored in a single folder.</li>
			</ol>
			</li>
			<li>Pixel labelling of validation frames 
			NOTE: To quantitatively validate the accuracy of the trained classifier, an additional set of labeled frames should be created.
			<ol>
				<li>Select frames from the original video sequence not already chosen to be included in the training set. For a 5 minute video, these need not be more than 20 frames, but should be sampled uniformly from across the video sequence to ensure sufficient coverage.</li>
				<li>Label pixels in each frame using the same procedure as outlined for preparing Training Frames (6.2.2). However, this time be as precise and as comprehensive as possible with labelling as this will be used as a ground truth comparison.</li>
				<li>When labelling of a frame is complete, use the same naming convention as for training, however ensure files are saved in a separate validation frames folder.</li>
			</ol>
			</li>
		</ol>
		</li>
		<li>Automatic pixel labelling of the video sequence
		<ol>
			<li>Download the DARWIN software library from http://drwn.anu.edu.au. 
			NOTE: The system described in this paper was implemented within a Linux environment, and so the Linux release of DARWIN should be downloaded and installed, following the instructions from the website.</li>
			<li>Launch the Darwin GUI</li>
			<li>In the Darwin GUI, click on Load Training Labels.</li>
			<li>Using the file explorer dialogue box that appears, select the folder containing the labeled training images for the relevant video sequence.</li>
			<li>Click on Load Video Frames and follow the same procedure as 6.2.3.2 to select the folder containing all original frames of the video sequence. This should be a folder of images, in which each frame filename is the number of that frame in the video sequence (e.g., frame 1234 would be named 1234.png)</li>
			<li>Click the Train button. The algorithm will now examine each labeled training frame and learn a model of appearance for classifying pixels into any of the specified object classes of interest.</li>
		</ol>
		</li>
		<li>Validating the trained classifier
		<ol>
			<li>Once training is complete, click the Validate Training button.</li>
			<li>Using the file explorer dialogue box, select the folder containing all labeled validation images for the relevant video sequence. The trained classifier will now be used to classify pixels in each of the frames referred to in the validation set. Compare this to the ground truth labels provided in the validation folder. 
			NOTE: Once complete, the GUI will display accuracy statistics, including the percentage of pixels correctly labeled for each frame in the validation set, as well as across the entire validation set.</li>
			<li>To visually validate the generated labels, click the Visual Validation button. If clicked, each generated labeled image is displayed next to the original validation frame. 
			NOTE: This may be instructive in identifying where mis-classifications are occurring on the object. For example, visual inspection may indicate errors are consistently occurring across a particular part of an object class, suggesting improved labelling in the training set is required. Alternatively, inspection may show that errors occur only at a particular time point in the video, suggesting more training examples are required from that time period in the video.</li>
			<li>If the accuracy observed in either quantitative or qualitative validation falls below acceptable levels, then include further training examples. In this case, repeat all steps from 6.2.2 to include additional training frames, re-train the classifier following steps in 6.2.3, and re-validate following steps in 6.2.4.</li>
		</ol>
		</li>
		<li>Video pixel labelling
		<ol>
			<li>Once the classifier training and validation phase is complete, click Label Frames on the Darwin GUI to begin the full labelling of all frames in the video sequence using the trained classifier.</li>
			<li>Follow the prompt to select a destination folder for all output frames, which will be in the form of labeled images using the same color palette as used in training.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
7. Registering the Eye-tracking Data to the Video Content
<ol>
	<li>In the eye-tracking software click on File &#62; Export&#8230; Export the eye-tracking file as a CSV file.</li>
	<li>Open the file in a spreadsheet program.</li>
	<li>From the video sequence identify the time at which the white cross on the top left hand of the frame (see 2.3) disappears. Use this time to identify the eye-tracking by using the relevant column on the eye-tracking result. This is the start of the eye-tracking data collection.</li>
	<li>Add a column to the eye-tracking data.</li>
	<li>Using this new column label each row or eye-tracking data point with a frame number starting at the number 1.</li>
</ol>
8. Displaying the Amount of Time that Participants Examined Different Classes of Objects in the Videos
NOTE: Due to the huge size of the eye-tracking data, Python programming language is better used for steps through to 8.4, although a data processing program can also be used.
<ol>
	<li>Using the included Python code, calculate the amount of overlap between the objects of interest and the eye-tracking fixation time and frequency.</li>
	<li>Sum this data to understand the amount of time that participants spent looking at different objects.</li>
	<li>Employ a histogram to show the total amount of time that the objects occupied the screen.</li>
	<li>Compare this with the total amount of time that participants looked at different objects.</li>
	<li>To produce a heatmap in the eye-tracking software click on one of the park walk videos using the tick box.</li>
	<li>Click the Heat Map tab.</li>
	<li>Adjust the color and other features of the Heat Map as desired using the variables of &#8216;Count&#8217; (total number of fixations made over the time window), &#8216;Absolute duration&#8217; (accumulated fixation duration), and &#8216;Relative duration&#8217; (the amount of time spent looking at an object divided by the amount of time spent looking at a scene).</li>
	<li>Export the image or video that includes the heat map as a JPEG file or video file.</li>
</ol>

<ol>
	<li>Patrik, P., Stigsdotter, U. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+relation+between+perceived+sensory+dimensions+of+urban+green+space+and+stress+restoration.">The relation between perceived sensory dimensions of urban green space and stress restoration.</a> Landscape and Urban Planning. 94 (3-4), 264-275 (2010).</li><li>Bj&#248;rn, G., Patil, G. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Biophilia:+does+visual+contact+with+nature+impact+on+health+and+well-being?.">Biophilia: does visual contact with nature impact on health and well-being?.</a> International Journal of Environmental Research and Public Health. 6 (9), 2332-2343 (2009).</li><li>Velarde, M. a. D., Fry, G., Tveit, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Health+effects+of+viewing+landscapes-Landscape+types+in+environmental+psychology.">Health effects of viewing landscapes-Landscape types in environmental psychology.</a> Urban Forestry &amp; Urban Greening. 6 (4), 199-212 (2007).</li><li>Polat, A. T., Ahmet, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Relationships+between+the+visual+preferences+of+urban+recreation+area+users+and+various+landscape+design+elements.">Relationships between the visual preferences of urban recreation area users and various landscape design elements.</a> Urban Forestry &amp; Urban Greening. 14 (3), 573-582 (2015).</li><li>Peter, P., Giannopoulos, I., Raubal, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Where+am+I?+Investigating+map+matching+during+self-localization+with+mobile+eye+tracking+in+an+urban+environment.">Where am I? Investigating map matching during self-localization with mobile eye tracking in an urban environment.</a> Transactions in GIS. 18 (5), 660-686 (2014).</li><li>Berto, R., Massaccesi, S., Pasini, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+Eye+Movements+Measured+across+High+and+Low+Fascination+Photographs+Differ?+Addressing+Kaplan's+Fascination+Hypothesis.">Do Eye Movements Measured across High and Low Fascination Photographs Differ? Addressing Kaplan's Fascination Hypothesis.</a> Journal of Environmental Psychology. 28 (2), 185-191 (2008).</li><li>Kaplan, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+restorative+benefits+of+nature:+Towards+an+integrative+framework.">The restorative benefits of nature: Towards an integrative framework.</a> Journal of Environmental Psychology. 15, 169-182 (1995).</li><li>Duchowski, A. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Eye Tracking Methodology: Theory and Practice. , (2017).</li><li>Amati, M., Ghanbari Parmehr, E., McCarthy, C., Sita, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+eye-catching+are+natural+features+when+walking+through+a+park?+Eye-+tracking+responses+to+videos+of+walks?.">How eye-catching are natural features when walking through a park? Eye- tracking responses to videos of walks?.</a> Urban Forestry and Urban Greening. 31, 67-78 (2018).</li><li>Gould, S. D. A. R. W. I. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Framework+for+Machine+Learning+and+Computer+Vision+Research+and+Development.">A Framework for Machine Learning and Computer Vision Research and Development.</a> Journal of Machine Learning Research. (Dec), 3533-3537 (2012).</li><li>Richardson, D., Matlock, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+integration+of+figurative+language+and+static+depictions:+an+eye+movement+study+of+fictive+motion.">The integration of figurative language and static depictions: an eye movement study of fictive motion.</a> Cognition. 102 (1), 129-138 (2007).</li><li>Bojko, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Eye Tracking the User Experience: A Practical Guide to Research. , (2013).</li></ol>

Generally, in standard software packages for analyzing eye-tracking data, a vector AOI is used. Even for a single still image, the size of the vector cannot be easily measured. Furthermore, including all AOIs in an image and calculating the relative amounts of AOIs is laborious. It is almost impossible to do this manually on a video without a machine learning technique such as the one described. This was a relatively simple statement that infers a free viewing situation. A much more precise scenario can be used and diffe...

Our daily lived experiences of urban environments greatly impact on our health and well-being. Our well-being can depend on the amount of green spaces that we view and experience1,2,3, and these views can be quantified using eye-tracking equipment to guide decision making about park design. However, an issue arises with the volume of eye tracking data that is generated and making sense of this data. As the equipment for recording gaze data in a lab or natural setting becomes easier to use and more powerful, researchers need to consider how we can collect and analyze data validly to help with decision-making questions.
So far, a great deal of eye tracking research has used photographs in a survey or laboratory setting4. While this methodology allows for a great deal of reproducibility and control over results, it is unable to take advantage of the latest advances in eye-tracking technology which include the use of video and wearable mobile eye-trackers. Furthermore, we would argue that the act of walking and relaxing is necessarily dynamic particularly when oriented towards a task such as wayfinding5. Therefore a fully scientific understanding of these settings should take place outside of the laboratory. However, at the moment, eye-tracking in a real-life naturalistic setting makes comparing the experience between subjects very difficult. For example, if we wanted to compare whether one respondent looks at trees more than another, how could we control for the fact their point of view would be constantly changing compared to others or that their head might have turned. Detailed analysis in these conditions is near impossible with current analysis techniques. We would argue that it is important to control the viewing areas available to the individual being studied and in the analysis to be able to account for the total scene being viewed at any one point in time.
There are a set of theories linking stress levels and perceptions of safety to landscape views and well-evolved measures of stress6,7. There has also been a rapid increase in the sophistication of eye-tracking equipment to measure gaze8. Eye-tracking is important because involuntary eye-movements may be more reliably connected to preference, stress and other traditional measures than surveys and intrusive, physiological tests such as salivary cortisol levels. The objective of this research is to develop tools that enable a more precise measurement of eye-tracking data applied to more naturalistic settings, so as to provide further evidence for or refute long-standing landscape theories that have informed park design for decades.
The aim of this project is to develop and test a novel analysis technique that can generate relevant eye-tracking data for different videos of park walk simulations. Our work reported here and elsewhere9 represents a half-way point between the naturalistic setting of a fully mobile eye-tracking system and the lab-based photo studies referred to above. In particular, we concentrate on using videos as the stimulus material, exploring how this material can be used to test the amount of fascination that different parks generate in the City of Melbourne. Our work is based on the assumption that detailed analysis of videos is a necessary step to breach before undertaking a fuller, more naturalistic assessment of the potential of parks to provide restoration from stress.
In this study, we employed a desktop eye-tracker with videos of walks through urban parks and asked participants to imagine that they were taking a relaxing walk through a park. We describe a method to allow the amount of time that participants spent looking at different objects to be comparable between parks. Desktop studies are generally easier to control compared to mobile ET studies and allow comparative analysis of each subject.
Standard eye-tracking software uses a manual area of interest tool in which an operator can manually draw boundaries around objects of interest in each scene. This enables the amount of time that participants spent looking at different objects to be automatically counted. For video data, this process is labour intensive and subject to operator subjectivity and error. In later versions of eye-tracking analysis software, AOIs can automatically track objects across frames when they are the same size in the video. This is an improvement, however, this is only intended to be used for a small number of stimuli in each image and each image must be checked and confirmed.
Manual labelling of objects in an image is common and supported by image editing software such as GNU Image Manipulation Program (GIMP). Given that 1 s produces 30 frames or images, the manual labelling of videos is impractical. In addition, AOI labelling by drawing vector polygons around the edge of complex objects such as tree canopies is very time consuming. Finally, while it is conceivably possible to calculate the size of objects in a field of view using vector labelling, this feature is not currently available.
The method that we report on below deals with these limitations. This study employed automatic labelling of objects. This is possible using an image processing technique known as semantic labelling, in which every pixel in every frame of the video is assigned a label indicating an object class. Machine learning is used to derive pixel classifiers for each object class of interest. These classifiers provide a probabilistic label for each pixel (known as unary potentials), which are then refined in a subsequent optimization process to achieve the final labelled output. These classifiers learn statistical decision boundaries between each object classes in the space of features extracted from the image, including texture, histogram of edge orientations, RGB color values, and the normalized image coordinates. An appropriate technique for this is implemented in the DARWIN machine learning toolbox10 and is described below.

<table><tbody><tr><td>12 mm lens</td><td>Olympus</td><td></td><td>Lens</td></tr><tr><td>Panasonic GH4&amp;nbsp;</td><td>Panasonic</td><td></td><td>Video Camera</td></tr><tr><td>Tobii Studio version (2.1.14)&amp;nbsp;&amp;nbsp;</td><td>Tobii</td><td></td><td>Software</td></tr><tr><td>Tobii x120 desktop eye-tracker</td><td>Tobii</td><td></td><td>Eye-tracker</td></tr></tbody></table>

combining eye-tracking data with an analysis of video content from free-viewing a video of a walk in an urban park environment

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. Eye-tracking informatics are known to connect to a range of feelings, health conditions, mental states and actions. But because vision is the result of constant eye-movements, teasing out what is important from what is noise is complex and data intensive. Furthermore, a significant challenge is controlling for what people look at compared to what is presented to them.
The following presents a methodology for combining and analyzing eye-tracking on a video of a natural and complex scene with a machine learning technique for analyzing the content of the video. In the protocol we focus on analyzing data from filmed videos, how a video can be best used to record participants&#39; eye-tracking data, and importantly how the content of the video can be analyzed and combined with the eye-tracking data. We present a brief summary of the results and a discussion of the potential of the method for further studies in complex environments.

Figure 1 and Figure 2 show the result of taking all eye-tracking data for the whole video across all participants and producing a heat map; this is the standard approach available in eye-tracking software packages. By comparing Figure 1 and Figure 2 it is possible to identify that on average participants scanned left and right on the x coordinate of the video in ...

Watch this Scientific Journal Video about Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment at JoVE.com

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

The objective of the protocol is to detail how to collect video data for use in the laboratory; how to record eye-tracking data of participants looking at the data and how to efficiently analyze the content of the videos that they were looking at using a machine learning technique.

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. ...

combining-eye-tracking-data-with-an-analysis-video-content-from-free

Research

JoVE Journal

Environment

8.9K Views.  Università degli Studi di Bari. Veel eye tracking studies vertrouwen op complexe video stimuli en real world instellingen maken analyse van de gegevens zeer complex. Deze analysetechniek zorgt voor een veel rijkere en geautomatiseerde benadering van het analyseren van videogebaseerde gegevens dan de momenteel beschikbare methoden die leiden tot een rijkere extractie van complexere gegevens. Deze methode kan worden gebruikt in veel verschillende eye tracking toepassingen met name in de echte wereld situaties of die video geb...

Video: Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

Using Eye Movements to Evaluate the Cognitive Processes Involved in Text Comprehension

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Measuring eye movements during reading is one of the most precise methods for measuring moment-by-moment (online) processing demands during text comprehension. Cognitive processing demands are reflected by several aspects of eye movement behavior, such as fixation duration, number of fixations, and number of regressions (returning to prior parts of a text). Important properties of eye tracking equipment that researchers need to consider are described, including how frequently the eye position is measured (sampling rate), accuracy of determining eye position, how much head movement is allowed, and ease of use. Also described are properties of stimuli that influence eye movements that need to be controlled in studies of text comprehension, such as the position, frequency, and length of target words. Procedural recommendations related to preparing the participant, setting up and calibrating the equipment, and running a study are given. Representative results are presented to illustrate how data can be evaluated. Although the methodology is described in terms of reading comprehension, much of the information presented can be applied to any study in which participants read verbal stimuli.

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Descriptions of eye tracking equipment, how to develop experimental stimuli, and procedural recommendations are included. The information presented can be applied to most any study using verbal stimuli.

The present article describes how to use eye  tracking methodologies to study the cognitive processes involved in text  comprehension. Measuring eye ...

Behavior

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

Eye-tracking to Distinguish Comprehension-based and Oculomotor-based Regressive Eye Movements During Reading

Regressive eye movements are eye movements that move backwards through the text and comprise approximately 10-25% of eye movements during reading. As such, understanding the causes and mechanisms of regressions plays an important role in understanding eye movement behavior. Inhibition of return (IOR) is an oculomotor effect that results in increased latency to return attention to a previously attended target versus a target that was not previously attended. Thus, IOR may affect regressions. This paper describes how to design materials to distinguish between regressions caused by comprehension-related and oculomotor processes; the latter is subject to IOR. The method allows researchers to identify IOR and control the causes of regressions. While the method requires tightly controlled materials and large numbers of participants and materials, it allows researchers to distinguish and control the types of regressions that occur in their reading studies.

The method was designed to investigate the role of inhibition of return (IOR) in regressive eye movements during reading. The focus is on differentiating between regressions triggered as a result of comprehension difficulty versus those triggered from oculomotor error, including the role of IOR in the two types of regressions.

Regressive eye movements are eye movements that move backwards through the text and comprise approximately 10-25% of eye movements during reading. As ...

Gaze in Action: Head-mounted Eye Tracking of Children's Dynamic Visual Attention During Naturalistic Behavior

Young children's visual environments are dynamic, changing moment-by-moment as children physically and visually explore spaces and objects and interact with people around them. Head-mounted eye tracking offers a unique opportunity to capture children's dynamic egocentric views and how they allocate visual attention within those views. This protocol provides guiding principles and practical recommendations for researchers using head-mounted eye trackers in both laboratory and more naturalistic settings. Head-mounted eye tracking complements other experimental methods by enhancing opportunities for data collection in more ecologically valid contexts through increased portability and freedom of head and body movements compared to screen-based eye tracking. This protocol can also be integrated with other technologies, such as motion tracking and heart-rate monitoring, to provide a high-density multimodal dataset for examining natural behavior, learning, and development than previously possible. This paper illustrates the types of data generated from head-mounted eye tracking in a study designed to investigate visual attention in one natural context for toddlers: free-flowing toy play with a parent. Successful use of this protocol will allow researchers to collect data that can be used to answer questions not only about visual attention, but also about a broad range of other perceptual, cognitive, and social skills and their development.

Young children do not passively observe the world, but rather actively explore and engage with their environment. This protocol provides guiding principles and practical recommendations for using head-mounted eye trackers to record infants' and toddlers' dynamic visual environments and visual attention in the context of natural behavior.

Young children's visual environments are dynamic, changing moment-by-moment as children physically and visually explore spaces and objects and interact ...

Comparing Eye-tracking Data of Children with High-functioning ASD, Comorbid ADHD, and of a Control Watching Social Videos

Children with autism spectrum disorders (ASD) are known to have sensory-perceptual processing deficits that weaken their abilities to attend and perceive social stimuli in daily living contexts. Since daily social episodes consist of subtle dynamic changes in social information, any failure to attend to or process subtle human nonverbal cues, such as facial expression, postures, and gestures, might lead to inappropriate social interaction. Traditional behavioral rating scales or assessment tools based on static social scenes have limitations in capturing the moment-to-moment changes in social scenarios. An eye-tracking assessment, which can be administered in a video-based mode, is therefore preferred, to augment clinical observation. In this study, using the single-case comparison design, the eye-tracking data of three participants, a child with autism spectrum disorder (ASD), another with comorbid attention deficit-hyperactive disorder (ADHD), and a neurotypical control, are captured while they view a video of social scenarios. The eye-tracking experiment has helped answer the research question: How does social attention differ between the three participants? By predefining areas of interest (AOIs), their visual attention on relevant or irrelevant social stimuli, how fast each participant attends to the first social stimuli appearing in the videos, for how long each participant continues to attend to those stimuli within the AOIs, and the gaze shifts between multiple social stimuli appearing concurrently in the same social scene are captured, compared, and analyzed in a video-based eye-tracking experiment.

This is a qualitative comparative case study analysis of eye-tracking data on the first moments of social video scenes as viewed by three participants: one with autism spectrum disorder, one with comorbid attention deficit-hyperactive disorder, and one neurotypical control.

Children with autism spectrum disorders (ASD) are known to have sensory-perceptual processing deficits that weaken their abilities to attend and perceive ...

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage video where viewers&#8217; comments are overlaid over visual content, challenges past perspectives. This study proposes an activation and match satisfaction model to study viewing behaviors of lonely people and to challenge previous claims. It presents a protocol to examine the mechanism of how loners use barrage videos by combining eye tracking and self-report measures. Eye tracking documents the audience&#39;s conscious and subconscious watching behaviors in real time and allows for inference of the amount of allocated cognitive resources in response to rational and emotional content. The self-report gauges the amount of satisfaction obtained. Overall, results from the measures supported an activation and match satisfaction model regarding loners and their barrage video viewing behaviors. Implications are discussed.

The study proposes an activation-match model to study how loneliness is mitigated when a lonely audience watches barrage videos of rational and emotional appeals. The protocol uses eye tracking to document duration and fixation, accounting for the degree of satisfaction when emotional needs are appeased by content and barrage.

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage ...

Eye-tracking Technology and Data-mining Techniques used for a Behavioral Analysis of Adults engaged in Learning Processes

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous technological changes and scientific advances, there is a need for life-long learning and education within both formal and non-formal educational environments. In response to this challenge, the use of eye-tracking technology and data-mining techniques, respectively, for supervised (mainly prediction) and unsupervised (specifically cluster analysis) learning, provide methods for the detection of forms of learning among users and/or the classification of their learning styles. In this study, a protocol is proposed for the study of learning styles among adults with and without previous knowledge at different ages (18 to 69-year-old) and at different points throughout the learning process (start and end). Statistical analysis-of-variance techniques mean that differences may be detected between the participants by type of learner and previous knowledge of the task. Likewise, the use of&#160;unsupervised learning clustering techniques throws light on similar forms of learning among the participants across different groups. All these data will facilitate personalized proposals from the teacher for the presentation of each task at different points in the chain of information processing. It will likewise be easier for the teacher to adapt teaching materials to the learning needs of each student or group of students with similar characteristics.

We present a protocol for a behavioral analysis of adults (ages 18 to 70-year-old) engaged in learning processes, undertaking tasks designed for Self-Regulated Learning (SRL). The participants, university teachers and students, and adults from the University of Experience, were monitored with eye-tracking devices and the data were analyzed with data-mining techniques.

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous ...

Eye Tracking During A Complex Aviation Task For Insights Into Information Processing

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill performance. Previous work has shown that traditional and advanced gaze metrics reliably demonstrate robust differences in pilot expertise, cognitive load, fatigue, and even situation awareness (SA).
This study describes the methodology for using a wearable eye tracker and gaze mapping algorithm that captures naturalistic head and eye movements (i.e., gaze) in a high-fidelity flight motionless simulator. The method outlined in this paper describes the area of interest (AOI)-based gaze analyses, which provides more context related to where participants are looking, and dwell time duration, which indicates how efficiently they are processing the fixated information. The protocol illustrates the utility of a wearable eye tracker and computer vision algorithm to assess changes in gaze behavior in response to an unexpected in-flight emergency.
Representative results demonstrated that gaze was significantly impacted when the emergency event was introduced. Specifically, attention allocation, gaze dispersion, and gaze sequence complexity significantly decreased and became highly concentrated on looking outside the front window and at the airspeed gauge during the emergency scenario (all p values &#60; 0.05). The utility and limitations of employing a wearable eye tracker in a high-fidelity motionless flight simulation environment to understand the spatiotemporal characteristics of gaze behavior and its relation to information processing in the aviation domain are discussed.

Eye tracking is a non-invasive method to probe information processing. This article describes how eye tracking can be used to study gaze behavior during a flight simulation emergency task in low-time pilots (i.e., &lt;350 flight hours).

Eye tracking has been used extensively as a proxy to gain insight into the cognitive, perceptual, and sensorimotor processes that underlie skill ...

Characterizing the Relationship Between Eye Movement Parameters and Cognitive Functions in Non-demented Parkinson's Disease Patients with Eye Tracking

Cognitive impairment is a common phenomenon in Parkinson&#8217;s disease that has implications on the prognosis. A simple, noninvasive and objective proxy measurement of cognitive function in Parkinson&#8217;s disease will be helpful in detecting early cognitive decline. As a physiological metric, eye movement parameter is not confounded by the subject&#39;s attributes and intelligence and can function as a proxy marker if it correlates with cognitive functions. To this end, this study explored the relationship between the eye movement parameters and performance in cognitive tests in multiple domains. In the experiment, a visual search task with eye tracking was set up, where subjects were asked to look for a number embedded in an array of alphabets scattered randomly on a computer screen. The differentiation between the number and the alphabet is an overlearned task such that the confounding effect of cognitive ability on the eye movement parameters is minimized. The average saccadic amplitude and fixation duration were captured and calculated during the visual search task. The cognitive assessment battery covered domains of frontal-executive functions, attention, verbal and visual memory. It was found that prolonged fixation duration was associated with poorer performance in verbal fluency, visual and verbal memory, allowing further exploration on the use of eye movement parameters as proxy markers for cognitive function in Parkinson&#8217;s disease patients. The experimental paradigm has been found to be highly tolerable in our group of Parkinson&#39;s disease patients and could be applied transdiagnostically to other disease entities for similar research questions.

Here, we present a protocol to study the relationship between the eye movement parameters and cognitive functions in non-demented Parkinson&#39;s disease patients. The experiment used an eye tracker to measure the saccadic amplitude and fixation duration in a visual search task. The correlation with performance in multi-domain cognitive tasks was subsequently measured.

Cognitive impairment is a common phenomenon in Parkinson&#8217;s disease that has implications on the prognosis. A simple, noninvasive and objective proxy ...

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

Samenvatting

Meer video's verkennen