This research was supported by Science Foundation of Beijing Language and Cultural University under the Fundamental Research Funds for the Central Universities (Approval number 15YJ050003).

All subjects must give informed written consent before the administration of the experimental protocols. All procedures, consent forms, and the experimental protocol were approved by the Research Ethics Committee of the Beijing Language and Culture University.
NOTE: A comprehension study using the visual world paradigm normally consists of the following steps: Introduce the theoretical problems to be explored; Form an experimental design; Prepare the visual and auditory stimuli; Frame the theo...

The author declares that he has no competing financial interests.

To conduct a visual world study, there are several critical steps to follow. First, researchers intend to deduce the interpretation of the auditorily presented language via participants&#39; eye movements in the visual world. Henceforth, in designing the layout of the visual stimuli, the properties of eye movements in a natural task that potentially affect participants&#39; eye movements should be controlled. The effect of the spoken language on participants&#39; eye movements can then be recognized. Second, acoustic cue...

Spoken language is a fast, ongoing information flow, which disappears right away. It is a challenge to experimentally study this temporal, rapidly change speech signal. Eye movements recorded in the visual world paradigm can be used to overcome this challenge. In a typical eye tracking study using the visual world paradigm, participants&#39; eye movements to pictures in a display or to real objects in a visual workspace are monitored as they listen to, or produce, spoken language depicting the contents of the visual world1,2,3,4. The basic logic, or the linking hypothesis, behind this paradigm is that comprehending or planning an utterance will (overtly or covertly) shift participants&#39; visual attention to a certain object in the visual world. This attention shift will have a high probability to initiate a saccadic eye movement to bring the attended area into the foveal vision. With this paradigm, researchers intend to determine at what temporal point, with respect to some acoustic landmark in the speech signal, a shift in the participant&#39;s visual attention occurs, as measured by a saccadic eye movement to an object or a picture in the visual world. When and where saccadic eye movements are launched in relation to the speech signal are then used to deduce the online language processing. The visual world paradigm can be used to study both the spoken language comprehension1,2 and production5,6. This methodological article will focus on comprehension studies. In a comprehension study using the visual world paradigm, participants&#39; eye movements on the visual display are monitored as they listen to the spoken utterances talking about the visual display.
Different eye tracking systems have been designed in history. The simplest, least expensive, and most portable system is just a normal video camera, which records an image of the participant&#39;s eyes. Eye movements are then manually coded through frame-by-frame examination of the video recording. However, the sampling rate of such an eye-tracker is relatively low, and the coding procedure is time consuming. Thus, a contemporary commercial eye tracking system normally uses optical sensors measuring the orientation of the eye in its orbit7,8,9. To understand how a contemporary commercial eye-tracking system works, the following points should be considered. First, to correctly measure the direction of the foveal vision, an infrared illuminator (normally with the wavelength around 780-880 nm) is normally laid along or off the optical axis of the camera, making the image of the pupil distinguishably brighter or darker than the surrounding iris. The image of the pupil and/or of the pupil corneal reflection (normally the first Purkinje image) is then used to calculate the orientation of the eye in its orbit. Second, the gaze location in the visual world is actually contingent not only on the eye orientation with respect to the head but also on the head orientation with respect to the visual world. To accurately infer the gaze of regard from the eye orientation, the light source and the camera of the eye-trackers either are fixed with respect to participants&#39; head (head-mounted eye-trackers) or are fixed with respect to the visual world (table-mounted or remote eye-trackers). Third, the participants&#39; head orientation must either be fixed with respect to the visual world or be computationally compensated if participants&#39; head is free to move. When a remote eye-tracker is used in a head-free-to-move mode, the participants&#39; head position is typically recorded by placing a small sticker on participants&#39; forehead. The head orientation is then computationally subtracted from the eye orientation to retrieve the gaze location in the visual world. Fourth, a calibration and a validation process are then required to map the orientation of the eye to the gaze of regard in the visual world. In the calibration process, participants&#39; fixation samples from known target points are recorded to map the raw eye data to gaze position in the visual world. In the validation process, participants are presented with the same target points as the calibration process. The difference existing between the computed fixation position from the calibrated results and the actual position of the fixated target in the visual world are then used to judge the accuracy of the calibration. To further reconfirm the accuracy of the mapping process, a drift check is normally applied on each trial, where a single fixation target is presented to participants to measure the difference between the computed fixation position&#160;and the actual position of the current target.
The primary data of a visual world study is a stream of gaze locations in the visual world recorded at the sampling rate of the eye-tracker, ranging over the whole or part of the trial duration. The dependent variable used in a visual world study is typically the proportion of samples that participants&#39; fixations are situated at certain spatial region in the visual world across a certain time window. To analyze the data, a time window has firstly to be selected, often referred to as periods of interest. The time window is typically time-locked to the presentation of some linguistic events in the auditory input. Furthermore, the visual world is also needed to split into several regions of interest (ROIs), each of which is associated with one or more objects. One such region contains the object corresponding to the correct comprehension of the spoken language, and thus&#160;is often called the target area. A typical way to visualize the data is a proportion-of-fixation plot, where at each bin in a time window, the proportion of samples with a look to each region of interest are averaged across participants and items.
Using the data obtained from a visual world study, different research questions can be answered: a) On the coarse-grain level, are participants&#39; eye movements in the visual world affected by different auditory linguistic input?&#160;b) If there is an effect, what is the trajectory of the effect over the course of the trial? Is it a linear effect or high-order effect?&#160;and c) If there is an effect, then on the fine-grain level, when is the earliest temporal point where such an effect emerges and how long does this effect last?
To statistically analyze the results, the following points should be considered. First, the response variable, i.e., proportions of fixations, is both below and above bounded (between 0 and 1), which will follow a multinomial distribution rather than a normal distribution. Henceforth, traditional statistical methods based on normal distribution such as t-test, ANOVA, and linear (mixed-effect) models10, cannot be directly utilized until the proportions have been transformed to unbounded variables such as with empirical logit formula11 or have been replaced with unbounded dependent variables such as Euclidean distance12. Statistical techniques that do not require the assumption of normal distribution such generalized linear (mixed-effect) models13 can also be used. Second, to explore the changing trajectory of the observed effect, a variable denoting the time-series has to be added into the model. This time-series variable is originally the eye-tracker&#8217;s sampling points realigned to the onset of the language input. Since the changing trajectory typically is not linear, a high-order polynomial function of the time-series is normally added into the (generalized) linear (mixed-effect) model, i.e., growth curve analyses14. Furthermore, participants&#8217; eye positions in the current sampling point is highly dependent on previous sampling point(s), especially when the recording frequency is high, resulting in the problem of autocorrelation. To reduce the autocorrelation between the adjacent sampling points, original data are often down-sampled or binned. In recent years, the generalized additive mixed effect models (GAMM) have also been used to tackle the autocorrelated errors12,15,16. The width of bins varies among different studies, ranging from several milliseconds to several hundred milliseconds. The narrowest bin a study can choose is restricted by the sampling rate of the eye tracker used in the specific study. For example, if an eye tracker has a sampling rate of 500 Hz, then the width of the time window cannot be smaller than 2 ms = 1000/500. Third, when a statistical analysis is repeatedly applied to each time bin of the periods of interest, the familywise error induced from these multiple comparisons should be tackled. As we described earlier, the trajectory analysis informs the researcher whether the effect observed on the coarse-grain level is linear with respect to the changing of the time, but does not show when the observed effect begins to emerge and how long the observed effect lasts. To determine the temporal position when the observed difference starts to diverge, and to figure out the duration of the temporal period that the observed effect lasts, a statistic analysis has to be repeatedly applied to each time bin. These multiple comparisons will introduce the so-called familywise error, no matter what statistical method is used. The familywise error is traditionally corrected with Bonferroni adjustment17. Recently, a method called nonparametric permutation test originally used in neuroimaging field18 has been applied to the visual word paradigm19 to control for the familywise error.
Researchers using the visual world paradigm intend to infer the comprehension of some spoken language from participants&#8217; eye movements in the visual world. To ensure the validity of this deduction, other factors that possibly influence the eye movements should be either ruled out or controlled. The following two factors are among the common ones that need to be considered. The first factor involves some systematic patterns in participants&#8217; explanatory fixations independent of the language input, such as the tendency to fixate on the top left quadrat of the visual world, and moving eyes in the horizontal direction being easier than in the vertical direction, etc.12,20 To make sure that the observed fixation patterns are related to the objects, not to the spatial locations where the objects are situated, the spatial positions of an object should be counterbalanced across different trials or across different participants. The second factor that might affect participants&#8217; eye movements is the basic image features of the objects in the visual world, such as luminance contrast, color and edge orientation, among others21. To diagnose this potential confounding, the visual display is normally presented prior to the onset of the spoken language or prior to the onset of the critical acoustic marker of the spoken language, for about 1000 ms. During the temporal period from the onset of the test image to the onset of the test audio, the language input or the disambiguation point of the language input has not been heard yet. Any difference observed between different conditions should be deduced to other confounding factors such as the visual display per se, rather than the language input. Henceforth, eye movements observed in this preview period provide a baseline for determining the effect of the linguistic input. This preview period also allows participants to get familiarized with the visual display, and to reduce the systematic bias of the explanatory fixations when the spoken language is presented.
To illustrate how a typical eye tracking study using the visual world paradigm is conducted, the following protocol describes an experiment adapted from L. Zhan&#160;17&#160;to explore the online processing of semantically complex statements, i.e., disjunctive statements (S1 or S2), conjunctive statements (S1 and S2), and but-statements (S1 but not-S2). In ordinary conservation, the information expressed by some utterances is actually stronger than its literal meaning. Disjunctive statements like Xiaoming&#39;s box contains a cow or a rooster are such utterances. Logically, the disjunctive statement is true as long as the two disjuncts Xiaoming&#39;s box contains a cow and Xiaoming&#39;s box contains a rooster are not both false. Therefore, the disjunctive statement is true when the two disjuncts are both true, where the corresponding conjunctive statement Xiaoming&#39;s box contains a cow and a rooster is also true. In ordinary conversation, however, hearing the disjunctive statement often suggests that the corresponding conjunctive statement is false (scalar implicature); and suggests that the truth values of the two disjuncts are unknown by the speaker (ignorance inference). Accounts in the literature differ in whether two inferences are grammatical or pragmatic processes22,23,24,25,26. The experiment shows how the visual world paradigm can be used to adjudicate between these accounts, by exploring the online processing of three complex statements.

<table border="0" cellpadding="0" cellspacing="0" style="width:1101px;" width="1101">
	<tbody>
		<tr height="42">
			<td height="42" style="height:42px;width:216px;">Pixelmator</td>
			<td style="width:129px;">Pixelmator Team</td>
			<td style="width:119px;">http://www.pixelmator.com/pro/</td>
			<td style="width:637px;">image editing app</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;">Praat</td>
			<td style="width:129px;">Open Sourse</td>
			<td style="width:119px;">http://www.fon.hum.uva.nl/praat/</td>
			<td style="width:637px;">Sound analyses and editting software</td>
		</tr>
		<tr height="84">
			<td height="84" style="height:84px;width:216px;">Eyelink 1000plus</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/products/eyelink-1000-plus/</td>
			<td style="width:637px;">remote infrared eye tracker&#160;</td>
		</tr>
		<tr height="84">
			<td height="84" style="height:84px;width:216px;">Experimental Builder</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/experiment-builder/</td>
			<td style="width:637px;">eye tracker software&#160;</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;width:216px;">Data Viewer</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/data-viewer/</td>
			<td style="width:637px;">eye tracker software&#160;</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:216px;">R</td>
			<td style="width:129px;">Open Sourse</td>
			<td style="width:119px;">https://www.r-project.org</td>
			<td style="width:637px;">free software environment for statistical computing and graphics</td>
		</tr>
	</tbody>
</table>

using eye movements recorded in the visual world paradigm to explore the online processing of spoken language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

Participants&#39; behavioral responses are summarized in Figure 4. As we described earlier, the correct response to a conjunctive statement (S1 and S2) is the big open box, such as Box A in Figure 1. The correct response to a but-statement (S1 but not S2) is the small open box containing the first mentioned animal, such as Box D in Figure 1. Critically, which box is chosen ...

Watch this Scientific Journal Video about Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language at JoVE.com

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

This method can help answer key questions in online processing of spoken language, including semantically complex sentences, such as disjunctions and conditionals. The main advantage of this technique is that it can be performed on a wide range of populations to study most topics in areas of language processing. Demonstrating the procedure will be Xu Jing, a post graduate student.Cao Yifan, Wang Meng, Yang Xiaotong, Zhan Min, and Zhou Xuehan are the students from my research group. To prepare the visual stimuli, download 60 copyright free clip art images of animals from the Internet and open the images in an appropriate image editor. Click Tools and use quick selection to select and delete the background from each image.Click Image and Image Size to resize the images to 120 by 120 pixels and draw one big open 320 x 240 pixels box, one small closed 160 x 160 pixels box, and two small, open 160 x 240 pixels boxes. Open a new file to create a 1, 024 by 768 pixels template for the first test image, and drag the boxes to the indicated locations. Drag two clip art images into the big open box and one each of the same two images into the two small open boxes.Then create 59 more test images as just demonstrated with each animal image being used twice per test image, counterbalancing the spatial locations of the four boxes among the images. To prepare the spoken language stimuli, design four test sentences corresponding to each test image in the native language of the participants for a total of 240 test sentences to be recorded, with at least three of the four sentences and one of each of the forms as illustrated. Recruit a female native speaker to record four example statements, as well as audio for all of the animals being used in the experiment.Divide the 240 test sentences into four groups, with each group containing 15 conjunctive statements, 15 disjunctive statements, 15 but statements, and 15 filler sentences. Then save all of the important information regarding the test stimuli into a tab delimited text file, with each row corresponding to each of the 240 trials. To build the experiment sequence, open the experiment builder and create a new project.Drag a display screen object into the workspace, and rename the workspace Instruction. Click Insert Multi Line Text Resource and double click to open the block sequence to build the block sequence. Drag an EI Camera Setup node into the block sequence to bring up a camera setup screen on EyeLink host PC to facilitate the camera setup calibration and validation.Click the Calibration Type field in the properties panel and select HV5 from the drop down list. To build the trial sequence drag a Display Screen node into the trial sequence and rename it Animal_1_Image. Drag a Play Sound node into the trial sequence and rename it Animal_1_Audio.Drag a Timer node into the trial sequence and rename it Animal_1_Audio_Length. Continue dragging and renaming additional Display Screen, Play Sound, and Timer nodes into the trial sequence until three nodes have been created for each clip art image. Then drag a Drift Correct node into the trial sequence to introduce the drift correction.To build the recording sequence, drag a new Display Screen node into the record sequence. Rename the node Test_Image. Double click the Display Screen node to open the screen builder, click the Insert Rectangle Interest Area Region, and draw four rectangular areas of interest.Drag a Timer node into the workspace, rename the node Pause, and change the duration property to 500 milliseconds. Drag a Play Sound node into the workspace and rename the node Test_Audio, and drag a Timer node into the workspace and rename it Test_Audio_Length. Then add a Keyboard node into the workspace, rename the node Behavioral Responses, and change the Acceptable Keys property to Up, Down, Right, Left.To conduct an experiment, boot the system on the host PC to start the host application of the camera, and click the executable version of the experimental project on the display PC.Input the participant's name and click Select Condition Value to run to select a group from the prompt window. Place a small target sticker on the participant's forehead to track the head position even when the pupil image is lost such as during blinks or sudden movements. Seat the participant approximately 60 centimeters from a 21-inch 4:3 color monitor with a 1, 024 by 769 pixel resolution, for which 27 pixels equals one degree angle.Adjust the height of the display PC monitor to ensure that when the participant is looking straight ahead, they are looking vertically at the middle to top 75%of the monitor, then rotate the focusing arm on the desk mount to bring the eye image into focus. Next, click Calibrate on the host PC and ask the participant to fixate on a grid of five fixation targets, in random succession with no overt behavioral responses, to map the participant's eye movements to the gaze of regard in the visual world. Click Validate and ask the participant to fixate on the same grid of fixation targets to validate the calibration results.Perform a drift check by asking the participant to press the space bar on the keyboard while fixating on the black dot in the center of the screen. Then present the visual stimuli via the display PC monitor while playing the auditory stimuli via a pair of external speakers situated to the left and right of the monitor. For data coding and analysis open the data viewer and click File, Import File, and import Multiple EyeLink Data Files to import all of the recorded eye tracker files.Save the files in a single evs file, then open the evs file and click Analysis, Reports, and Sample Report to export the raw sample data with no aggregation. The correct response to a conjunctive statement is the big open box, while the correct response to a but statement is the small open box containing the first mentioned animal within the statement. Critically, which box is chosen for the conjunctive statements depends on how a conditional statement is processed.For example, the small closed box is chosen only when the scalar implicature and the ignorance inferences are both computed, which results from the comprehension of a disjunctive statement. As illustrated in the panel, eye fixation on the small closed box doesn't increase unless the sentential connective is the disjunctive connective or with the increase beginning no later than the offset of the disjunctive connective. The Visual World Paradigm is a versatile eye tracking technique for inferring participant's online comprehension of spoken language from their eye movements in the real world.When recruiting participants remember they should have normal or correct normal visual ability and a normal hearing ability. To design an eligible real world study, other factors that might affect participants'eye movements should be controlled or ruled out. When statistically analyzing the results, problem that may arise from bounded responses, all auto correlation or automatic comparisons should be considered.

using-eye-movements-recorded-visual-world-paradigm-to-explore-online

Research

JoVE Journal

Behavior

9.7K Views.  Beijing Language and Culture University. This method can help answer key questions in online processing of spoken language, including semantically complex sentences, such as disjunctions and conditionals. The main advantage of this technique is that it can be performed on a wide range of populations to study most topics in areas of language processing. Demonstrating the procedure will be Xu Jing, a post graduate student.Cao Yifan, Wang Meng, Yang Xiaotong, Zhan Min, and Zhou Xuehan are the students from my research group. To ...