Name: using eye movements recorded in the visual world paradigm to explore the online processing of spoken language
Uploaded: 2018-10-13T12:30:00
Description: Watch this Scientific Journal Video about Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language at JoVE.com

This research was supported by Science Foundation of Beijing Language and Cultural University under the Fundamental Research Funds for the Central Universities (Approval number 15YJ050003).

All subjects must give informed written consent before the administration of the experimental protocols. All procedures, consent forms, and the experimental protocol were approved by the Research Ethics Committee of the Beijing Language and Culture University.
NOTE: A comprehension study using the visual world paradigm normally consists of the following steps: Introduce the theoretical problems to be explored; Form an experimental design; Prepare the visual and auditory stimuli; Frame the theo...

To conduct a visual world study, there are several critical steps to follow. First, researchers intend to deduce the interpretation of the auditorily presented language via participants&#39; eye movements in the visual world. Henceforth, in designing the layout of the visual stimuli, the properties of eye movements in a natural task that potentially affect participants&#39; eye movements should be controlled. The effect of the spoken language on participants&#39; eye movements can then be recognized. Second, acoustic cue...

Spoken language is a fast, ongoing information flow, which disappears right away. It is a challenge to experimentally study this temporal, rapidly change speech signal. Eye movements recorded in the visual world paradigm can be used to overcome this challenge. In a typical eye tracking study using the visual world paradigm, participants&#39; eye movements to pictures in a display or to real objects in a visual workspace are monitored as they listen to, or produce, spoken language depicting the contents of the visual world1,2,3,4. The basic logic, or the linking hypothesis, behind this paradigm is that comprehending or planning an utterance will (overtly or covertly) shift participants&#39; visual attention to a certain object in the visual world. This attention shift will have a high probability to initiate a saccadic eye movement to bring the attended area into the foveal vision. With this paradigm, researchers intend to determine at what temporal point, with respect to some acoustic landmark in the speech signal, a shift in the participant&#39;s visual attention occurs, as measured by a saccadic eye movement to an object or a picture in the visual world. When and where saccadic eye movements are launched in relation to the speech signal are then used to deduce the online language processing. The visual world paradigm can be used to study both the spoken language comprehension1,2 and production5,6. This methodological article will focus on comprehension studies. In a comprehension study using the visual world paradigm, participants&#39; eye movements on the visual display are monitored as they listen to the spoken utterances talking about the visual display.
Different eye tracking systems have been designed in history. The simplest, least expensive, and most portable system is just a normal video camera, which records an image of the participant&#39;s eyes. Eye movements are then manually coded through frame-by-frame examination of the video recording. However, the sampling rate of such an eye-tracker is relatively low, and the coding procedure is time consuming. Thus, a contemporary commercial eye tracking system normally uses optical sensors measuring the orientation of the eye in its orbit7,8,9. To understand how a contemporary commercial eye-tracking system works, the following points should be considered. First, to correctly measure the direction of the foveal vision, an infrared illuminator (normally with the wavelength around 780-880 nm) is normally laid along or off the optical axis of the camera, making the image of the pupil distinguishably brighter or darker than the surrounding iris. The image of the pupil and/or of the pupil corneal reflection (normally the first Purkinje image) is then used to calculate the orientation of the eye in its orbit. Second, the gaze location in the visual world is actually contingent not only on the eye orientation with respect to the head but also on the head orientation with respect to the visual world. To accurately infer the gaze of regard from the eye orientation, the light source and the camera of the eye-trackers either are fixed with respect to participants&#39; head (head-mounted eye-trackers) or are fixed with respect to the visual world (table-mounted or remote eye-trackers). Third, the participants&#39; head orientation must either be fixed with respect to the visual world or be computationally compensated if participants&#39; head is free to move. When a remote eye-tracker is used in a head-free-to-move mode, the participants&#39; head position is typically recorded by placing a small sticker on participants&#39; forehead. The head orientation is then computationally subtracted from the eye orientation to retrieve the gaze location in the visual world. Fourth, a calibration and a validation process are then required to map the orientation of the eye to the gaze of regard in the visual world. In the calibration process, participants&#39; fixation samples from known target points are recorded to map the raw eye data to gaze position in the visual world. In the validation process, participants are presented with the same target points as the calibration process. The difference existing between the computed fixation position from the calibrated results and the actual position of the fixated target in the visual world are then used to judge the accuracy of the calibration. To further reconfirm the accuracy of the mapping process, a drift check is normally applied on each trial, where a single fixation target is presented to participants to measure the difference between the computed fixation position&#160;and the actual position of the current target.
The primary data of a visual world study is a stream of gaze locations in the visual world recorded at the sampling rate of the eye-tracker, ranging over the whole or part of the trial duration. The dependent variable used in a visual world study is typically the proportion of samples that participants&#39; fixations are situated at certain spatial region in the visual world across a certain time window. To analyze the data, a time window has firstly to be selected, often referred to as periods of interest. The time window is typically time-locked to the presentation of some linguistic events in the auditory input. Furthermore, the visual world is also needed to split into several regions of interest (ROIs), each of which is associated with one or more objects. One such region contains the object corresponding to the correct comprehension of the spoken language, and thus&#160;is often called the target area. A typical way to visualize the data is a proportion-of-fixation plot, where at each bin in a time window, the proportion of samples with a look to each region of interest are averaged across participants and items.
Using the data obtained from a visual world study, different research questions can be answered: a) On the coarse-grain level, are participants&#39; eye movements in the visual world affected by different auditory linguistic input?&#160;b) If there is an effect, what is the trajectory of the effect over the course of the trial? Is it a linear effect or high-order effect?&#160;and c) If there is an effect, then on the fine-grain level, when is the earliest temporal point where such an effect emerges and how long does this effect last?
To statistically analyze the results, the following points should be considered. First, the response variable, i.e., proportions of fixations, is both below and above bounded (between 0 and 1), which will follow a multinomial distribution rather than a normal distribution. Henceforth, traditional statistical methods based on normal distribution such as t-test, ANOVA, and linear (mixed-effect) models10, cannot be directly utilized until the proportions have been transformed to unbounded variables such as with empirical logit formula11 or have been replaced with unbounded dependent variables such as Euclidean distance12. Statistical techniques that do not require the assumption of normal distribution such generalized linear (mixed-effect) models13 can also be used. Second, to explore the changing trajectory of the observed effect, a variable denoting the time-series has to be added into the model. This time-series variable is originally the eye-tracker&#8217;s sampling points realigned to the onset of the language input. Since the changing trajectory typically is not linear, a high-order polynomial function of the time-series is normally added into the (generalized) linear (mixed-effect) model, i.e., growth curve analyses14. Furthermore, participants&#8217; eye positions in the current sampling point is highly dependent on previous sampling point(s), especially when the recording frequency is high, resulting in the problem of autocorrelation. To reduce the autocorrelation between the adjacent sampling points, original data are often down-sampled or binned. In recent years, the generalized additive mixed effect models (GAMM) have also been used to tackle the autocorrelated errors12,15,16. The width of bins varies among different studies, ranging from several milliseconds to several hundred milliseconds. The narrowest bin a study can choose is restricted by the sampling rate of the eye tracker used in the specific study. For example, if an eye tracker has a sampling rate of 500 Hz, then the width of the time window cannot be smaller than 2 ms = 1000/500. Third, when a statistical analysis is repeatedly applied to each time bin of the periods of interest, the familywise error induced from these multiple comparisons should be tackled. As we described earlier, the trajectory analysis informs the researcher whether the effect observed on the coarse-grain level is linear with respect to the changing of the time, but does not show when the observed effect begins to emerge and how long the observed effect lasts. To determine the temporal position when the observed difference starts to diverge, and to figure out the duration of the temporal period that the observed effect lasts, a statistic analysis has to be repeatedly applied to each time bin. These multiple comparisons will introduce the so-called familywise error, no matter what statistical method is used. The familywise error is traditionally corrected with Bonferroni adjustment17. Recently, a method called nonparametric permutation test originally used in neuroimaging field18 has been applied to the visual word paradigm19 to control for the familywise error.
Researchers using the visual world paradigm intend to infer the comprehension of some spoken language from participants&#8217; eye movements in the visual world. To ensure the validity of this deduction, other factors that possibly influence the eye movements should be either ruled out or controlled. The following two factors are among the common ones that need to be considered. The first factor involves some systematic patterns in participants&#8217; explanatory fixations independent of the language input, such as the tendency to fixate on the top left quadrat of the visual world, and moving eyes in the horizontal direction being easier than in the vertical direction, etc.12,20 To make sure that the observed fixation patterns are related to the objects, not to the spatial locations where the objects are situated, the spatial positions of an object should be counterbalanced across different trials or across different participants. The second factor that might affect participants&#8217; eye movements is the basic image features of the objects in the visual world, such as luminance contrast, color and edge orientation, among others21. To diagnose this potential confounding, the visual display is normally presented prior to the onset of the spoken language or prior to the onset of the critical acoustic marker of the spoken language, for about 1000 ms. During the temporal period from the onset of the test image to the onset of the test audio, the language input or the disambiguation point of the language input has not been heard yet. Any difference observed between different conditions should be deduced to other confounding factors such as the visual display per se, rather than the language input. Henceforth, eye movements observed in this preview period provide a baseline for determining the effect of the linguistic input. This preview period also allows participants to get familiarized with the visual display, and to reduce the systematic bias of the explanatory fixations when the spoken language is presented.
To illustrate how a typical eye tracking study using the visual world paradigm is conducted, the following protocol describes an experiment adapted from L. Zhan&#160;17&#160;to explore the online processing of semantically complex statements, i.e., disjunctive statements (S1 or S2), conjunctive statements (S1 and S2), and but-statements (S1 but not-S2). In ordinary conservation, the information expressed by some utterances is actually stronger than its literal meaning. Disjunctive statements like Xiaoming&#39;s box contains a cow or a rooster are such utterances. Logically, the disjunctive statement is true as long as the two disjuncts Xiaoming&#39;s box contains a cow and Xiaoming&#39;s box contains a rooster are not both false. Therefore, the disjunctive statement is true when the two disjuncts are both true, where the corresponding conjunctive statement Xiaoming&#39;s box contains a cow and a rooster is also true. In ordinary conversation, however, hearing the disjunctive statement often suggests that the corresponding conjunctive statement is false (scalar implicature); and suggests that the truth values of the two disjuncts are unknown by the speaker (ignorance inference). Accounts in the literature differ in whether two inferences are grammatical or pragmatic processes22,23,24,25,26. The experiment shows how the visual world paradigm can be used to adjudicate between these accounts, by exploring the online processing of three complex statements.

<table border="0" cellpadding="0" cellspacing="0" style="width:1101px;" width="1101">
	<tbody>
		<tr height="42">
			<td height="42" style="height:42px;width:216px;">Pixelmator</td>
			<td style="width:129px;">Pixelmator Team</td>
			<td style="width:119px;">http://www.pixelmator.com/pro/</td>
			<td style="width:637px;">image editing app</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;">Praat</td>
			<td style="width:129px;">Open Sourse</td>
			<td style="width:119px;">http://www.fon.hum.uva.nl/praat/</td>
			<td style="width:637px;">Sound analyses and editting software</td>
		</tr>
		<tr height="84">
			<td height="84" style="height:84px;width:216px;">Eyelink 1000plus</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/products/eyelink-1000-plus/</td>
			<td style="width:637px;">remote infrared eye tracker&#160;</td>
		</tr>
		<tr height="84">
			<td height="84" style="height:84px;width:216px;">Experimental Builder</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/experiment-builder/</td>
			<td style="width:637px;">eye tracker software&#160;</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;width:216px;">Data Viewer</td>
			<td style="width:129px;">SR-Research, Inc</td>
			<td style="width:119px;">https://www.sr-research.com/data-viewer/</td>
			<td style="width:637px;">eye tracker software&#160;</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:216px;">R</td>
			<td style="width:129px;">Open Sourse</td>
			<td style="width:119px;">https://www.r-project.org</td>
			<td style="width:637px;">free software environment for statistical computing and graphics</td>
		</tr>
	</tbody>
</table>

using eye movements recorded in the visual world paradigm to explore the online processing of spoken language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

Participants&#39; behavioral responses are summarized in Figure 4. As we described earlier, the correct response to a conjunctive statement (S1 and S2) is the big open box, such as Box A in Figure 1. The correct response to a but-statement (S1 but not S2) is the small open box containing the first mentioned animal, such as Box D in Figure 1. Critically, which box is chosen ...

Watch this Scientific Journal Video about Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language at JoVE.com

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

This method can help answer key questions in online processing of spoken language, including semantically complex sentences, such as disjunctions and conditionals. The main advantage of this technique is that it can be performed on a wide range of populations to study most topics in areas of language processing. Demonstrating the procedure will be Xu Jing, a post graduate student.Cao Yifan, Wang Meng, Yang Xiaotong, Zhan Min, and Zhou Xuehan are the students from my research group. To prepare the visual stimuli, download 60 copyright free clip art images of animals from the Internet and open the images in an appropriate image editor. Click Tools and use quick selection to select and delete the background from each image.Click Image and Image Size to resize the images to 120 by 120 pixels and draw one big open 320 x 240 pixels box, one small closed 160 x 160 pixels box, and two small, open 160 x 240 pixels boxes. Open a new file to create a 1, 024 by 768 pixels template for the first test image, and drag the boxes to the indicated locations. Drag two clip art images into the big open box and one each of the same two images into the two small open boxes.Then create 59 more test images as just demonstrated with each animal image being used twice per test image, counterbalancing the spatial locations of the four boxes among the images. To prepare the spoken language stimuli, design four test sentences corresponding to each test image in the native language of the participants for a total of 240 test sentences to be recorded, with at least three of the four sentences and one of each of the forms as illustrated. Recruit a female native speaker to record four example statements, as well as audio for all of the animals being used in the experiment.Divide the 240 test sentences into four groups, with each group containing 15 conjunctive statements, 15 disjunctive statements, 15 but statements, and 15 filler sentences. Then save all of the important information regarding the test stimuli into a tab delimited text file, with each row corresponding to each of the 240 trials. To build the experiment sequence, open the experiment builder and create a new project.Drag a display screen object into the workspace, and rename the workspace Instruction. Click Insert Multi Line Text Resource and double click to open the block sequence to build the block sequence. Drag an EI Camera Setup node into the block sequence to bring up a camera setup screen on EyeLink host PC to facilitate the camera setup calibration and validation.Click the Calibration Type field in the properties panel and select HV5 from the drop down list. To build the trial sequence drag a Display Screen node into the trial sequence and rename it Animal_1_Image. Drag a Play Sound node into the trial sequence and rename it Animal_1_Audio.Drag a Timer node into the trial sequence and rename it Animal_1_Audio_Length. Continue dragging and renaming additional Display Screen, Play Sound, and Timer nodes into the trial sequence until three nodes have been created for each clip art image. Then drag a Drift Correct node into the trial sequence to introduce the drift correction.To build the recording sequence, drag a new Display Screen node into the record sequence. Rename the node Test_Image. Double click the Display Screen node to open the screen builder, click the Insert Rectangle Interest Area Region, and draw four rectangular areas of interest.Drag a Timer node into the workspace, rename the node Pause, and change the duration property to 500 milliseconds. Drag a Play Sound node into the workspace and rename the node Test_Audio, and drag a Timer node into the workspace and rename it Test_Audio_Length. Then add a Keyboard node into the workspace, rename the node Behavioral Responses, and change the Acceptable Keys property to Up, Down, Right, Left.To conduct an experiment, boot the system on the host PC to start the host application of the camera, and click the executable version of the experimental project on the display PC.Input the participant's name and click Select Condition Value to run to select a group from the prompt window. Place a small target sticker on the participant's forehead to track the head position even when the pupil image is lost such as during blinks or sudden movements. Seat the participant approximately 60 centimeters from a 21-inch 4:3 color monitor with a 1, 024 by 769 pixel resolution, for which 27 pixels equals one degree angle.Adjust the height of the display PC monitor to ensure that when the participant is looking straight ahead, they are looking vertically at the middle to top 75%of the monitor, then rotate the focusing arm on the desk mount to bring the eye image into focus. Next, click Calibrate on the host PC and ask the participant to fixate on a grid of five fixation targets, in random succession with no overt behavioral responses, to map the participant's eye movements to the gaze of regard in the visual world. Click Validate and ask the participant to fixate on the same grid of fixation targets to validate the calibration results.Perform a drift check by asking the participant to press the space bar on the keyboard while fixating on the black dot in the center of the screen. Then present the visual stimuli via the display PC monitor while playing the auditory stimuli via a pair of external speakers situated to the left and right of the monitor. For data coding and analysis open the data viewer and click File, Import File, and import Multiple EyeLink Data Files to import all of the recorded eye tracker files.Save the files in a single evs file, then open the evs file and click Analysis, Reports, and Sample Report to export the raw sample data with no aggregation. The correct response to a conjunctive statement is the big open box, while the correct response to a but statement is the small open box containing the first mentioned animal within the statement. Critically, which box is chosen for the conjunctive statements depends on how a conditional statement is processed.For example, the small closed box is chosen only when the scalar implicature and the ignorance inferences are both computed, which results from the comprehension of a disjunctive statement. As illustrated in the panel, eye fixation on the small closed box doesn't increase unless the sentential connective is the disjunctive connective or with the increase beginning no later than the offset of the disjunctive connective. The Visual World Paradigm is a versatile eye tracking technique for inferring participant's online comprehension of spoken language from their eye movements in the real world.When recruiting participants remember they should have normal or correct normal visual ability and a normal hearing ability. To design an eligible real world study, other factors that might affect participants'eye movements should be controlled or ruled out. When statistically analyzing the results, problem that may arise from bounded responses, all auto correlation or automatic comparisons should be considered.

using-eye-movements-recorded-visual-world-paradigm-to-explore-online

Research

JoVE Journal

Behavior

Using Eye Movements to Evaluate the Cognitive Processes Involved in Text Comprehension

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Measuring eye movements during reading is one of the most precise methods for measuring moment-by-moment (online) processing demands during text comprehension. Cognitive processing demands are reflected by several aspects of eye movement behavior, such as fixation duration, number of fixations, and number of regressions (returning to prior parts of a text). Important properties of eye tracking equipment that researchers need to consider are described, including how frequently the eye position is measured (sampling rate), accuracy of determining eye position, how much head movement is allowed, and ease of use. Also described are properties of stimuli that influence eye movements that need to be controlled in studies of text comprehension, such as the position, frequency, and length of target words. Procedural recommendations related to preparing the participant, setting up and calibrating the equipment, and running a study are given. Representative results are presented to illustrate how data can be evaluated. Although the methodology is described in terms of reading comprehension, much of the information presented can be applied to any study in which participants read verbal stimuli.

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Descriptions of eye tracking equipment, how to develop experimental stimuli, and procedural recommendations are included. The information presented can be applied to most any study using verbal stimuli.

The present article describes how to use eye  tracking methodologies to study the cognitive processes involved in text  comprehension. Measuring eye ...

Methods to Explore the Influence of Top-down Visual Processes on Motor Behavior

Kinesthetic awareness is important to successfully navigate the environment. When we interact with our daily surroundings, some aspects of movement are deliberately planned, while others spontaneously occur below conscious awareness. The deliberate component of this dichotomy has been studied extensively in several contexts, while the spontaneous component remains largely under-explored. Moreover, how perceptual processes modulate these movement classes is still unclear. In particular, a currently debated issue is whether the visuomotor system is governed by the spatial percept produced by a visual illusion or whether it is not affected by the illusion and is governed instead by the veridical percept. Bistable percepts such as 3D depth inversion illusions (DIIs) provide an excellent context to study such interactions and balance, particularly when used in combination with reach-to-grasp movements. In this study, a methodology is developed that uses a DII to clarify the role of top-down processes on motor action, particularly exploring how reaches toward a target on a DII are affected in both deliberate and spontaneous movement domains.

It is unclear how top-down signals from the ventral visual stream affect movement. We developed a paradigm to test motor behavior towards a target on a 3D depth inversion illusion. Significant differences are reported in both deliberate, goal-directed movements and automatic actions under illusory and veridical viewing conditions.

Kinesthetic awareness is important to successfully navigate the environment. When we interact with our daily surroundings, some aspects of movement are ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

A Method to Quantify Visual Information Processing in Children Using Eye Tracking

Visual problems that occur early in life can have major impact on a child's development. Without verbal communication and only based on observational methods, it is difficult to make a quantitative assessment of a child's visual problems. This limits accurate diagnostics in children under the age of 4 years and in children with intellectual disabilities. Here we describe a quantitative method that overcomes these problems. The method uses a remote eye tracker and a four choice preferential looking paradigm to measure eye movement responses to different visual stimuli. The child sits without head support in front of a monitor with integrated infrared cameras. In one of four monitor quadrants a visual stimulus is presented. Each stimulus has a specific visual modality with respect to the background, e.g., form, motion, contrast or color. From the reflexive eye movement responses to these specific visual modalities, output parameters such as reaction times, fixation accuracy and fixation duration are calculated to quantify a child's viewing behavior. With this approach, the quality of visual information processing can be assessed without the use of communication. By comparing results with reference values obtained in typically developing children from 0-12 years, the method provides a characterization of visual information processing in visually impaired children. The quantitative information provided by this method can be advantageous for the field of clinical visual assessment and rehabilitation in multiple ways. The parameter values provide a good basis to: (i) characterize early visual capacities and consequently to enable early interventions; (ii) compare risk groups and follow visual development over time; and (iii), construct an individual visual profile for each child.

A method is described to quantify the quality of visual information processing based on reflexive eye movements in response to specific visual modalities. Reaction times and fixation output parameters are used to characterize visual performance in children with and without visual impairments from 6 months of age.

Visual problems that occur early in life can have major impact on a child's development. Without verbal communication and only based on observational ...

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test&#160;theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to&#160;comprehension informed by&#160;comprehenders&#39; experience. Among the examined aspects of the visual context are actions, events, a speaker&#39;s gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eye-tracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension.

The present article reviews an eye-tracking methodology for studies on language comprehension. To obtain reliable data, key steps of the protocol must be followed. Among these are the correct set-up of the eye tracker (e.g., ensuring good quality of the eye and head images) and accurate calibration.

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing ...

Using the Visual World Paradigm to Study Sentence Comprehension in Mandarin-Speaking Children with Autism

Sentence comprehension relies on the ability to rapidly integrate different types of linguistic and non-linguistic information. However, there is currently a paucity of research exploring how preschool children with autism understand sentences using different types of cues. The mechanisms underlying sentence comprehension remains largely unclear. The present study presents a protocol to examine the sentence comprehension abilities of preschool children with autism. More specifically, a visual world paradigm of eye-tracking is used to explore the moment-to-moment sentence comprehension in the children. The paradigm has multiple advantages. First, it is sensitive to the time course of sentence comprehension and thus can provide rich information about how sentence comprehension unfolds over time. Second, it requires minimal task and communication demands, so it is ideal for testing children with autism. To further minimize the computational burden of children, the present study measures eye movements that arise as automatic responses to linguistic input rather than measuring eye movements that accompany conscious responses to spoken instructions.

We present a protocol to examine the use of morphological cues during real-time sentence comprehension by children with autism.

Sentence comprehension relies on the ability to rapidly integrate different types of linguistic and non-linguistic information. However, there is ...

Defining the Role Of Language in Infants' Object Categorization with Eye-tracking Paradigms

Assessing infant category learning is a challenging but vital aspect of studying infant cognition. By employing a familiarization-test paradigm, we straightforwardly measure infants&#8217; success in learning a novel category while relying only on their looking behavior. Moreover, the paradigm can directly measure the impact of different auditory signals on the infant categorization across a range of ages. For instance, we assessed how 2-year-olds learn categories in a variety of labeling environments: in our task, 2-year-olds successfully learned categories when all exemplars were labeled&#160;or the first two exemplars were labeled, but they failed to categorize when no exemplars were labeled or only the final two exemplars were labeled. To determine infants&#8217; success in such tasks, researchers can examine both the overall preference displayed by infants in each condition and infants&#8217; pattern of looking over the course of the test phase, using an eye-tracker to provide fine-grained time-course data. Thus, we present a powerful paradigm for identifying the role of language, or any auditory signal, in infants&#8217; object category learning.

Here we present a protocol for familiarization-test paradigms which provide a direct test of infant categorization and help to define the role of language in early category learning.

Assessing infant category learning is a challenging but vital aspect of studying infant cognition. By employing a familiarization-test paradigm, we ...

Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm

We discuss the use of the preferential looking paradigm in eye tracking studies in order to study how infants develop, understand, and attend to the world around them. Eye tracking is a safe and non-invasive way to collect gaze data from infants, and the preferential looking paradigm is simple to design and only requires the infant to be attending to the screen. By simultaneously showing two visual stimuli that differ in one dimension, we can assess whether infants show different looking behavior for either stimulus, thus demonstrating sensitivity to that difference. The challenges in such experimental approaches are that experiments must be kept brief (no more than 10 min) and be carefully controlled such that the two stimuli differ in only one way. The interpretation of null results must also be carefully considered. In this paper, we illustrate a successful example of an infant eye tracking study with a preferential looking paradigm to discover that 6-month-olds are sensitive to linguistic cues in a signed language despite having no prior exposure to signed language, suggesting that infants possess intrinsic or innate sensitivities to these cues.

Eye tracking studies using a preferential looking paradigm can be used to study infants&#39;&#160;emerging understanding of, and attention to, their external visual world.

We discuss the use of the preferential looking paradigm in eye tracking studies in order to study how infants develop, understand, and attend to the world ...

Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks

Working memory (WM) plays a central role in the comprehension of complex sentences. Its function in the processing of spoken complex sentences is especially evident because spoken complex sentence processing is memory-intensive. The dual-modal interference paradigm has been used to examine how the WM system is involved in complex syntactic processing. This article presents two exemplar experiments involving auditory processing with either intra- or extrasentential interferences. In the first experiment, auditory stimuli [spoken-Chinese relative clause (RC) sentences with two syntactic types: subject-gapped (SRC) vs. object-gapped (ORC)] are interfered with via a visually presented lexical decision task within a sentence and manipulated using three different interference timepoints. In the second experiment, the same auditory stimuli, presented via an auditory window moving technique, are interfered with via a visually presented digital recall task beyond the sentence and manipulated using three digital memory loads. By assessing how the primary task of comprehending the RC sentences is affected by the secondary task, we can tackle the controversial issue concerning Chinese RC processing asymmetry. Our results reveal different patterns of RC processing compared to those reported in previous studies. Experiment 1 manifests no clear RC processing advantages in either SRC or ORC; however, a preference for ORC is observed at the ends of sentences, and a preference for SRC is found at the main verb site. Likewise, Experiment 2 presents a dynamic pattern. Under a no-digit load, SRCs show processing advantages in the RC marker region. However, under higher-digit-load interference, ORCs show processing advantages in the same region. These results lead to the conjecture that no obvious or intrinsic processing asymmetry exists in the processing of Chinese RCs. Using the approach of assessing specific interference during syntactic processing, these experiments demonstrate future research applications that explore the processing metrics of spoken sentences involving working memory.

Here, we present a protocol employing dual-modal interference tasks to examine online processing of spoken-Chinese relative clause sentences. Two exemplar experiments involving auditory processing with intra- and extrasentential interferences are described. The paradigm provides a methodology for addressing the nature of working memory and its effects on sentence processing.

Working memory (WM) plays a central role in the comprehension of complex sentences. Its function in the processing of spoken complex sentences is ...

Transcranial Direct Current Stimulation for Online Gamers

Transcranial direct current stimulation (tDCS) is a noninvasive brain stimulation technique that applies a weak electric current to the scalp to modulate neuronal membrane potentials. Compared to other brain stimulation methods, tDCS is relatively safe, simple, and inexpensive to administer.
Since excessive online gaming can negatively affect mental health and daily functioning, developing treatment options for gamers is necessary. Although tDCS over the dorsolateral prefrontal cortex (DLPFC) has demonstrated promising results for various addictions, it has not been tested in gamers. This paper describes a protocol and a feasibility study for applying repeated tDCS over the DLPFC and neuroimaging to examine the underlying neural correlates in gamers.
At baseline, individuals who play online games report average weekly hours spent on games, complete questionnaires on addiction symptoms and self-control, and undergo brain 18F-&#64258;uoro-2-deoxyglucose positron emission tomography (FDG-PET). The tDCS protocol consists of 12 sessions over the DLPFC for 4 weeks (anode F3/cathode F4, 2 mA for 30 min per session). Then, a follow-up is conducted using the same protocol as the baseline. Individuals who do not play online games receive only baseline FDG-PET scans without tDCS. Changes of clinical characteristics and asymmetry of regional cerebral metabolic rate of glucose (rCMRglu) in the DLPFC are examined in gamers. In addition, asymmetry of rCMRglu is compared between gamers and non-gamers at baseline.
In our experiment, 15 gamers received tDCS sessions and completed baseline and follow-up scans. Ten non-gamers underwent FDG-PET scans at the baseline. The tDCS reduced addiction symptoms, time spent on games, and increased self-control. Moreover, abnormal asymmetry of rCMRglu in the DLPFC at baseline was alleviated after tDCS.
The current protocol may be useful for assessing treatment efficacy of tDCS and its underlying brain changes in gamers. Further randomized sham-controlled studies are warranted. Moreover, the protocol can be applied to other neurological and psychiatric disorders.

We present a protocol and a feasibility study for applying transcranial direct current stimulation (tDCS) and neuroimaging assessment in online gamers.

Transcranial direct current stimulation (tDCS) is a noninvasive brain stimulation technique that applies a weak electric current to the scalp to modulate ...