JoVE Logo
Faculty Resource Center

Sign In





Representative Results






Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Published: October 13th, 2018



1Institute for Speech Pathology and the Brain Science, School of Communication Science, Beijing Language and Culture University

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

Spoken language is a fast, ongoing information flow, which disappears right away. It is a challenge to experimentally study this temporal, rapidly change speech signal. Eye movements recorded in the visual world paradigm can be used to overcome this challenge. In a typical eye tracking study using the visual world paradigm, participants' eye movements to pictures in a display or to real objects in a visual workspace are monitored as they listen to, or produce, spoken language depicting the contents of the visual world1,2,3,4. The basic l....

Log in or to access full content. Learn more about your institution’s access to JoVE content here

All subjects must give informed written consent before the administration of the experimental protocols. All procedures, consent forms, and the experimental protocol were approved by the Research Ethics Committee of the Beijing Language and Culture University.

NOTE: A comprehension study using the visual world paradigm normally consists of the following steps: Introduce the theoretical problems to be explored; Form an experimental design; Prepare the visual and auditory stimuli; Frame the theo.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

Participants' behavioral responses are summarized in Figure 4. As we described earlier, the correct response to a conjunctive statement (S1 and S2) is the big open box, such as Box A in Figure 1. The correct response to a but-statement (S1 but not S2) is the small open box containing the first mentioned animal, such as Box D in Figure 1. Critically, which box is chosen .......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

To conduct a visual world study, there are several critical steps to follow. First, researchers intend to deduce the interpretation of the auditorily presented language via participants' eye movements in the visual world. Henceforth, in designing the layout of the visual stimuli, the properties of eye movements in a natural task that potentially affect participants' eye movements should be controlled. The effect of the spoken language on participants' eye movements can then be recognized. Second, acoustic cue.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

This research was supported by Science Foundation of Beijing Language and Cultural University under the Fundamental Research Funds for the Central Universities (Approval number 15YJ050003).


Log in or to access full content. Learn more about your institution’s access to JoVE content here

Name Company Catalog Number Comments
Pixelmator Pixelmator Team image editing app
Praat Open Sourse Sound analyses and editting software
Eyelink 1000plus SR-Research, Inc remote infrared eye tracker 
Experimental Builder SR-Research, Inc eye tracker software 
Data Viewer SR-Research, Inc eye tracker software 
R Open Sourse free software environment for statistical computing and graphics

  1. Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., Sedivy, J. C. Integration of visual and linguistic information in spoken language comprehension. Science. 268 (5217), 1632-1634 (1995).
  2. Cooper, R. M. The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology. 6 (1), 84-107 (1974).
  3. Salverda, A. P., Tanenhaus, M. K., de Groot, A. M. B., Hagoort, P. . Research methods in psycholinguistics and the neurobiology of language: A practical guide. , (2017).
  4. Huettig, F., Rommers, J., Meyer, A. S. Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica. 137 (2), 151-171 (2011).
  5. Meyer, A. S., Sleiderink, A. M., Levelt, W. J. M. Viewing and naming objects: Eye movements during noun phrase production. Cognition. 66 (2), B25-B33 (1998).
  6. Griffin, Z. M., Bock, K. What the eyes say about speaking. Psychological Science. 11 (4), 274-279 (2000).
  7. Young, L. R., Sheena, D. Survey of eye movement recording methods. Behavior Research Methods & Instrumentation. 7 (5), 397-429 (1975).
  8. Conklin, K., Pellicer-Sánchez, A., Carrol, G. . Eye-tracking: A guide for applied linguistics research. , (2018).
  9. Duchowski, A. . Eye tracking methodology: Theory and practice. , (2007).
  10. Baayen, R. H., Davidson, D. J., Bates, D. M. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language. 59 (4), 390-412 (2008).
  11. Barr, D. J. Analyzing 'visual world' eyetracking data using multilevel logistic regression. Journal of Memory and Language. 59 (4), 457-474 (2008).
  12. Nixon, J. S., van Rij, J., Mok, P., Baayen, R. H., Chen, Y. The temporal dynamics of perceptual uncertainty: eye movement evidence from Cantonese segment and tone perception. Journal of Memory and Language. 90, 103-125 (2016).
  13. Bolker, B. M., et al. Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology and Evolution. 24 (3), 127-135 (2009).
  14. Mirman, D., Dixon, J. A., Magnuson, J. S. Statistical and computational models of the visual world paradigm: Growth curves and individual differences. Journal of Memory and Language. 59 (4), 475-494 (2008).
  15. Baayen, H., Vasishth, S., Kliegl, R., Bates, D. The cave of shadows: Addressing the human factor with generalized additive mixed models. Journal of Memory and Language. 94, 206-234 (2017).
  16. Baayen, R. H., van Rij, J., de Cat, C., Wood, S., Speelman, D., Heylen, K., Geeraerts, D. . Mixed-Effects Regression Models in Linguistics. 4, 49-69 (2018).
  17. Zhan, L. Scalar and ignorance inferences are both computed immediately upon encountering the sentential connective: The online processing of sentences with disjunction using the visual world paradigm. Frontiers in Psychology. 9, (2018).
  18. Maris, E., Oostenveld, R. Nonparametric statistical testing of EEG- and MEG-data. Journal of Neuroscience Methods. 164 (1), 177-190 (2007).
  19. Barr, D. J., Jackson, L., Phillips, I. Using a voice to put a name to a face: The psycholinguistics of proper name comprehension. Journal of Experimental Psychology-General. 143 (1), 404-413 (2014).
  20. Dahan, D., Tanenhaus, M. K., Salverda, A. P., van Gompel, R. P. G., Fischer, M. H., Murray, W. S., Hill, R. L. . Eye movements: A window on mind and brain. , 471-486 (2007).
  21. Parkhurst, D., Law, K., Niebur, E. Modeling the role of salience in the allocation of overt visual attention. Vision Research. 42 (1), 107-123 (2002).
  22. Grice, H. P., Cole, P., Morgan, J. L. Vol. 3 Speech Acts. Syntax and semantics. , 41-58 (1975).
  23. Sauerland, U. Scalar implicatures in complex sentences. Linguistics and Philosophy. 27 (3), 367-391 (2004).
  24. Chierchia, G. Scalar implicatures and their interface with grammar. Annual Review of Linguistics. 3 (1), 245-264 (2017).
  25. Fox, D., Sauerland, U., Stateva, P. . Presupposition and Implicature in Compositional Semantics. , 71-120 (2007).
  26. Meyer, M. C. . Ignorance and grammar. , (2013).
  27. SR Research Ltd. . SR Research Experiment Builder User Manual (Version 2.1.140). , (2017).
  28. SR Research Ltd. . EyeLink® 1000 Plus Technical Specifications. , (2017).
  29. SR Research Ltd. . EyeLink-1000-Plus-Brochure. , (2017).
  30. SR Research Ltd. . EyeLink® 1000 Plus User Manual (Version 1.0.12). , (2017).
  31. SR Research Ltd. . EyeLink® Data Viewer User’s Manual (Version 3.1.97). , (2017).
  32. McQueen, J. M., Viebahn, M. C. Tracking recognition of spoken words by tracking looks to printed words. The Quarterly Journal of Experimental Psychology. 60 (5), 661-671 (2007).
  33. Altmann, G. T. M., Kamide, Y. Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition. 73 (3), 247-264 (1999).
  34. Altmann, G. T. M., Kamide, Y. The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language. 57 (4), 502-518 (2007).
  35. Snedeker, J., Trueswell, J. C. The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adult sentence processing. Cognitive Psychology. 49 (3), 238-299 (2004).
  36. Allopenna, P. D., Magnuson, J. S., Tanenhaus, M. K. Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language. 38 (4), 419-439 (1998).
  37. Zhan, L., Crain, S., Zhou, P. The online processing of only if and even if conditional statements: Implications for mental models. Journal of Cognitive Psychology. 27 (3), 367-379 (2015).
  38. Zhan, L., Zhou, P., Crain, S. Using the visual-world paradigm to explore the meaning of conditionals in natural language. Language, Cognition and Neuroscience. 33 (8), 1049-1062 (2018).
  39. Brown-Schmidt, S., Tanenhaus, M. K. Real-time investigation of referential domains in unscripted conversation: A targeted language game approach. Cognitive Science. 32 (4), 643-684 (2008).
  40. Fernald, A., Pinto, J. P., Swingley, D., Weinberg, A., McRoberts, G. W. Rapid gains in speed of verbal processing by infants in the 2nd year. Psychological Science. 9 (3), 228-231 (1998).
  41. Trueswell, J. C., Sekerina, I., Hill, N. M., Logrip, M. L. The kindergarten-path effect: studying on-line sentence processing in young children. Cognition. 73 (2), 89-134 (1999).
  42. Zhou, P., Su, Y., Crain, S., Gao, L. Q., Zhan, L. Children's use of phonological information in ambiguity resolution: a view from Mandarin Chinese. Journal of Child Language. 39 (4), 687-730 (2012).
  43. Zhou, P., Crain, S., Zhan, L. Grammatical aspect and event recognition in children's online sentence comprehension. Cognition. 133 (1), 262-276 (2014).
  44. Zhou, P., Crain, S., Zhan, L. Sometimes children are as good as adults: The pragmatic use of prosody in children's on-line sentence processing. Journal of Memory and Language. 67 (1), 149-164 (2012).
  45. Moscati, V., Zhan, L., Zhou, P. Children's on-line processing of epistemic modals. Journal of Child Language. 44 (5), 1025-1040 (2017).
  46. Helfer, K. S., Staub, A. Competing speech perception in older and younger adults: Behavioral and eye-movement evidence. Ear and Hearing. 35 (2), 161-170 (2014).
  47. Dickey, M. W., Choy, J. W. J., Thompson, C. K. Real-time comprehension of wh-movement in aphasia: Evidence from eyetracking while listening. Brain and Language. 100 (1), 1-22 (2007).
  48. Magnuson, J. S., Nusbaum, H. C. Acoustic differences, listener expectations, and the perceptual accommodation of talker variability. Journal of Experimental Psychology-Human Perception and Performance. 33 (2), 391-409 (2007).
  49. Reinisch, E., Jesse, A., McQueen, J. M. Early use of phonetic information in spoken word recognition: Lexical stress drives eye movements immediately. Quarterly Journal of Experimental Psychology. 63 (4), 772-783 (2010).
  50. Chambers, C. G., Tanenhaus, M. K., Magnuson, J. S. Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology-Learning Memory and Cognition. 30 (3), 687-696 (2004).
  51. Tanenhaus, M. K., Trueswell, J. C., Trueswell, J. C., Tanenhaus, M. K. . Approaches to Studying World-Situated Language Use: Bridging the Language-as-Product and Language-as-Action Traditions. , (2005).

This article has been published

Video Coming Soon

JoVE Logo


Terms of Use





Copyright © 2024 MyJoVE Corporation. All rights reserved