Name: measuring attention and visual processing speed by model-based analysis of temporal-order judgments
Uploaded: 2017-01-23T14:00:00
Description: Watch this Scientific Journal Video about Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments at JoVE.com

Parts of this work have been supported by the German Research Foundation (DFG) via grants 1515/1-2 and 1515/6-1 to Ingrid Scharlau.

NOTE: Some steps in this protocol can be accomplished using custom software provided (along with installation instructions) at http://groups.upb.de/viat/TVATOJ. In the protocol, this collection of programs and scripts is referred to as &#8220;TVATOJ&#8221;.
1. Selection of Stimulus Material
<ol>
	<li>Select stimuli according to the research question. 
	NOTE: In general, two targets are shown at different locations on the screen. Stimuli that have been used with the pres...

<ol>
	<li>Posner, M. I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Orienting+of+attention.">Orienting of attention.</a> Quarterly Journal of Experimental Psychology. 32 (1), 3-25 (1980).</li><li>Van der Heijden, A., Wolters, G., Groep, J., Hagenaar, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Single-letter+recognition+accuracy+benefits+from+advance+cuing+of+location.">Single-letter recognition accuracy benefits from advance cuing of location.</a> Perception &amp; Psychophysics. 42 (5), 503-509 (1987).</li><li>Bundesen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+theory+of+visual+attention.">A theory of visual attention.</a> Psychological Review. 97 (4), 523-547 (1990).</li><li>Bublak, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Staged+decline+of+visual+processing+capacity+in+mild+cognitive+impairment+and+Alzheimer's+disease.">Staged decline of visual processing capacity in mild cognitive impairment and Alzheimer's disease.</a> Neurobiology of Aging. 32 (7), 1219-1230 (2011).</li><li>Petersen, A., Kyllingsb&#230;k, S., Bundesen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measuring+and+modeling+attentional+dwell+time.">Measuring and modeling attentional dwell time.</a> Psychonomic Bulletin &amp; Review. 19 (6), 1029-1046 (2012).</li><li>Vroomen, J., Keetels, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perception+of+intersensory+synchrony:+A+tutorial+review.">Perception of intersensory synchrony: A tutorial review.</a> Attention, Perception, &amp; Psychophysics. 72 (4), 871-884 (2010).</li><li>T&#252;nnermann, J., Petersen, A., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Does+attention+speed+up+processing?+Decreases+and+increases+of+processing+rates+in+visual+prior+entry.">Does attention speed up processing? Decreases and increases of processing rates in visual prior entry.</a> Journal of Vision. 15 (3), 1-27 (2015).</li><li>Kr&#252;ger, A., T&#252;nnermann, J., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+and+conspicuous?+Quantifying+salience+with+the+Theory+of+Visual+Attention.">Fast and conspicuous? Quantifying salience with the Theory of Visual Attention.</a> Advances in Cognitive Psychology. 12 (1), 20 (2016).</li><li>Bundesen, C., Habekost, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Principles of Visual Attention: Linking Mind and Brain. , (2008).</li><li>Plummer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=JAGS:+A+program+for+analysis+of+Bayesian+graphical+models+using+Gibbs+sampling.">JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling.</a> Proceedings of the 3rd international workshop on distributed statistical computing. , 124-125 (2003).</li><li>Kruschke, J. K., Vanpaemel, W., Busemeyer, J., Townsend, J., Wang, Z. J., Eidels, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bayesian+estimation+in+hierarchical+models.">Bayesian estimation in hierarchical models.</a> The Oxford Handbook of Computational and Mathematical Psychology. , 279-299 (2015).</li><li>Math&#244;t, S., Schreij, D., Theeuwes, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=OpenSesame:+An+open-source,+graphical+experiment+builder+for+the+social+sciences.">OpenSesame: An open-source, graphical experiment builder for the social sciences.</a> Behavior Research Methods. 44 (2), 314-324 (2012).</li><li>Kruschke, J. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan. , (2015).</li><li>Rensink, R. A., O'Regan, J. K., Clark, J. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=To+see+or+not+to+see:+The+need+for+attention+to+perceive+changes+in+scenes.">To see or not to see: The need for attention to perceive changes in scenes.</a> Psychological Science. 8 (5), 368-373 (1997).</li><li>T&#252;nnermann, J., Kr&#252;ger, N., Mertsching, B., Mustafa, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Affordance+estimation+enhances+artificial+visual+attention:+Evidence+from+a+change-blindness+study.">Affordance estimation enhances artificial visual attention: Evidence from a change-blindness study.</a> Cognitive Computation. 7 (5), 525-538 (2015).</li><li>Shore, D. I., Klein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+effects+of+scene+inversion+on+change+blindness.">The effects of scene inversion on change blindness.</a> The Journal of General Psychology. 127 (1), 27-43 (2000).</li><li>Scharlau, I., Neumann, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+parameters+and+time+course+of+perceptual+latency+priming.">Temporal parameters and time course of perceptual latency priming.</a> Acta Psychologica. 113 (2), 185-203 (2003).</li><li>Schneider, K. A., Bavelier, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Components+of+visual+prior+entry.">Components of visual prior entry.</a> Cognitive Psychology. 47 (4), 333-366 (2003).</li><li>Scharlau, I., Neumann, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perceptual+latency+priming+by+masked+and+unmasked+stimuli:+Evidence+for+an+attentional+interpretation.">Perceptual latency priming by masked and unmasked stimuli: Evidence for an attentional interpretation.</a> Psychological Research. 67 (3), 184-196 (2003).</li><li>Shore, D. I., Spence, C., Klein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Visual+prior+entry.">Visual prior entry.</a> Psychological Science. 12 (3), 205-212 (2001).</li><li>Alcal&#225;-Quintana, R., Garc&#237;a-P&#233;rez, M. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fitting+model-based+psychometric+functions+to+simultaneity+and+temporal-order+judgment+data:+MATLAB+and+R+routines.">Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines.</a> Behavior Research Methods. 45 (4), 972-998 (2013).</li><li>Hoffman, M. D., Gelman, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+No-U-turn+sampler:+adaptively+setting+path+lengths+in+Hamiltonian+Monte+Carlo.">The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.</a> Journal of Machine Learning Research. 15 (1), 1593-1623 (2014).</li><li>Lee, M. D., Wagenmakers, E. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Bayesian cognitive modeling: A practical course. , (2014).</li><li>Vangkilde, S., Bundesen, C., Coull, J. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Prompt+but+inefficient:+Nicotine+differentially+modulates+discrete+components+of+attention.">Prompt but inefficient: Nicotine differentially modulates discrete components of attention.</a> Psychopharmacology. 218 (4), 667-680 (2011).</li><li>T&#252;nnermann, J., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Peripheral+Visual+Cues:+Their+Fate+in+Processing+and+Effects+on+Attention+and+Temporal-order.">Peripheral Visual Cues: Their Fate in Processing and Effects on Attention and Temporal-order.</a> Front. Psychol. 7 (1442), (2016).</li></ol>

The protocol in this article describes how to conduct simple TOJs and fit the data with models based on fundamental stimulus encoding. Three experiments demonstrated how the results can be evaluated in a hierarchical Bayesian estimation framework to assess the influence of attention in highly different stimulus material. Salience in pop-out displays led to increased attentional weights. Also, increased weights were estimated for action space objects in natural images. However, due to the persisting advantage when spatial...

How attention is distributed in space and time is one of the most important factors in human visual perception. Objects that capture attention because of their conspicuity or importance are typically processed faster and with higher accuracy. In behavioral research, such performance benefits have been demonstrated in a variety of experimental paradigms. For instance, allocating attention to the target location speeds up the reaction in probe detection tasks1. Similarly, the accuracy of reporting letters is improved by attention2. Such findings prove that attention enhances processing, but they remain hopelessly mute about how this enhancement is established.
The present paper shows that low-level mechanisms behind attentional advantages can be assessed by measuring the processing speed of individual stimuli in a model-based framework that relates the measurements to fine-grained components of attention. With such a model, the overall processing capacity and its distribution among the stimuli can be inferred from processing speed measurements.
Bundesen&#39;s Theory of Visual Attention (TVA)3 provides a suitable model for this endeavor. It is typically applied to data from letter report tasks. In the following, the fundamentals of TVA are explained and it is shown how they can be extended to model temporal-order judgment (TOJ) data obtained with (almost) arbitrary stimuli. This novel method provides estimates of processing speed and resource distribution which can be readily interpreted. The protocol in this article explains how to plan and conduct such experiments and details how the data can be analyzed.
As mentioned above, the usual paradigm in TVA-based modeling and estimation of attention parameters is the letter report task. Participants report the identities of a set of letters which is briefly flashed and typically masked after a varying delay. Among other parameters, the rate at which visual elements are encoded into visual short-term memory can be estimated. The method has been successfully applied to questions in fundamental and clinical research. For instance, Bublak and colleagues4 assessed which attentional parameters are affected in different stages of age-related cognitive deficits. In fundamental attention research, Petersen, Kyllingsb&#230;k, and Bundesen5 used TVA to model the attentional dwell time effect, the observer&#39;s difficulty in perceiving the second of two targets at certain time intervals. A major drawback of the letter report paradigm is that it requires sufficiently overlearned and maskable stimuli. This requirement limits the method to letters and digits. Other stimuli would require heavy training of participants.
The TOJ paradigm requires neither specific stimuli nor masking. It can be used with any kind of stimuli for which the order of appearance can be judged. This extends the stimulus range to pretty much everything that could be of interest, including direct cross-modal comparisons6.
Investigating attention with TOJs is based on the phenomenon of attentional prior entry which is a measure of how much earlier an attended stimulus is perceived compared to an unattended one. Unfortunately, the usual method for analyzing TOJ data, fitting observer performance psychometric functions (such as cumulative Gaussian or logistic functions), cannot distinguish whether attention increases the processing rate of the attended stimulus or if it decreases the rate of the unattended stimulus7. This ambiguity is a major problem because the question whether the perception of a stimulus is truly enhanced or if it benefits because of the withdrawal of resources from a competing stimulus is a question of both fundamental and practical relevance. For instance, for the design of human-machine interfaces it is highly relevant to know if increasing the prominence of one element works at the expense of another one.
The TOJ task usually proceeds as follows: A fixation mark is presented for a brief delay, typically a randomly drawn interval shorter than a second. Then, the first target is presented, followed after a variable stimulus onset asynchrony (SOA) by the second target. At negative SOAs, the probe, the attended stimulus, is shown first. At positive SOAs, the reference, the unattended stimulus, leads. At an SOA of zero, both targets are shown simultaneously.
Typically, presenting the target refers to switching the stimulus on. Under certain conditions, however, other temporal events, such as a flicker of an already present target or offsets are used8.
In TOJs, responses are collected in an unspeeded manner, usually by keys mapped to the stimulus identities and presentation orders (e.g., if stimuli are squares and diamonds, one key indicates &#34;square first&#34; and another one &#34;diamond first&#34;). Importantly, for the evaluation, these judgments must be converted to &#34;probe first&#34; (or &#34;reference first&#34;) judgments.
In the present work, a combination of the processing model of TVA and the TOJ experimental paradigm is used to eliminate the problems in either individual domain. With this method, readily interpretable speed parameters can be estimated for almost arbitrary visual stimuli, enabling to infer how the observer&#39;s attention is allocated to competing visual elements.
The model is based on TVA&#39;s equations for the processing of individual stimuli, which will be shortly explained in the following. The probability that one stimulus is encoded into visual short-term memory before the other is interpreted as the probability of judging this stimulus as appearing first. The individual encoding durations are exponentially distributed9:
<img alt="Equation 1" src="/files/ftp_upload/54856/54856eq1.jpg" /> (1)
The maximum ineffective exposure duration t0 is a threshold before which nothing is encoded at all. According to TVA, the rate vx,i at which object x is encoded as member of a perceptual category i (such as color or a shape) is given by the rate equation,
<img alt="Equation 2" src="/files/ftp_upload/54856/54856eq2.jpg" />&#160;. (2)
The strength of the sensory evidence that x belongs to category i is expressed in &#951;x,i, and &#946;i is a decision bias for categorizing stimuli as members of category i. This is multiplied by attentional weights. Individual attentional weights wx are divided by the attentional weights of all objects in the visual field. Hence, the relative attentional weight is calculated as
<img alt="Equation 3" src="/files/ftp_upload/54856/54856eq3.jpg" />&#160; (3)
where R represents all categories and &#951;x,i represents the sensory evidence that object x belongs to category j. The value &#960;j is called pertinence of category j and reflects a bias to make categorizations in j. The overall processing capacity C is the sum of all processing rates for all stimuli and categorizations. For a more detailed description of TVA, refer to Bundesen and Habekost&#39;s book9.
In our novel method, Equation 1, which describes the encoding of individual stimuli, is transformed into a model of TOJs. Assuming that selection biases and report categories are constant within an experimental task, the processing rates vp and vr of the two target stimuli probe (p) and reference (r) depend on C and the attentional weights in the form vp= C &#183; wp and vr = C &#183; wr, respectively. The new TOJ model expresses the success probability Pp1st that a participant judges the probe stimulus to be first as a function of the SOA and the processing rates. It can be formalized as follows:
<img alt="Equation 4" src="/files/ftp_upload/54856/54856eq4.jpg" /> (4)
A more detailed description of how this equation is derived from the basic TVA equations is described by T&#252;nnermann, Petersen, and Scharlau7.
For the sake of simplicity, the parameter t0 is omitted in the model in Equation 1. According to the original TVA, t0 should be identical for both targets in the TOJ task, and, therefore, it cancels out. However, this assumption may sometimes be violated (see section Discussion).
For fitting this equation to TOJ data, a hierarchical Bayesian estimation scheme11 is suggested. This approach allows to estimate the attentional weights wp and wr of the probe and reference stimuli and the overall processing rate C. These parameters, the resulting uptake rates vp and vr, and attention-induced differences between them, can be assessed on the subject and group levels along with estimated uncertainties. The hierarchical model is illustrated in Figure 1. During the planning stage for an experiment, convenient Bayesian power analysis can be conducted.
The following protocol describes how to plan, execute and analyze TOJ experiments from which processing speed parameters and attentional weights for visual stimuli can be obtained. The protocol assumes that the researcher is interested in how an attentional manipulation influences the processing speeds of some targets of interest.
<img alt="Figure 1" src="/files/ftp_upload/54856/54856fig1.jpg" />
Figure 1: Graphical model used in the Bayesian estimation procedure. Circles indicate estimated distributions; double circles indicate deterministic nodes. Squares indicate data. The relations are given on the right side of the figure. The nodes outside the rounded frames (&#8220;plates&#8221;) represent mean and dispersion estimates of TVA parameters (see Introduction) on the group level. In the &#8220;j Subjects&#8221; plate, it can be seen how attentional weights (w) are combined with the overall processing rates (C) to from stimulus processing rates (v) on the subject level. Plate &#8220;i SOAs&#8221; shows how these TVA parameters are then transformed (via the function Pp1st described in the Introduction) into the success probability (&#952;) for the binomially distributed responses at each SOA. Therefore, the &#952; together with the repetitions of the SOA (n) describe the data points (y). For more details on notation and interpretation of graphical models, refer to Lee and Wagenmakers23. Note that for the sake of clarity, the nodes that represent differences of parameters have been omitted. These deterministic parameters are indicated in the figures of the experimental results instead. <a href="http://cloudflare.jove.com/files/ftp_upload/54856/54856fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table border="0" cellpadding="0" cellspacing="0" width="336">
	<tbody>
		<tr height="17">
			<td height="17" style="height:17px;width:336px;">Personal Computer</td>
		</tr>
		<tr height="17">
			<td height="17" style="height:17px;">(Open Source) Experimentation and evaluation software</td>
		</tr>
	</tbody>
</table>

measuring attention and visual processing speed by model-based analysis of temporal-order judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen&#39;s Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level.
To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

In the following, results obtained with the proposed method are reported. Three experiments measured the influence of different attentional manipulations with three highly different types of stimulus material. The stimuli are simple line segments in pop-out patterns, action space objects in natural images, and cued letter targets.
Experiment 1: Salience in pop-out displays 
Experiment 1 aimed at measuring the influence of v...

Watch this Scientific Journal Video about Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments at JoVE.com

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

Research

JoVE Journal

Behavior

The 5-Choice Serial Reaction Time Task: A Task of Attention and Impulse Control for Rodents

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. ...

Topographical Estimation of Visual Population Receptive Fields by fMRI

Visual cortex is retinotopically organized so that neighboring populations of cells map to neighboring parts of the visual field. Functional magnetic ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

Measurement of Neurophysiological Signals of Ignoring and Attending Processes in Attention Control

Attention control is the ability to selectively attend to some sensory signals while ignoring others. This ability is thought to involve two processes: ...

A Method to Quantify Visual Information Processing in Children Using Eye Tracking

Visual problems that occur early in life can have major impact on a child's development. Without verbal communication and only based on observational ...

Intracortical Inhibition Within the Primary Motor Cortex Can Be Modulated by Changing the Focus of Attention

It is well recognized that an external focus (EF) compared with an internal focus (IF) of attention improves motor learning and performance. Studies have ...

Central and Divided Visual Field Presentation of Emotional Images to Measure Hemispheric Differences in Motivated Attention

Two dominant theories on lateralized processing of emotional information exist in the literature. One theory posits that unpleasant emotions are processed ...

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

Gaze in Action: Head-mounted Eye Tracking of Children's Dynamic Visual Attention During Naturalistic Behavior

Young children's visual environments are dynamic, changing moment-by-moment as children physically and visually explore spaces and objects and interact ...

Spotlighting Customers' Visual Attention at the Stock, Shelf and Store Levels with the 3S Model

Several models of the in-store search process exist in the fields of retailing, marketing, and consumer-based research. The present article presents a new ...