Testing Sensory and Multisensory Function in Children with Autism Spectrum Disorder

Summary

We describe how to implement a battery of behavioral tasks to examine the processing and integration of sensory stimuli in children with ASD. The goal is to characterize individual differences in temporal processing of simple auditory and visual stimuli and relate these to higher order perceptual skills like speech perception.

Abstract

In addition to impairments in social communication and the presence of restricted interests and repetitive behaviors, deficits in sensory processing are now recognized as a core symptom in autism spectrum disorder (ASD). Our ability to perceive and interact with the external world is rooted in sensory processing. For example, listening to a conversation entails processing the auditory cues coming from the speaker (speech content, prosody, syntax) as well as the associated visual information (facial expressions, gestures). Collectively, the “integration” of these multisensory (i.e., combined audiovisual) pieces of information results in better comprehension. Such multisensory integration has been shown to be strongly dependent upon the temporal relationship of the paired stimuli. Thus, stimuli that occur in close temporal proximity are highly likely to result in behavioral and perceptual benefits – gains believed to be reflective of the perceptual system's judgment of the likelihood that these two stimuli came from the same source. Changes in this temporal integration are expected to strongly alter perceptual processes, and are likely to diminish the ability to accurately perceive and interact with our world. Here, a battery of tasks designed to characterize various aspects of sensory and multisensory temporal processing in children with ASD is described. In addition to its utility in autism, this battery has great potential for characterizing changes in sensory function in other clinical populations, as well as being used to examine changes in these processes across the lifespan.

Introduction

Traditional neuroscience research has often approached understanding sensory perception by focusing on the individual sensory modalities. However, the environment consists of a wide array of sensory inputs that are integrated into a unified perceptual view of the world in a seemingly effortless manner. The fact that we exist in such a rich multisensory environment requires that we better understand the way in which the brain combines information across the different sensory systems. The need for this understanding is further amplified by the fact that the presence of multiple pieces of sensory information often results in substantial improvements in behavior and perception^1-3. For example, there is a large improvement (up to 15 dB in the signal-to-noise ratio) in the ability to understand speech in a noisy environment if the observer can also see the speaker’s lip movements^4-7.

One of the major factors that affects how the different sensory inputs are combined and integrated is their relative temporal proximity. If two sensory cues occur close together in time, a temporal structure that suggests common origin, they are highly likely to be integrated as evidenced by changes in behavior and perception^8-12. One of the most powerful experimental tools for examining the impact of multisensory temporal structure on behavioral and perceptual responses is simultaneity judgment (SJ) tasks^13-16. In such a task, multisensory (e.g., visual and auditory) stimuli are paired at various stimulus onset asynchronies (SOAs) ranging from objectively simultaneous (i.e., a temporal offset of 0 msec) to highly asynchronous (e.g., 400 msec). Participants are asked to judge the stimuli as simultaneous or not via a simple button press. In such a task, even when the visual and auditory stimuli are presented at SOAs of 100 msec or more, subjects report that the pair was simultaneous on a large proportion of trials. The window of time in which two inputs can occur and have a high probability of being perceived as occurring simultaneously is known as the temporal binding window (TBW)^17-19.

The TBW is a highly ethological construct, in that it represents the statistical regularities of the world around us¹⁹. The “window” provides flexibility for the specification of events of common origin; one that allows for stimuli occurring at different distances with different propagation times (both physical and neural) to still be “bound” to one another. However, although the TBW is a probabilistic construct, changes that expand (or contract) the size of this window are likely to have cascading and potentially detrimental effects on perception^20,21.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that has been classically diagnosed on the basis of deficits in social communication and the presence of restricted interests and repetitive behaviors²². In addition, and as recently codified in the DSM-5, children with ASD frequently exhibit alterations in their responses to sensory stimuli. Rather than being restricted to a single sense, these deficits often encompass multiple senses including hearing, touch, balance, taste and vision. Along with such a “multisensory” presentation, individuals with ASD often exhibit deficits in the temporal realm. Collectively, these observations suggest that multisensory temporal function may be preferentially altered in autism^17,23-25. Although concordant with the view of altered sensory function in ASD, changes in multisensory temporal function may also be an important contributor to the deficits in social communication in ASD, given the importance of rapid and accurate binding of multisensory stimuli for social and communication functions. Take as an example the speech exchange described above in which important information is contained in both the auditory and visual modalities. Indeed, these tasks have been used to demonstrate significant differences in the width of the multisensory TBW in high functioning children with autism^26-28.

Due to its importance for normal perceptual function, its potential implications for higher order processes such as social communication (and other cognitive abilities), and its clinical relevance, a battery of tasks designed to assess multisensory temporal function in children with ASD is described.

Protocol

Ethics statement: All subjects must provide informed consent prior to the experiment. The research described here has been approved by the Vanderbilt University Medical Center’s Institutional Review Board.

1. Experiment Set Up

1. Ask the participants to complete the tasks in a dimly lit, sound controlled room.
   NOTE: Consider implementing a visual schedule^29,30 as part of the study design. Although each task in this battery is relatively short, performing several tasks in a row can cause fatigue in some children, both with typical development (TD) and with ASD. A visual schedule should include all the planned activities (both tasks and hearing/vision screening), as well as short breaks between tasks. This structure will help contribute to an overall positive research experience for the participant, and has even been shown to elicit more accurate responses in some tasks³¹.
2. Affix a chin rest to the table where the participant will sit while completing the task, with the computer monitor placed 60 cm away from the participant. This is to ensure that the stimuli are the same intensity for each participant. Use noise-cancelling headphones or speakers for auditory stimulus delivery.
3. Due to differences in individual experimental platforms (sound card, graphics card, operating system, etc.), verify the stimulus duration and stimulus onset asynchronies (SOAs) with an oscilloscope, photovoltaic cell, and microphone on each computer for the experiment.
   NOTE: Depending on the individual platform (for example, a slow sound card), make adjustments to the experiment code so that the timing of stimulus presentation is accurate.

2. Stimuli

1. Generate 2 .wav or .mp4 files with duration of 16 msec (including a 2 - 3 msec up ramp and down ramp) at 500 Hz and 1,000 Hz. Do this by specifying a sine wave of the desired frequency with a gradual ramp up to full amplitude, followed by a down ramp at the end of the tone. Save the sine wave as an auditory file. Test the volume of each tone with a sound pressure level meter to verify that it is played at 60 dB. If speakers are used to present auditory stimuli, the sound should be tested at 60 cm away from the screen (where the participant will sit). If headphones will be used, measure the volume directly next to each headphone.
   NOTE: It is easier to keep the computer at a standard volume and adjust the volume of the tone accordingly by adjusting the code used to generate the stimulus or an audio editing program.
2. Create visual stimuli by either specifying the size and location of the flash in the experiment code, or by generating a JPEG or bitmap image with a black background and a white ring centered around a fixation crosshairs, and displaying at the appropriate time. Set the duration of the visual flash to 16 ms in the experiment code.
3. Record speech stimuli by a native speaker in a quiet room against a plain white background from the shoulders up with the speaker in the center of the frame. Record the the video stimuli with the highest resolution video camera available. Alternatively, publicly available stimulus videos may be utilized if desired.
   NOTE: Video and audio of the speaker saying the syllables “ba” and “ga” are required for this experiment.
   1. Using any video editing program, export the auditory component of each track and save as a separate .wav file. Do this by going into export settings window and select “wav audio file” from the “Format” drop-down menu. Check “Export Audio” and then click “Export” to the bottom of the Export Settings window. 
   2. Next, export the visual component (i.e., silent video) of each track and save as a separate .avi file. Do this by going into export settings window and select “Uncompressed AVI” from the “Format” drop-down menu. Check “Export Video” and then click “Export” to the bottom of the Export Settings window.
   3. Finally, remove the auditory component of the “ga” track and replace it with the auditory component of the “ba” track to make the McGurk stimulus. To do this select the “.avi” file from the desktop into the as the video source by selecting “Source”, in this case “ga_VOnly.avi”. Similarly select the other video “ba_Aonly.avi”. In the Program sequence menu, ensure that the video (V1) “Video 1” source is “ga_VOnly.avi” and the audio (A1) “Audio 1” source is “ba_Aonly.avi”. Verify that the onset of the visual stimulus “ga” is temporally aligned with the auditory stimulus “ba”.
      NOTE: It is important that the auditory stimuli are the exact same recording in both the audiovisual and auditory only stimulus (not just the same syllable) so that the only difference between an audiovisual “ba” and the McGurk stimulus is the video component. This will ensure that one can make a proper comparison to examine the influence of the visual stimulus on the perceived auditory syllable.

3. Task Battery

NOTE: This task requires that all participants are able to understand and comply with verbal instructions from the experimenter.

1. Ensure that all participants have normal vision by conducting a simple screening prior to testing. Use a Snellen eye chart at 20 feet and ask the participant to read each line with both eyes open (participants will be viewing stimuli with both eyes open). Record the lowest line that participants accurately report the visual stimuli. Participants should have 20/40 vision or better.
2. Ensure that all participants have normal hearing by testing hearing thresholds at 500, 1,000, 2,000, and 4,000 Hz in each ear. Auditory testing should be completed in a sound controlled room with an audiometer.
   1. To find a participant’s threshold, instruct the participant to raise their hand each time they detect a tone. Play a pulsed 500 Hz tone routed to the right ear starting at 35 dB and decrease the volume in 5 dB steps. Once a participant no longer detects a tone, increase the volume in 5 dB steps to verify the lowest perceptible volume. Repeat this procedure with each frequency, and then repeat all tone frequencies in the left ear. Participants should have thresholds of 20 dB or lower.
3. Ensure that participants are able to understand and comply with verbal instructions by measuring both IQ and receptive language skills with standardized neuropsychological measures prior to testing. Participants should have a measured IQ of 70 or more. If desired, additional neuropsychological testing can be completed at this time.

4. Simultaneity Judgment (SJ)

NOTE: The SJ task is a two alternative forced-choice task (2-AFC) and consists of a visual ring and 1,000 Hz auditory tone presented at various SOAs (negative = auditory preceding visual, positive = visual proceeding auditory) presented in random order.

1. Be sure to include fairly large SOAs (at least -400 to +400 msec) to get an accurate measurement of the full width of the TBW (typical stimulus set: -400, -300, -200, -150, -100, -50, 0, 50, 100, 150, 200, 300, 400 msec SOA). Use the same set of SOAs for each participant, which allows for easier comparison of task performance across participants. Present a minimum of 20 trials per SOA for a precise estimate. The task takes approximately 15 - 20 min to complete. Provide a short break every 100 trials to reduce participant fatigue.
2. Instruct the participant to observe a flash and a beep and explain that their task is to decide if the flash and beep occurred at the same time or at different times. Instruct the participant to press “1” on the number pad, if the stimuli occurred at the same time, or “2”, if the stimuli occurred at a different time.
   NOTE: If a response box is available, this may also be used for collecting responses. Include these same instructions on the response screen after each trial.
3. As an alternative substitute the flash and beep with a visual and auditory speech token (mouthed “ba” and voiced “ba”) and present at the same SOAs with the same task instruction (“Same time or different time?”). In this manner, compare the TBW for stimuli varying in complexity and social content within individual subjects²⁷.

5. Temporal Order Judgment (TOJ)

NOTE: The auditory TOJ task is a 2-AFC task used to examine the temporal acuity of auditory processing. The visual TOJ task is a 2-AFC task used to examine the temporal acuity of visual processing. The multisensory TOJ task is a 2-AFC task used to examine temporal acuity across audition and vision. Each task takes approximately 10 - 15 min to complete.

1. In the auditory TOJ task, instruct the participant to listen to two beeps presented (500 Hz and 1,000 Hz) at various delays and ask the participant to press “1” if the higher tone is played first or press “2” if the lower tone is played first. Present 20 trials for each SOA in random order.
   NOTE: Compared to the SJ task, there is a much smaller dynamic range of SOAs over which perception of unisensory auditory and visual temporal order changes, therefore use a stimulus set where smaller SOAs are more heavily represented (typical stimulus set: -250, -200, -150, -75, -50, -35, -20, -10, 10, 20, 35, 50, 75, 150, 200, 250 msec SOA, where negative = higher tone preceding lower tone, positive = lower tone proceeding higher tone) for the unisensory TOJ tasks.
2. In the visual TOJ task, instruct the participant to observe two circles (above and below a central fixation crosshair) at various delays and ask the participant to press “1” if the top circle appears first or press “2” if the bottom circle appears first. Present 20 trials for each SOA in random order.
   NOTE: In this task, negative SOAs indicate that the top circle was presented first and positive SOAs indicate that the bottom circle was presented first.
3. In the audiovisual TOJ task, instruct the participant to observe a small central flash and listen to a single tone (1,000 Hz) at various delays and ask the participant to press “1”, if the beep was presented first or press “2” if the flash was presented first. Present 20 trials for each SOA in random order.
   NOTE: Accuracy in the audiovisual TOJ is typically much worse compared to the unisensory auditory TOJ and visual TOJ tasks. This requires a wider range of SOAs compared to the unisensory TOJ tasks (typical stimulus set: -300, -250, -200, -150, -100, -80, -50, -20, 0, 20, 50, 80, 100, 150, 200, 250, 300). In this task, negative SOAs indicate that the auditory was presented first and positive SOAs indicated that the visual was presented first.
   NOTE: As with the SJ task, the TOJ task can be adapted to examine temporal processing across multiple kinds of stimuli. Here the TOJ task was completed with simple stimuli (auditory beeps and visual flashes), but this can be expanded to look at other stimulus pairs like speech and biological motion²⁴.

6. McGurk Task

NOTE: The McGurk illusion consists of a video of the visual syllable “ga” paired with an auditory recording of the syllable “ba”. Many subjects will actually fuse the visual and auditory syllables and perceive this pair as the syllable “da” or “tha”³².

1. Instruct the participant to observe different syllables and ask the participant to report the syllable that they perceived. In one block present 20 trials each of the unisensory syllables (auditory only syllables (A-“ba”, A-“ga”) and visual only syllables (V-“ba”, V-“ga”) in random order. In a second block, present 20 trials each of the audiovisual syllables (AV-“ba”, AV-“ga”, and the A-“ba”/V-“ga” McGurk stimulus) in random order. Ask the participant to press the letter on the keyboard corresponding to the perceived syllable (“press b for ba, press g for ga, press d for da, press t for tha”). This task takes approximately 5 - 10 min total to complete.
2. A more conservative estimate consists of an open response format³³ in which the participant reports out loud the perceived syllable and the response is recorded by the experimenter.

Results

This task battery has proven very successful in measuring individual differences in temporal processing in individuals with and without ASD^17,18,23,27. For the SJ task, plot the resulting data from each individual subject by first calculating the proportion of responses at each SOA that subject responded “synchronous” and then fitting the resulting response curve with a Gaussian curve. As illustrated in Figure 1A, there is a window of time in which visual-auditory stimuli pairs can...

Discussion

The manuscript describes elements of a psychophysical task battery that are used to assess temporal processing and acuity in sensory and multisensory systems research. The battery has wide applicability for a number of populations and has been used by our laboratory in order to characterize audiovisual temporal performance in typical adults¹⁸, children^10,39, and in children and adults with autism^17,23. In addition, it has been used to examine how various facets of the battery relate to on...

Materials

Name	Company	Catalog Number	Comments
Oscilloscope
Photovoltaic cell
Microphone
Noise-cancelling headphones
Chin rest
Audiometer