Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Summary

Presented here is a protocol introducing a set of child-friendly statistical learning tasks geared towards examining children’s learning of temporal statistical patterns across domains and sensory modalities. The developed tasks collect behavioral data using the web-based platform and task-based functional magnetic resonance imaging (fMRI) data for examining neural engagement during statistical learning.

Abstract

Statistical learning, a fundamental skill to extract regularities in the environment, is often considered a core supporting mechanism of the first language development. While many studies of statistical learning are conducted within a single domain or modality, recent evidence suggests that this skill may differ based on the context in which the stimuli are presented. In addition, few studies investigate learning as it unfolds in real-time, rather focusing on the outcome of learning. In this protocol, we describe an approach for identifying the cognitive and neural basis of statistical learning, within an individual, across domains (linguistic vs. non-linguistic) and sensory modalities (visual and auditory). The tasks are designed to cast as little cognitive demand as possible on participants, making it ideal for young school-aged children and special populations. The web-based nature of the behavioral tasks offers a unique opportunity for us to reach more representative populations nationwide, to estimate effect sizes with greater precision, and to contribute to open and reproducible research. The neural measures provided by the functional magnetic resonance imaging (fMRI) task can inform researchers about the neural mechanisms engaged during statistical learning, and how these may differ across individuals on the basis of domain or modality. Finally, both tasks allow for the measurement of real-time learning, as changes in reaction time to a target stimulus is tracked across the exposure period. The main limitation of using this protocol relates to the hour-long duration of the experiment. Children might need to complete all four statistical learning tasks in multiple sittings. Therefore, the web-based platform is designed with this limitation in mind so that tasks may be disseminated individually. This methodology will allow users to investigate how the process of statistical learning unfolds across and within domains and modalities in children from different developmental backgrounds.

Introduction

Statistical learning is an elementary skill supporting the acquisition of rule-governed combinations in language inputs¹. Successful statistical learning ability in infants predicts later language learning success^2,3. Variability in statistical learning skills in school-aged children has also been associated with vocabulary⁴ and reading^5,6. Difficulty in statistical learning has been proposed as one etiological mechanism underlying language impairment⁷. Despite the association between statistical learning and language outcomes in both neurotypical and atypical populations, the cognitive and the neural mechanisms underlying statistical learning remain poorly understood. In addition, previous literature has revealed that, within an individual, statistical learning ability is not uniform but independent across domains and modalities^6,8,9. The developmental trajectory of statistical learning abilities may further vary across domains and modalities¹⁰. These findings emphasize the importance of assessing individual differences in statistical learning across multiple tasks throughout the course of development. However, the field first requires a more systematic investigation of the relationship between statistical learning and first language development. To address these questions, we apply innovative methods including a web-based testing platform¹¹ that reaches a large number of children, and laboratory-based neuroimaging techniques (functional magnetic resonance imaging, or fMRI) that examine the real-time encoding of statistical information.

Standard measures of statistical learning begin with a familiarization phase and are followed by a two-alternative forced choice (2-AFC) task^12,13. The familiarization phase introduces a continuous stream of stimuli embedded with statistical regularities, where some stimuli are more likely to co-occur than others. The presentation of these co-occurring stimuli follows a fixed temporal order. Participants are passively exposed to the stream during the familiarization phase, followed by a 2-AFC task that tests whether the participant successfully extracted the patterns. The 2-AFC accuracy task presents two consecutive sequences: one sequence has been presented to the participant during the familiarization phase, while the other is a novel sequence, or contains part of the sequence. Above-chance accuracy on the 2-AFC would indicate successful learning at the group level. Traditional behavioral tasks assessing statistical learning generally rely upon accuracy as the outcome measure of learning. However, accuracy fails to account for the natural learning of information as it unfolds in time. A measure of real-time learning is necessary to tap into the implicit learning process of statistical learning during which children are still encoding the regularities from the inputs^14,15,16. Various adaptations across paradigms have been developed in an effort to move away from the 2-AFC measure, towards measures of on-line learning through behavioral responses during the exposure¹⁶. Studies utilizing these adaptations which measure reaction time during the exposure phase found they were related to post-learning accuracy¹⁷ with better test-retest reliability compared to that of the accuracy in adult learners¹⁸.

Neural measures are also foundational to our understanding of how learning unfolds over time, as the implicit process by which language learning occurs likely recruits different neural resources from those used once language is learned¹⁹. Neural measures also provide insights into differences in cognitive specializations underlying language ability across special populations²⁰. How the condition contrast is designed in an fMRI study is crucial for how we interpret patterns of neural activation during learning. One common practice is to compare brain responses during the familiarization phase between sequences containing regular patterns versus those containing the same stimuli which are ordered randomly. However, previous research implementing such a random control condition found no evidence for learning in behavior, despite neural differences between structured and random sequences. This might be due to the interference of random sequences on learning of structured sequences, as both were constructed from the same stimuli^21,22. Other fMRI studies which utilized backward speech or earlier learning blocks as the control condition confirmed learning took place behaviorally^19,23. However, each of these paradigms introduced its own confounding factor, such as the effect of language processing for the former case and the effect of the experimental order for the latter case. Our paradigm uses the random sequence as the control condition but mitigates their interference on participants’ learning of the structured sequences. Our fMRI paradigm also implements a mixed block/event-related design, which allows for the simultaneous modeling of transient trial-related and sustained task-related BOLD signals²⁴. Lastly, and more broadly, neural measures allow for the measurement of learning in populations where eliciting an explicit behavioral response may be difficult (e.g., developmental and special populations)²⁵.

The current protocol adopts a response time measure, in addition to traditional accuracy measures, and examines brain activation during the familiarization phase. The combination of these methods aims to provide a rich dataset for the investigation of real-time learning processes. The web-based platform offers a set of learning measures by including both response time during the exposure phase and accuracy of the 2-AFC task during the test phase. The neuroimaging protocol allows for the investigation of the underlying neural mechanisms supporting statistical learning across domains and modalities. While it is optimal to measure statistical learning within an individual using both the web-based and fMRI protocols, the tasks are designed so that they may be disseminated independently, and therefore, as two independent measures of statistical learning. The fMRI experiments included in the current protocol can help clarify how stimulus encoding, pattern extraction, and other constituent components of statistical learning are represented by particular brain regions and networks.

Protocol

All participants gave written consent to participate and study was conducted in accordance with the Institutional Review Board.

1. Overview of the statistical learning paradigm utilized in the web-based protocol

1. Include four tasks in the current paradigm: image (visual-nonlinguistic), letter (visual-linguistic), tone (auditory-nonlinguistic), and syllable (auditory-linguistic).
   1. Construct stimuli for visual tasks using 12 standalone alien cartoon images (image) and 12 letter images (letter; B, J, K, A, H, C, F, E, J, G, D, M) showing the same alien holding up 12 signs with capital letters written on them.
   2. Construct auditory stimuli using 12 English syllables (syllable; pi,pu,pa,ti,tu.ta,di,du,da,bi,bu,ba) and 12 musical tones within the same octave (tone; F,G,D,G#,C#,B,C,F#,D#,E,A,A#). The syllable stimuli can be made using an artificial speech synthesizer, and can be recorded as separate files in Praat^26,27.
2. In the familiarization phase, present stimuli in a structured stream (see Figure 1). Feedback is not provided at any point during the familiarization or test phase.
   NOTE: Within each task, a familiarization phase is immediately followed by a test phase.
   1. For the Image (visual-nonlinguistic) task, structure 12 images into four target triplets. In the familiarization phase, repeat each of the four target triplets 24 times for a total of 96 triplets.
      NOTE: The 96 triplets are randomly concatenated into a continuous stream, with the constraint that no triplet can be immediately repeated. Images are presented one at a time in the center of the screen. Each image is presented for 800 ms with 200 ms of inter-stimulus interval. The whole familiarization phase will last for 4 min 48 s.
   2. Ensure the test phase always follows the familiarization phase and is composed of 32 two-alternative forced-choice (2AFC) questions. For each question, include 2 options: a target triplet from the familiarization phase and a triplet that was not included in the familiarization phase, referred to as a foil triplet.
      NOTE: Foil triplets are constructed so that the relative position of each image in the foil triplet is the same as the target triplet. Each target and foil triplet are presented 8 times total in a test, and each foil-target pair is repeated. The test phase consists of 32 (4 target triplets x 4 foil triplets x 2 repetitions) randomly ordered trials.
   3. For the letter (visual-linguistic) task include 12 images of capitalized letters that are organized into four target triplets (GJA, FKC, LBE, and MDH).  For the test phase, create 4 foil triplets (GDE, FJH, LKA, and MBC) and pair them with the target triplets to form the 32 2AFC test trials. No letter triplet can contain any words, common acronyms, or initialisms.
   4. For the tone (auditory-nonlinguistic) task include 12 musical pure tones within the same octave (a full chromatic scale starting from middle C) and concatenate them into four target triplets (F#DE, ABC, C#A#F, and GD#G#). Unlike in the visual tasks, presentation speed is faster due to differences in auditory perceptual preference^6,28,29.
      NOTE: Each of the four target triplets is repeated 48 times for a total of 192 triplets (twice as much as the visual conditions). All triplets are concatenated into a sound stream with no triplet being repeated twice in a row. Pure tones are presented one at a time while participants view a blank screen. The duration of each tone is 460 ms with a 20 ms inter-stimulus interval. The whole stream lasts about 4 min and 36 s. As in the visual tasks, a test phase of 32 2AFC trials with pairs of target and foil triplets (F#BF, AA#G#, C#D#E, GDC) immediately follows the familiarization phase.
   5. For the syllable (auditory-linguistic) task use 12 consonant vowel (CV) syllables created and grouped into four target triplets (pa-bi-ku, go-la-tu, da-ro-pi, and ti-bu-do). The duration of each syllable and the inter-stimulus interval is the same as the tone condition. Pair four foil triplets (pa-ro-do, go-bu-ku, da-bi-tu, and ti-la-pi) with the target triplets in the test phase.
3. Randomize the order of the four statistical learning tasks across participants.

2. Participant recruitment

NOTE: While the web-based protocol and the fMRI protocol are best implemented together within a single participant, here we outline the best practices for participant recruitment for each task independently.

1. Web based participant recruitment
   1. Recruit participants age 6 years and above. Participants of any sex, race, and ethnicity may participate; however, study sample should be representative of the population.
   2. Recruit participants who are a native English speaker and have been exposed to no languages besides English before the age of 5.
   3. Ensure they report no known psychological (including ADD, depression, PTSD, and clinical anxiety) and/or neurological condition (including stroke, seizure, brain tumor, or closed head injury).
   4. Ensure participants have normal or corrected-to-normal vision (glasses or contacts are okay), normal color vision and normal hearing (no hearing aid or cochlear implant devices).
2. Task based fMRI participant recruitment
   1. Recruit participants age 6 years and above. Participants of any sex, race, and ethnicity may participate; however, study sample should be representative of the population.
   2. To be eligible, recruit participants who are native English speaker and have never been exposed to any languages besides English before the age of 5.
   3. Recruit right-handed individuals, with no known psychological (including ADD, depression, PTSD, and clinical anxiety) and neurological condition (including stroke, seizure, brain tumor, or closed head injury).
   4. Exclude participants who are pregnant, claustrophobic, taking psychoactive drugs, or have any metal in the body (including pacemakers, neural implants, metal plates or joints, shrapnel, and surgical staples).
   5. Ensure participants have normal or corrected-to-normal vision (glasses or contacts are okay), normal color vision and normal hearing (no hearing aid or cochlear implant devices).
   6. Determine the eligibility to participate in the MRI by having participants (or parents if the participant is a minor) complete an MRI Safety Screening Form.

3. Web based protocol

NOTE: The web-based statistical learning paradigm is hosted on a secure website (https://www.cogscigame.co¹¹) and developed using jsPsych, a JavaScript library for creating behavioral experiments online³⁰.

1. To reproduce tasks, go to DOI: 10.5281/zenodo.3820620. All scripts and materials are publicly available. Researchers can modify the scripts and run the experiments locally on any web browser as long as all the paths for the output files are set up appropriately.
2. Have participants complete a cover task where they are told to press a button when they see a certain target during all familiarization phases of each statistical learning task.
3. Target stimulus assignment for each task
   1. In the image, letter and syllable tasks, randomly choose one of the four triplets and assign the target to the third stimulus of the triplet. In the tone task, constrain the target stimulus to only the lowest or the highest tones of the third stimulus in the triplets, and assign the target to the third stimulus of the triplet. This is done because tone stimuli are relatively harder to discriminate than other types of stimuli.
   2. In the syllable and tone tasks, introduce participants to an alien and the favorite word/note in its alien language/folk music. Tell participants that they will listen to the alien’s language/music and to remember to press the spacebar whenever they hear the favorite word/note” by deleting the space between / and note.
   3. In the image task, tell participants to keep track of a special alien as a group of aliens line up to enter a spaceship. In the letter task, tell participants to keep track of the alien’s favorite sign as the alien holds up signs for a parade. Give participant’s a practice trial in both the image and letter tasks.
   4. Do not provide explicit instructions about the presence of triplets.
   5. Measure response time over the 24 trials in the visual tasks and over the 48 trials in the auditory tasks to assess online learning.
   6. During the test phase, both a target (included in familiarization phase) and foil triplet (not included in familiarization phase) are presented to the participant. Instruct participants to then choose which one of the two is more similar to what they saw or heard in the familiarization phase. Each trial must end with a response.
4. Behavioral measures of statistical learning in the web-based protocol
   1. Measure the real time learning during the familiarization phase via the linear slope of reaction time (change in reaction time throughout the familiarization phase).
   2. To be considered a valid response to the target, check that the keypress must be in the time window of one stimulus before and one stimulus after the target stimulus. That is -480 ms to +960 ms relative to the onset of the target in the auditory tasks and -1000 ms to +2000 ms in the visual tasks. A keypress prior to the target is considered as anticipation and thus yields a negative reaction time.
   3. To compare reaction times across conditions, transform the reaction times of each participant for each task into z scores. This normalizes the reaction times of an individual so that scores across tasks can be compared.
   4. Calculate a reaction time slope of each participant for each condition using linear regression. Input the z-normed reaction times as the dependent variable and the target trial order as the independent variable (visual: 1 to 24; auditory: 1 to 48). The slope of the linear regression line (Beta coefficient/Estimate) is the reaction time slope (RT slope).
   5. Measure offline accuracy of each participant for each condition by dividing the number of correct trials from the test phase by the total number of trials (32 trials).

4. Task based fMRI protocol

1. Modifications to the statistical learning paradigm (Figure 2).
   1. For each task, present both a structured sequence (containing statistical regularities) and a random sequence (no statistical regularities).
      NOTE: Structured sequences are identical to those described for the web-based protocol (see Figure 1). In contrast, random sequences contain the same 12 stimuli as presented in the structured sequences but are ordered pseudo-randomly. No combinations of any three stimuli are repeated more than once.
   2. Divide each sequence into six smaller blocks of equal length (24 stimuli for the visual tasks and 48 stimuli for the auditory tasks).
   3. Concatenate three structured blocks, 3 random blocks, and 6 resting blocks (silence with a blank screen) in a pseudorandom order to create four runs of auditory stimuli and four runs of visual stimuli. To maximize learning of the structured sequences, ensure that the random blocks in each run contain a different domain from the structured sequence (e.g., syllable structured sequences are presented together with tone random sequences in one run, and syllable random sequences are presented together with tone structured sequences in another run).
   4. Include 288 images to be presented in each run for the visual task lasting approximately 4.77 min. Include 576 sounds to be presented in the auditory task which lasts approximately 4.42 min. At the beginning of each block, present a cue about the target with a verbal and visual probe: “Now listen/look for the [TARGET]”.
   5. Among the four runs of the visual task, ensure that two contain structured sequences of images and the other two contain structured sequences of letters. Among the four runs of the auditory task, ensure that two contain structured sequences of syllables and the other two contain structured sequences of tones.
2. fMRI statistical learning procedure
   1. To help make participants, especially children, comfortable in the scanner, practice the MRI scanning session first using a mock scanner³¹. A mock scanner provides a naturalistic experience similar to the actual scanning session but is typically situated in a more child-friendly environment.
   2. First introduce the child to the mock scanner, i.e., brain camera, and ensure they are comfortable before putting them in the scanner.
   3. Introduce them to their “scan-buddy” and explain that the purpose of the scan buddy is to keep them accompanied and help them if they need anything. The scan buddy will gently remind the participant to keep still if too much motion is detected by the “camera”.
   4. Once they are in the scanner, play child-friendly videos to help them acclimate to the sound and video. When they are ready, play a few pre-recorded scanner sound clips to prepare them for the noises produced by the real MRI. During this time have them practice staying still and working with the scan buddy.
   5. Introduce children to the statistical learning paradigm and have them practice outside of the scanner. This is done by having children complete a brief portion of the task on a computer, similar to the web-based protocol by performing steps 3.2.2 and 3.2.3 mentioned above.
      NOTE: The practice stimuli are the same as those utilized in the task; however, children are only exposed to the random sequence and not the structured sequences, allowing for brief habituation to the stimuli and task demands without enabling learning of particular sequences.
   6. Ensure the fMRI data collection protocol is appropriately set up on the MRI acquisition computer.
      NOTE: The acquisition parameters follow the recommendations of the Adolescent Brain Cognitive Development (ABCD) Study³².
   7. Begin the scanning session with high resolution T1-weighted scans. Acquire these using a 176-slice 3D MPRAGE (Magnetization Prepared Rapid Gradient Echo) volume scan with TR (Repetition Time) = 2500 ms, TE (Echo Time) = 2.9 ms, flip angle = 8˚, FOV (Field of View) = 25.6 cm, 256 X 256 matrix size, and 1 mm slice thickness. This acquisition will last 7.2 min.
   8. To acquire functional data, use T2*-weighted echo-planar imaging with simultaneous multi-slice scans acquisition with TR= 800 ms, TE = 32 ms, flip angle = 61˚, FOV = 21 cm, and matrix = 64 x 64. In this experiment, 60 adjacent slices are acquired in an interleaved sequence with 2.5 mm slice thickness, a 21 cm FOV, and a 64 X 64 matrix, resulting in an in-plane resolution of 2.5 mm x 2.5 mm x 2.5 mm.
   9. Have participants lie comfortably on the bed of the fMRI scanner with headphones that protect their ears from the scanner noise and a response pad/button box in their hand (both headphones and button box must be scanner compatible).
   10. Place additional padding around their head to ensure limited head motion during data collection. Give the button response box to the participant ahead of time to record responses and counterbalance whether the left or right hand is used to press buttons across participants.
   11. Give every child an option of a scan buddy. For older, neurotypical children who are comfortable without a scan buddy, give them a squeeze ball to notify the experimenter if they are distressed or need to stop. Give younger children and special populations a squeeze ball but also provide them with a scan buddy to assist them (described in 4.2.3).
   12. Place the head coil over the participant’s head and align the patient’s position in the bed.
   13. On the acquisition computer register a new participant. Enter their participant ID, date of birth, weight and height. The participant may now be inserted into the bore of the MRI.
   14. Acquire T1-weighted scan while showing participants a movie.
   15. Before beginning the statistical learning paradigm, give participants the instructions of each task by speaking to them through an intercom system connected to their headphones.
   16. In the auditory tasks, tell participants: “Now we’re going to play a button-pressing game. You will hear the aliens say words and play music. Remember to press the button in your LEFT/RIGHT hand whenever you hear the sound you are listening for. There will be 4 parts, and each part will last about 5 min.”
   17. In the visual tasks, tell participants: “Now you are going to see the pictures of the aliens and the letters. Whenever you see the picture you are looking for, press the button in your LEFT/RIGHT hand. You will play this 4 times in a row. It will take about 5 minutes each time.”
   18. Start the statistical learning paradigm on the presentation computer and acquire the task fMRI data.
   19. Once the participant has completed the paradigm, stop the MRI, safely remove them from inside the scanner, and remove the head coil.
   20. After data collection, transfer all MRI data from the acquisition computer to a secured server for further analyses.
3. fMRI data analyses
   1. Analyze in-scanner reaction time during the fMRI task similarly to the web-based calculation of reaction time during the familiarization phase. Normalize reaction time to compare across conditions, and calculate a linear slope using the normalized reaction time for each condition of an individual.
   2. When analyzing the fMRI data, first organize and convert data to Brain Imaging Data Structure³³ (BIDS) formatting using HeuDiConv³⁴ (https://github.com/nipy/heudiconv).
   3. Preprocess these data using fMRIPrep^35,36. This automated preprocessing pipeline combines methodology from AFNI³⁷, ANTs³⁸, Freesurfer³⁹, FSL⁴⁰, and Mindboggle⁴¹ to provide scientifically rigorous and reproducible data for use in data analysis.
      NOTE: The current study implements a mixed block/event-related design. The representative results (below) treat each mini block as an event (e.g., random sequence is an event, structured sequence is an event, etc.). However, the task is also designed so that one can model each stimulus as an event.
   4. Include two task regressors for each run (“image” and “letter” for the visual condition, and “syllable” and “tone” for the auditory condition) in the first-level model design. Determine task regressors by convolving a vector of event onset times with their durations with a canonical hemodynamic response function. Compute differences and means between runs within each subject for higher-level model designs. This will result in a contrast between structured and random sequences within each type of stimuli.
   5. Create a group mean of activation for structured blocks compared to random blocks within each modality/domain.

Results

Web-based Behavioral Results
Given the current protocol is designed for easy dissemination with developmental populations, we have included preliminary web-based results based on data from 22 developing school-aged children (Mean (M) age = 9.3 years, Standard Deviation (SD) age = 2.04 years, range = 6.2-12.6 years, 13 girls). In the web-based statistical learning task, children performed significantly better than 0.5 chance-level on all conditions, indicating successful statistical learning at the ...

Discussion

The methods presented in the current protocol provide a multimodal paradigm for understanding the behavioral and neural indices of statistical learning across the course of development. The current design allows for the identification of individual differences in statistical learning ability across modalities and domains, which can be used for future investigation of the relationship between statistical learning and language development. Since an individuals’ statistical learning ability is found to vary across dom...

Materials

Name	Company	Catalog Number	Comments
4 Button Inline Response Device	Cambridge Research Systems	SKU: N1348	An fMRI reponse pad used for measuring in-scanner response time
Short/Slim Canal Tips	Comply Foam	SKU: 40-15028-11	Short & slim in-ear canal tips are recommended for children to protect hearing and allow for them to hear the stimuli while in the scanner.
jsPsych	jsPsych	https://www.jspsych.org/	jsPsych is a JavaScript library for running behavioral experiments in a web browser.
Speech Synthesizer	Praat	Version 6.1.14	This program is an artificial speech synthesizer which was used to create the syllable stimuli.
Web-based statistical learning tasks	Zenodo	http://doi.org/10.5281/zenodo.3820620 (2020).	All web-based statistical learning tasks are available for free access on Zenodo.