Development and validation of the L2 word learning protocol was supported by a National Defense Science and Engineering Graduate (NDSEG) Fellowship (32 CFR 168a) issued by the US Department of Defense Air Force Office of Scientific Research. Development and validation of the cartoon retelling protocol with adolescents with ASD was supported by a Ruth S. Kirschstein Institutional National Research Service Award (T32) from the National Institutes of Mental Health. The author thanks Rachel Fader, Theo Haugen, Andrew Lynn, Ashlie Caputo, and Marco Pilotta with assistance with data collection and coding.

All participants provided written consent, and all protocols were approved by the Institutional Review Boards at the host institution. The L2 word learning and cartoon retelling protocols were implemented in the studies on which the representative results are based21,33. Although these protocols have been conducted only in in-person contexts to date, a related protocol manipulating the visibility of the interlocutor and the participant independently not described here is currently being conducted to provide insight into replicability in this context, with plans to expand it to populations with communication challenges, such as individuals with ASD.
1. Participants
NOTE: While respecting necessary constraints on participant characteristics for each study, samples should be representative of the larger population insofar as possible.
<ol>
	<li>Participant recruitment for L2 word learning protocol
	<ol>
		<li>Recruit participants aged 18 years and above with normal or corrected-to-normal vision and normal hearing who are L1 speakers of English. To ensure that the study examines the impact of gesture on initial-stage L2 word learning, recruit participants without any knowledge of Hungarian, the target language.</li>
	</ol>
	</li>
	<li>Participant recruitment for cartoon retelling protocol
	<ol>
		<li>Recruit participants of an age capable of completing the experimental task (10-20 years of age in Morett et al.21.) with a diagnosis of ASD or who are typically developing (TD).
		<ol>
			<li>To ensure that participants can complete the experimental task successfully, recruit participants who are L1 speakers of English, have normal or corrected-to-normal vision and normal hearing, and have a full-scale IQ of 85 or above, as measured before or at the time of the study via a standardized assessment46.</li>
			<li>To ensure that the study examines the specific effects of ASD on gesture production, recruit participants with ASD without epilepsy, meningitis, encephalitis, diabetes, childhood disintegrative disorder, or Pervasive developmental disorder-not otherwise specified (PDD-NOS).</li>
			<li>To ensure that gesture production in ASD can be compared to typical gesture production, recruit TD participants without comorbid psychiatric disorders or first-degree relatives diagnosed with pervasive developmental disorders.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
2. Overview of visibility manipulation
<ol>
	<li>Recruit participants in pairs if the effects of producing and observing spontaneous gestures will be examined, as in the L2 word learning protocol. Recruit participants individually in coordination with a research assistant acting as a participant if only the effects of producing spontaneous gestures will be examined, as in the cartoon retelling protocol.</li>
	<li>Provide an informed consent form with information about the study to participants to sign and answer any questions about it, taking care not to reveal the focus of the study on gesture production. Collect the form after participants have signed it.</li>
	<li>For participants recruited in pairs, use a selection method such as a coin toss to determine which participant will serve as the speaker, who will do most of the talking in the experimental task, and which participant will serve as the interlocutor, who will do most of the listening and occasionally interact with the speaker in the experimental task. For participants recruited individually with a research assistant acting as a participant, use a fixed selection method such as a rigged coin toss to ensure that the participant always serves as the speaker and that the research assistant always serves as the interlocutor.</li>
	<li>In half of the experimental sessions, place an opaque divider between the speaker and the interlocutor to ensure they cannot see one another&#39;s face or hands.</li>
	<li>Explain to participants what they will do in the experimental task and answer any questions about it. Inform participants that their speech will be audio recorded, pointing out the microphone used for this purpose. To avoid any effects of awareness of video recording on gesture production, do not inform participants that they will be video recorded and ensure that the video camera used for this purpose is not visible to participants.</li>
	<li>Initiate video and audio recording by pressing the appropriate buttons. Implement the experimental task to elicit speech and gesture from the speaker and the participant interlocutor, if there is one.</li>
	<li>If a within-participants design is employed, remove the opaque divider if used in the first iteration or place it between the speaker and interlocutor if not used in the first iteration.</li>
	<li>Repeat the experimental task as needed until all stimuli have been exhausted.</li>
	<li>Stop video and audio recording by pressing the appropriate buttons. Inform participants that the study focused on gestures and that they were video recorded and offer to delete their videos prior to data analysis if desired.</li>
	<li>Answer any questions from participants and thank them for participating before dismissing them.</li>
</ol>
3. L2 word learning protocol
<ol>
	<li>Video and audio record the speaker and interlocutor so that speech and gesture production during the experimental task can be analyzed later.
	<ol>
		<li>Place a table microphone between the speaker and the interlocutor for audio recording. Place a video camera to one side between the speaker and interlocutor at a 90&#176; angle and ensure it is not visible to participants.</li>
		<li>Additionally, ensure that both the speaker and interlocutor are encompassed within the scope of view of the video camera and that the computer screen is not visible in the scope of view of the video camera to maintain blindness to the condition in which L2 words are presented in the analysis.</li>
	</ol>
	</li>
	<li>Using a computer, present an L2 word from the target language aurally to the speaker via earphones to ensure that it is inaudible to the interlocutor so that the speaker can subsequently teach it to the interlocutor. At the same time, present either a video of an L1 speaker of the target language saying the L2 word and producing a representational gesture conveying its meaning with the text of the L2 word at the bottom of the screen or the L2 word as text in the middle of the screen without a video to the speaker.</li>
	<li>After a 500 ms pause, present the L1 translation of the L2 word as text in the middle of the screen to the speaker for 2000 ms.</li>
	<li>Prompt the explainer to teach the L2 word and its meaning to the interlocutor via speech and/or gesture.</li>
	<li>Instruct the speaker to press a button to move on when ready. After the speaker does so, present another L2 word. Repeat this process until all 20 L2 words have been presented in random order.</li>
	<li>Individually test the speaker&#39;s and interlocutor&#39;s knowledge of the L2 words presented in the experimental task by presenting each L2 word as text and prompting them to say its L1 translation or skip the word if they do not recall its translation.</li>
</ol>
4. Cartoon retelling protocol
<ol>
	<li>If the participant&#39;s IQ has not been measured via a standardized IQ assessment previously, administer one to ensure that their IQ is 85 or above.</li>
	<li>Video and audio record participants so that their speech and gesture production can be analyzed later. Place a table microphone in front of the participant for audio recording. Place a camera on one side of the participant at a 90&#176; angle and ensure that the camera is not visible to the participant.</li>
	<li>Present the first half of the Looney Tunes cartoon Canary Row to the participant. This half of the cartoon contains several vignettes in which Sylvester the cat attempts and fails to catch Tweety bird.</li>
	<li>Instruct the participant to retell the events of the first half of the cartoon to the interlocutor in as much detail as they can recall. Instruct the interlocutor to listen to the participant&#39;s retelling, prompting the participant to say more if it is brief.</li>
	<li>Repeat steps 4.1. and 4.2. for the second half of the Canary Row video, which contains several more vignettes in which Sylvester the cat attempts and fails to catch Tweety bird.</li>
</ol>
5. Transcription and coding
NOTE: To minimize bias, when possible, transcribe and code data blind without knowing which participants and trials were assigned to which conditions. Ideally, transcribers and coders should not be involved in data collection.
<ol>
	<li>Transcribe all participant speech orthographically by hand using appropriate software according to a standardized scheme, such as CHAT47, to produce a word count. Transcribe all verbal information, such as fillers and corrections, but do not transcribe pauses.</li>
	<li>Identify and code all participant gestures to produce a count for each gesture type. Identify gestures from rest to rest and categorize them into four types based on McNeill&#39;s taxonomy2,3: iconic, metaphorical, beat, and deictic. Note that some gestures may be classified as more than one type.</li>
	<li>Compute the rate of gestures per hundred words by dividing the number of representational gestures by the number of words and multiplying it by 100 for each vignette.</li>
	<li>Have an independent coder code participant gestures for 25% of the data. Compute percent agreement for data coded by both coders and ensure that it is at least 80% to ensure sufficient interrater reliability.</li>
</ol>

Protocol for Assessing Gestures Under Communication Challenges

The author has no competing financial interests to declare.

The current protocol manipulates the visibility of the speaker and interlocutor to one another, providing insight into its impact on gesture production under challenging circumstances: L2 word learning and narrative retelling by adolescents with ASD. This protocol can be implemented either in-person or virtually, permitting participant and interlocutor visibility to be manipulated in tandem or independently. It can accommodate a wide variety of experimental tasks, gestures, and populations, providing them with the flexib...

Co-speech gestures (heretofore, gestures) - meaningful hand movements produced concurrently with speech - contribute to interpersonal communication by conveying information complementing verbal content1. According to the most widely used taxonomy2,3, gestures can be divided into three categories: representational gestures, which convey referents via their form and motion (e.g., flapping the hands back and forth together to convey flying); beat gestures, which convey emphasis via simple punctate movements (e.g., moving the dominant hand downward slightly in conjunction with each word in the phrase &#34;right now&#34;); and deictic gestures, which draw attention to the presence or absence of an entity via pointing (e.g., swinging the thumb backward to indicate something behind oneself). Representational gestures can be further divided into two additional categories: iconic gestures, which convey concrete referents (e.g., a bird), and metaphorical gestures, which convey metaphorical referents (e.g., ecstasy). Because gesture and speech arise from the same conceptual content, they are closely related in meaning4,5. By serving as an iconic visual medium for communication, gestures can help compensate when challenges in spoken language comprehension arise6, whether due to individual factors such as limited proficiency in the language spoken7,8 or environmental factors such as difficulty in hearing speech9,10. Thus, gestures are integral to understanding interpersonal communication in face-to-face and virtual contexts, providing insight into how speakers and listeners convey and comprehend information multimodally.
Several structured interactive tasks have been developed to measure how gesture affects interpersonal communication. These tasks include interviews, in which participants respond to questions by describing their personal experiences11,12; image description, in which a static image is presented to participants to describe13,14; puzzle solving, in which participants describe how to solve a puzzle by spatially orienting components correctly15,16; direction provision, in which participants are instructed to give directions to a location unfamiliar to listeners17,18,19; and narrative retelling, in which participants view a cartoon depicting a sequence of events and subsequently recount them20,21,22. Many of the tasks employed in the studies cited above incorporate spatial content and action, which gesture conveys with particular efficacy15,23,24,25. In the tasks for the studies cited above, participants typically produce gestures in conjunction with language, and information conveyed via gesture and its relationship to concurrently produced language is assessed. Alternatively, participants may view a recording of someone gesturing (or not gesturing) and producing language when completing one of these tasks, after which participants&#39; comprehension is assessed. All of these tasks may be conducted virtually as well as face-to-face, enabling data collection from a wide range of participants and comparison across modalities.
The impact of gestures on interpersonal communication has been implemented with a wide range of participants. Of particular interest have been populations with communication challenges, including young children26,27,28,29,30, second language (L2) users8,31,32,33, individuals with autism spectrum disorder (ASD)20,21,22,34, individuals with specific language impairment35,36,37, individuals with aphasia12,38, individuals with brain injury39, and individuals who stutter40,41. This work has revealed that, while many of these populations can utilize gestures to facilitate communication, some, such as individuals with autism spectrum disorder, may encounter difficulty leveraging gestures to communicate effectively. The findings suggest that this may be due to the extent to which these populations take audience design and environmental cues into consideration, highlighting the practical implications of these explanations for the impact of gestures on communication.
A key feature providing insight into the impact of gestures on interpersonal communication is interlocutor visibility. A question of great importance in the field of gesture research is the extent to which participants gesture for their own benefit, indicating the offloading of cognitive operations onto the body, vs. the benefit of their interlocutors, indicating the use of gesture to communicate. This question has been investigated by examining the impact of interlocutor (non-)visibility on gesture production in face-to-face contexts via the use of an opaque partition26,42, as well as in telephone contexts13 and virtual contexts43. Overall, the results of this work indicate that, although speakers gesture for their own benefit, they produce more representational gestures when communicating with visible than non-visible interlocutors, whereas beat gesture production is similar regardless of interlocutor visibility. Thus, they suggest that representational gesture facilitates communication across a variety of contexts, suggesting that speakers take the perspective of their interlocutors into account and modify their gesture production accordingly. Although previous research examining the effect of interlocutor visibility has been instrumental in providing insight into the contributions of gesture to interpersonal communication, it has focused on English as a first language (L1) in typically developing populations, so it is unclear whether the findings can be extended to populations with communication challenges. Two such populations are L2 learners, who may struggle to communicate via speech in the target language, and children with ASD, whose verbal and nonverbal communication are abnormal. Furthermore, little research has examined the impact of interlocutor visibility on gesture production in virtual contexts, which allow the effect of the interlocutor&#39;s visibility on the participant to be disentangled from the effect of the participant&#39;s visibility to the interlocutor, so the replicability of findings from these contexts is currently unclear. Finally, some research has focused on the impact of interlocutor visibility on the production of specific types of gestures, so it is unclear whether the production of other types of gestures is similarly affected by interlocutor visibility.
The protocols described below manipulate interlocutor visibility to examine gesture production in challenging circumstances: L2 word learning and narrative retelling by individuals with ASD. The L2 word learning protocol bridges research examining the impact of observing gestures on L2 word learning with research examining the contribution of gestures to communication via an interactive word learning paradigm. In this paradigm, participants unfamiliar with the target language learn words in it and then teach these words to other participants unfamiliar with the target language, permitting the impact of interlocutor visibility on gesture production to be examined in the earliest stages of L2 acquisition in a conversational context. The cartoon retelling paradigm employs a widely used narrative retelling task in which differences in gesture production have been observed when interlocutor visibility is manipulated with a new population: adolescents with ASD. This population is of interest because language development, including gesture production, is mature by adolescence, and ASD entails difficulty with verbal and nonverbal communication, including gestures, as well as a lack of sensitivity to the communicative needs of interlocutors. Together, these protocols provide insight into the extent to which gesture production is dependent on - and, conversely, can compensate for - speech when interpersonal communication is challenging.
Based on findings demonstrating how listener visibility affects gesture production by L1 English speakers13,26,42,43, it was hypothesized that participants would produce more overall gestures and more representational gestures when discussing L2 words with a visible than a non-visible interlocutor. Based on findings demonstrating abnormalities in gesture production&#160;in ASD20,44, it was hypothesized that adolescents with ASD would produce fewer overall gestures as well as fewer representational and deictic gestures than typically developing (TD) adolescents. Moreover, based on findings showing that ASD entails difficulty with perspective taking45, it was hypothesized that gesture production by adolescents with ASD would not differ significantly in the presence of visible and non-visible interlocutors. Finally, an interaction between diagnosis and visibility was predicted, such that gesture production would not differ by visibility for adolescents with ASD but that it would for TD adolescents.

<table><tbody><tr><td>Computer</td><td>Apple</td><td>Z131</td><td>24&quot; iMac, 8-core CPU &amp;amp; GPU, M3 chip</td></tr><tr><td>Conference USB microphone</td><td>Tonor</td><td>B07GVGMW59</td><td></td></tr><tr><td>ELAN</td><td>The Language Archive</td><td></td><td>Software application used to transcribe speech and gesture</td></tr><tr><td>Video Recorder</td><td>Vjianger</td><td>B07YBCMXJJ</td><td>FHD 2.7K 30 FPS 24 MP 16X digital zoom 3&quot; touch screen video recorder with renote control and tripod</td></tr><tr><td>Weschler Abbreviated Scale of Intelligence</td><td>Pearson</td><td>158981561</td><td>Used to verify full scale IQ &amp;ge; 80 in Morett et al. (2016)</td></tr></tbody></table>

examining gesture production in the presence of communication challenges

Understanding why speakers modify their co-speech hand gestures when speaking to interlocutors provides valuable insight into how these gestures contribute to interpersonal communication in face-to-face and virtual contexts. The current protocols manipulate the visibility of speakers and their interlocutors in tandem in a face-to-face context to examine the impact of visibility on gesture production when communication is challenging. In these protocols, speakers complete tasks such as teaching words from an unfamiliar second language or recounting the events of cartoon vignettes to an interlocutor who is either another participant or a confederate. When performing these tasks, speakers are visible or non-visible to their interlocutor, and the speaker is visible or non-visible to the participant. In the word learning task, speakers and interlocutors visible to one another produce more representational gestures, which convey meaning via handshape and motion, and deictic (pointing) gestures than speakers and interlocutors who are not visible to one another. In the narrative retelling protocol, adolescents with autism spectrum disorder (ASD) produced more gestures when speaking to visible interlocutors than non-visible interlocutors. A major strength of the current protocol is its flexibility in terms of the tasks, populations, and gestures examined, and the current protocol can be implemented in videoconferencing as well as face-to-face contexts. Thus, the current protocol has the potential to advance the understanding of gesture production by elucidating its role in interpersonal communication in populations with communication challenges.

L2 word learning 
Implementation: The results reported below are based on data collected from 52 healthy participants (21 males, 31 females; age: M = 20.15, SD = 1.73, range = 18-28) according to the protocol outlined above. All speech and gesture were coded by a primary coder, who could not be blind to visibility condition as data from both the speaker and interlocutor were recorded using a single camera, as described in step 3.4. To establish inter-rater reliability, a se...

Read this Scientific Article Protocol about Author Spotlight: Deciphering the Cognitive and Neural Mechanisms of Gesture in Communication at JoVE.com

Author Spotlight: Deciphering the Cognitive and Neural Mechanisms of Gesture in Communication  

Here, we present a protocol that manipulates interlocutor visibility to examine its impact on gesture production in interpersonal communication. This protocol is flexible to tasks implemented, gestures examined, and communication modality. It is ideal for populations with communication challenges, such as second language learners and individuals with autism spectrum disorder.

Examining Gesture Production in the Presence of Communication Challenges

Understanding why speakers modify their co-speech hand gestures when speaking to interlocutors provides valuable insight into how these gestures ...

examining-gesture-production-presence-communication

Research

JoVE Journal

Behavior

818 Views.  University of Missouri. Mijn onderzoek onderzoekt de cognitieve en neurale mechanismen van de impact van het lichaam op communicatie via handgebaren en gebarentaal. Een belangrijk aandachtspunt van mijn werk is hoe de relatie tussen gebaar en spraak verschilt in typische en atypische taalontwikkeling en -verwerking. In vivo neurowetenschappelijke technieken zoals elektro-encefalografie of EEG, functionele magnetische resonantiebeeldvorming of fMRI en functionele nabij-infraroodspect...

Video: Auteur in de schijnwerpers: het ontcijferen van de cognitieve en neurale mechanismen van gebaren in communicatie  

Corticospinal Excitability Modulation During Action Observation

This study used the transcranial magnetic stimulation/motor evoked potential (TMS/MEP) technique to pinpoint when the automatic tendency to mirror someone else's action becomes anticipatory simulation of a complementary act. TMS was delivered to the left primary motor cortex corresponding to the hand to induce the highest level of MEP activity from the abductor digiti minimi (ADM; the muscle serving little finger abduction) as well as the first dorsal interosseus (FDI; the muscle serving index finger flexion/extension) muscles. A neuronavigation system was used to maintain the position of the TMS coil, and electromyographic (EMG) activity was recorded from the right ADM and FDI muscles. Producing original data with regard to motor resonance, the combined TMS/MEP technique has taken research on the perception-action coupling mechanism a step further. Specifically, it has answered the questions of how and when observing another person's actions produces motor facilitation in an onlooker's corresponding muscles and in what way corticospinal excitability is modulated in social contexts.

Single-pulse transcranial magnetic stimulation over the primary motor cortex, neuronavigation, and registration of electromyographic activity of hand muscles were used in this study to explore corticospinal excitability while participants were observing action sequences.

This study used the transcranial magnetic stimulation/motor evoked potential (TMS/MEP) technique to pinpoint when the automatic tendency to mirror someone ...

A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis (ALS)

Improved methods for assessing bulbar impairment are necessary for expediting diagnosis of bulbar dysfunction in ALS, for predicting disease progression across speech subsystems, and for addressing the critical need for sensitive outcome measures for ongoing experimental treatment trials. To address this need, we are obtaining longitudinal profiles of bulbar impairment in 100 individuals based on a comprehensive instrumentation-based assessment that yield objective measures. Using instrumental approaches to quantify speech-related behaviors is very important in a field that has primarily relied on subjective, auditory-perceptual forms of speech assessment1. Our assessment protocol measures performance across all of the speech subsystems, which include respiratory, phonatory (laryngeal), resonatory (velopharyngeal), and articulatory. The articulatory subsystem is divided into the facial components (jaw and lip), and the tongue. Prior research has suggested that each speech subsystem responds differently to neurological diseases such as ALS. The current protocol is designed to test the performance of each speech subsystem as independently from other subsystems as possible. The speech subsystems are evaluated in the context of more global changes to speech performance. These speech system level variables include speaking rate and intelligibility of speech.
The protocol requires specialized instrumentation, and commercial and custom software. The respiratory, phonatory, and resonatory subsystems are evaluated using pressure-flow (aerodynamic) and acoustic methods. The articulatory subsystem is assessed using 3D motion tracking techniques. The objective measures that are used to quantify bulbar impairment have been well established in the speech literature and show sensitivity to changes in bulbar function with disease progression. The result of the assessment is a comprehensive, across-subsystem performance profile for each participant. The profile, when compared to the same measures obtained from healthy controls, is used for diagnostic purposes. Currently, we are testing the sensitivity and specificity of these measures for diagnosis of ALS and for predicting the rate of disease progression. In the long term, the more refined endophenotype of bulbar ALS derived from this work is expected to strengthen future efforts to identify the genetic loci of ALS and improve diagnostic and treatment specificity of the disease as a whole. The objective assessment that is demonstrated in this video may be used to assess a broad range of speech motor impairments, including those related to stroke, traumatic brain injury, multiple sclerosis, and Parkinson disease.

A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis ALS

Objective assessments of the physiological mechanisms that support speech are needed to monitor disease onset and progression in persons with ALS and to quantify treatment effects in clinical trials. In this video, we present a comprehensive, instrumentation-based protocol for quantifying speech motor performance in clinical populations.

Improved methods for assessing bulbar impairment are necessary for expediting diagnosis of bulbar dysfunction in ALS, for predicting disease progression ...

Medicine

One Dimensional Turing-Like Handshake Test for Motor Intelligence

In the Turing test, a computer model is deemed to "think intelligently" if it can generate answers that are not distinguishable from those of a human. However, this test is limited to the linguistic aspects of machine intelligence. A salient function of the brain is the control of movement, and the movement of the human hand is a sophisticated demonstration of this function. Therefore, we propose a Turing-like handshake test, for machine motor intelligence. We administer the test through a telerobotic system in which the interrogator is engaged in a task of holding a robotic stylus and interacting with another party (human or artificial). Instead of asking the interrogator whether the other party is a person or a computer program, we employ a two-alternative forced choice method and ask which of two systems is more human-like. We extract a quantitative grade for each model according to its resemblance to the human handshake motion and name it "Model Human-Likeness Grade" (MHLG). We present three methods to estimate the MHLG. (i) By calculating the proportion of subjects' answers that the model is more human-like than the human; (ii) By comparing two weighted sums of human and model handshakes we fit a psychometric curve and extract the point of subjective equality (PSE); (iii) By comparing a given model with a weighted sum of human and random signal, we fit a psychometric curve to the answers of the interrogator and extract the PSE for the weight of the human in the weighted sum. Altogether, we provide a protocol to test computational models of the human handshake. We believe that building a model is a necessary step in understanding any phenomenon and, in this case, in understanding the neural mechanisms responsible for the generation of the human handshake.

We present a Turing-like Handshake test administered through a telerobotic system in which the interrogator is holding a robotic stylus and interacting with another party (human or artificial). We use a forced choice method, and extract a measure for the similarity of the artificial model to a human handshake.

In the Turing test, a computer model is deemed to "think intelligently" if it can generate answers that are not distinguishable from those of a human.  ...

Neuroscience

Examining Recall Memory in Infancy and Early Childhood Using the Elicited Imitation Paradigm

The ability to recall the past allows us to report on details of previous experiences, from the everyday to the significant. Because recall memory is commonly assessed using verbal report paradigms in adults, studying the development of this ability in preverbal infants and children proved challenging. Over the past 30 years, researchers have developed a non-verbal means of assessing recall memory known as the elicited or deferred imitation paradigm. In one variant of the procedure, participants are presented with novel three-dimensional stimuli for a brief baseline period before a researcher demonstrates a series of actions that culminate in an end- or goal-state. The participant is allowed to imitate the demonstrated actions immediately, after a delay, or both. Recall performance is then compared to baseline or to performance on novel control sequences presented at the same session; memory can be assessed for the individual target actions and the order in which they were completed. This procedure is an accepted analogue to the verbal report techniques used with adults, and it has served to establish a solid foundation of the nature of recall memory in infancy and early childhood. In addition, the elicited or deferred imitation procedure has been modified and adapted to answer questions relevant to other aspects of cognitive functioning. The broad utility and application of imitation paradigms is discussed, along with limitations of the approach and directions for future research.

The elicited imitation procedure was established to examine the development of recall memory in infancy and early childhood. This procedure has been widely used to establish a solid foundation of the nature of recall memory in infancy and early childhood.

The ability to recall the past allows us to report on details of previous experiences, from the everyday to the significant. Because recall memory is ...

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test&#160;theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to&#160;comprehension informed by&#160;comprehenders&#39; experience. Among the examined aspects of the visual context are actions, events, a speaker&#39;s gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eye-tracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension.

The present article reviews an eye-tracking methodology for studies on language comprehension. To obtain reliable data, key steps of the protocol must be followed. Among these are the correct set-up of the eye tracker (e.g., ensuring good quality of the eye and head images) and accurate calibration.

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing ...

Virtual Hand with Ambiguous Movement between the Self and Other Origin: Sense of Ownership and 'Other-Produced' Agency

The feeling that a body part is one&#8217;s own body (sense of ownership; SoO) and the feeling based on the causal relationship between one&#8217;s will and action (sense of agency; SoA) have been recognized as the basis of our bodily self-consciousness. Previously, the illusory SoO over a fake body part (e.g., rubber hand) was introduced as the rubber hand illusion (RHI). Furthermore, it was determined that one could also evoke a SoA over an object with movements linked to the one&#8217;s prior intention. On the other hand, the postdictivity of our spontaneity implies that it is essentially inseparable whether actions originate from self or others. In other words, our SoA or daily experiences are obtained in such as inseparable scenario. Previous research, however, has maintained the premise that self- and other-origin movements are perceptually distinguishable. Here, we implement a protocol to make these aspects ambiguous for the participants and to estimate whether they can feel SoO and/or SoA and how. To this end, we employ an experiment using virtual reality, under which participants observe virtual fingers moving very slowly (or quickly or not moving) while their own fingers do not move. For evaluation of the illusory SoO, measurements of skin conductance responses against a knife threat are adopted. Additionally, we introduce face-to-face interviews to determine whether the feelings regarding the slow movement match the conventional SoA definition. Our representative results suggest that the SoO is evoked over the hand, and various attitudes to accept its movement as the participant&#8217;s own with awareness that they did not originate it are reported by the majority. As the results show, the novelty of this protocol is discovering that in such a situation, the SoO cooperates with an externally produced SoA to establish one&#8217;s own bodily experience rather than the independence of the SoO and SoA.

While previous research on bodily self-consciousness assumed that self- and other-origin movements were perceptually distinguishable, this protocol allows them to be ambiguous on a virtual hand with unintentional slight movements. This enables us to observe one&#8217;s experience formed by SoO and other-produced SoA, rather than the absence of SoA.

The feeling that a body part is one&#8217;s own body (sense of ownership; SoO) and the feeling based on the causal relationship between one&#8217;s will ...

EEG Study on Recognizing Familiar Speakers in Expressive Speech

Memorization-Based Training and Testing Paradigm for Robust Vocal Identity Recognition in Expressive Speech Using Event-Related Potentials Analysis

Recognizing familiar speakers from vocal streams is a fundamental aspect of human verbal communication. However, it remains unclear how listeners can still discern the speaker's identity in expressive speech. This study develops a memorization-based individual speaker identity recognition approach and an accompanying electroencephalogram (EEG) data analysis pipeline, which monitors how listeners recognize familiar speakers and tell unfamiliar ones apart. EEG data captures online cognitive processes during new versus old speaker distinction based on voice, offering a real-time measure of brain activity, overcoming limits of reaction times and accuracy measurements. The paradigm comprises three steps: listeners establish associations between three voices and their names (training); listeners indicate the name corresponding to a voice from three candidates (checking); listeners distinguish between three old and three new speaker voices in a two-alternative forced-choice task (testing). The speech prosody in testing was either confident or doubtful. EEG data were collected using a 64-channel EEG system, followed by preprocessing and imported into RStudio for ERP and statistical analysis and MATLAB for brain topography. Results showed an enlarged late positive component (LPC) was elicited in the old-talker compared to the new-talker condition in the 400-850 ms window in the Pz and other wider range of electrodes in both prosodies. Yet, the old/new effect was robust in central and posterior electrodes for doubtful prosody perception, whereas the anterior, central, and posterior electrodes are for confident prosody condition. This study proposes that this experiment design can serve as a reference for investigating speaker-specific cue-binding effects in various scenarios (e.g., anaphoric expression) and pathologies in patients like phonagnosia.

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

The study introduces a training-testing paradigm to investigate old/new effects of event-related potentials in confident and doubtful prosodic scenarios. Data reveals an enhanced late positive component between 400-850 ms at Pz and other electrodes. This pipeline can explore factors beyond speech prosody and their influence on cue-binding target identification.

Recognizing familiar speakers from vocal streams is a fundamental aspect of human verbal communication. However, it remains unclear how listeners can ...

Capturing Representative Hand Use at Home Using Egocentric Video in Individuals with Upper Limb Impairment

Impaired hand function after neurological injuries can have a major impact on independence and quality of life. Most existing upper limb assessments are carried out in person, which is not always indicative of hand use in the community. Novel approaches to capture hand function in daily life are required to measure the true impact of rehabilitation interventions. Egocentric video combined with computer vision for automated analysis has been proposed to evaluate hand use at home. However, there are limitations to the duration of continuous recordings. We present a protocol designed to ensure that the videos obtained are representative of daily routines while respecting participant privacy. 
A representative recording schedule is selected through a collaborative process between the researchers and participants, to ensure that the videos capture natural tasks and performance, while being useful for hand assessment. Use of the equipment and procedures is demonstrated to the participants. A total of 3 h of video recordings are scheduled over two weeks. To reduce privacy concerns, participants have full control to start and stop recordings, and the opportunity to edit the videos before returning them to the research team. Reminders are provided, as well as help calls and home visits if necessary. 
The protocol was tested with 9 stroke survivors and 14 individuals with cervical spinal cord injury. The videos obtained contained a variety of activities, such as meal preparation, dishwashing, and knitting. An average of 3.11 &plusmn; 0.98 h of video were obtained. The recording periods varied from 12-69 d, due to illness or unexpected events in some cases. Data was successfully obtained from twenty-two out of 23 participants, with 6 participants requiring assistance from the investigators during the home recording period. The protocol was effective for collecting videos that contained valuable information about hand function at home after neurological injuries.

A protocol is proposed to capture natural hand function of individuals with hand impairments during their daily routines using an egocentric camera. The goal of the protocol is to ensure that the recordings are representative of an individual&#39;s typical hand use during activities of daily living at home.

Impaired hand function after neurological injuries can have a major impact on independence and quality of life. Most existing upper limb assessments are ...

Bioengineering

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for state-of-the-art prosthetics and optimized rehabilitation. Surface electromyography (sEMG), in conjunction with deep learning methods, is considered a promising method in this domain. However, current methods often rely on cumbersome recording setups and the identification of static hand positions, limiting their effectiveness in real-world applications. The protocol we report here presents an advanced approach combining a wearable surface EMG and finger&#160;tracking system to capture comprehensive data during dynamic hand movements. The method records muscle activity from soft printed electrode arrays (16 electrodes) placed on the forearm as subjects perform gestures in different hand positions and during movement. Visual instructions prompt subjects to perform specific gestures while EMG and finger positions are recorded. The integration of synchronized EMG recordings and finger tracking data enables comprehensive analysis of muscle activity patterns and corresponding gestures. The reported approach demonstrates the potential of combining EMG and visual tracking technologies as an important resource for developing intuitive and responsive gesture recognition systems with applications in prosthetics, rehabilitation, and interactive technologies. This protocol aims to guide researchers and practitioners, fostering further innovation and application of gesture recognition in dynamic and real-world scenarios.

The paper presents a comprehensive protocol for simultaneously recording hand-electromyography (EMG) and visual finger tracking during natural finger gesturing. The visual data is designed to serve as the ground truth for the development of accurate EMG-based computational models for finger gesture recognition.

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for ...

Group Synchronization During Collaborative Drawing Using Functional Near-Infrared Spectroscopy

Functional near-infrared spectroscopy (fNIRS) is a noninvasive method particularly suitable for measuring cerebral cortex activation in multiple subjects, which is relevant for studying group interpersonal interactions in ecological settings. Although many fNIRS systems technically offer the possibility to monitor more than two individuals simultaneously, establishing easy-to-implement setup procedures and reliable paradigms to track hemodynamic and behavioral responses in group interaction is still required. The present protocol combines fNIRS and video-based observation to measure interpersonal synchronization in quartets during a cooperative task.This protocol provides practical recommendations for data acquisition and paradigm design, as well as guiding principles for an illustrative data analysis example. The procedure is designed to assess differences in brain and behavior interpersonal responses between social and non-social conditions inspired by a well-known ice-breaker activity, the Collaborative Face Drawing Task. The described procedures can guide future studies to adapt group naturalistic social interaction activities to the fNIRS environment.

The present protocol combines functional near-infrared spectroscopy (fNIRS) and video-based observationto measure interpersonal synchronization in quartets during a collaborative drawing task.

Functional near-infrared spectroscopy (fNIRS) is a noninvasive method particularly suitable for measuring cerebral cortex activation in multiple subjects, ...

Examining Gesture Production in the Presence of Communication Challenges

Samenvatting

Meer video's verkennen