cartoon retelling protocol to study gesture production

Protocol for Assessing Gestures Under Communication Challenges

Co-speech gestures (heretofore, gestures) - meaningful hand movements produced concurrently with speech - contribute to interpersonal communication by conveying information complementing verbal content1. According to the most widely used taxonomy2,3, gestures can be divided into three categories: representational gestures, which convey referents via their form and motion (e.g., flapping the hands back and forth together to convey flying); beat gestures, which convey emphasis via simple punctate movements (e.g., moving the dominant hand downward slightly in conjunction with each word in the phrase &#34;right now&#34;); and deictic gestures, which draw attention to the presence or absence of an entity via pointing (e.g., swinging the thumb backward to indicate something behind oneself). Representational gestures can be further divided into two additional categories: iconic gestures, which convey concrete referents (e.g., a bird), and metaphorical gestures, which convey metaphorical referents (e.g., ecstasy). Because gesture and speech arise from the same conceptual content, they are closely related in meaning4,5. By serving as an iconic visual medium for communication, gestures can help compensate when challenges in spoken language comprehension arise6, whether due to individual factors such as limited proficiency in the language spoken7,8 or environmental factors such as difficulty in hearing speech9,10. Thus, gestures are integral to understanding interpersonal communication in face-to-face and virtual contexts, providing insight into how speakers and listeners convey and comprehend information multimodally.
Several structured interactive tasks have been developed to measure how gesture affects interpersonal communication. These tasks include interviews, in which participants respond to questions by describing their personal experiences11,12; image description, in which a static image is presented to participants to describe13,14; puzzle solving, in which participants describe how to solve a puzzle by spatially orienting components correctly15,16; direction provision, in which participants are instructed to give directions to a location unfamiliar to listeners17,18,19; and narrative retelling, in which participants view a cartoon depicting a sequence of events and subsequently recount them20,21,22. Many of the tasks employed in the studies cited above incorporate spatial content and action, which gesture conveys with particular efficacy15,23,24,25. In the tasks for the studies cited above, participants typically produce gestures in conjunction with language, and information conveyed via gesture and its relationship to concurrently produced language is assessed. Alternatively, participants may view a recording of someone gesturing (or not gesturing) and producing language when completing one of these tasks, after which participants&#39; comprehension is assessed. All of these tasks may be conducted virtually as well as face-to-face, enabling data collection from a wide range of participants and comparison across modalities.
The impact of gestures on interpersonal communication has been implemented with a wide range of participants. Of particular interest have been populations with communication challenges, including young children26,27,28,29,30, second language (L2) users8,31,32,33, individuals with autism spectrum disorder (ASD)20,21,22,34, individuals with specific language impairment35,36,37, individuals with aphasia12,38, individuals with brain injury39, and individuals who stutter40,41. This work has revealed that, while many of these populations can utilize gestures to facilitate communication, some, such as individuals with autism spectrum disorder, may encounter difficulty leveraging gestures to communicate effectively. The findings suggest that this may be due to the extent to which these populations take audience design and environmental cues into consideration, highlighting the practical implications of these explanations for the impact of gestures on communication.
A key feature providing insight into the impact of gestures on interpersonal communication is interlocutor visibility. A question of great importance in the field of gesture research is the extent to which participants gesture for their own benefit, indicating the offloading of cognitive operations onto the body, vs. the benefit of their interlocutors, indicating the use of gesture to communicate. This question has been investigated by examining the impact of interlocutor (non-)visibility on gesture production in face-to-face contexts via the use of an opaque partition26,42, as well as in telephone contexts13 and virtual contexts43. Overall, the results of this work indicate that, although speakers gesture for their own benefit, they produce more representational gestures when communicating with visible than non-visible interlocutors, whereas beat gesture production is similar regardless of interlocutor visibility. Thus, they suggest that representational gesture facilitates communication across a variety of contexts, suggesting that speakers take the perspective of their interlocutors into account and modify their gesture production accordingly. Although previous research examining the effect of interlocutor visibility has been instrumental in providing insight into the contributions of gesture to interpersonal communication, it has focused on English as a first language (L1) in typically developing populations, so it is unclear whether the findings can be extended to populations with communication challenges. Two such populations are L2 learners, who may struggle to communicate via speech in the target language, and children with ASD, whose verbal and nonverbal communication are abnormal. Furthermore, little research has examined the impact of interlocutor visibility on gesture production in virtual contexts, which allow the effect of the interlocutor&#39;s visibility on the participant to be disentangled from the effect of the participant&#39;s visibility to the interlocutor, so the replicability of findings from these contexts is currently unclear. Finally, some research has focused on the impact of interlocutor visibility on the production of specific types of gestures, so it is unclear whether the production of other types of gestures is similarly affected by interlocutor visibility.
The protocols described below manipulate interlocutor visibility to examine gesture production in challenging circumstances: L2 word learning and narrative retelling by individuals with ASD. The L2 word learning protocol bridges research examining the impact of observing gestures on L2 word learning with research examining the contribution of gestures to communication via an interactive word learning paradigm. In this paradigm, participants unfamiliar with the target language learn words in it and then teach these words to other participants unfamiliar with the target language, permitting the impact of interlocutor visibility on gesture production to be examined in the earliest stages of L2 acquisition in a conversational context. The cartoon retelling paradigm employs a widely used narrative retelling task in which differences in gesture production have been observed when interlocutor visibility is manipulated with a new population: adolescents with ASD. This population is of interest because language development, including gesture production, is mature by adolescence, and ASD entails difficulty with verbal and nonverbal communication, including gestures, as well as a lack of sensitivity to the communicative needs of interlocutors. Together, these protocols provide insight into the extent to which gesture production is dependent on - and, conversely, can compensate for - speech when interpersonal communication is challenging.
Based on findings demonstrating how listener visibility affects gesture production by L1 English speakers13,26,42,43, it was hypothesized that participants would produce more overall gestures and more representational gestures when discussing L2 words with a visible than a non-visible interlocutor. Based on findings demonstrating abnormalities in gesture production&#160;in ASD20,44, it was hypothesized that adolescents with ASD would produce fewer overall gestures as well as fewer representational and deictic gestures than typically developing (TD) adolescents. Moreover, based on findings showing that ASD entails difficulty with perspective taking45, it was hypothesized that gesture production by adolescents with ASD would not differ significantly in the presence of visible and non-visible interlocutors. Finally, an interaction between diagnosis and visibility was predicted, such that gesture production would not differ by visibility for adolescents with ASD but that it would for TD adolescents.

Author Spotlight: Deciphering the Cognitive and Neural Mechanisms of Gesture in Communication  

Examining Gesture Production in the Presence of Communication Challenges

My research examines the cognitive and neural mechanisms of the body's impact on communication via hand gesture and sign language. A major focus of my work is how the relationship between gesture and speech differs in typical and atypical language development and processing. In vivo neuroscience techniques such as electroencephalography or EEG, functional magnetic resonance imaging or fMRI, and functional near infrared spectroscopy or fNIRS have been instrumental in revealing the neural mechanisms of hand gesture and sign language.In addition, eye tracking provides insight into real-time processing of hand gesture and sign language. My work was the first to show that observing hand gestures conveying pitch contours helps speakers of atonal languages, such as English, learn lexical tone and tonal languages such as Mandarin. I'm currently using EEG and fNIRS to investigate the neural signatures of this effect.This protocol provides a means to investigate gesture production in the presence of communication challenges. I've used it successfully to investigate gesture production in second language learners and individuals with autism spectrum disorders in in-person settings, and I plan to expand it to additional populations and virtual settings. My findings provide insight into how observing and producing hand gestures can facilitate language acquisition and processing.Moreover, they're helping to reveal potential biomarkers of language atypicalities as well as treatment efficacy.

Cartoon Retelling Protocol to Study Gesture Production

To begin, administer consent to participants. Then administer a standardized IQ assessment to participants, ensuring their IQ is 85 or above. To record participants for their speech and gesture production, set up a table microphone in front of the participant and a hidden camera at a 90 degree angle on one side of the participant.Explain the setup to the participant. Present the first half of the Looney Tunes Cartoon, Canary Row, to the participant, focusing on vignettes when Sylvester the Cat tries to catch Tweety Bird. Instruct the participant to retell the events of the cartoon's first half in detail to the interlocutor.Guide the interlocutor to encourage more details if the participant's account is brief. Then place a divider. Repeat these steps for the second half of Canary Row, covering additional vignettes of Sylvester's attempts to catch Tweety Bird.A study conducted on 41 adolescents, of which 18 were diagnosed with autism spectrum disorder, showed no overall difference in gesture production compared to typically developed adolescents. However, adolescents with autism spectrum disorder produced fewer metaphorical and beat gestures. Also, both groups gestured more to visible interlocutors, with no significant interaction between diagnosis and listener visibility.

cartoon-retelling-protocol-to-study-gesture-production

Research

JoVE Journal

Behavior

Understanding why speakers modify their co-speech hand gestures when speaking to interlocutors provides valuable insight into how these gestures contribute to interpersonal communication in face-to-face and virtual contexts. The current protocols manipulate the visibility of speakers and their interlocutors in tandem in a face-to-face context to examine the impact of visibility on gesture production when communication is challenging. In these protocols, speakers complete tasks such as teaching words from an unfamiliar second language or recounting the events of cartoon vignettes to an interlocutor who is either another participant or a confederate. When performing these tasks, speakers are visible or non-visible to their interlocutor, and the speaker is visible or non-visible to the participant. In the word learning task, speakers and interlocutors visible to one another produce more representational gestures, which convey meaning via handshape and motion, and deictic (pointing) gestures than speakers and interlocutors who are not visible to one another. In the narrative retelling protocol, adolescents with autism spectrum disorder (ASD) produced more gestures when speaking to visible interlocutors than non-visible interlocutors. A major strength of the current protocol is its flexibility in terms of the tasks, populations, and gestures examined, and the current protocol can be implemented in videoconferencing as well as face-to-face contexts. Thus, the current protocol has the potential to advance the understanding of gesture production by elucidating its role in interpersonal communication in populations with communication challenges.

Here, we present a protocol that manipulates interlocutor visibility to examine its impact on gesture production in interpersonal communication. This protocol is flexible to tasks implemented, gestures examined, and communication modality. It is ideal for populations with communication challenges, such as second language learners and individuals with autism spectrum disorder.

Understanding why speakers modify their co-speech hand gestures when speaking to interlocutors provides valuable insight into how these gestures ...

Protocol for Visibility Manipulation to Study Gesture Production

To begin, provide an informed consent form to the participants explaining the study without revealing its focus on gesture production. Ensure the participants sign the form and answer any questions they have. Collect the signed form.For pair participants, use a coin toss to assign roles. One as the speaker and the other is the interlocutor. For individual participants with a research assistant, used a rigged coin toss to assign the participant as the speaker and the research assistant as the interlocutor.In half of the sessions, place an opaque divider between the speaker and interlocutor to ensure they cannot see one another's face or hands. Explain the experimental task to participants answering any questions. Inform them about audio recording, showing the microphone, but do not disclose video recording and ensure that the video camera is not visible to the participants.Start audio and video recording, and implement the experimental task to elicit speech and gesture. If using a within participants design, remove the opaque divider according to the session iteration. Repeat the experimental task until all stimuli are covered.Stop audio and video recording. Inform participants about the study's focus on gestures and that they were video recorded. Answer any final questions from participants and thank them before dismissal.

L2 Word Learning Protocol to Study Gesture Production

To begin, recruit paired participants and use a coin toss to assign roles. One is the speaker, and the other is the interlocutor. Place a table microphone between the speaker and interlocutor for audio recording.Also place a video camera at a 90-degree angle to the side, ensuring it remains hidden from the participants. Ensure both the speaker and interlocutor are within the view of the video camera and the computer screen remains out of the frame to maintain condition blindness. Use a computer to present an L2 word from the target language orally to the speaker via earphones and visually in one of two formats, either with a representational gesture video of an L1 speaker or as text only.After a 500-millisecond pause, display the L1 translation of the L2 word to the speaker for 2, 000 milliseconds. Prompt the speaker to teach the L2 word and its meaning to the interlocutor using speech or gesture. Instruct the speaker to press a button when ready to proceed.Present the next L2 word and continue until all 20 L2 words are covered. Present each L2 word as text, and prompt them to say its L1 translation, or skip the word if they do not recall its translation. The study examined the impact of visibility on gesture production and communication.It was found that primary speakers made more gestures when interlocutors were visible with significant increases in representational and deictic gestures. No significant difference was noted for beat gestures for the primary speakers. Interlocutors also gestured more when visible to primary speakers, but without statistical significance.