This work was supported by the Natural Science Foundation of China (Grant No. 31971037); the Shuguang Program supported by the Shanghai Education Development Foundation and Shanghai Municipal Education Committee (Grant No. 20SG31); the Natural Science Foundation of Shanghai (22ZR1460200); the Supervisor Guidance Program of Shanghai International Studies University (2022113001); and the Major Program of the National Social Science Foundation of China (Grant No. 18ZDA293).

The Ethics Committee of the Institute of Linguistics, Shanghai International Studies University, has approved the experiment design described below. Informed consent was obtained from all participants for this study.
1. Preparation and validation of the audio library
<ol>
	<li>Audio recording and editing
	<ol>
		<li>Create a Chinese vocal database following the standard procedure of making a previous English version while making adaptations where needed to fit into the conte...

EEG Study on Recognizing Familiar Speakers in Expressive Speech

The study presents a pipeline for EEG data collection and analysis, focusing on recognizing previously learned speaker identities. This study addresses variations between learning and recognition phases, including differences in speech content22 and prosody10. The design is adaptable to a range of research fields, including psycholinguistics, such as pronoun and anaphoric processing41. 
The training-testing paradigm is a c...

Human vocal streams are rich in information, such as emotion1,2, health status3,4, biological sex5, age6, and, more importantly, the individual vocal identity7,8. Studies have suggested that human listeners have a robust capacity to recognize and differentiate their peers' identities through voices, overcoming within-speaker variations around speaker identity's average-based representation in the acoustic space9. Such variations are brought about by acoustic manipulation (fundamental frequency and vocal tract length, i.e., F0 and VTL) that corresponds to no clear pragmatic intentions9, emotion prosodies10, and vocal confidence that conveys speakers' feeling of knowing11. Behavioral experiments have focused on many factors that influence listeners' performance in recognizing the talkers, including language-related manipulations8,12,13, participants-related characteristics such as music experience or reading ability14,15, and stimuli-related adaptions like backward speech or nonwords16,17; more can be found in literature reviews18,19. A few recent experiments have investigated how individual variation of the speaker identity representation might undermine the recognition accuracy, considering aspects including high versus low emotional expressiveness16 and neutral versus fearful prosodies5; more possible scenarios open for further investigation, as suggested by a review20. 
For the first research gap, the study proposes that the neurological underpinnings of speaker identification have yet to fully explore how within-speaker variation challenges listeners' brain activities. For example, in an fMRI-based speaker recognition task by Z&#228;ske et al., participants' right posterior superior temporal gyrus (pSTG), right inferior/middle frontal gyrus (IFG/MFG), right medial frontal gyrus, and left caudate showed reduced activation when correctly identified as old versus new talkers, regardless of the linguistic content being the same or different21. However, an earlier electroencephalography (EEG) study by Z&#228;ske et al. did not observe this old/new effect when speaker identity variation was introduced through different texts22. Specifically, a larger, late positive component (LPC) ranging from 300 to 700 ms, detected at the Pz electrode when listeners encountered their familiar trained talker expressing the same text (i.e., hearing a replay with non-varied linguistic content), was absent when the talkers delivered new texts. 
In support of the assertion made by Z&#228;ske et al.21, this study suspects that an old/new effect can still be observed despite differences in linguistic content between training and testing sessions in event-related potential (ERP) analyses. This rationale stems from the notion that the absence of the old/new effect in Z&#228;ske et al.22, under conditions where different texts were used, may be attributed to the lack of an additional check session during the training task to ensure thorough and effective identity learning, as suggested by Lavan et al.23. Consequently, the first objective of the study is to examine and validate this hypothesis. This study aims to test this by adding a checking session to the training-testing paradigm22. 
Another key question this study aims to address is the robustness of speaker identification in the presence of speech prosody. Previous behavioral studies have suggested that listeners particularly struggle to recognize talkers across different prosodies, which indicates a modulatory role of prosodic context - listeners underperformed in the different training-testing prosody conditions. This study aims to test this by exposing listeners to recognize familiar talkers in either confident or doubtful prosodies24. This study expects that the observed ERP differences will help explain how speech prosody influences identity recognition. 
The core objective of the current study is to investigate the robustness of the old/new effect in speaker recognition, specifically examining whether there are differences in recognizing talkers in confident versus doubtful prosodies. Xu and Armony10 performed a behavioral study using a training-testing paradigm, and their findings suggest that listeners cannot overcome prosodic differences (e.g., trained to recognize a talker in neutral prosody and tested on fearful prosody) and can only achieve accuracy lower than chance level10. Acoustic analysis indicates that speakers expressing varied emotive states are associated with VTL/F0 modulation; for example, confident prosody is characterized by lengthened VTL and lower F0, whereas the opposite is true for doubtful prosody11,24. Another piece of evidence comes from the study by Lavan et al.23, which confirmed that listeners can adapt to VTL and F0 changes of the speaker and form average-based representations of the talkers. This study reconciles that, from a behavioral data perspective, listeners are likely to still recognize the talker's identity across prosodies (e.g., trained to recognize one in confident prosody but tested in doubtful prosody; reported in a separate manuscript in preparation). Yet, the neural correlates of speaker identification, specifically the generalizability of the old/new effect observed by Z&#228;ske et al.22, remain unclear. Hence, the current study is committed to validating the robustness of the old/new effect in confident versus doubtful prosodies as contexts for testing. 
The study introduces a departure from previous research paradigms in old/new effects studies. While past research focused on how old/new talker recognition influences perception, this study extends this by incorporating two confidence levels (confident versus doubtful) into the paradigm (thus, a 2+2 study). This allows us to investigate speaker recognition within the contexts of confident and doubtful speech prosodies. The paradigm enables the exploration of the robustness of old/new effects. The analyses of memory effects and regions of interest (ROI) within both confident and doubtful speech contexts serve as evidence for this investigation. 
Altogether, the study aims to update the understanding of the EEG correlates of voice recognition, with the hypotheses that the enlarged LPC of the EEG old/new effect is observable even when 1) the linguistic content is not the same, and 2) with the presence of confident versus doubtful prosody. This study investigated the hypotheses through a three-step paradigm. Firstly, during the training phase, participants established associations between three voices and their corresponding names. Subsequently, in the checking phase, they were tasked with identifying the name corresponding to a voice from a selection of three candidates. This checking, following Lavan et al.23, aims to overcome insufficient old speaker familiarization, which led to the unobserved old/new effect when the text in the training and testing phases differed6, and talkers could not recognize talkers across neutral and fearful prosodies10. Finally, in the testing phase, participants distinguished between three old and three new speaker voices in a two-alternative forced-choice task, with speech prosody presented as either confident or doubtful. EEG data were collected using a 64-channel EEG system and underwent preprocessing before analysis. Statistical analysis and event-related potential (ERP) analysis were conducted in RStudio, while MATLAB was utilized for brain topography analysis.
Regarding design details, this study proposes a speaker identity learning experiment that controls for the talker's height, which is related to VTL and influences impressions of who is talking23. This aspect also influences social impressions, such as perceived dominance25, and such higher-level impression formation might interact with decoding speaker identity26.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>64Ch Standard BrainCap for BrainAmp</td>
			<td>Easycap GmbH</td>
			<td>Steingrabenstrasse 14 DE-82211</td>
			<td>https://shop.easycap.de/products/64ch-standard-braincap</td>
		</tr>
		<tr>
			<td>Abrasive Electrolyte-Gel</td>
			<td>Easycap GmbH</td>
			<td>Abralyt 2000</td>
			<td>https://shop.easycap.de/products/abralyt-2000</td>
		</tr>
		<tr>
			<td>actiCHamp Plus</td>
			<td>Brain Products GmbH</td>
			<td>64 channels + 8 AUX</td>
			<td>https://www.brainproducts.com/solutions/actichamp/</td>
		</tr>
		<tr>
			<td>Audio Interface</td>
			<td>Native Instruments GmbH</td>
			<td>Komplete audio 6</td>
			<td>https://www.native-instruments.com/en/products/komplete/audio-interfaces/komplete-audio-6/</td>
		</tr>
		<tr>
			<td>Foam Eartips</td>
			<td>Neuronix</td>
			<td>ER3-14&#160;</td>
			<td>https://neuronix.ca/products/er3-14-foam-eartips</td>
		</tr>
		<tr>
			<td>Gel-based passive electrode system</td>
			<td>Brain Products GmbH</td>
			<td>BC 01453</td>
			<td>https://www.brainproducts.com/solutions/braincap/</td>
		</tr>
		<tr>
			<td>High-Viscosity Electrolyte Gel&#160;</td>
			<td>Easycap GmbH</td>
			<td>SuperVisc</td>
			<td>https://shop.easycap.de/products/supervisc</td>
		</tr>
	</tbody>
</table>

memorization-based training and testing paradigm for robust vocal identity recognition in expressive speech using event-related potentials analysis

Recognizing familiar speakers from vocal streams is a fundamental aspect of human verbal communication. However, it remains unclear how listeners can still discern the speaker's identity in expressive speech. This study develops a memorization-based individual speaker identity recognition approach and an accompanying electroencephalogram (EEG) data analysis pipeline, which monitors how listeners recognize familiar speakers and tell unfamiliar ones apart. EEG data captures online cognitive processes during new versus old speaker distinction based on voice, offering a real-time measure of brain activity, overcoming limits of reaction times and accuracy measurements. The paradigm comprises three steps: listeners establish associations between three voices and their names (training); listeners indicate the name corresponding to a voice from three candidates (checking); listeners distinguish between three old and three new speaker voices in a two-alternative forced-choice task (testing). The speech prosody in testing was either confident or doubtful. EEG data were collected using a 64-channel EEG system, followed by preprocessing and imported into RStudio for ERP and statistical analysis and MATLAB for brain topography. Results showed an enlarged late positive component (LPC) was elicited in the old-talker compared to the new-talker condition in the 400-850 ms window in the Pz and other wider range of electrodes in both prosodies. Yet, the old/new effect was robust in central and posterior electrodes for doubtful prosody perception, whereas the anterior, central, and posterior electrodes are for confident prosody condition. This study proposes that this experiment design can serve as a reference for investigating speaker-specific cue-binding effects in various scenarios (e.g., anaphoric expression) and pathologies in patients like phonagnosia.

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Memorization-Based Training and Testing Paradigm for Robust Vocal Identity Recognition in Expressive Speech Using Event-Related Potentials Analysis

For over 50 years, scientists have been intrigued by how we recognize voices. Our recent study explores how emotional prosodies affect this process. Despite differences in prosodies, we found that the overall old-new effect is robust, indicating that our brains consistently distinguish familiar and unfamiliar voices.Current experimental challenges in speaker identification arise from the commonly used training testing paradigm in the past, which is not suitable for studying detailed variables affecting speaker identity processing. Ensuring that the training is thorough and effective is crucial to address this challenge. Our fundings reveal that while the LPC, or the old-new effect, shows differences between confident and doubtful processes, it remains robust overall.Our paradigm is well suited for investigating the processing of familiar and unfamiliar objects with internal variations. My lab focuses on voice perception, including how pragmatically marked emotions are expressed and perceived and how vocal information has an impact on listeners. We use EEGs and fMRI to measure these factors with a key focus on social factors such as accent or whether the talker is AI.This study on JoVE supports our future research in these areas.

The classic old/new effect is characterized by a significant increase in listeners&#39; brain activity on the Pz electrode (between 300 to 700 ms) when the speech content of the testing session matches that of the training session, particularly in the old talker condition compared to the new talker condition22. The protocol unveils an updated version of this effect: Firstly, observing larger positive trends in the Pz electrode and across the entire brain region for the old condition compared to th...

The study introduces a training-testing paradigm to investigate old/new effects of event-related potentials in confident and doubtful prosodic scenarios. Data reveals an enhanced late positive component between 400-850 ms at Pz and other electrodes. This pipeline can explore factors beyond speech prosody and their influence on cue-binding target identification.

Recognizing familiar speakers from vocal streams is a fundamental aspect of human verbal communication. However, it remains unclear how listeners can ...

memorization-based-training-testing-paradigm-for-robust-vocal

Research

JoVE Journal

Behavior

Preparing the Venue and Participants for Electroencephalography (EEG)

To begin, assign participant IDs and ask participants to wash and dry their hair within one hour before participating in the experiment. Mix the electrolyte gel and abrasive electrolyte gel in a one-to-three ratio, adding a small amount of water. Stir the mixture evenly in a container with a spoon.Prepare fine-tipped cotton swabs and a dry EEG cap. Now ask the participant to sit comfortably in a chair and inform them that the experimenter will apply the EEG cap. Provide the participant with instructions about the experimental tasks and an informed consent form for the experiment.Obtain the participant's signature before proceeding with the preparation phase. Next, connect the EEG cap to the amplifier, which in turn connects to the EEG data acquisition computer. Use an additional monitor to check the color indicators on the 64 electrodes, as this study uses a passive cap.Open BrainVision Recorder and import a customized workspace file that has defined the recording parameters. Click on Monitor to check the impedance. Ask the participant to sit upright in a chair.Select an appropriately sized gel-based passive electrode system for the participant's head. Now dip a disposable cotton swab into the conductive paste and apply it to the holes of the cap, making sure to rub against the participant's scalp. An electrode's corresponding indicator turning green on the EEG data collection computer signifies that it is successfully collecting optimal data.Once the indicative color for all electrodes on the screen except for the two independent sided electrodes turn green, apply the side electrodes to attach the left electrode near the participant's left eye and the right electrode near the right temple. Once all electrodes are green, place an elastic net over the participant's head to help the EEG cap fit more securely and stably against the participant's head. Then equip the participant with wired headphones.Close the electromagnetic shielding door and guide the participant's actions through a microphone that allows communication inside and outside. Additionally, monitor the participant's movements through an external monitor, reminding them not to move their body significantly. Also, monitor the participant's progress in behavioral tasks through a behavioral data monitor.Ask the participant to wear earphones connected to the behavioral collection computer through an audio interface.

Performing the Encephalography Experiment Block-by-Block Independently

To begin, create a new EEG recording file on the EEG data collection computer. Open brain vision recorder and click on monitor to double check the impedance followed by stat or resume recording. Open the PsychoPy program's run experiment for the behavioral experiment.Enter the participant's ID and the corresponding block number and click okay to start the experiment. Closely monitor the accuracy of data reported on the screen after the participant completes the check phase on the behavioral data computer. Pay close attention to the final reported accuracy of old versus new recognition.After the participant has completed all blocks, invite them to wash their hair. To clean the EEG cap, remove the residual conductive paste with a toothbrush and wrap them in plastic bags. Once cleaned, hang the EEG cap in a well-ventilated area to dry.Copy the EEG and behavioral data onto a portable hard drive, ensuring that the EEG data and behavioral data correspond.

407 Views.  Shanghai International Studies University.  The study introduces a training-testing paradigm to investigate old/new effects of event-related potentials in confident and doubtful prosodic scenarios. Data reveals an enhanced late positive component between 400-850 ms at Pz and other electrodes. This pipeline can explore factors beyond speech prosody and their influence on cue-binding target identification. 

EEG Study on Recognizing Familiar Speakers in Expressive Speech

Summary