Name: cross-modal multivariate pattern analysis
Uploaded: 2011-11-09T13:00:00
Duration: 13 min 51 s
Description: Watch this Scientific Journal Video about Cross-Modal Multivariate Pattern Analysis at JoVE.com

This work was supported by grants by the Mathers Foundation and the National Institutes of Health (grant number 5P50NS019632-27) to Antonio and Hanna Damasio.

<ol>
	<li>Haynes, J. -. D., Rees, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Decoding+mental+states+from+brain+activity+in+humans.">Decoding mental states from brain activity in humans.</a> Nat. Rev. Neurosci. 7, 523-534 (2006).</li><li>Norman, K. A., Polyn, S. M., Detre, G. J., Haxby, J. V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Beyond+mind-reading:+multi-voxel+pattern+analysis+of+fMRI+data.">Beyond mind-reading: multi-voxel pattern analysis of fMRI data.</a> Trends. Cogn. Sci. 10, 424-430 (2006).</li><li>Kriegeskorte, N., Bandettini, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Analyzing+for+information,+not+activation,+to+exploit+high-resolution+fMRI.">Analyzing for information, not activation, to exploit high-resolution fMRI.</a> NeuroImage. 38, 649-662 (2007).</li><li>Mur, M., Bandettini, P. A., Kriegeskorte, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Revealing+representational+content+with+pattern-information+fMRI+-+an+introductory+guide.">Revealing representational content with pattern-information fMRI - an introductory guide.</a> Soc. Cogn. Affect. Neurosci. 4, 101-109 (2009).</li><li>Kamitani, Y., Tong, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Decoding+the+visual+and+subjective+contents+of+the+human+brain.">Decoding the visual and subjective contents of the human brain.</a> Nat. Neurosci. 8, 679-685 (2005).</li><li>Formisano, E., De Martino, F., Bonte, M., Goebel, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term="Who"+is+saying+"what"?+Brain-based+decoding+of+human+voice+and+speech?.">"Who" is saying "what"? Brain-based decoding of human voice and speech?.</a> Science. 322, 970-973 (2008).</li><li>Meyer, K., Kaplan, J. T., Essex, R., Webber, C., Damasio, H., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+visual+stimuli+on+the+basis+of+activity+in+auditory+cortices.">Predicting visual stimuli on the basis of activity in auditory cortices.</a> Nat. Neurosci. 13, 667-668 (2010).</li><li>Meyer, K., Kaplan, J. T., Essex, R., Damasio, H., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Seeing+touch+is+correlated+with+content-specific+activity+in+primary+somatosensory+cortex.">Seeing touch is correlated with content-specific activity in primary somatosensory cortex.</a> Cereb. Cortex. 21, 2113-2121 (2011).</li><li>Damasio, A. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Time-locked+multiregional+retroactivation:+a+systems-level+proposal+for+the+neural+substrates+of+recall+and+recognition.">Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition.</a> Cognition. 33, 25-62 (1989).</li><li>Meyer, K., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Convergence+and+divergence+in+a+neural+architecture+for+recognition+and+memory.">Convergence and divergence in a neural architecture for recognition and memory.</a> Trends. Neurosci. 32, 376-382 (2009).</li><li>Friston, K. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Modalities,+modes,+and+models+in+functional+neuroimaging.">Modalities, modes, and models in functional neuroimaging.</a> Science. 326, 399-403 (2009).</li><li>Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., Pollmann, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=PyMVPA:+a+Python+toolbox+for+multivariate+pattern+analysis+of+fMRI+data.">PyMVPA: a Python toolbox for multivariate pattern analysis of fMRI data.</a> Neuroinformatics. 7, 37-53 (2009).</li><li>Harrison, S. A., Tong, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Decoding+reveals+the+contents+of+visual+working+memory+in+early+visual+areas.">Decoding reveals the contents of visual working memory in early visual areas.</a> Nature. 458, 632-635 (2009).</li><li>Serences, J. T., Ester, E. F., Vogel, E. K., Awh, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stimulus-specific+delay+activity+in+human+primary+visual+cortex.">Stimulus-specific delay activity in human primary visual cortex.</a> Psych Sci. 20, 207-214 (2009).</li></ol>

No conflicts of interest declared.

The findings of our previous studies demonstrate that cross-modal MVPA is a useful tool to study the neural correlates of mental images experienced in the "mind's ear" and the "mind's touch". Specifically, the results show that the content of such images is correlated with neural activity in the early auditory and somatosensory cortices, respectively, providing direct empirical support for Damasio's framework of convergence-divergence zones. 
The basic paradigm we describe can be extended in a number of ways. Most obviously, similar studies can be carried out using different combinations of sensory modalities. In this respect, it is possible that establishing certain cross-modal associations prior to the experiment may increase the chances of success. For instance, to study cross-modal representations in olfactory cortices, one could prime subjects by exposing them simultaneously to the sight and the smell of a number of food items. Shortly after, inside the fMRI scanner, the olfactory memories of the smells would be triggered by visual cues, and MVPA could be used in an attempt to assign the trials to the correct food items based solely on activity in olfactory cortices. 
Another question of interest is whether the cross-modally induced patterns of activity in the early sensory cortices bear similarities with the patterns that are induced when sound or touch is actually experienced; in other words, does seeing the glass vase shatter invoke a similar pattern of neural activity in early auditory cortices as actually hearing the same event? This question, again, can be addressed through MVPA: is a classifier that has been trained on data recorded while subjects heard certain sounds able to correctly discriminate data recorded while the subjects watched corresponding video clips? In our auditory study, we did attempt such a classification, but the results were marginal (see Fig. 3 in ref. 7). In that study, however, the auditory associations of the participants were not controlled in any way; in other words, we do not know how similar the auditory images each subject associated with videos were to the audio tracks used to train the classifier. Again, it might be interesting to address the same question after specific cross-modal associations have been primed in the subjects, as described above for a visuo-olfactory association. This would permit to control more reliably the mental experience of the subjects during the video trials and thus might increase prediction performance of the classifier.
To conclude, we have introduced an extension to the classical MVPA paradigm by showing that stimuli can be predicted not only within, but also across sensory modalities. Thus, we show that the use of MVPA is not limited to investigating the correlates of perceptual representations induced directly by external sensory stimulation. Rather, MVPA can also assess the neural substrate of mental images that are triggered internally: in accordance with Damasio's convergence-divergence zone framework, our findings suggest that the conscious experience of mental images that are reconstructed based on memories is correlated with content-specific neural representations in the early sensory cortices.

cross-modal multivariate pattern analysis

Multivariate pattern analysis (MVPA) is an increasingly popular method of analyzing functional magnetic resonance imaging (fMRI) data1-4. Typically, the method is used to identify a subject's perceptual experience from neural activity in certain regions of the brain. For instance, it has been employed to predict the orientation of visual gratings a subject perceives from activity in early visual cortices5 or, analogously, the content of speech from activity in early auditory cortices6.
Here, we present an extension of the classical MVPA paradigm, according to which perceptual stimuli are not predicted within, but across sensory systems. Specifically, the method we describe addresses the question of whether stimuli that evoke memory associations in modalities other than the one through which they are presented induce content-specific activity patterns in the sensory cortices of those other modalities. For instance, seeing a muted video clip of a glass vase shattering on the ground automatically triggers in most observers an auditory image of the associated sound; is the experience of this image in the "mind's ear" correlated with a specific neural activity pattern in early auditory cortices? Furthermore, is this activity pattern distinct from the pattern that could be observed if the subject were, instead, watching a video clip of a howling dog?
In two previous studies7,8, we were able to predict sound- and touch-implying video clips based on neural activity in early auditory and somatosensory cortices, respectively. Our results are in line with a neuroarchitectural framework proposed by Damasio9,10, according to which the experience of mental images that are based on memories - such as hearing the shattering sound of a vase in the "mind's ear" upon seeing the corresponding video clip - is supported by the re-construction of content-specific neural activity patterns in early sensory cortices.

Watch this Scientific Journal Video about Cross-Modal Multivariate Pattern Analysis at JoVE.com

Cross-Modal Multivariate Pattern Analysis

Classical multivariate pattern analysis predicts sensory stimuli a subject perceives from neural activity in the corresponding cortices (e.g. visual stimuli from activity in visual cortex). Here, we apply pattern analysis cross-modally and show that sound- and touch-implying visual stimuli can be predicted from activity in auditory and somatosensory cortices, respectively.

Multivariate pattern analysis (MVPA) is an increasingly popular method of analyzing functional magnetic resonance imaging (fMRI) data1-4. Typically, the ...

Research

JoVE Journal

Neuroscience

Basics of Multivariate Analysis in Neuroimaging Data

Multivariate analysis techniques for neuroimaging data have recently received increasing attention as they have many attractive features that cannot be ...

Brain Imaging Investigation of the Impairing Effect of Emotion on Cognition

Emotions can impact cognition by exerting both enhancing (e.g., better memory for emotional events) and impairing (e.g., increased emotional ...

Transcriptome Analysis of Single Cells

Many gene expression analysis techniques rely on material isolated from heterogeneous populations of cells from tissue homogenates or cells in ...

T-maze Forced Alternation and Left-right Discrimination Tasks for Assessing Working and Reference Memory in Mice

Forced alternation and left-right discrimination tasks using the T-maze have been widely used to assess working and reference memory, respectively, in ...

Creating Objects and Object Categories for Studying Perception and Perceptual Learning

In order to quantitatively study object perception, be it perception by biological systems or by machines, one needs to create objects and object ...

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

In utero electroporation (IUE) has become a powerful technique to study the development of different regions of the embryonic nervous system 1-5. To date ...

The Swimmeret System of Crayfish: A Practical Guide for the Dissection of the Nerve Cord and Extracellular Recordings of the Motor Pattern

Here we demonstrate the dissection of the crayfish abdominal nerve cord. The preparation comprises the last two thoracic ganglia (T4, T5) and the chain of ...

Peering at Brain Polysomes with Atomic Force Microscopy

The translational machinery, i.e., the polysome or polyribosome, is one of the biggest and most complex cytoplasmic machineries in cells. Polysomes, ...

Analyzing Synaptic Modulation of Drosophila melanogaster Photoreceptors after Exposure to Prolonged Light

Analyzing Synaptic Modulation of Drosophila melanogaster Photoreceptors after Exposure to Prolonged Light

The nervous system has the remarkable ability to adapt and respond to various stimuli. This neural adjustment is largely achieved through plasticity at ...

Visualization of the Axonal Projection Pattern of Embryonic Motor Neurons in Drosophila

Visualization of the Axonal Projection Pattern of Embryonic Motor Neurons in Drosophila

The establishment of functional neuromuscular circuits relies on precise connections between developing motor axons and target muscles. Motor neurons ...