Name: 크로스 모달 변수 패턴 분석 (동영상) | JoVE Journal
Uploaded: 2011-11-09T13:00:00.000+00:00
Duration: 13 min 51 s
Description: 학습하세요 다중변량 패턴 분석(MVPA)을 활용한 인간 뇌의 다중감각 처리 메커니즘. fMRI 데이터로 시각 자극이 청각 및 체감각 피질에서 신경 활동을 유발하는 과정을 탐구합니다.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

playerTrialDesktop

This work was supported by grants by the Mathers Foundation and the National Institutes of Health (grant number 5P50NS019632-27) to Antonio and Hanna Damasio.

<ol><li>Haynes, J. -D., Rees, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Decoding+mental+states+from+brain+activity+in+humans%5BTitle%5D)+AND+%22Nat.+Rev.+Neurosci%22%5BJournal%5D)">Decoding mental states from brain activity in humans.</a> Nat. Rev. Neurosci. 7, 523-534 (2006).</li>
<li>Norman, K. A., Polyn, S. M., Detre, G. J., Haxby, J. V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Beyond+mind-reading%3A+multi-voxel+pattern+analysis+of+fMRI+data%5BTitle%5D)+AND+%22Trends.+Cogn.+Sci%22%5BJournal%5D)">Beyond mind-reading: multi-voxel pattern analysis of fMRI data.</a> Trends. Cogn. Sci. 10, 424-430 (2006).</li>
<li>Kriegeskorte, N., Bandettini, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Analyzing+for+information%2C+not+activation%2C+to+exploit+high-resolution+fMRI%5BTitle%5D)+AND+%22NeuroImage%22%5BJournal%5D)">Analyzing for information, not activation, to exploit high-resolution fMRI.</a> NeuroImage. 38, 649-662 (2007).</li>
<li>Mur, M., Bandettini, P. A., Kriegeskorte, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Revealing+representational+content+with+pattern-information+fMRI+-+an+introductory+guide%5BTitle%5D)+AND+%22Soc.+Cogn.+Affect.+Neurosci%22%5BJournal%5D)">Revealing representational content with pattern-information fMRI - an introductory guide.</a> Soc. Cogn. Affect. Neurosci. 4, 101-109 (2009).</li>
<li>Kamitani, Y., Tong, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Decoding+the+visual+and+subjective+contents+of+the+human+brain%5BTitle%5D)+AND+%22Nat.+Neurosci%22%5BJournal%5D)">Decoding the visual and subjective contents of the human brain.</a> Nat. Neurosci. 8, 679-685 (2005).</li>
<li>Formisano, E., De Martino, F., Bonte, M., Goebel, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((%22Who%22+is+saying+%22what%22%3F+Brain-based+decoding+of+human+voice+and+speech%3F%5BTitle%5D)+AND+%22Science%22%5BJournal%5D)">"Who" is saying "what"? Brain-based decoding of human voice and speech?</a> Science. 322, 970-973 (2008).</li>
<li>Meyer, K., Kaplan, J. T., Essex, R., Webber, C., Damasio, H., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Predicting+visual+stimuli+on+the+basis+of+activity+in+auditory+cortices%5BTitle%5D)+AND+%22Nat.+Neurosci%22%5BJournal%5D)">Predicting visual stimuli on the basis of activity in auditory cortices.</a> Nat. Neurosci. 13, 667-668 (2010).</li>
<li>Meyer, K., Kaplan, J. T., Essex, R., Damasio, H., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Seeing+touch+is+correlated+with+content-specific+activity+in+primary+somatosensory+cortex%5BTitle%5D)+AND+%22Cereb.+Cortex%22%5BJournal%5D)">Seeing touch is correlated with content-specific activity in primary somatosensory cortex.</a> Cereb. Cortex. 21, 2113-2121 (2011).</li>
<li>Damasio, A. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Time-locked+multiregional+retroactivation%3A+a+systems-level+proposal+for+the+neural+substrates+of+recall+and+recognition%5BTitle%5D)+AND+%22Cognition%22%5BJournal%5D)">Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition.</a> Cognition. 33, 25-62 (1989).</li>
<li>Meyer, K., Damasio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Convergence+and+divergence+in+a+neural+architecture+for+recognition+and+memory%5BTitle%5D)+AND+%22Trends.+Neurosci%22%5BJournal%5D)">Convergence and divergence in a neural architecture for recognition and memory.</a> Trends. Neurosci. 32, 376-382 (2009).</li>
<li>Friston, K. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Modalities%2C+modes%2C+and+models+in+functional+neuroimaging%5BTitle%5D)+AND+%22Science%22%5BJournal%5D)">Modalities, modes, and models in functional neuroimaging.</a> Science. 326, 399-403 (2009).</li>
<li>Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., Pollmann, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((PyMVPA%3A+a+Python+toolbox+for+multivariate+pattern+analysis+of+fMRI+data%5BTitle%5D)+AND+%22Neuroinformatics%22%5BJournal%5D)">PyMVPA: a Python toolbox for multivariate pattern analysis of fMRI data.</a> Neuroinformatics. 7, 37-53 (2009).</li>
<li>Harrison, S. A., Tong, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Decoding+reveals+the+contents+of+visual+working+memory+in+early+visual+areas%5BTitle%5D)+AND+%22Nature%22%5BJournal%5D)">Decoding reveals the contents of visual working memory in early visual areas.</a> Nature. 458, 632-635 (2009).</li>
<li>Serences, J. T., Ester, E. F., Vogel, E. K., Awh, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Stimulus-specific+delay+activity+in+human+primary+visual+cortex%5BTitle%5D)+AND+%22Psych+Sci%22%5BJournal%5D)">Stimulus-specific delay activity in human primary visual cortex.</a> Psych Sci. 20, 207-214 (2009).</li>
</ol>

No conflicts of interest declared.

The findings of our previous studies demonstrate that cross-modal MVPA is a useful tool to study the neural correlates of mental images experienced in the "mind's ear" and the "mind's touch". Specifically, the results show that the content of such images is correlated with neural activity in the early auditory and somatosensory cortices, respectively, providing direct empirical support for Damasio's framework of convergence-divergence zones. 
The basic paradigm we describe can be extended in a number of ways. Most obviously, similar studies can be carried out using different combinations of sensory modalities. In this respect, it is possible that establishing certain cross-modal associations prior to the experiment may increase the chances of success. For instance, to study cross-modal representations in olfactory cortices, one could prime subjects by exposing them simultaneously to the sight and the smell of a number of food items. Shortly after, inside the fMRI scanner, the olfactory memories of the smells would be triggered by visual cues, and MVPA could be used in an attempt to assign the trials to the correct food items based solely on activity in olfactory cortices. 
Another question of interest is whether the cross-modally induced patterns of activity in the early sensory cortices bear similarities with the patterns that are induced when sound or touch is actually experienced; in other words, does seeing the glass vase shatter invoke a similar pattern of neural activity in early auditory cortices as actually hearing the same event? This question, again, can be addressed through MVPA: is a classifier that has been trained on data recorded while subjects heard certain sounds able to correctly discriminate data recorded while the subjects watched corresponding video clips? In our auditory study, we did attempt such a classification, but the results were marginal (see Fig. 3 in ref. 7). In that study, however, the auditory associations of the participants were not controlled in any way; in other words, we do not know how similar the auditory images each subject associated with videos were to the audio tracks used to train the classifier. Again, it might be interesting to address the same question after specific cross-modal associations have been primed in the subjects, as described above for a visuo-olfactory association. This would permit to control more reliably the mental experience of the subjects during the video trials and thus might increase prediction performance of the classifier.
To conclude, we have introduced an extension to the classical MVPA paradigm by showing that stimuli can be predicted not only within, but also across sensory modalities. Thus, we show that the use of MVPA is not limited to investigating the correlates of perceptual representations induced directly by external sensory stimulation. Rather, MVPA can also assess the neural substrate of mental images that are triggered internally: in accordance with Damasio's convergence-divergence zone framework, our findings suggest that the conscious experience of mental images that are reconstructed based on memories is correlated with content-specific neural representations in the early sensory cortices.

cross-modal multivariate pattern analysis

Cross-Modal Multivariate Pattern Analysis

Multivariate pattern analysis (MVPA) is an increasingly popular method of analyzing functional magnetic resonance imaging (fMRI) data1-4. Typically, the method is used to identify a subject's perceptual experience from neural activity in certain regions of the brain. For instance, it has been employed to predict the orientation of visual gratings a subject perceives from activity in early visual cortices5 or, analogously, the content of speech from activity in early auditory cortices6.
Here, we present an extension of the classical MVPA paradigm, according to which perceptual stimuli are not predicted within, but across sensory systems. Specifically, the method we describe addresses the question of whether stimuli that evoke memory associations in modalities other than the one through which they are presented induce content-specific activity patterns in the sensory cortices of those other modalities. For instance, seeing a muted video clip of a glass vase shattering on the ground automatically triggers in most observers an auditory image of the associated sound; is the experience of this image in the "mind's ear" correlated with a specific neural activity pattern in early auditory cortices? Furthermore, is this activity pattern distinct from the pattern that could be observed if the subject were, instead, watching a video clip of a howling dog?
In two previous studies7,8, we were able to predict sound- and touch-implying video clips based on neural activity in early auditory and somatosensory cortices, respectively. Our results are in line with a neuroarchitectural framework proposed by Damasio9,10, according to which the experience of mental images that are based on memories - such as hearing the shattering sound of a vase in the "mind's ear" upon seeing the corresponding video clip - is supported by the re-construction of content-specific neural activity patterns in early sensory cortices.

Classical multivariate pattern analysis predicts sensory stimuli a subject perceives from neural activity in the corresponding cortices (e.g. visual stimuli from activity in visual cortex). Here, we apply pattern analysis cross-modally and show that sound- and touch-implying visual stimuli can be predicted from activity in auditory and somatosensory cortices, respectively.

Multivariate pattern analysis (MVPA) is an increasingly popular method of analyzing functional magnetic resonance imaging (fMRI) data1-4. Typically, the ...

The goal of the following experiment is to use multivariate pattern analysis of functional MRI data to investigate multimodal processing in the human brain. This is achieved by first creating a specific kind of sensory stimuli, which are presented to the subjects through a single sensory modality. For instance, the visual, but implies strong associations in a different modality.For instance, the auditory while the subjects perceive such stimuli neural activity in the parts of the brain pertaining to the modality of the association. The auditory cortices in this example is recorded using FMRI. The recorded activity is analyzed using multi-variate pattern analysis with the goal of predicting which of several stimuli the subject perceived in the original modality.The results show that unimodal stimuli containing information pertinent to more than one modality can induce content specific neural activity in the early sensory cortices of modalities other than the one through which they are presented. The technique we introduce today, which we refer to as Crossmodal multivariate pattern analysis, represents a natural extension of conventional multivariate pattern analysis with the difference that sensory STEMI are classified across rather than within sensory modalities. We first had the idea for this method when trying to produce empirical evidence for a neuro architectural framework introduced by Demio more than two decades ago.In this video, we'll introduce you not only to the technical aspects of multivariate pattern analysis or MVPA for short, but also to this theoretical framework. We will first point out some key differences between MVPA and conventional univariate FMRI analysis. Consider the following.Example, if a subject is presented with two different visual stimuli like an apple and an orange, then both stimuli averaged across a number of trials induce a specific pattern of neural activity. In the primary visual cortex symbolized here by the activation levels of six hypothetical voxels. In conventional FMRI analysis, there are essentially two methods to analyze these patterns.The first is to compare the average level of activity the stimuli induced in the entire region of interest. The difference between the averages, however, may not be significant. The second method is to establish a subtraction contrast for each voxel.The activation level during the apple condition is subtracted from the activation level during the orange condition, and the resulting difference for each voxel can be visualized on a whole brain contrast image. Again, however, these differences may be small and may reach the required statistical criterion only for a few voxels. However, unlike univariate analysis methods, MVPA is able to detect patterns within voxels by considering the activation levels of all the voxels simultaneously.While only a few of the activation differences may be significant in isolation, the two patterns when considered in their entirety may indeed be statistically different. Another major difference between conventional FMRI analysis and MVPA is that the latter method uses what is called reverse inference. In conventional FMRI analysis, the researcher typically asks a question of the type will two different visual stimuli.For instance, the picture of a face and the picture of a house lead to different activity levels in a specific region of interest, such as the fusiform face area. By contrast, MVPA is typically expressed in terms of reverse inference or decoding, and asks, based on the pattern of neural activity in a specific brain region, will we be able to predict which of two visual stimuli a subject perceived. It is important to note, however, that from a statistical point of view, it is equivalent to say that two stimuli lead to distinct activity patterns in a given brain region and to say that the activity pattern in that brain region permits prediction of the inducing stimulus.In other words, the sensitivity of MVPA is superior to that of univariate analyses because it considers several voxel simultaneously and not because it proceeds in an inverse direction. To return to the earlier example, let's consider a typical MVPA paradigm that assesses whether seeing an apple induces a different pattern of neural activity in primary visual cortex than seeing an orange. In a first step, FMRI data are recorded while a subject sees a large number of the stimuli to be discriminated.The acquired data are then divided into a training data set and a testing data set. The data from the training set are entered into a pattern classifier, which attempts to detect features in the neural patterns that distinguish the two types of trials from one another. Next, the classifier is presented with unlabeled data from the testing set and based on the patterns it detected in the training data set attributes the most likely label to each of the testing trials for each stimulus.The classifier guess is then compared to the correct stimulus label and classifier performance is calculated as the percentage of correct guesses. As we've seen the classifier in the example reduce the correct stimulus labels in nine out of 12 cases, or 75%as chance performance. In such a two way discrimination would be 50%This suggests that there are indeed consistent differences between the neural patterns induced in V one by the orange and the apple stimuli.Of course, the significance of this result would have to be proven statistically. One important issue to keep in mind in such a classification experiment is that the training and the testing data sets be completely independent from one another because only if that's the case can any conclusions be drawn as to the generalizability of the patterns that have been learned during the training session. For this reason, MVPA paradigms often use a so-called cross validation paradigm.While this procedure serves to maximize the number of training and testing trials that can be gained from a given data set, it also assures at the same time that the training and testing sets are do not overlap during the individual classification steps. Consider the following MVPA experiment with eight functional runs. In a first cross validation step, the classifier is trained on the data from runs one through seven and tested on the data from run eight.In the second step, the classifier is then trained on runs one through six as well as on run eight, and subsequently tested on run seven. Following this schema, eight cross validation steps are carried out with each run serving as the test run exactly. Once classifier performance is obtained for each cross validation step and averaging these results yields overall performance.There are freely available software packages on the internet to perform MVPA such as pi MVPA and the toolbox offered by the Princeton Neuroscience Institute. Experimental paradigms such as the one just described, have been used successfully to predict perceptual stimuli from neural activity in the corresponding parts of the cerebral cortex, so for example, to predict visual stimuli based on the activity in visual cortices or auditory stimuli. Based on the activity in auditory cortices, we would now like to introduce an extension of this basic idea in which perceptual stimuli are predicted not only within but across modalities.Our idea draws on the fact that perception is intricately linked to the recall of memories. For example, a visual stimulus that has a strong auditory implication such as the site of a glass base shattering on the ground. It's gonna automatically trigger in our mind's ear an auditory image that shares similarities with the auditory images we experienced on previous encounters with breaking glass.According to a framework introduced by Theo in the late 1980s, the memory association between the site of the shattering base and the corresponding sound is stored in so-called convergence divergence zones or CDs for short convergence. Divergence zones are conceptualized as ensembles of neurons that are located at the various hierarchical levels of the sensory systems. As their name implies.CDs at each level receive convergent bottom up projections from lower order cortices and turn. They sent back divergent top-down projections to those same lower order cortices Due to the convergent bottom-up projections. CDs can be activated by perceptual representations in multiple modalities, for instance, both by the site and the sound of a shattering base due to the divergent top-down projections.They can then promote the reconstruction of associated images by signaling back to the early sensory cortices of additional modalities. Consider the sequence of activation a purely visual, but sound implying stimulus will induce according to this framework. The stimulus first induces a specific pattern of neural activity in the early visual cortices via convergent bottom-up projections.The early visual cortices activate the first level of CDs, the CDZ ones. Depending on the exact pattern of activity in the corresponding early cortical sector, A CDZ may become activated or it may remain inactive. CDZ ones project upward to cdz twos just as cdz ones detected.Activity patterns in the early visual cortices, cdz twos detect patterns of activity among cdz ones, several cdz twos may become activated, but for simplicity only one is depicted here via top down projections. Cdz ones may at the same time complete the activity pattern in the early visual cortices via several additional levels of c DZs CDZ twos. Twos project forward to CDZ ends in multimodal association.Cortices.Again, multiple CDZ ends may become activated, but only one is depicted for reasons of simplicity. Again, CD Z twos also signal backward to CDZ ones, which in turn may further modify the pattern originally induced in the early visual cortices. The CDZ ends signal back to the CZ twos of all modalities.In the auditory cortices, a neural pattern will be constructed, which permits the subject to experience in the mind's ear, an auditory image associated with the visually presented stimulus. The depicted top-down signaling to the somatosensory modality reflects the fact that almost any visual stimulus also implies some tactile associations. Thus, the framework predicts that a sound implying visual stimulus would lead to a content specific pattern of neural activity.In the early auditory cortices. Analyzing this neural pattern using MVPA, it should therefore be possible to predict which of several sound implying visual stimuli a subject has seen. In the first experiment, neural activity was recorded from the early auditory cortices while subjects watched nine different video clips of objects and events that implied sound.The region of interest in this experiment was a restricted area on the Semal plane, which comprised primary auditory cortex and very early auditory association cortices. In a second experiment, neural activity was recorded from primary somatosensory cortex while subjects viewed five different video clips that implied touch. In this experiment, the region of interest comprised the primary somatosensory cortex located in the postcentral gyrus.Both studies involved eight subjects in the auditory study. An MVPA classifier performed above the chance level of 50%for all possible two-way discriminations between pairs of stimuli. In 26 out of the 36 discriminations classifier performance was significantly different from the chance level of 0.5.Likewise, in the somatosensory study, the classifier performed above chance in all two-way discriminations, and eight out of the 10 discriminations reached statistical significance. As you can see, using crossmodal multivariate pattern analysis, we've been able to demonstrate that the perception of visual stimuli that implies sound or touch leads to content specific neural representations in early auditory and somatosensory cortices in accordance with the theoretical framework introduced earlier. Clearly, the experimental paradigm we have presented does not have to remain restricted to the modalities involved in our own experiments, but can be extended to other sensory modalities as well.We hope therefore, that other groups will join us in extending this kind of research in an attempt to broaden our knowledge about how to bring in processes multimodal stimuli from the environment.

cross-modal-multivariate-pattern-analysis

Research

JoVE Journal

Neuroscience

The current article describes the basics of multivariate analysis and contrasts it to the more commonly used voxel-wise univariate analysis. Both types of analysis are applied to a clinical-neuroscience data set. Supplementary split-half simulations show better replication of the multivariate results in independent data sets.

Basics of Multivariate Analysis in Neuroimaging Data

Multivariate techniques including principal component analysis (PCA) have been used to identify signature patterns of regional change in functional brain images. We have developed an algorithm to identify reproducible network biomarkers for the diagnosis of neurodegenerative disorders, assessment of disease progression, and objective evaluation of treatment effects in patient populations.

Identification of Disease-related Spatial Covariance Patterns using Neuroimaging Data

Informational connectivity measures the correspondence between time courses of multi-voxel information across different brain regions. Multi-voxel pattern discriminability time series are extracted from regions and compared, revealing networks that are not identified in a typical functional connectivity approach.

Using Informational Connectivity to Measure the Synchronous Emergence of fMRI Multi-voxel Information Across Time

The protocol presents an experimental psychophysics paradigm to obtain large quantities of similarity judgments, and an accompanying analysis workflow. The paradigm probes context effects and enables modeling of similarity data in terms of Euclidean spaces of at least five dimensions.

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

We present a flexible, extendible Jupyter-lab-based workflow for the unsupervised analysis of complex multi-omics datasets that combines different pre-processing steps, estimation of the multi-omics factor analysis model, and several downstream analyses.

Application of Unsupervised Multi-Omic Factor Analysis to Uncover Patterns of Variation and Molecular Processes Linked to Cardiovascular Disease

In this article we describe a simple method for the harvesting of single cells from rat primary neuronal cultures and subsequent transcriptome analysis using aRNA amplification. This approach is generalizable to any cell type.

Transcriptome Analysis of Single Cells

This article presents the protocol of T-maze tests using a modified automated apparatus for assessing the learning and memory functions in mice.

T-maze Forced Alternation and Left-right Discrimination Tasks for Assessing Working and Reference Memory in Mice

We describe a novel methodology for creating naturalistic 3-D objects and object categories with precisely defined feature variations.  We use simulations of the biological processes of morphogenesis and phylogenesis to create novel, naturalistic virtual 3-D objects and object categories that can then be rendered as visual images or haptic objects.

Creating Objects and Object Categories for Studying Perception and Perceptual Learning

This article describes in detail a protocol to electroporate in utero the cerebral cortex and the hippocampus at E14.5 in mice. We also show that this is a valuable method to study dendrites and spines in these two cerebral regions.

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

Here we describe the dissection of the crayfish abdominal nerve cord. We also demonstrate an electrophysiological technique to record fictive locomotion from swimmeret motor neurons.

The Swimmeret System of Crayfish: A Practical Guide for the Dissection of the Nerve Cord and Extracellular Recordings of the Motor Pattern

학습하세요 다중변량 패턴 분석(MVPA)을 활용한 인간 뇌의 다중감각 처리 메커니즘. fMRI 데이터로 시각 자극이 청각 및 체감각 피질에서 신경 활동을 유발하는 과정을 탐구합니다.

Cross-Modal Multivariate Pattern Analysis

더 많은 동영상 탐색