Cortical mechanisms of seeing and hearing speech

No Thumbnail Available
Journal Title
Journal ISSN
Volume Title
Doctoral thesis (article-based)
Checking the digitized thesis and permission for publishing
Instructions for the author
Degree programme
56, [44]
Helsinki University of Technology Laboratory of Computational Engineering publications. Report B, 43
In face-to-face communication speech is perceived through eyes and ears. The talker's articulatory gestures are seen and the speech sounds are heard simultaneously. Whilst acoustic speech can be often understood without visual information, viewing articulatory gestures aids hearing substantially in noisy conditions. On the other hand, speech can be understood, to some extent, by solely viewing articulatory gestures (i.e., by speechreading). In this thesis, electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) were utilized to disclose cortical mechanisms of seeing and hearing speech. One of the major challenges of modern cognitive neuroscience is to find out how the brain integrates inputs from different senses. In this thesis, integration of seen and heard speech was investigated using EEG and MEG. Multisensory interactions were found in the sensory-specific cortices at early latencies and in the multisensory regions at late latencies. Viewing other person's actions activate regions belonging to the human mirror neuron system (MNS) which are also activated when subjects themselves perform actions. Possibly, the human MNS enables simulation of other person's actions, which might be important also for speech recognition. In this thesis, it was demonstrated with MEG that seeing speech modulates activity in the mouth region of the primary somatosensory cortex (SI), suggesting that also the SI cortex is involved in simulation of other person's articulatory gestures during speechreading. The question whether there are speech-specific mechanisms in the human brain has been under scientific debate for decades. In this thesis, evidence for the speech-specific neural substrate in the left posterior superior temporal sulcus (STS) was obtained using fMRI. Activity in this region was found to be greater when subjects heard acoustic sine wave speech stimuli as speech than when they heard the same stimuli as non-speech.
auditory cortex, electroencephalography, functional magnetic resonance imaging, magnetoencephalography, multisensory, speech, SI, somatosensory cortex, superior temporal sulcus
  • Möttönen, R., Krause, C. M., Tiippana, K., and Sams, M., 2002. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13, number 3, pages 417-425.
  • Klucharev, V., Möttönen, R., and Sams, M., 2003. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 18, number 1, pages 65-75.
  • Möttönen, R., Schürmann, M., and Sams, M., 2004. Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neuroscience Letters 363, number 2, pages 112-115.
  • Möttönen, R., Järveläinen, J., Sams, M., and Hari, R., Viewing speech modulates activity in the left SI mouth cortex. NeuroImage, in press.
  • Möttönen, R., Calvert, G. A., Jääskeläinen, I. P., Matthews, P. M., Thesen, T., Tuomainen, J., and Sams, M., 2004. Perception of identical acoustic stimuli as speech or non-speech modifies activity in left posterior superior temporal sulcus. Helsinki University of Technology, Laboratory of Computational Engineering publications, Technical Report B42.
Permanent link to this item