Neural correlates of the processing of self-adaptors, emblems, and iconic gestures with speech: an fMRI study

Abstract

Various types of arm-and-hand movements co-occurring with speech give rise to diverse cross-modal semantic relations. This study investigated how the brain processes self-adaptors, emblems, and iconic gestures with the same speech by using fMRI. Gestures with speech evoked bilateral fusiform gyrus and left supramarginal gyrus in visual and multisensory processing. These regions were involved in processing conventional forms and meanings in emblems with speech. Iconic gestures uniquely activated right supramarginal gyrus associated with spatial processing of unconventional configurations of event knowledge of lexical concepts. Self-adaptors without meaning particularly involved the left superior parietal lobule in directing spatial attention to hand movements and processing spatio-motoric attributes with motor representations. Without gestures, bilateral superior temporal gyrus and calcarine sulcus were engaged in processing sounds and meanings of audiovisual speech. The overall results demonstrate the brain’s sensitivity to gesture-speech semantic variation, and reveal the nature of knowledge in the involved neural regions.