Semantic processing of self-adaptors, emblems, and iconic gestures: An ERP study

Abstract

The study investigates how the brain processes self-adaptors, semantically-unrelated emblems, and iconic gestures along with speech. The three types of gestures give rise to a continuum of semantic distinctions in relation to the accompanying speech. The overall N400 component occurred between 500 and 800 msec after the simultaneous gesture and speech onsets. In comparison to the speech-only condition, the reduced N400 evidenced the facilitation effect of iconic gestures at the centro-parietal sites. The meaningful yet non-speech-related emblems elicited enhanced N400 s at the left frontal-parietal sites; the meaningless self-adaptors produced the largest N400 effect over the scalp at the frontal-parietal sites. Self-adaptors had produced a larger negativity of N400 than emblems did at the centro-parietal regions. The results evidence the automatic integration of gesture and speech, and the diverse influence of gesture on processing. Only iconic gestures facilitate the semantic integration with speech. For a linguistic meaning to integrate with a semantically-unrelated emblem is less effortful than with a self-adaptor, suggesting that the processing of meanings proceeds more readily than the processing of a meaningless gesture occurring at the same time with speech.