Beat gestures and speech processing: When prosody extends to the speaker’s hands

TitleBeat gestures and speech processing: When prosody extends to the speaker’s hands
Publication TypeThesis
Year of Publication2015
AuthorsBiau E, Soto-Faraco S
Academic DepartmentDepartment of Experimental and Health Sciences
Number of Pages221
Date Published11/2015
UniversityPompeu Fabra
Thesis TypePhD
KeywordsAV speech, Beat gestures, ERPs, fMRI, Oscillations

Speakers naturally accompany their speech with hand gestures. In particular, they spontaneously extend the auditory prosody to visual modality through rapid and biphasic beat gestures, helping them to structure their narrative and emphasize relevant information. The present thesis aimed to increment the relatively less documented beat gestures and their neural correlates on the listener’s side. We developed a naturalistic approach combining political discourse presentations with neuroimaging techniques (ERPs, EEG and fMRI) to investigate beats correlates in both temporal and spatial dimensions. We also set experimental procedures to determine behavioral measures indexing the influence of beat gestures on audiovisual speech processing. The main findings of the thesis first revealed that beat-speech processing engaged language-related areas, suggesting that gestures and auditory speech are part of the same language system. Second, the time course analyses revealed that the presence of beats modulated the auditory processing of affiliated words around their onsets and later at phonological stages. We concluded that listeners perceive beats as visual prosody and rely on their predictive value to anticipate relevant acoustic cues of their corresponding words, engaging local attentional processes. The present dissertation confirmed that, even if simple, spontaneous beats presented in continuous audiovisual speeches are a good alternative to investigate the neural correlates of gesture-speech processing.