Fulltext

Multimodal Integration of Dynamic Audio-Visual Cues in the Communication of Agreement and Disagreement

Mehu, Marc ; van der Maaten, Laurens

In: Journal of Nonverbal Behavior, 2014, vol. 38, no. 4, p. 569-597

Aggiungi alla tua lista
    Titre
    • Multimodal Integration of Dynamic Audio-Visual Cues in the Communication of Agreement and Disagreement
    Auteur
    • Mehu, Marc. Swiss Center for Affective Sciences, University of Geneva, 7 rue des Battoirs, 1205, Geneva, Switzerland
    • van der Maaten, Laurens. Department of Computer Science, Delft University of Technology, 4 Mekelweg, 2628 CD, Delft, The Netherlands
    Type de document
    • Postprint
    Langue
    • Inglese
    Publié dans
    • Journal of Nonverbal Behavior, 2014, vol. 38, no. 4, p. 569-597. Springer US; http://www.springer-ny.com
    Autre version électronique
    Identifiant OAI-PMH
    • oai:doc.rero.ch:325095
    Summary
    Recent research has stressed the importance of using multimodal and dynamic features to investigate the role of nonverbal behavior in social perception. This paper examines the influence of low-level visual and auditory cues on the communication of agreement, disagreement, and the ability to convince others. In addition, we investigate whether the judgment of these attitudes depends on ratings of socio-emotional dimensions such as dominance, arousal, and valence. The material we used consisted of audio-video excerpts that represent statements of agreement and disagreement, as well as neutral utterances taken from political discussions. Each excerpt was rated on a number of dimensions: agreement, disagreement, dominance, valence, arousal, and convincing power in three rating conditions: audio-only, video-only, and audio-video. We extracted low-level dynamic visual features using optical flow. Auditory features consisted of pitch measurements, vocal intensity, and articulation rate. Results show that judges were able to distinguish statements of disagreement from agreement and neutral utteranceson the basis of nonverbal cues alone, in particular, when both auditory and visual information were present. Visual features were more influential when presented along with auditory features. Perceivers mainly used changes in pitch and the maximum speed of vertical movements to infer agreement and disagreement, and targets appeared more convincing when they showed consistent and rapid movements on the vertical plane. The effect of nonverbal features on ratings of agreement and disagreement was completely mediated by ratings of dominance, valence, and arousal, indicating that the impact of low-level audio-visual features on the perception of agreement and disagreement depends on the perception of fundamental socio-emotional dimensions.