Virtual Social Interaction Conference Speaker
Glasgow, UK
July 2021*
Abstract: Co-speech gestures of the hands transform mental constructs into physical forms and actions, and are widely shown to play a powerful role in face-to-face interaction. Our interest lies specifically in metaphoric gestures, which embody abstract concepts and reflect the relationship between thought, speech, and motion. These gestures not only increase speaker interpretability and viewer comprehension in both humans and virtual agents, but importantly can alter how the viewer qualitatively understands information presented by the speaker. Gesture behavior and interpretations of some gestures differ across cultures. For instance, emblematic gestures, such as the “thumbs-up,” are entirely culturally dependent. Similarly, frequency and amplitude of gesture performances vary widely across cultures. However, metaphoric gestures ground abstract concepts in physical motion. By the Embodied Cognition hypothesis, interpretations of these gestures may therefore be consistent across cultures, as all individuals experience the same physical world. Similar interpretations of the same metaphoric gestures would imply the same conceptual metaphors are driving gestures in individuals from different cultures. By decomposing metaphoric gestures into physical components, we hope to use cross-cultural interpretations of combinations of these components as a tool to study the potential for universality of physical embodiment of abstract concepts. Our interest is in modeling both the underlying processes that go from mental construct to gesture, as well as in modeling the perception of the behavior, specifically how those gestures then influence observers. We can in turn use the assessment of the latter perceptual model’s effectiveness in an interaction to determine a virtual human’s generation model, leading to more effective virtual human performances. The presented study focuses on a key step along this path, the perception of metaphoric gestures. In this crowd-sourced study, we present metaphoric gestures with origins in American and Japanese speakers to viewers of each culture and ask viewers to self-report interpretations of abstract notions seen in these gestures. These notions were gathered using thematic analysis of free-response interpretations of these gestures. Results from human studies indicate that interpretations of abstract notions that may be represented by these gestures, such as conflict and togetherness, differ significantly across cultures. This indicates that embodied signifiers of these concepts differ both between individuals and cultures, discouraging the idea of a universal mapping from physical motion to abstract concept.
*Note: as this Conference did not publish official proceedings, this is not a formal publication.