Multimodal Coordination Dynamics During Conversation

A dynamical systems approach to how acoustic, linguistic, and movement behavior couple together over the course of a conversation.

Award: NSF 1103356 (Role: PI)

When two people interact, complex patterns of behavior emerge quite spontaneously. These patterns are organized around multiple types of movement that simultaneously co-occur with little to no conscious awareness. Nevertheless, they form a stable network of associations that guide how people converge on meaning and respond to higher-level communicative goals. In this research, we take a dynamical systems approach to understand how various components of behavior, including acoustic, linguistic, and movement components, interact over time.

This focus on the interdependence of behavior can also be characterized as a focus on behavioral coordination, whereby individual’s cognitive processes and behavioral patterns are integrated into a coupled system. How this coordination changes under various high-level conditions, such as when people lie to each other, persuade each other, or attempt to understand each other, is currently being explored in the lab. To do so, we use automated motion analysis from video, motion tracking tools like the Microsoft Kinect, and high- powered infrared motion tracking cameras.

Selected papers

  • Duran, N. D., Paxton, A., & Fusaroli, R. (2019). ALIGN: Analyzing linguistic interactions with generalizable techNiques—a Python library. Psychological Methods. [PDF]
  • Duran, N. D., & Fusaroli, R. (2017). Conversing with a devil’s advocate: Interpersonal coordination in deception and disagreement. PLOS ONE, 12, e0178140. [PDF]
  • Moulder, R. G., Duran, N. D., & D’Mello, S. K. (2022). Assessing multimodal dynamics in multi-party collaborative interactions with multi-level vector autoregression. Proceedings of the ACM International Conference on Multimodal Interaction (ICMI ‘22). [PDF]
  • Eloy, L., Stewart, A. E. B., Amon, M. J., Reinhardt, C., Michaels, A., Sun, C., Shute, V., Duran, N. D., & D’Mello, S. K. (2019). Modeling team-level multimodal dynamics during multiparty collaboration. Proceedings of the ACM International Conference on Multimodal Interaction (ICMI ‘19). [PDF]
  • Grafsgaard, J. F., Duran, N. D., Randall, A. K., Tao, C., & D’Mello, S. (2018). Generative multimodal models of nonverbal synchrony in close relationships. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). [PDF]

See the full list on the publications page.