A Social Cognition-driven Approach to Dialogue Management

Research Areas: 

Dialogue Management constitutes a core component in any Spoken Dialogue System (SDS), controlling the progression of the interaction by keeping a dialogue history, estimating the user's goals and generally deciding “what to say next”. This problem is imbued with uncertainties, which state-of-the-art approaches often cope with at the level of single utterances. In our project, we aim to model higher-level processes that relate to social cognition (Mutual Beliefs, Intention Recognition, Theory of Mind), to build an improved Dialogue management framework for communicative virtual agents.

Methods and Research Questions: 

Dialogues are prone to misunderstandings and often suffer from uncertainty of Speech Recognition and Semantic Analysis. Drawing inspiration from how humans reduce these uncertainties through building "mental models" of each other, we aim to improve human-agent interaction by recognizing and generating relevant cues using an integrated Dialogue Management Component.

A Dialogue Manager acts a high-level control component in any Dialogue System or Conversational Agent, taking some form of abstract input (mostly some form of speech-act) from a input processing track and generating output (in the same form) for a behavior generation track. Common approaches to Dialogue Management make often use of deterministic models (Finite-State Machines, Slot-Filling Models), while more recent developments try to address the uncertainty in human-machine dialogue by utilizing probabilistic models (Bayesian Networks, POMDPs). Although the latter approaches proved to be useful, most of them concentrate solely on the speech recognition process as the main source of uncertainty. While this is reasonable, we investigate whether taking into account a greater variability of social cues and their underlying mechanisms related to social cognition (engagement, gaze tracking, shared attention, mutual beliefs, intention recognition) can help to further reduce uncertainty and enable a more natural and life-like interaction between humans and virtual agents. The main research questions are:

  1. How to build a mental model of the interlocutor that comprises but exceeds the perceptual and conceptual common ground, by also entailing predictions about possible beliefs, goals, and intentions of the user?
  2. How to construct and maintain this model based on social cues (e.g. gaze, feedback, etc.)?
  3. How to reduce uncertainty in dialog management through using this model for own action planning and managing floor (topic, initiative, and turn taking)?


As our first step we focus on dialogue initiative. Generally speaking, the interaction partner having the initiative makes topic proposals, takes the turn more often, and drives the interlocution towards a goal. Many existing systems rely on „system initiative“, by asking questions to the human interaction partner until sufficient information has been accumulated. A mixed-initiative approach, in which initiative switches between human and system, is seen as a better approximation to human dialogue and needed for mutual cooperation. We analyze a data corpus on how human interlocutors interact and coordinate themselves when solving conflicts in a calendar management setting (finding alternative time slots for conflicting appointments), to gain insights into the process of initiative-taking. Results will be transferred to the dialogue management component of the virtual agent „BILLIE“ acting as a “secretary” for the human user.


Comparison between different approaches and assessment of available methods; collection of a data corpus of human partners interacting in a calendar management task. Basic technical layout of a dialogue manager component for the virtual agent BILLIE based on a combination of a rule-based production system with slot-filling.