Symbiosis of Language and Action

Research Areas: 

SyLA investigates the ways in which language and action support the cognitive and linguistic development of a child, taking into account situative and intrapersonal factors. By combining observational and experimental studies of caregiver-child interactions and tutor-learner scenarios, the aim of the project is to specify parameters of caregiver or tutor behavior modification, which can be implemented in artificial systems as “top down constraints”, to allow learning, recognition and execution of human actions.


Methods and Research Questions: 
  • How does the symbiosis of language and action support the cognitive and linguistic development of a child depending on situative intrapersonal factors, and
  • How can this symbiosis be used for technical systems (e.g. social robots), so that they can learn to recognize and carry out human actions?

The project addresses these questions by focusing on three particular aspects:

  1. Goal-directed action as it becomes evident in the case of object manipulation. The focus lies in the variability and stability in the movement and speech of caregivers and how they convey knowledge about actions and relevant aspects of objects.
  2. Meaning-oriented action in the form of manual gestures. Gestures support (mostly in preverbal infants) or supplement the semantic force of a verbal utterance. The research focus here is on situative and intrapersonal factors.
  3. Conceptual development. The focus here is on the interplay between language and action as a mechanism promoting the development of meaning and the acquisition of new words. Depending on the language system, an action can be picked up conceptually and processed in memory in different ways. Everyday caregiver-infant interactions are considered by describing the ways in which a) actions make language more salient and b) language structures actions. How does the symbiosis develop over time? Another focus is on children‘s existing event knowledge and ways of associating it to their word learning processes. Can the association between children‘s event knowledge and a new word be conveyed in child-directed input and facilitate children‘s word learning?

 A combination of methods is used to explore the above aspects in four different parallel projects:

  1. Goal-directed action: In this study, computational analyses of hand trajectories are used to account for the movement adjustments that adults do in tutoring scenarios with children.
  2. Gesture studies: In this longitudinal study, we elicit mother-child interactions in two different situations: reading a book and free play. We annotate and code gestural behavior of the mother and analyze it with respect to the specifics of the situation and the age of the child.
  3. Language structures action: This study uses a naturalistic, longitudinal design to describe the interplay between language and action within the interactional context in which it occurs and to track how it is shaped in time by the developing infant. The subproject uses fine-grained qualitative analyses to extract behavioral features that are then used to describe the data quantitatively.
  4. Associative event knowledge: In this project, a systematic combination of qualitative and quantitative methods is combined. In a first explorative study, the nature of the parental use of associative event knowledge of their children in child-directed input is determined. The effect of parental associative event knowledge on children‘s word learning is finally systematically investigated in a training study.



The symbiosis of language and action emerges as a teaching strategy as we found that – across different modalities (speech, gesture, actions and synchronized action-speech packages), behavioral modifications occur when addressing children and that robots also receive such modified input from which they can benefit. These modifications are present in the input within early interactions with infants. We found that German mothers vocalize in a tight relationship with their actions which effect is perceivable in language and tangible to the infants. The symbiosis of action can be brought into a situation by associations. We found that embedding a new word in an action sequence (as an event unfolds) will facilitate the learning process. Our findings indicate a benefit of this associative input for children whose productive vocabulary was reported to be more advanced.