Workshop on Benchmarking and Evaluation of Interactive Cognitive Systems

07. Februar 2011 - 08. Februar 2011

Supported by Excellence Cluster Cognitive Interaction Technology (CITEC), Bielefeld University

In the last years, the overall progress in fields like cognitive robotics, social agents, human-robot interaction, has become manifest in impressive achievements. However, it is still very difficult to compare and judge different systems, to find appropriate measurements, to generate appropriate corpora for evaluation and training. The workshop will review recent developments in the field of system evaluation and benchmarking and discuss future requirements.

Currently, we experience an increase of competitions and challenges that are set up at different events or conferences. There are initiatives to establish standard robotic platforms and to exchange multimodal corpora. There is an increasing research community that does empirical studies in human-robot or human-agent interaction coming up with qualitative and quantitative measurements. We feel that there is substantial work in different fields, that will be very fruitful to exchange and discuss.The workshop will tackle questions along the following lines

  • What is the impact of existing challenges and competitions on the development of technical systems? How should they be adapted or changed in order to meet future needs?
  • What are the requirements that enable the comparison of approaches and techniques on a system level? What is needed in terms of standard platforms, software frameworks, interfaces, and multi-modal corpora?
  • Can we measure the "cognitive skills" of a system? If so, what are appropriate metrics and measurements? How can they be related to system design decisions and possible bottlenecks?

Confirmed Invited Speakers

  • Tijn van der Zant (University of Groningen)
  • Minoru Asada (Osaka University)
  • Martin Riedmiller (University of Freiburg)
  • Alexander Koller (Saarland University)
  • Jim Little (University of British Columbia)
  • David Traum (University of Southern California)
  • David House (KTH, Stockholm)
  • Astrid Weiss (University of Salzburg)
  • Sebastian Möller (TU Berlin/T-Laboratories)
  • Fabio Bonsignorio (University Carlos III of Madrid)
  • Gorgio Metta (Italian Institute of Technology)
  • Martin Buss (TU München)
  • Rüdiger Dillmann (Karlsruhe Institute of Technology)
  • Tamim Asfour (Karlsruhe Institute of Technology)
  • Rudolphe Gelin (Aldebaran Robotics)
  • Vincent C. Müller (Amerikaniko Kollegio Anatolia, Greece)

Organization committee

  • Sven Wachsmuth
  • Helge Ritter
  • Manja Lohse
  • Stefan Kopp
  • Jochen Steil
  • Ulrich Rückert