Deep Familiarization and Learning
An important foundation of our cognitive skills is our familiarity with the many things that surround us. How did we gain this rich fabric of knowledge about the appearance and interaction patterns with objects, and how can we endow robots with similar capabilities that will enable them to quickly familiarize themselves with novel objects grounded in manual exploration, cooperative action and interactive exposure to language?
To approach these scientific questions, the FAMULA project pursues an interdisciplinary, synthetic-analytic approach in order to develop a "minimal cognitive architecture." This architecture will be iteratively refined through the interplay of experimental studies with human subjects and the progressive implementation of putative learning and interaction mechanisms on a bimanual head-arm system with anthropomorphic hands aided by vision and tactile sensing.
- Gaining a deep understanding of how a cognitive agent can become familiar with new objects and concepts through the interplay of manual action and language.
- Gathering concrete empirical results about human manual action and its use in creating improved or novel technologies and algorithms for haptic processing and advanced manual action control and learning.
- Transferring these insights and advances to a robotic system that can be both easily and naturally guided to expand its knowledge about objects, how we use them, and how we speak about them.
- Compile the essential elements of rapid familiarization into a portable Cognitive Interaction Toolkit that can be used to in new cognitive interaction technology applications.
- WP1 Development of Haptic-Centered Interaction Technology and Processing Strategies to study and implement the necessary sensorimotor interactions between the robotic hand(s) and objects.
- WP2 Situated Dialogue about Manual Actions to develop the necessary representations for connecting the concepts of physical and linguistic control, which will enable dialogue to be deeply grounded in manual action.
- WP3 Manual Interaction to experimentally investigate and implement the necessary structures for flexible hand/eye control along with interfaces for language, learning and (long-term) memory.
- WP4 Memory Architecture and Learning to create and store novel experiences with objects through guided exploration, and to reuse and generalize stored experiences in novel situations.
- WP5 Platform-Centered Integration provides the required software technology for continuous system integration and compiles the reusable functionality in the Cognitive Interaction Toolkit.
Year 1: Initial Demonstrator: This will involve setting up the demonstrator platform and experiment scenarios and adjusting all software interfaces to prepare for tight integration. The robot will be able to inspect a rigid object from various angles, to kinesthetically feel its shape and to connect the two input modalities. Furthermore, the robot will be able to integrate verbal comments about object characteristics or parts.
Year 2: Guidable System: The robot will now be able to benefit from kinesthetic and language-based guidance to familiarize itself with objects that have moveable parts. This will lead the robot to new categories, such as caps, lids or flaps, based on the similarity of movement possibilities. At this stage, the focus will be on hand-object relations.
Year 3: Relation Understanding: The robot will have advanced from hand-object to general object-object relations, enabling the robot to familiarize itself with how functional properties can arise from certain object arrangements. The robot will begin to recognize analogies and to engage in dialogues that reflect an understanding at this level of abstraction.
The Final Demonstrator fluently combines the capabilities developed during years 1–3. It will be able to strongly generalize about novel objects and will display high robustness for mechanical tolerances. It will also exhibit only graceful degradation when taxed beyond its capabilities. Furthermore, it will be capable of "cognitive" failure resolution modes, involving dialogues with questions, excuses or refusals that reflect that the system has a self-representation that includes important aspects of its own limits.