The lab layout

The CSRA consists of three rooms with a total space of 60m². One large room has kitchen, living room and hallway areas. Additionally, there are a bathroom and a multi-functional room, the gym, that can be changed according to changing research projects and further needs. Besides a developer access, there is a smart door connecting to a public corridor to allow the investigation of arrival and departure situations. The area in front of this door and a control room where researchers operate experiments or develop new components are non-apartment parts of the lab where special privacy considerations apply.

Sensors

The CSRA features a number of sensors. For visual tracking and recording, there are 12 RGB-D cameras in the ceiling used for person tracking, looking straight down to cover the whole apartment area approximately at waist level, 4 high-quality cameras for uncompressed video, and 4 lower-quality cameras that can stream H.264 video. The bathroom is exempt from any video surveillance for privacy reasons. As non-visual means for tracking and activity detection, there are 16 Fibaro motion sensors (using the Z-Wave protocol) and two areas oufitted with capacitive flooring. For audio recording and speech recognition, there are 12 cardioid microphones and 5 omnidirectional microphones installed (details see below).

Actuators

Output-wise, the apartment has a range of devices at its disposal, including a Samsung Surface and a 4K TV, a pan-tilt projector and two fixed projectors, 18 studio monitors plus 2 subwoofers and 50 Philips Hue LED light bulbs with configurable hue and intensity (using the ZigBee protocol), as well as smart drawer and cupboard handles that can show different light colors and patterns for targeted attention control. With the help of a KNX system, 66 plug sockets can be controlled and it can be sensed how much power is drawn from each socket. Furthermore, 56 wall buttons, 5 radiator and 2 temperature controllers, 2 shutters and 4 sun-blinds are integrated via KNX. The windows and doors of the apartment are equipped with wireless reed switches. There are also reed switches for the cupboard doors and drawers in the kitchen.

The smart entrance door was equipped with a number of sensors as part of the KogniHome project, including an outwardfacing camera, a radar sensor and pickup microphones. It also has a motor to open automatically, a remote-controlled lock, and an RGB frame, as well as an e-ink display on the outside.

Interaction devices

There are two instances of a virtual agent displayed on two monitors, one in the hallway and one in the kitchen with webcams to track users with their eyes and recognize learned faces. The CSRA is also home to the mobile service robot Floka which is based on the Meka-Bot M1 and has an anthropomorphic upper body. It features multiple cameras, a laser range nder and microphones. The sensor head can be easily exchanged with a human-like robotic head that was adapted from the robot head Flobi and that is able to show expressive social cues. While it is realized as an independent autonomous system, it is also integrated with the apartment. The four-fingered hands feature compliant force controlled actuators. In total, the robot is equipped with 37 motor-powered joints: 7 per arm, 5 per hand, 2 in the head, 2 in the torso and 9 joints to actuate the base including a z-lift.

Network

Beyond KNX, we use three parallel gigabit Ethernet networks in the CSRA: one is used for IP-based connectivity of system devices and services while the other one is for transmission of introspection data and video streams that are recorded as ground truth in experimental studies. There is also a WLAN access point that allows wireless components to be integrated into the CSRA infrastructure, including the robot.

Audio system

The CSRA audio input/output system consists of 20 installed speakers and 20 microphones, the latter consisting of 12 cardioid, 5 omnidirectional and 3 pickup microphones. These are centrally accessible by the common RSB middleware and can be combined into different configurations for different purposes (such as different stereo setups, 4.0, 5.1 or higher multichannel setups). Both input and output can handle different sample rates and bit depths. The loose coupling of the RSB middleware was somewhat of a challenge, as communication is basically stateless and the audio subsystem needs to be able to handle both event-based and streaming audio output. Our custom-built audio server, however, handles any combination of these cases with an arbitrary number of senders and mixes them transparently. Integration with the BCO platform handles location- and type-based service discovery.