In automated driving, the vehicle decides what it needs to do — it steers, brakes and accelerates. However, until vehicles can completely dispense with a driver, semi-automated vehicles will support whomever is at the wheel to give them increasingly more freedom. In the case of semi-automated vehicles, it goes without saying that handovers from the vehicle to the driver are still required, for example, if there is construction work on the highway or when transitioning from driving on the highway to the city. The vehicle must therefore be intelligent on two levels: it must be able to interpret traffic and monitor what is going on inside the vehicle, as well as interact with the driver. What is the driver doing? How quickly could they take control of the vehicle? Although driver monitoring systems already exist, so far these use hardly any camera image data and are mainly limited to fatigue detection.

Artificial Intelligence detects what the driver is doing

Researchers at Fraunhofer IOSB are focusing on the interaction between the driver and the vehicle — and working to bridge the gaps. “Our technology not only enables us to identify facial features but also the current poses of the driver and the passengers,” says Michael Voit, Group Manager at Fraunhofer IOSB. “We can then determine from these poses what the driver and passengers are currently doing.”

At the core of the development lie machine learning algorithms and processes, such as Artificial Intelligence (AI). Algorithms analyze the camera data in real time to find out whether the driver is on the phone, playing with children or looking at a passenger’s cellphone. IOSB technology therefore goes beyond image recognition and interprets the activities in context. The researchers first trained the system by manually annotating numerous camera shots: Where are people’s hands, feet and shoulders? Where are objects such as smartphones, books and other items located? They then evaluated the algorithms using new images and corrected or verified their results.

The system abstracts images of the driver or passengers to form a digital skeleton — a type of stick figure that replicates the person’s body poses. It consequently deduces the activity using the skeletal movement and supplementary object recognition. “The algorithms can thereby tell whether someone is sleeping or looking at the street, how distracted the person is and how long it will take them to focus back on the road,” explains Voit. The system supports both traditional video cameras and infrared cameras that can see in the dark, as well as 3D cameras that measure the distance between objects and the camera. The system even gives interior designers freedom in terms of camera placement.

The researchers have been dealing with issues related to activity detection in vehicle interiors in numerous research projects, collaborating with renowned car manufacturers such as Audi and Volkswagen, as well as with suppliers such as Bosch and Continental. The projects have been supported by the German Federal Ministry of Education and Research (BMBF), the German Federal Ministry for Economic Affairs and Energy (BMWi) or the Federal Ministry of Transport and Digital Infrastructure (BMVI). “We can not only detect the activities of the driver but those of all passengers, too — both in the front and back of the vehicle,” confirms Voit. “The technology is ready for pilot production. We are already in contact with companies who want to use our technology.” Linking the developments is the institute’s own driving simulator, which industry customers can also benefit from in the context of individual R&D projects. Simulating traffic situations, it forms the basis for collecting relevant driving and behavioral data. Furthermore, its extensive sensor technology means that the behavior of all passengers can be studied.

Data protection is a priority

As far as data protection and security aspects are concerned, the researchers made this a priority from the start. “The camera data are analyzed in real time, not saved and do not leave the vehicle at any point. Personalized models are also not needed for this, so no personal data is collected,” says Pascal Birnstill, Senior Scientist at Fraunhofer IOSB, who specializes in data security, data protection and transparency. The technology respects privacy from the outset and thus complies with the strict regulations and high level of data protection awareness in the EU.

Numerous applications — even beyond activity detection

A new EU regulation highlights how important activity detection is; driver monitoring is to become mandatory in automated cars, no matter their level of automation. Using Fraunhofer IOSB technology, vehicle manufacturers can not only meet this requirement but also make numerous visions in terms of autonomous driving a reality. Let’s take an example: voice recognition for communication between people and cars is fast approaching its limits. The command “Park there” is not meaningful in itself. However, when used in conjunction with body pose identification and activity detection, the system can determine which parking space the user is pointing to at that very moment. The system can also help with safety aspects of driverless vehicles. While it is currently still the driver’s responsibility to ensure that all passengers comply with safety regulations and fasten their seatbelts, for example, this will soon be the job of the driverless vehicle — one example being autonomous taxis or buses. Reliable monitoring of the vehicle’s interior is also essential for this.