Automated driving & Incabin AI: ­Automotive topics at Fraunhofer IOSB

Future technologies, especially in the automotive and mobility sectors, are increasingly located in broad areas of tension. They connect people with technical systems, including mechanical engineering, electronics, digitalization, and AI. Their success on the market is determined by acceptance, legal conditions, technical performance, and costs for development and operation.

Contract research enables companies to benefit from current developments and trends – from an early assessment of which technologies are relevant and which innovations hold promising potential for the future, to competent development and testing. Our priority is to offer our clients fast and reliable guidance in a complex environment and to deliver concrete, applicable solutions. These solutions are tailored to specific needs and can thus be put directly into practical use.

In addition to decades of experience in the industry, we are supported by an excellent research infrastructure and a strong network of partners. Fraunhofer IOSB currently operates three test vehicles for automated and connected driving that are approved for road use, as well as several driving simulators and test facilities. These can be used to test experimental technologies at a low threshold and put them into practice.

As a partner of the Autonomous Driving Test Field Baden-Württemberg and operator of platforms and methods for traffic data collection and mapping in public spaces, we provide an essential data basis for AI and machine learning on the one hand, and are in constant exchange with data protection officers on the other.

As a partner of the KAMO: Karlsruhe Mobility High-Performance Center and the Fraunhofer Transport Alliance, Fraunhofer IOSB works closely and actively with key experts from Karlsruhe (such as KIT, FZI, HKA) and the Fraunhofer-Gesellschaft to offer our clients an even broader range of solutions for individual problems.

From low-threshold preliminary studies, research, benchmarks, and experiments to the overall development and testing of innovative mobility technologies - with our competencies and laboratories, we are your contact for research and development.

Safety applications / Automotive safety

With the increasing importance of AI and rising levels of automation, new requirements are emerging for safety and sensors in the exterior and interior of vehicles. New laws (General Safety Regulation II, GSR II) and test procedures (including those from EuroNCAP) take this into account and promote, in particular, sensor fusion between many, often optical, sensors. Data fusion, including the latest approaches to artificial intelligence, holds untapped potential and is firmly anchored in our research and development. To meet future legislative and EuroNCAP requirements, we at Fraunhofer IOSB are conducting research into the required detection of drowsiness and sleep, distraction and cognitive distraction, motion sickness, and level-compliant driver behavior. We rely on AI-driven technology development, varied AI data collection, and standardized user studies.

In line with the new requirements, concepts and standards for function development are also evolving. Classic standards, in particular ISO 26262 (functional safety), represent the focus on stochastic, technical function failures, graded according to risks in accordance with Automotive Safety Integrity Levels (ASIL). Even here, considerable testing and validation was required. The increasing complexity of functions, which affects not only technical failures but also operating errors or the behavior of other road users, is reflected in ISO 21448 (safety of the intended function, SOTIF). This covers all risks that arise even though all technical components are nominally functioning. The area of safety and AI, in turn, is covered by ISO 8800. Overall, this results in a complex structure of safety considerations for proving that a driving function in public transportation is actually safe. According to the current status, concrete verification requires real-world test drives as well as test bench tests and simulations in the areas of hardware-in-the-loop (HIL) or software-in-the-loop (SIL). Fraunhofer offers this service with extensive laboratories and efficient processes, and enables virtual testing using the Fraunhofer OCTANE/OCTAS simulation platform.

User Experience

The development and use of new, especially optical, sensors in vehicles enable new and innovative human-machine and human-AI interactions in vehicles. The user experience often determines market success – this is especially true for a technically mature product such as the passenger car. In the future, competition will be decided in the area of user experience. New functions are also continuing to be added to vehicles in large numbers, increasing their everyday value – even for activities that are not related to mobility. Electric vehicles can also offer full functionality when stationary, with a comfortable interior and optimized operating and display elements. Cameras in the exterior and interior offer a great deal of untapped potential. At Fraunhofer IOSB, we are therefore conducting user-centered research into parking space detection, person recognition, the integration of video games, and new camera-based operating concepts for the vehicle and in the vehicle of the future.

Collecting data in vehicles requires a deep understanding of the quality criteria for AI training data. Efficient collection requires experience with measurement technology, processes for GDPR-compliant data storage and processing, and optimized laboratory equipment. Fraunhofer can be commissioned to collect data in Germany, Europe, and worldwide. The necessary processes and impressive laboratory equipment with vehicles, measurement technology, and servers are available for this purpose.

Our research units advise you on the implementation of your data-driven AI projects and develop proof-of-concept demonstrators and customized applications for you. We provide our proven processes and laboratory equipment for AI performance evaluations, user experience evaluations, and optimizations.

The development and testing of vehicle technologies are increasingly shifting to the virtual realm. Simulations and synthetic data are already helping to test and evaluate functions at an early stage and with low thresholds – often before the real systems are completed.

However, due to the increasing scope of validation, which can hardly be covered with real data, the critical importance of safety, and the increased data requirements of AI and ML, simulation and virtual testing are also taking on additional roles. It is therefore assumed that, in the future, automated vehicle functions will have to be tested and approved in simulation for the most part in order to remain economical.

Fraunhofer IOSB has therefore been developing the open simulation platform OCTANE/OCTAS since 2014, which aims to enable the modular interaction of different components. Thanks to its plugin-based architecture, simulation modules from different providers can work together seamlessly and be recombined for any application. This includes the simulation of driving dynamics, sensors, communication (V2V, V2I, V2X), and the behavior of other road users. The result is a comprehensive and flexibly expandable digital twin of the entire system.

In addition, this enables the generation of synthetic data sets, including ground truth, which can be used to train AI processes and test their performance and system limits. Open data sets that have been proven to achieve the quality of real data for AI training are available at synset.de. Customized data sets for customer-specific applications provide a realistic, data protection-compliant, and high-quality source for training and test data.

A significant proportion of accidents are attributable to so-called human error. Despite, or perhaps because of, increasing automation in vehicles, human factors play a decisive role in vehicle safety. Current and future EU legislation and EuroNCAP test protocols therefore require effective systems for monitoring the condition of drivers (driver monitoring systems (DMS) and occupant monitoring systems (OMS)).

Interior sensors, such as the Advanced Occupant Monitoring System (link) from Fraunhofer IOSB, can also be used for many innovative entertainment functions. They enable completely new functions and increase usability and user experience in the vehicle. Thanks to decades of research into current developments, Fraunhofer supports the implementation and evaluation of new applications that enhance the brand experience in the vehicle. We support the use of all types of optical sensors such as mono and stereo cameras, ToF and FIR, and sensor fusion with additional data. Thanks to the latest AI methods, such as multimodal AI models, vision language models (VLM), and local LLM, our functional developments are fast and extremely innovative. We set up proof-of-concept demonstrators and evaluate them in user studies and performance tests.

At Fraunhofer IOSB, we offer our customers many years of experience in the areas of vehicle interior analysis, vehicle exterior analysis, and analyses of the entire traffic system. This bundled competence enables successful projects that require the fusion of these data sources. Fraunhofer IOSB's focus on optronics, systems engineering, and image processing enables the fusion of these disciplines for an optimized overall vehicle system. Uniform optical detection systems unlock concrete potential through the unique combination of core competencies at Fraunhofer IOSB.

As a renowned research institution, Fraunhofer IOSB has established processes that meet all GDPR requirements. This enables us to carry out a wide range of surveys that exclude any personal references from the outset. In addition, we are also able to process personal data and can correctly assess data minimization. Thanks to our wide range of experience and a whole series of our own research projects specifically on these topics, we are your reliable partner for consulting and the processing of personal data. In the Anymos project (link), we are developing a data processing architecture that processes image data from vehicles in a partially anonymized form and also makes it available for research via a privacy broker. In other projects, we develop customized solutions for collecting data in the automotive and mobility sectors, as well as for anonymizing video images while retaining relevant information.