Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB 
DRUM – drone-based support for mass events
Our technologies for authorities and organizations
Crowds in the entrance area of a major event.
Heatmap representation of a concentration of people in the access area of a major event.
For authorities and organizations that are securing mass events such as festivals or concerts, it is essential to estimate the number of people in a particular area – be it the access area or a so-called wave breaker – and to recognize any unwanted mass movements or compressions (jostling, mass panic, etc.) that must be counteracted.
The emergency services usually use stationary or mobile video sensors (mast cameras) to create an up-to-date picture of the situation. Due to their flexible use and relatively low costs, drones are increasingly being used as sensor carrier systems. However, the permanent viewing and analysis of transmitted video and audio data by human operators is strenuous, time-consuming, exhausting, and inaccurate. The fact that the information often has to be radioed to the head of operations makes the challenge of accurately describing the situation even more difficult.
The DRUM project aims to create tools to support drone teams and operations managers of authorities and organizations in securing mass events with drones. An essential task is the automation of the sensor data exploitation, as well as the transmission and visualization of the results, which would enable a quick and easy assessment of the situation, detection of dangerous situations, and localization of relevant events by the operations management.
The project team
At Fraunhofer IOSB and Fraunhofer FKIE we have been researching various methods for automated sensor data exploitation, information processing, and presentation for many years. Our expertise includes image and video exploitation methods, sensor data fusion, situation display and command and control systems, human-machine interaction, and cloud-based data processing. The cooperation of several research groups from both institutes in the DRUM project created a synergy that enabled us to solve the above-mentioned problem based on the preliminary work within the Project IDEAL
Solution
Data processing chain
DRUM data processing chain: Evaluation of image, audio and telemetry data with the help of AI processes enables the detection and localization of security-relevant incidents at mass events.
The developed solution is based on the exploitation of video and audio data using the latest artificial intelligence (AI) methods in combination with the exploitation of sensor metadata (telemetry data, such as altitude and sensor orientation). AI methods make it possible to estimate the number of people and the crowd density in drone images. Analyzing the audio signals from a microphone array makes it possible to filter out ambient noise and detect and take a bearing of loud noises (e.g., gunshots, screams, etc.). Evaluating the telemetry data allows us to assign each image pixel and each audio source a real geocoordinate. This enables events detected in the sensor data to be located geographically and displayed live on a map making it possible to quickly identify dangerous situations at mass events.
Schematic representation of the envisaged solution: Video images and audio signals captured by the drone are sent to the ground control station (usually a smartphone), from where they are forwarded to the evaluation station via Wi-Fi. There, the data is processed automatically. The results can be displayed directly or, for example, forwarded to the incident command.
Implementation as a cloud solution (data exploitation as a service)
The cloud solution involves users uploading video images to the cloud, where they are analyzed. The results can be distributed to the situation display and command systems.
In addition to the stand-alone system operation mode, a cloud solution was implemented as part of the project. This offers the potential to provide an estimate of the number of people and the crowd density as a service for project partners from the security authorities and organizations. The specific way of providing this service will be examined in the further course of the project. The way it works is that the user uploads a drone image and receives an estimated number of people and the georeferenced density data. The results can also be distributed to the situation display and command and control systems.
The stand-alone operation mode is more robust than the cloud solution since it has fewer requirements regarding data transmission. On the other side, the computational power and AI models are centrally available in the cloud operation mode, so no special computer hardware or software are required on-site.
Work done within the project
- ● Both live and offline access and correct interpretation of telemetry data
- ● Adaptation of existing AI models to usage with drone footage (video: different viewing perspective compared to mast cameras; audio: motor sounds)
- Acquisition of large amounts of training and test data
- Creation of an interface for the situation map software
- Implementation of a stand-alone solution and a cloud-based solution:
- Distributed data processing
- Interfaces for realizing data exploitation as a service
- Addressing issues of data and privacy protection related to handling video data
- Establishing collaborations with security authorities and organizations as well as industry
Result
Within the project, we established cooperations with several first responder organizations and successfully tested the developed procedures using operational data. The resulting demonstration is to be extensively evaluated in user studies and then further developed to enable security authorities and organizations to access this technology. The ability to estimate the number and density of people in relevant areas in real-time offers security authorities and organizations a great deal of added value.