IDEBOS - Automated drone image and video exploitation for emergency service organizations

Inital situation

In any operation conducted by emergency services such as firefighters, police, or rescue services, the first steps are situation reconnaissance and risk assessment. Drones that transmit live video to the ground control station, can offer a great advantage, especially in complex incidents that are spread over large areas. Hence, drones are becoming an established operational tool for many emergency services. In addition to reconnaissance, surveillance, and situation update, drones are also suitable for various other deployment scenarios: monitoring critical infrastructure (e.g., dykes), detecting changes in the spread of fires or water, searching for pockets of embers, searching for people and animals, and documenting rescue operations.

Problem definition

Unfortunately, in such scenarios the drones are often reduced to being an airborne CCTV camera. Images and videos from drones are usually examined by image analysts. The gathered information is then passed on to the incident management, e.g., to the head of operations. Often, this is done by radio.

It is strenuous, time-consuming, and exhausting for human operators to continuously view and analyze transmitted video and audio data. Due to the fact, that the reporting is done by radio, the situation description might be inaccurate, not concise, and not up to date. Creating a correct and up to date situation map based on such reports poses a great challenge.

Solutions

At Fraunhofer IOSB, we have been researching various methods for automated sensor data exploitation, information processing, and presentation for many years. Our expertise includes methods for image and video exploitation, from image quality enhancement and image mosaicing to object recognition, classification, and tracking and up to activity analysis using the latest artificial intelligence (AI) methods.

Exploiting the telemetry data, such as altitude, camera position, and orientation, allows us to assign a real geocoordinate to each image pixel and each detection. This means that objects or events detected in the sensor data can be geographically located and displayed live on a situation map. Most of the methods are real-time capable. The results can therefore be examined and used live.

• image quality enhancement
  • video stabilization, image optimization
  • denoising, superresolution
• generation of overview images
  • multiresolution images
  • mosaicing (image stitching)
• automatic object and activity recognition, geographical location of objects and events
  • moving object detection and change detection
  • detection, categorization, tracking, and counting of various objects, e.g., people, vehicles, drones, etc.
  • crowd density estimation, detection of dangerous compressions and anomalies
  • location and display on the situation map (2D and 3D)
• Tools for live and offline exploitation of videos and individual images
  • parallel recording and displaying multiple video streams
  • extended video player functionality with the ability to navigate in a (live) video stream
  • ability to bookmark time points and intervals and add annotations
  • exporting individual images and video clips

Image quality enhancement

In poor visibility conditions, some image processing methods can drastically improve the perception of image content. Examples of such processes are histogram spreading, noise reduction, and superresolution.

Detect, classify, and track relevant objects in video data using AI methods

Deep learning methods can be used to detect, classify, and track relevant objects in a video stream, even in poor visibility conditions. This can make search and area screening missions much more efficient and significantly less exhausting for human operators.

In addition to RGB and thermal cameras, drones can carry other payloads, e.g., multispectral cameras, laser scanners, and radars – individually or in combination. This allows to cover many use cases, such as

• people search, animal search
• detection of pockets of embers
• drone detection
• surface condition
• vegetation condition
• 3D reconstruction, height estimation, etc.