Semantic sensor fusion: from camera to sparse lidar information

by   Julie Stephany Berrio, et al.

To navigate through urban roads, an automated vehicle must be able to perceive and recognize objects in a three-dimensional environment. A high-level contextual understanding of the surroundings is necessary to plan and execute accurate driving maneuvers. This paper presents an approach to fuse different sensory information, Light Detection and Ranging (lidar) scans and camera images. The output of a convolutional neural network (CNN) is used as classifier to obtain the labels of the environment. The transference of semantic information between the labelled image and the lidar point cloud is performed in four steps: initially, we use heuristic methods to associate probabilities to all the semantic classes contained in the labelled images. Then, the lidar points are corrected to compensate for the vehicle's motion given the difference between the timestamps of each lidar scan and camera image. In a third step, we calculate the pixel coordinate for the corresponding camera image. In the last step we perform the transfer of semantic information from the heuristic probability images to the lidar frame, while removing the lidar information that is not visible to the camera. We tested our approach in the Usyd Dataset <cit.>, obtaining qualitative and quantitative results that demonstrate the validity of our probabilistic sensory fusion approach.


page 1

page 2

page 3

page 5

page 7


Camera-Lidar Integration: Probabilistic sensor fusion for semantic mapping

An automated vehicle operating in an urban environment must be able to p...

Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection

We propose a new method for fusing a LIDAR point cloud and camera-captur...

LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment

3D panoptic segmentation is a challenging perception task that requires ...

Probabilistic Egocentric Motion Correction of Lidar Point Cloud and Projection to Camera Images for Moving Platforms

The fusion of sensor data from heterogeneous sensors is crucial for robu...

VANETs Meet Autonomous Vehicles: A Multimodal 3D Environment Learning Approach

In this paper, we design a multimodal framework for object detection, re...

SeDAR - Semantic Detection and Ranging: Humans can localise without LiDAR, can robots?

How does a person work out their location using a floorplan? It is proba...

Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle

It is difficult to perform 3D reconstruction from on-vehicle gathered vi...

Please sign up or login with your details

Forgot password? Click here to reset