Bridged Transformer for Vision and Point Cloud 3D Object Detection

by   Yikai Wang, et al.

3D object detection is a crucial research topic in computer vision, which usually uses 3D point clouds as input in conventional setups. Recently, there is a trend of leveraging multiple sources of input data, such as complementing the 3D point cloud with 2D images that often have richer color and fewer noises. However, due to the heterogeneous geometrics of the 2D and 3D representations, it prevents us from applying off-the-shelf neural networks to achieve multimodal fusion. To that end, we propose Bridged Transformer (BrT), an end-to-end architecture for 3D object detection. BrT is simple and effective, which learns to identify 3D and 2D object bounding boxes from both points and image patches. A key element of BrT lies in the utilization of object queries for bridging 3D and 2D spaces, which unifies different sources of data representations in Transformer. We adopt a form of feature aggregation realized by point-to-patch projections which further strengthen the correlations between images and points. Moreover, BrT works seamlessly for fusing the point cloud with multi-view images. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.


page 3

page 7

page 9

page 10


An Overview Of 3D Object Detection

Point cloud 3D object detection has recently received major attention an...

MLOD: A multi-view 3D object detection based on robust feature fusion method

This paper presents Multi-view Labelling Object Detector (MLOD). The det...

3D Object Detection with Pointformer

Feature learning for 3D object detection from point clouds is very chall...

Deep Hough Voting for 3D Object Detection in Point Clouds

Current 3D object detection methods are heavily influenced by 2D detecto...

Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

3D object detection in point clouds is a challenging vision task that be...

PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds

3D scene understanding from point clouds plays a vital role for various ...

Generalized Convolutional Neural Networks for Point Cloud Data

The introduction of cheap RGB-D cameras, stereo cameras, and LIDAR devic...

Please sign up or login with your details

Forgot password? Click here to reset