Leveraging Geometry for Shape Estimation from a Single RGB Image

by   Florian Langer, et al.

Predicting 3D shapes and poses of static objects from a single RGB image is an important research area in modern computer vision. Its applications range from augmented reality to robotics and digital content creation. Typically this task is performed through direct object shape and pose predictions which is inaccurate. A promising research direction ensures meaningful shape predictions by retrieving CAD models from large scale databases and aligning them to the objects observed in the image. However, existing work does not take the object geometry into account, leading to inaccurate object pose predictions, especially for unseen objects. In this work we demonstrate how cross-domain keypoint matches from an RGB image to a rendered CAD model allow for more precise object pose predictions compared to ones obtained through direct predictions. We further show that keypoint matches can not only be used to estimate the pose of an object, but also to modify the shape of the object itself. This is important as the accuracy that can be achieved with object retrieval alone is inherently limited to the available CAD models. Allowing shape adaptation bridges the gap between the retrieved CAD model and the observed shape. We demonstrate our approach on the challenging Pix3D dataset. The proposed geometric shape prediction improves the AP mesh over the state-of-the-art from 33.2 to 37.8 on seen objects and from 8.2 to 17.1 on unseen objects. Furthermore, we demonstrate more accurate shape predictions without closely matching CAD models when following the proposed shape adaptation. Code is publicly available at https://github.com/florianlanger/leveraging_geometry_for_shape_estimation .


page 2

page 4

page 6

page 8

page 9

page 10

page 14

page 16


SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB image

Estimating 3D shapes and poses of static objects from a single image has...

CNOS: A Strong Baseline for CAD-based Novel Object Segmentation

We propose a simple three-stage approach to segment unseen objects in RG...

Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve

Object recognition has seen significant progress in the image domain, wi...

Uncertainty-aware 3D Object-Level Mapping with Deep Shape Priors

3D object-level mapping is a fundamental problem in robotics, which is e...

Marr Revisited: 2D-3D Alignment via Surface Normal Prediction

We introduce an approach that leverages surface normal predictions, alon...

Integrating Objects into Monocular SLAM: Line Based Category Specific Models

We propose a novel Line based parameterization for category specific CAD...

Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database

We present a new solution to the fine-grained retrieval of clean CAD mod...

Please sign up or login with your details

Forgot password? Click here to reset