Large Scale Joint Semantic Re-Localisation and Scene Understanding via Globally Unique Instance Coordinate Regression

by   Ignas Budvytis, et al.

In this work we present a novel approach to joint semantic localisation and scene understanding. Our work is motivated by the need for localisation algorithms which not only predict 6-DoF camera pose but also simultaneously recognise surrounding objects and estimate 3D geometry. Such capabilities are crucial for computer vision guided systems which interact with the environment: autonomous driving, augmented reality and robotics. In particular, we propose a two step procedure. During the first step we train a convolutional neural network to jointly predict per-pixel globally unique instance labels and corresponding local coordinates for each instance of a static object (e.g. a building). During the second step we obtain scene coordinates by combining object center coordinates and local coordinates and use them to perform 6-DoF camera pose estimation. We evaluate our approach on real world (CamVid-360) and artificial (SceneCity) autonomous driving datasets. We obtain smaller mean distance and angular errors than state-of-the-art 6-DoF pose estimation algorithms based on direct pose regression and pose estimation from scene coordinates on all datasets. Our contributions include: (i) a novel formulation of scene coordinate regression as two separate tasks of object instance recognition and local coordinate regression and a demonstration that our proposed solution allows to predict accurate 3D geometry of static objects and estimate 6-DoF pose of camera on (ii) maps larger by several orders of magnitude than previously attempted by scene coordinate regression methods, as well as on (iii) lightweight, approximate 3D maps built from 3D primitives such as building-aligned cuboids.


page 1

page 2

page 3

page 5

page 8

page 10


Decoupling Features and Coordinates for Few-shot RGB Relocalization

Cross-scene model adaption is a crucial feature for camera relocalizatio...

YOLOff: You Only Learn Offsets for robust 6DoF object pose estimation

Estimating the 3D translation and orientation of an object is a challeng...

3D Robot Pose Estimation from 2D Images

This paper considers the task of locating articulated poses of multiple ...

A Dynamic Keypoints Selection Network for 6DoF Pose Estimation

6 DoF poses estimation problem aims to estimate the rotation and transla...

A Pose-only Solution to Visual Reconstruction and Navigation

Visual navigation and three-dimensional (3D) scene reconstruction are es...

A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Images

3D pose estimation from a single 2D image is an important and challengin...

Scene Coordinate Regression with Angle-Based Reprojection Loss for Camera Relocalization

Image-based camera relocalization is an important problem in computer vi...

Please sign up or login with your details

Forgot password? Click here to reset