Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

by   Xiang Guo, et al.

Novel view synthesis is a challenging problem in computer vision and robotics. Different from the existing works, which need the reference images or 3D models of the scene to generate images under novel views, we propose a novel paradigm to this problem. That is, we synthesize the novel view from only a 6-DoF camera pose directly. Although this setting is the most straightforward way, there are few works addressing it. While, our experiments demonstrate that, with a concise CNN, we could get a meaningful parametric model that could reconstruct the correct scenery images only from the 6-DoF pose. To this end, we propose a two-stage learning strategy, which consists of two consecutive CNNs: GenNet and RefineNet. GenNet generates a coarse image from a camera pose. RefineNet is a generative adversarial network that refines the coarse image. In this way, we decouple the geometric relationship between mapping and texture detail rendering. Extensive experiments conducted on the public datasets prove the effectiveness of our method. We believe this paradigm is of high research and application value and could be an important direction in novel view synthesis.


page 1

page 3

page 4

page 5

page 6

page 7


Free View Synthesis

We present a method for novel view synthesis from input images that are ...

Fast View Synthesis with Deep Stereo Vision

Novel view synthesis is an important problem in computer vision and grap...

Comparing two- and three-view Computer Vision

To reconstruct the points in three dimensional space, we need at least t...

PoseGAN: A Pose-to-Image Translation Framework for Camera Localization

Camera localization is a fundamental requirement in robotics and compute...

EpipolarNVS: leveraging on Epipolar geometry for single-image Novel View Synthesis

Novel-view synthesis (NVS) can be tackled through different approaches, ...

Pose-guided Generative Adversarial Net for Novel View Action Synthesis

We focus on the problem of novel-view human action synthesis. Given an a...

Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models

Novel view synthesis from a single input image is a challenging task, wh...

Please sign up or login with your details

Forgot password? Click here to reset