Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field

by   Leheng Li, et al.

This work explores the use of 3D generative models to synthesize training data for 3D vision tasks. The key requirements of the generative models are that the generated data should be photorealistic to match the real-world scenarios, and the corresponding 3D attributes should be aligned with given sampling labels. However, we find that the recent NeRF-based 3D GANs hardly meet the above requirements due to their designed generation pipeline and the lack of explicit 3D supervision. In this work, we propose Lift3D, an inverted 2D-to-3D generation framework to achieve the data generation objectives. Lift3D has several merits compared to prior methods: (1) Unlike previous 3D GANs that the output resolution is fixed after training, Lift3D can generalize to any camera intrinsic with higher resolution and photorealistic output. (2) By lifting well-disentangled 2D GAN to 3D object NeRF, Lift3D provides explicit 3D information of generated objects, thus offering accurate 3D annotations for downstream tasks. We evaluate the effectiveness of our framework by augmenting autonomous driving datasets. Experimental results demonstrate that our data generation framework can effectively improve the performance of 3D object detectors. Project page:


page 2

page 4

page 5

page 6

page 7

page 8


Multiscale Generative Models: Improving Performance of a Generative Model Using Feedback from Other Dependent Generative Models

Realistic fine-grained multi-agent simulation of real-world complex syst...

Radio Generation Using Generative Adversarial Networks with An Unrolled Design

As a revolutionary generative paradigm of deep learning, generative adve...

Unpaired Multi-Domain Image Generation via Regularized Conditional GANs

In this paper, we study the problem of multi-domain image generation, th...

Unbiased Auxiliary Classifier GANs with MINE

Auxiliary Classifier GANs (AC-GANs) are widely used conditional generati...

On the "steerability" of generative adversarial networks

An open secret in contemporary machine learning is that many models work...

Interpreting Spatially Infinite Generative Models

Traditional deep generative models of images and other spatial modalitie...

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Given a small training data set and a learning algorithm, how much more ...

Please sign up or login with your details

Forgot password? Click here to reset