Cross-Modal 3D Shape Generation and Manipulation

by   Zezhou Cheng, et al.

Creating and editing the shape and color of 3D objects require tremendous human effort and expertise. Compared to direct manipulation in 3D interfaces, 2D interactions such as sketches and scribbles are usually much more natural and intuitive for the users. In this paper, we propose a generic multi-modal generative model that couples the 2D modalities and implicit 3D representations through shared latent spaces. With the proposed model, versatile 3D generation and manipulation are enabled by simply propagating the editing from a specific 2D controlling modality through the latent spaces. For example, editing the 3D shape by drawing a sketch, re-colorizing the 3D surface via painting color scribbles on the 2D rendering, or generating 3D shapes of a certain category given one or a few reference images. Unlike prior works, our model does not require re-training or fine-tuning per editing task and is also conceptually simple, easy to implement, robust to input domain shifts, and flexible to diverse reconstruction on partial 2D inputs. We evaluate our framework on two representative 2D modalities of grayscale line sketches and rendered color images, and demonstrate that our method enables various shape manipulation and generation tasks with these 2D modalities.


page 1

page 2

page 3

page 4


Intuitive Shape Editing in Latent Space

The use of autoencoders for shape generation and editing suffers from ma...

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

We present CLIP-NeRF, a multi-modal 3D object manipulation method for ne...

XDGAN: Multi-Modal 3D Shape Generation in 2D Space

Generative models for 2D images has recently seen tremendous progress in...

Collaborative Diffusion for Multi-Modal Face Generation and Editing

Diffusion models arise as a powerful generative tool recently. Despite t...

Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model

Generating images from hand-drawings is a crucial and fundamental task i...

SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation

In this work, we present a novel framework built to simplify 3D asset ge...

Generative Visual Manipulation on the Natural Image Manifold

Realistic image manipulation is challenging because it requires modifyin...

Please sign up or login with your details

Forgot password? Click here to reset