NCHO: Unsupervised Learning for Neural 3D Composition of Humans and Objects

by   Taeksoo Kim, et al.

Deep generative models have been recently extended to synthesizing 3D digital humans. However, previous approaches treat clothed humans as a single chunk of geometry without considering the compositionality of clothing and accessories. As a result, individual items cannot be naturally composed into novel identities, leading to limited expressiveness and controllability of generative 3D avatars. While several methods attempt to address this by leveraging synthetic data, the interaction between humans and objects is not authentic due to the domain gap, and manual asset creation is difficult to scale for a wide variety of objects. In this work, we present a novel framework for learning a compositional generative model of humans and objects (backpacks, coats, scarves, and more) from real-world 3D scans. Our compositional model is interaction-aware, meaning the spatial relationship between humans and objects, and the mutual shape change by physical contact is fully incorporated. The key challenge is that, since humans and objects are in contact, their 3D scans are merged into a single piece. To decompose them without manual annotations, we propose to leverage two sets of 3D scans of a single person with and without objects. Our approach learns to decompose objects and naturally compose them back into a generative human model in an unsupervised manner. Despite our simple setup requiring only the capture of a single subject with objects, our experiments demonstrate the strong generalization of our model by enabling the natural composition of objects to diverse identities in various poses and the composition of multiple objects, which is unseen in training data.


page 1

page 3

page 6

page 7


Compositional Human-Scene Interaction Synthesis with Semantic Control

Synthesizing natural interactions between virtual humans and their 3D en...

A Case for Object Compositionality in Deep Generative Models of Images

Deep generative models seek to recover the process with which the observ...

gCoRF: Generative Compositional Radiance Fields

3D generative models of objects enable photorealistic image synthesis wi...

gDNA: Towards Generative Detailed Neural Avatars

To make 3D human avatars widely available, we must be able to generate a...

AnthroNet: Conditional Generation of Humans via Anthropometrics

We present a novel human body model formulated by an extensive set of an...

Knowledge-Guided Object Discovery with Acquired Deep Impressions

We present a framework called Acquired Deep Impressions (ADI) which cont...

ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis

This paper presents a novel deep learning-based approach for automatical...

Please sign up or login with your details

Forgot password? Click here to reset