Generating 3D People in Scenes without People

by   Yan Zhang, et al.

We present a fully-automatic system that takes a 3D scene and generates plausible 3D human bodies that are posed naturally in that 3D scene. Given a 3D scene without people, humans can easily imagine how people could interact with the scene and the objects in it. However, this is a challenging task for a computer as solving it requires (1) the generated human bodies should be semantically plausible with the 3D environment, e.g. people sitting on the sofa or cooking near the stove; (2) the generated human-scene interaction should be physically feasible in the way that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions. To that end, we make use of the surface-based 3D human model SMPL-X. We first train a conditional variational autoencoder to predict semantically plausible 3D human pose conditioned on latent scene representations, then we further refine the generated 3D bodies using scene constraints to enforce feasible physical interaction. We show that our approach is able to synthesize realistic and expressive 3D human bodies that naturally interact with 3D environment. We perform extensive experiments demonstrating that our generative framework compares favorably with existing methods, both qualitatively and quantitatively. We believe that our scene-conditioned 3D human generation pipeline will be useful for numerous applications; e.g. to generate training data for human pose estimation, in video games and in VR/AR.


page 1

page 8

page 13

page 14

page 16

page 17

page 18

page 19


Populating 3D Scenes by Learning Human-Scene Interaction

Humans live within a 3D space and constantly interact with it to perform...

Generating Person-Scene Interactions in 3D Scenes

High fidelity digital 3D environments have been proposed in recent years...

Hallucinating Pose-Compatible Scenes

What does human pose tell us about a scene? We propose a task to answer ...

Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoning

Naturally controllable human-scene interaction (HSI) generation has an i...

Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views

Automatic perception of human behaviors during social interactions is cr...

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors

We introduce (HPS) Human POSEitioning System, a method to recover the fu...

FLEX: Full-Body Grasping Without Full-Body Grasps

Synthesizing 3D human avatars interacting realistically with a scene is ...

Please sign up or login with your details

Forgot password? Click here to reset