MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

by   Zhaoqi Su, et al.

We introduce MulayCap, a novel human performance capture method using a monocular video camera without the need for pre-scanning. The method uses "multi-layer" representations for geometry reconstruction and texture rendering, respectively. For geometry reconstruction, we decompose the clothed human into multiple geometry layers, namely a body mesh layer and a garment piece layer. The key technique behind is a Garment-from-Video (GfV) method for optimizing the garment shape and reconstructing the dynamic cloth to fit the input video sequence, based on a cloth simulation model which is effectively solved with gradient descent. For texture rendering, we decompose each input image frame into a shading layer and an albedo layer, and propose a method for fusing a fixed albedo map and solving for detailed garment geometry using the shading layer. Compared with existing single view human performance capture systems, our "multi-layer" approach bypasses the tedious and time consuming scanning step for obtaining a human specific mesh template. Experimental results demonstrate that MulayCap produces realistic rendering of dynamically changing details that has not been achieved in any previous monocular video camera systems. Benefiting from its fully semantic modeling, MulayCap can be applied to various important editing applications, such as cloth editing, re-targeting, relighting, and AR applications.


page 2

page 4

page 7

page 9

page 10

page 11

page 12

page 15


SimulCap : Single-View Human Performance Capture with Cloth Simulation

This paper proposes a new method for live free-viewpoint human performan...

Vehicle Reconstruction and Texture Estimation Using Deep Implicit Semantic Template Mapping

We introduce VERTEX, an effective solution to recover 3D shape and intri...

Neural Parameterization for Dynamic Human Head Editing

Implicit radiance functions emerged as a powerful scene representation f...

Modeling Clothing as a Separate Layer for an Animatable Human Avatar

We have recently seen great progress in building photorealistic animatab...

Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering

Due to inevitable noises introduced during scanning and quantization, 3D...

Shape and Material Capture at Home

In this paper, we present a technique for estimating the geometry and re...

MonoNHR: Monocular Neural Human Renderer

Existing neural human rendering methods struggle with a single image inp...

Please sign up or login with your details

Forgot password? Click here to reset