Rig-space Neural Rendering

by   Dominik Borer, et al.

Movie productions use high resolution 3d characters with complex proprietary rigs to create the highest quality images possible for large displays. Unfortunately, these 3d assets are typically not compatible with real-time graphics engines used for games, mixed reality and real-time pre-visualization. Consequently, the 3d characters need to be re-modeled and re-rigged for these new applications, requiring weeks of work and artistic approval. Our solution to this problem is to learn a compact image-based rendering of the original 3d character, conditioned directly on the rig parameters. Our idea is to render the character in many different poses and views, and to train a deep neural network to render high resolution images, from the rig parameters directly. Many neural rendering techniques have been proposed to render from 2d skeletons, or geometry and UV maps. However these require manual work, and to do not remain compatible with the animator workflow of manipulating rig widgets, as well as the real-time game engine pipeline of interpolating rig parameters. We extend our architecture to support dynamic re-lighting and composition with other 3d objects in the scene. We designed a network that efficiently generates multiple scene feature maps such as normals, depth, albedo and mask, which are composed with other scene objects to form the final image.


page 1

page 2

page 5

page 6

page 7

page 8

page 12


Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality

The representation of consistent mixed reality (XR) environments require...

Collaborative Neural Rendering using Anime Character Sheets

Drawing images of characters at desired poses is an essential but labori...

Real-time GeoAI for High-resolution Mapping and Segmentation of Arctic Permafrost Features

This paper introduces a real-time GeoAI workflow for large-scale image a...

EasyPBR: A Lightweight Physically-Based Renderer

Modern rendering libraries provide unprecedented realism, producing real...

Procedural Planetary Multi-resolution Terrain Generation for Games

Terrains are the main part of an electronic game. To reduce human effort...

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

The recent research explosion around implicit neural representations, su...

Deep UAV Localization with Reference View Rendering

This paper presents a framework for the localization of Unmanned Aerial ...

Please sign up or login with your details

Forgot password? Click here to reset