LANe: Lighting-Aware Neural Fields for Compositional Scene Synthesis

by   Akshay Krishnan, et al.

Neural fields have recently enjoyed great success in representing and rendering 3D scenes. However, most state-of-the-art implicit representations model static or dynamic scenes as a whole, with minor variations. Existing work on learning disentangled world and object neural fields do not consider the problem of composing objects into different world neural fields in a lighting-aware manner. We present Lighting-Aware Neural Field (LANe) for the compositional synthesis of driving scenes in a physically consistent manner. Specifically, we learn a scene representation that disentangles the static background and transient elements into a world-NeRF and class-specific object-NeRFs to allow compositional synthesis of multiple objects in the scene. Furthermore, we explicitly designed both the world and object models to handle lighting variation, which allows us to compose objects into scenes with spatially varying lighting. This is achieved by constructing a light field of the scene and using it in conjunction with a learned shader to modulate the appearance of the object NeRFs. We demonstrate the performance of our model on a synthetic dataset of diverse lighting conditions rendered with the CARLA simulator, as well as a novel real-world dataset of cars collected at different times of the day. Our approach shows that it outperforms state-of-the-art compositional scene synthesis on the challenging dataset setup, via composing object-NeRFs learned from one scene into an entirely different scene whilst still respecting the lighting variations in the novel scene. For more results, please visit our project website


page 4

page 7

page 12

page 13

page 14

page 15

page 16

page 17


Object-Centric Neural Scene Rendering

We present a method for composing photorealistic scenes from captured im...

RobustNeRF: Ignoring Distractors with Robust Losses

Neural radiance fields (NeRF) excel at synthesizing new views given mult...

3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue

This paper studies how to flexibly integrate reconstructed 3D models int...

Semantic-aware Occlusion Filtering Neural Radiance Fields in the Wild

We present a learning framework for reconstructing neural scene represen...

Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing

Modern deep neural networks tend to be evaluated on static test sets. On...

Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes

Recent breakthroughs in text-guided image generation have led to remarka...

ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation

There has been a recent surge in methods that aim to decompose and segme...

Please sign up or login with your details

Forgot password? Click here to reset