Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis

by   Angtian Wang, et al.

Human vision demonstrates higher robustness than current AI algorithms under out-of-distribution scenarios. It has been conjectured such robustness benefits from performing analysis-by-synthesis. Our paper formulates triple vision tasks in a consistent manner using approximate analysis-by-synthesis by render-and-compare algorithms on neural features. In this work, we introduce Neural Textured Deformable Meshes, which involve the object model with deformable geometry that allows optimization on both camera parameters and object geometries. The deformable mesh is parameterized as a neural field, and covered by whole-surface neural texture maps, which are trained to have spatial discriminability. During inference, we extract the feature map of the test image and subsequently optimize the 3D pose and shape parameters of our model using differentiable rendering to best reconstruct the target feature map. We show that our analysis-by-synthesis is much more robust than conventional neural networks when evaluated on real-world images and even in challenging out-of-distribution scenarios, such as occlusion and domain shift. Our algorithms are competitive with standard algorithms when tested on conventional performance measures.


page 1

page 4

page 5

page 6

page 11

page 12

page 13


Robust Shape Estimation for 3D Deformable Object Manipulation

Existing shape estimation methods for deformable object manipulation suf...

NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes

With the introduction of Neural Radiance Fields (NeRFs), novel view synt...

VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis

Differentiable rendering allows the application of computer graphics on ...

Robust 3D-aware Object Classification via Discriminative Render-and-Compare

In real-world applications, it is essential to jointly estimate the 3D o...

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

We study the problem of learning to estimate the 3D object pose from a f...

3D Shape Perception Integrates Intuitive Physics and Analysis-by-Synthesis

Many surface cues support three-dimensional shape perception, but people...

PhysXNet: A Customizable Approach for LearningCloth Dynamics on Dressed People

We introduce PhysXNet, a learning-based approach to predict the dynamics...

Please sign up or login with your details

Forgot password? Click here to reset