TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation

by   Xiangyun Meng, et al.

Effective use of camera-based vision systems is essential for robust performance in autonomous off-road driving, particularly in the high-speed regime. Despite success in structured, on-road settings, current end-to-end approaches for scene prediction have yet to be successfully adapted for complex outdoor terrain. To this end, we present TerrainNet, a vision-based terrain perception system for semantic and geometric terrain prediction for aggressive, off-road navigation. The approach relies on several key insights and practical considerations for achieving reliable terrain modeling. The network includes a multi-headed output representation to capture fine- and coarse-grained terrain features necessary for estimating traversability. Accurate depth estimation is achieved using self-supervised depth completion with multi-view RGB and stereo inputs. Requirements for real-time performance and fast inference speeds are met using efficient, learned image feature projections. Furthermore, the model is trained on a large-scale, real-world off-road dataset collected across a variety of diverse outdoor environments. We show how TerrainNet can also be used for costmap prediction and provide a detailed framework for integration into a planning module. We demonstrate the performance of TerrainNet through extensive comparison to current state-of-the-art baselines for camera-only scene prediction. Finally, we showcase the effectiveness of integrating TerrainNet within a complete autonomous-driving stack by conducting a real-world vehicle test in a challenging off-road scenario.


page 5

page 6

page 8

page 10

page 14

page 15

page 16

page 17


VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics

One of the key challenges in high speed off road navigation on ground ve...

Learning Inverse Kinodynamics for Accurate High-Speed Off-Road Navigation on Unstructured Terrain

This paper presents a learning-based approach to consider the effect of ...

PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation

Comprehensive modeling of the surrounding 3D world is key to the success...

Self-Supervised Simultaneous Multi-Step Prediction of Road Dynamics and Cost Map

While supervised learning is widely used for perception modules in conve...

How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability

Estimating terrain traversability in off-road environments requires reas...

Vision-Based High Speed Driving with a Deep Dynamic Observer

In this paper we present a framework for combining deep learning-based r...

NVAutoNet: Fast and Accurate 360^∘ 3D Perception For Self Driving

Robust real-time perception of 3D world is essential to the autonomous v...

Please sign up or login with your details

Forgot password? Click here to reset