Multimodal dynamics modeling for off-road autonomous vehicles

by   Jean-François Tremblay, et al.

Dynamics modeling in outdoor and unstructured environments is difficult because different elements in the environment interact with the robot in ways that can be hard to predict. Leveraging multiple sensors to perceive maximal information about the robot's environment is thus crucial when building a model to perform predictions about the robot's dynamics with the goal of doing motion planning. We design a model capable of long-horizon motion predictions, leveraging vision, lidar and proprioception, which is robust to arbitrarily missing modalities at test time. We demonstrate in simulation that our model is able to leverage vision to predict traction changes. We then test our model using a real-world challenging dataset of a robot navigating through a forest, performing predictions in trajectories unseen during training. We try different modality combinations at test time and show that, while our model performs best when all modalities are present, it is still able to perform better than the baseline even when receiving only raw vision input and no proprioception, as well as when only receiving proprioception. Overall, our study demonstrates the importance of leveraging multiple sensors when doing dynamics modeling in outdoor conditions.


page 4

page 5

page 6


A Multi-step Dynamics Modeling Framework For Autonomous Driving In Multiple Environments

Modeling dynamics is often the first step to making a vehicle autonomous...

DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability

Task and Motion Planning (TAMP) approaches are effective at planning lon...

Multimodal representation models for prediction and control from partial information

Similar to humans, robots benefit from interacting with their environmen...

Connected Autonomous Vehicle Motion Planning with Video Predictions from Smart, Self-Supervised Infrastructure

Connected autonomous vehicles (CAVs) promise to enhance safety, efficien...

GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments

We present a novel trajectory traversability estimation and planning alg...

Learning to Navigate Sidewalks in Outdoor Environments

Outdoor navigation on sidewalks in urban environments is the key technol...

Please sign up or login with your details

Forgot password? Click here to reset