Separating the World and Ego Models for Self-Driving

by   Vlad Sobal, et al.

Training self-driving systems to be robust to the long-tail of driving scenarios is a critical problem. Model-based approaches leverage simulation to emulate a wide range of scenarios without putting users at risk in the real world. One promising path to faithful simulation is to train a forward model of the world to predict the future states of both the environment and the ego-vehicle given past states and a sequence of actions. In this paper, we argue that it is beneficial to model the state of the ego-vehicle, which often has simple, predictable and deterministic behavior, separately from the rest of the environment, which is much more complex and highly multimodal. We propose to model the ego-vehicle using a simple and differentiable kinematic model, while training a stochastic convolutional forward model on raster representations of the state to predict the behavior of the rest of the environment. We explore several configurations of such decoupled models, and evaluate their performance both with Model Predictive Control (MPC) and direct policy learning. We test our methods on the task of highway driving and demonstrate lower crash rates and better stability. The code is available at


page 5

page 7

page 14


Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions

We focus on the problem of predicting future states of entities in compl...

Learning from All Vehicles

In this paper, we present a system to train driving policies from experi...

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Driving requires reacting to a wide variety of complex environment condi...

Model-Based Imitation Learning for Urban Driving

An accurate model of the environment and the dynamic agents acting in it...

Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

In this paper, we explore the potential of using a large language model ...

AutoPreview: A Framework for Autopilot Behavior Understanding

The behavior of self driving cars may differ from people expectations, (...

Learning to drive from a world on rails

We learn an interactive vision-based driving policy from pre-recorded dr...

Please sign up or login with your details

Forgot password? Click here to reset