Predicting Scene Parsing and Motion Dynamics in the Future

by   Xiaojie Jin, et al.

The ability of predicting the future is important for intelligent systems, e.g. autonomous vehicles and robots to plan early and make decisions accordingly. Future scene parsing and optical flow estimation are two key tasks that help agents better understand their environments as the former provides dense semantic information, i.e. what objects will be present and where they will appear, while the latter provides dense motion information, i.e. how the objects will move. In this paper, we propose a novel model to simultaneously predict scene parsing and optical flow in unobserved future video frames. To our best knowledge, this is the first attempt in jointly predicting scene parsing and motion dynamics. In particular, scene parsing enables structured motion prediction by decomposing optical flow into different groups while optical flow estimation brings reliable pixel-wise correspondence to scene parsing. By exploiting this mutually beneficial relationship, our model shows significantly better parsing and motion prediction results when compared to well-established baselines and individual prediction models on the large-scale Cityscapes dataset. In addition, we also demonstrate that our model can be used to predict the steering angle of the vehicles, which further verifies the ability of our model to learn latent representations of scene dynamics.


page 2

page 6


Dense Optical Flow Prediction from a Static Image

Given a scene, what is going to move, and in what direction will it move...

Novel Video Prediction for Large-scale Scene using Optical Flow

Making predictions of future frames is a critical challenge in autonomou...

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

Today's image prediction methods struggle to change the locations of obj...

Video Scene Parsing with Predictive Feature Learning

In this work, we address the challenging video scene parsing problem by ...

Recurrent Flow-Guided Semantic Forecasting

Understanding the world around us and making decisions about the future ...

Forecasting Future Instance Segmentation with Learned Optical Flow and Warping

For an autonomous vehicle it is essential to observe the ongoing dynamic...

Future Segmentation Using 3D Structure

Predicting the future to anticipate the outcome of events and actions is...

Please sign up or login with your details

Forgot password? Click here to reset