Video Generation from Text Employing Latent Path Construction for Temporal Modeling

by   Amir Mazaheri, et al.

Video generation is one of the most challenging tasks in Machine Learning and Computer Vision fields of study. In this paper, we tackle the text to video generation problem, which is a conditional form of video generation. Humans can listen/read natural language sentences, and can imagine or visualize what is being described; therefore, we believe that video generation from natural language sentences will have an important impact on Artificial Intelligence. Video generation is relatively a new field of study in Computer Vision, which is far from being solved. The majority of recent works deal with synthetic datasets or real datasets with very limited types of objects, scenes, and emotions. To the best of our knowledge, this is the very first work on the text (free-form sentences) to video generation on more realistic video datasets like Actor and Action Dataset (A2D) or UCF101. We tackle the complicated problem of video generation by regressing the latent representations of the first and last frames and employing a context-aware interpolation method to build the latent representations of in-between frames. We propose a stacking “upPooling” block to sequentially generate RGB frames out of each latent representation and progressively increase the resolution. Moreover, our proposed Discriminator encodes videos based on single and multiple frames. We provide quantitative and qualitative results to support our arguments and show the superiority of our method over well-known baselines like Recurrent Neural Network (RNN) and Deconvolution (as known as Convolutional Transpose) based video generation methods.


Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation

This paper proposes a novel model for video generation and especially ma...

Video Storytelling

Bridging vision and natural language is a longstanding goal in computer ...

Order Matters: Shuffling Sequence Generation for Video Prediction

Predicting future frames in natural video sequences is a new challenge t...

Adversarial Generation of Natural Language

Generative Adversarial Networks (GANs) have gathered a lot of attention ...

LIFI: Towards Linguistically Informed Frame Interpolation

In this work, we explore a new problem of frame interpolation for speech...

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Solving the visual symbol grounding problem has long been a goal of arti...

Joint Level Generation and Translation Using Gameplay Videos

Procedural Content Generation via Machine Learning (PCGML) faces a signi...

Please sign up or login with your details

Forgot password? Click here to reset