Generative Models for Low-Rank Video Representation and Reconstruction

by   Rakib Hyder, et al.

Finding compact representation of videos is an essential component in almost every problem related to video processing or understanding. In this paper, we propose a generative model to learn compact latent codes that can efficiently represent and reconstruct a video sequence from its missing or under-sampled measurements. We use a generative network that is trained to map a compact code into an image. We first demonstrate that if a video sequence belongs to the range of the pretrained generative network, then we can recover it by estimating the underlying compact latent codes. Then we demonstrate that even if the video sequence does not belong to the range of a pretrained network, we can still recover the true video sequence by jointly updating the latent codes and the weights of the generative network. To avoid overfitting in our model, we regularize the recovery problem by imposing low-rank and similarity constraints on the latent codes of the neighboring frames in the video sequence. We use our methods to recover a variety of videos from compressive measurements at different compression rates. We also demonstrate that we can generate missing frames in a video sequence by interpolating the latent codes of the observed frames in the low-dimensional space.


page 4

page 6

page 9

page 11

page 12

page 13

page 14

page 19


Latent Video Transformer

The video generation task can be formulated as a prediction of future vi...

Learning to Decompose and Disentangle Representations for Video Prediction

Our goal is to predict future video frames given a sequence of input fra...

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

Generative adversarial models (GANs) continue to produce advances in ter...

Consistent Generative Query Networks

Stochastic video prediction is usually framed as an extrapolation proble...

Low-Rank Phase Retrieval with Structured Tensor Models

We study the low-rank phase retrieval problem, where the objective is to...

Video (language) modeling: a baseline for generative models of natural videos

We propose a strong baseline model for unsupervised feature learning usi...

Brain-Inspired Inference on Missing Video Sequence

In this paper, we propose a novel end-to-end architecture that could gen...

Please sign up or login with your details

Forgot password? Click here to reset