Attentive Gaussian processes for probabilistic time-series generation

by   Kuilin Chen, et al.

The transduction of sequence has been mostly done by recurrent networks, which are computationally demanding and often underestimate uncertainty severely. We propose a computationally efficient attention-based network combined with the Gaussian process regression to generate real-valued sequence, which we call the Attentive-GP. The proposed model not only improves the training efficiency by dispensing recurrence and convolutions but also learns the factorized generative distribution with Bayesian representation. However, the presence of the GP precludes the commonly used mini-batch approach to the training of the attention network. Therefore, we develop a block-wise training algorithm to allow mini-batch training of the network while the GP is trained using full-batch, resulting in a scalable training method. The algorithm has been proved to converge and shows comparable, if not better, quality of the found solution. As the algorithm does not assume any specific network architecture, it can be used with a wide range of hybrid models such as neural networks with kernel machine layers in the scarcity of resources for computation and memory.


page 1

page 2

page 3

page 4


A Mini-Batch Method for Solving Nonlinear PDEs with Gaussian Processes

Gaussian processes (GPs) based methods for solving partial differential ...

Deep Neural Networks as Gaussian Processes

A deep fully-connected neural network with an i.i.d. prior over its para...

Scalable Hybrid HMM with Gaussian Process Emission for Sequential Time-series Data Clustering

Hidden Markov Model (HMM) combined with Gaussian Process (GP) emission c...

Measuring Uncertainty in Signal Fingerprinting with Gaussian Processes Going Deep

In indoor positioning, signal fluctuation is highly location-dependent. ...

Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification

Gaussian processes (GP) are one of the most successful frameworks to mod...

Gaussian Processes to speed up MCMC with automatic exploratory-exploitation effect

We present a two-stage Metropolis-Hastings algorithm for sampling probab...

Please sign up or login with your details

Forgot password? Click here to reset