Efficient Optimization of Echo State Networks for Time Series Datasets

by   Jacob Reinier Maat, et al.

Echo State Networks (ESNs) are recurrent neural networks that only train their output layer, thereby precluding the need to backpropagate gradients through time, which leads to significant computational gains. Nevertheless, a common issue in ESNs is determining its hyperparameters, which are crucial in instantiating a well performing reservoir, but are often set manually or using heuristics. In this work we optimize the ESN hyperparameters using Bayesian optimization which, given a limited budget of function evaluations, outperforms a grid search strategy. In the context of large volumes of time series data, such as light curves in the field of astronomy, we can further reduce the optimization cost of ESNs. In particular, we wish to avoid tuning hyperparameters per individual time series as this is costly; instead, we want to find ESNs with hyperparameters that perform well not just on individual time series but rather on groups of similar time series without sacrificing predictive performance significantly. This naturally leads to a notion of clusters, where each cluster is represented by an ESN tuned to model a group of time series of similar temporal behavior. We demonstrate this approach both on synthetic datasets and real world light curves from the MACHO survey. We show that our approach results in a significant reduction in the number of ESN models required to model a whole dataset, while retaining predictive performance for the series in each cluster.


page 1

page 2

page 3

page 4


Optimizing Hyperparameters in CNNs using Bilevel Programming in Time Series Data

Hyperparameter optimization has remained a central topic within the mach...

Recurrent Neural Processes

We extend Neural Processes (NPs) to sequential data through Recurrent NP...

Model-Coupled Autoencoder for Time Series Visualisation

We present an approach for the visualisation of a set of time series tha...

Model Compression for Dynamic Forecast Combination

The predictive advantage of combining several different predictive model...

Rethinking Defaults Values: a Low Cost and Efficient Strategy to Define Hyperparameters

Machine Learning (ML) algorithms have been successfully employed by a va...

Training Echo State Networks with Regularization through Dimensionality Reduction

In this paper we introduce a new framework to train an Echo State Networ...

Please sign up or login with your details

Forgot password? Click here to reset