DeepAI AI Chat
Log In Sign Up

An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks

by   S. Ivvan Valdez, et al.

The performance of deep neural networks, such as Deep Belief Networks formed by Restricted Boltzmann Machines (RBMs), strongly depends on their training, which is the process of adjusting their parameters. This process can be posed as an optimization problem over n dimensions. However, typical networks contain tens of thousands of parameters, making this a High-Dimensional Problem (HDP). Although different optimization methods have been employed for this goal, the use of most of the Evolutionary Algorithms (EAs) becomes prohibitive due to their inability to deal with HDPs. For instance, the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) which is regarded as one of the most effective EAs, exhibits the enormous disadvantage of requiring O(n^2) memory and operations, making it unpractical for problems with more than a few hundred variables. In this paper, we introduce a novel EA that requires O(n) operations and memory, but delivers competitive solutions for the training stage of RBMs with over one million variables, when compared against CMA-ES and the Contrastive Divergence algorithm, which is the standard method for training RBMs.


page 1

page 2

page 3

page 4


Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy

Restricted Boltzmann machines are undirected neural networks which have ...

Regularized Evolutionary Algorithm for Dynamic Neural Topology Search

Designing neural networks for object recognition requires considerable a...

GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization

Deep neural network learning can be formulated as a non-convex optimizat...

Optimal-margin evolutionary classifier

We introduce a novel approach for discriminative classification using ev...

Supervising Unsupervised Learning with Evolutionary Algorithm in Deep Neural Network

A method to control results of gradient descent unsupervised learning in...

Limited Evaluation Evolutionary Optimization of Large Neural Networks

Stochastic gradient descent is the most prevalent algorithm to train neu...

Block Layer Decomposition schemes for training Deep Neural Networks

Deep Feedforward Neural Networks' (DFNNs) weights estimation relies on t...