Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy

by   Marylou Gabrié, et al.

Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics known as the Thouless-Anderson-Palmer approach. We demonstrate that our algorithm provides performance equal to, and sometimes superior to, persistent contrastive divergence, while also providing a clear and easy to evaluate objective function. We believe that this strategy can be easily generalized to other models as well as to more accurate higher-order approximations, paving the way for systematic improvements in training Boltzmann machines with hidden units.


Restricted Boltzmann Machine with Multivalued Hidden Variables: a model suppressing over-fitting

Generalization is one of the most important issues in machine learning p...

A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

Restricted Boltzmann machines (RBMs) are energy-based neural-networks wh...

An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks

The performance of deep neural networks, such as Deep Belief Networks fo...

Self-regularizing restricted Boltzmann machines

Focusing on the grand-canonical extension of the ordinary restricted Bol...

Weighted Contrastive Divergence

Learning algorithms for energy based Boltzmann architectures that rely o...

Conditional Restricted Boltzmann Machines for Structured Output Prediction

Conditional Restricted Boltzmann Machines (CRBMs) are rich probabilistic...

Maximal Information Divergence from Statistical Models defined by Neural Networks

We review recent results about the maximal values of the Kullback-Leible...

Please sign up or login with your details

Forgot password? Click here to reset