Kernels and Submodels of Deep Belief Networks

by   Guido F. Montúfar, et al.
Penn State University

We study the mixtures of factorizing probability distributions represented as visible marginal distributions in stochastic layered networks. We take the perspective of kernel transitions of distributions, which gives a unified picture of distributed representations arising from Deep Belief Networks (DBN) and other networks without lateral connections. We describe combinatorial and geometric properties of the set of kernels and products of kernels realizable by DBNs as the network parameters vary. We describe explicit classes of probability distributions, including exponential families, that can be learned by DBNs. We use these submodels to bound the maximal and the expected Kullback-Leibler approximation errors of DBNs from above depending on the number of hidden layers and units that they contain.


page 2

page 13


Learning the Structure of Deep Sparse Graphical Models

Deep belief networks are a powerful way to model complex probability dis...

Stochastic Feedforward Neural Networks: Universal Approximation

In this chapter we take a look at the universal approximation question f...

Dimension of Marginals of Kronecker Product Models

A Kronecker product model is the set of visible marginal probability dis...

Quantitative Universal Approximation Bounds for Deep Belief Networks

We show that deep belief networks with binary hidden units can approxima...

de Finetti reductions for partially exchangeable probability distributions

We introduce a general framework for de Finetti reduction results, appli...

Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

We generalize recent theoretical work on the minimal number of layers of...

Maximal Information Divergence from Statistical Models defined by Neural Networks

We review recent results about the maximal values of the Kullback-Leible...

Please sign up or login with your details

Forgot password? Click here to reset