SuperNet – An efficient method of neural networks ensembling

by   Ludwik Bukowski, et al.

The main flaw of neural network ensembling is that it is exceptionally demanding computationally, especially, if the individual sub-models are large neural networks, which must be trained separately. Having in mind that modern DNNs can be very accurate, they are already the huge ensembles of simple classifiers, and that one can construct more thrifty compressed neural net of a similar performance for any ensemble, the idea of designing the expensive SuperNets can be questionable. The widespread belief that ensembling increases the prediction time, makes it not attractive and can be the reason that the main stream of ML research is directed towards developing better loss functions and learning strategies for more advanced and efficient neural networks. On the other hand, all these factors make the architectures more complex what may lead to overfitting and high computational complexity, that is, to the same flaws for which the highly parametrized SuperNets ensembles are blamed. The goal of the master thesis is to speed up the execution time required for ensemble generation. Instead of training K inaccurate sub-models, each of them can represent various phases of training (representing various local minima of the loss function) of a single DNN [Huang et al., 2017; Gripov et al., 2018]. Thus, the computational performance of the SuperNet can be comparable to the maximum CPU time spent on training its single sub-model, plus usually much shorter CPU time required for training the SuperNet coupling factors.


Understanding the Loss Surface of Neural Networks for Binary Classification

It is widely conjectured that the reason that training algorithms for ne...

Transfer learning for ensembles: reducing computation time and keeping the diversity

Transferring a deep neural network trained on one problem to another req...

Effective training-time stacking for ensembling of deep neural networks

Ensembling is a popular and effective method for improving machine learn...

Collective Learning by Ensembles of Altruistic Diversifying Neural Networks

Combining the predictions of collections of neural networks often outper...

Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation

Deep neural networks (DNNs) have made a revolution in numerous fields du...

A Note on "Assessing Generalization of SGD via Disagreement"

Jiang et al. (2021) give empirical evidence that the average test error ...

Please sign up or login with your details

Forgot password? Click here to reset