Money on the Table: Statistical information ignored by Softmax can improve classifier accuracy

01/26/2019
by   Charles B. Delahunt, et al.
22

Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an array D of class response distributions, where D_ij is the distribution of the j^th pre-Softmax readout neuron's responses to the i^th class. Given a test sample, Softmax implicitly uses only the row of this array D that corresponds to the readout neurons' responses to the sample's true class. Leveraging more of this array D can improve classifier accuracy, because the likelihoods of two competing classes can be encoded in other rows of D. To explore this potential resource, we develop a hybrid classifier (Softmax-Pooling Hybrid, SPH) that uses Softmax on high-scoring samples, but on low-scoring samples uses a log-likelihood method that pools the information from the full array D. We apply SPH to models trained on a vectorized MNIST dataset to varying levels of accuracy. SPH replaces only the final Softmax layer in the trained NN, at test time only. All training is the same as for Softmax. Because the pooling classifier performs better than Softmax on low-scoring samples, SPH reduces test set error by 6 same trained model, whatever the baseline Softmax accuracy. This reduction in error reflects hidden capacity of the trained NN that is left unused by Softmax.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/10/2023

A Hybrid of Generative and Discriminative Models Based on the Gaussian-coupled Softmax Layer

Generative models have advantageous characteristics for classification t...
research
06/09/2020

A t-distribution based operator for enhancing out of distribution robustness of neural network classifiers

Neural Network (NN) classifiers can assign extreme probabilities to samp...
research
09/11/2018

Heated-Up Softmax Embedding

Metric learning aims at learning a distance which is consistent with the...
research
07/24/2019

Sampled Softmax with Random Fourier Features

The computational cost of training with softmax cross entropy loss grows...
research
06/06/2023

A Quality Aware Sample-to-Sample Comparison for Face Recognition

Currently available face datasets mainly consist of a large number of hi...
research
08/02/2023

Global Hierarchical Neural Networks using Hierarchical Softmax

This paper presents a framework in which hierarchical softmax is used to...
research
10/18/2021

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Detecting out-of-distribution (OOD) samples is vital for developing mach...

Please sign up or login with your details

Forgot password? Click here to reset