Understanding VAEs in Fisher-Shannon Plane
share
In information theory, Fisher information and Shannon information (entropy) are respectively used to measure the ability in parameter estimation and the uncertainty among variables. The uncertainty principle asserts a fundamental relationship between Fisher information and Shannon information, i.e., the more Fisher information we get, the less Shannon information we gain, and vice versa. This enlightens us about the essence of the encoding/decoding procedure in variational auto-encoders (VAEs) and motivates us to investigate VAEs in the Fisher-Shannon plane. Our studies show that the performance of the latent representation learning and the log-likelihood estimation are intrinsically influenced by the trade-off between Fisher information and Shannon information. To flexibly adjust the trade-off, we further propose a variant of VAEs that can explicitly control Fisher information in encoding/decoding mechanism, termed as Fisher auto-encoder (FAE). Through qualitative and quantitative experiments, we show the complementary properties of Fisher information and Shannon information, and give a guide for Fisher information conditioning to achieve high resolution reconstruction, disentangle feature learning, over-fitting/over-regularization resistance, etc.
READ FULL TEXT