Dense and Sparse Coding: Theory and Architectures
share
The sparse representation model has been successfully utilized in a number of signal and image processing tasks; however, recent research has highlighted its limitations in certain deep-learning architectures. This paper proposes a novel dense and sparse coding model that considers the problem of recovering a dense vector 𝐱 and a sparse vector 𝐮 given linear measurements of the form 𝐲 = 𝐀𝐱+𝐁𝐮. Our first theoretical result proposes a new natural geometric condition based on the minimal angle between subspaces corresponding to the measurement matrices 𝐀 and 𝐁 to establish the uniqueness of solutions to the linear system. The second analysis shows that, under mild assumptions and sufficient linear measurements, a convex program recovers the dense and sparse components with high probability. The standard RIPless analysis cannot be directly applied to this setup. Our proof is a non-trivial adaptation of techniques from anisotropic compressive sensing theory and is based on an analysis of a matrix derived from the measurement matrices 𝐀 and 𝐁. We begin by demonstrating the effectiveness of the proposed model on simulated data. Then, to address its use in a dictionary learning setting, we propose a dense and sparse auto-encoder (DenSaE) that is tailored to it. We demonstrate that a) DenSaE denoises natural images better than architectures derived from the sparse coding model (𝐁𝐮), b) training the biases in the latter amounts to implicitly learning the 𝐀𝐱 + 𝐁𝐮 model, and c) 𝐀 and 𝐁 capture low- and high-frequency contents, respectively.
READ FULL TEXT