LASSO risk and phase transition under dependence
share
We consider the problem of recovering a k-sparse signal _0∈ℝ^p from noisy observations y= X_0+ w∈ℝ^n. One of the most popular approaches is the l_1-regularized least squares, also known as LASSO. We analyze the mean square error of LASSO in the case of random designs in which each row of X is drawn from distribution N(0,) with general . We first derive the asymptotic risk of LASSO in the limit of n,p→∞ with n/p→δ. We then examine conditions on n, p, and k for LASSO to exactly reconstruct _0 in the noiseless case w=0. A phase boundary δ_c=δ(ϵ) is precisely established in the phase space defined by 0≤δ,ϵ≤ 1, where ϵ=k/p. Above this boundary, LASSO perfectly recovers _0 with high probability. Below this boundary, LASSO fails to recover _0 with high probability. While the values of the non-zero elements of _0 do not have any effect on the phase transition curve, our analysis shows that δ_c does depend on the signed pattern of the nonzero values of _0 for general I_p. This is in sharp contrast to the previous phase transition results derived in i.i.d. case with = I_p where δ_c is completely determined by ϵ regardless of the distribution of _0. Underlying our formalism is a recently developed efficient algorithm called approximate message passing (AMP) algorithm. We generalize the state evolution of AMP from i.i.d. case to general case with I_p. Extensive computational experiments confirm that our theoretical predictions are consistent with simulation results on moderate size system.
READ FULL TEXT