The effect of smooth parametrizations on nonconvex optimization landscapes
share
We develop new tools to study the landscapes of nonconvex optimization problems. Specifically, we leverage the fact that many such problems can be paired with others via a smooth parametrization of their domains. The global optima of such pairs of problems correspond to each other, but their landscapes can be significantly different. We introduce a framework to relate the two landscapes. Applications include: optimization over low-rank matrices and tensors by optimizing over a factorization; the Burer-Monteiro approach to semidefinite programs; training neural networks by optimizing over their weights and biases; and quotienting out symmetries. In all these examples, one of the two problems is smooth, and hence can be tackled with algorithms for optimization on manifolds. These can find desirable points (e.g., critical points). We determine the properties that ensure these map to desirable points for the other problem via the smooth parametrization. These new tools enable us to strengthen guarantees for an array of optimization problems, previously obtained on a case-by-case basis in the literature.
READ FULL TEXT