The Landscape of Matrix Factorization Revisited

by   Hossein Valavi, et al.

We revisit the landscape of the simple matrix factorization problem. For low-rank matrix factorization, prior work has shown that there exist infinitely many critical points all of which are either global minima or strict saddles. At a strict saddle the minimum eigenvalue of the Hessian is negative. Of interest is whether this minimum eigenvalue is uniformly bounded below zero over all strict saddles. To answer this we consider orbits of critical points under the general linear group. For each orbit we identify a representative point, called a canonical point. If a canonical point is a strict saddle, so is every point on its orbit. We derive an expression for the minimum eigenvalue of the Hessian at each canonical strict saddle and use this to show that the minimum eigenvalue of the Hessian over the set of strict saddles is not uniformly bounded below zero. We also show that a known invariance property of gradient flow ensures the solution of gradient flow only encounters critical points on an invariant manifold M_C determined by the initial condition. We show that, in contrast to the general situation, the minimum eigenvalue of strict saddles in M_0 is uniformly bounded below zero. We obtain an expression for this bound in terms of the singular values of the matrix being factorized. This bound depends on the size of the nonzero singular values and on the separation between distinct nonzero singular values of the matrix.


page 1

page 2

page 3

page 4


Asymptotic Escape of Spurious Critical Points on the Low-rank Matrix Manifold

We show that the Riemannian gradient descent algorithm on the low-rank m...

Symmetry, Saddle Points, and Global Geometry of Nonconvex Matrix Factorization

We propose a general theory for studying the geometry of nonconvex objec...

Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm

Gradient-related first-order methods have become the workhorse of large-...

Embedding Principle: a hierarchical structure of loss landscape of deep neural networks

We prove a general Embedding Principle of loss landscape of deep neural ...

Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss

We consider a deep matrix factorization model of covariance matrices tra...

Analysis of Asymptotic Escape of Strict Saddle Sets in Manifold Optimization

In this paper, we provide some analysis on the asymptotic escape of stri...

Please sign up or login with your details

Forgot password? Click here to reset