Meta-learning for Multi-variable Non-convex Optimization Problems: Iterating Non-optimums Makes Optimum Possible

by   Jingyuan Xia, et al.

In this paper, we aim to address the problem of solving a non-convex optimization problem over an intersection of multiple variable sets. This kind of problems is typically solved by using an alternating minimization (AM) strategy which splits the overall problem into a set of sub-problems corresponding to each variable, and then iteratively performs minimization over each sub-problem using a fixed updating rule. However, due to the intrinsic non-convexity of the overall problem, the optimization can usually be trapped into bad local minimum even when each sub-problem can be globally optimized at each iteration. To tackle this problem, we propose a meta-learning based Global Scope Optimization (GSO) method. It adaptively generates optimizers for sub-problems via meta-learners and constantly updates these meta-learners with respect to the global loss information of the overall problem. Therefore, the sub-problems are optimized with the objective of minimizing the global loss specifically. We evaluate the proposed model on a number of simulations, including solving bi-linear inverse problems: matrix completion, and non-linear problems: Gaussian mixture models. The experimental results show that our proposed approach outperforms AM-based methods in standard settings, and is able to achieve effective optimization in some challenging cases while other methods would typically fail.


page 10

page 15


Non-Convex Optimizations for Machine Learning with Theoretical Guarantee: Robust Matrix Completion and Neural Network Learning

Despite the recent development in machine learning, most learning system...

Non-convex Global Minimization and False Discovery Rate Control for the TREX

The TREX is a recently introduced method for performing sparse high-dime...

Solving Inverse Problems by Joint Posterior Maximization with a VAE Prior

In this paper we address the problem of solving ill-posed inverse proble...

AGGLIO: Global Optimization for Locally Convex Functions

This paper presents AGGLIO (Accelerated Graduated Generalized LInear-mod...

Global Optimality in Bivariate Gradient-based DAG Learning

Recently, a new class of non-convex optimization problems motivated by t...

When compressive learning fails: blame the decoder or the sketch?

In compressive learning, a mixture model (a set of centroids or a Gaussi...

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

In this work we address the problem of solving ill-posed inverse problem...

Please sign up or login with your details

Forgot password? Click here to reset