Stochastic Variable Metric Proximal Gradient with variance reduction for non-convex composite optimization

01/02/2023
by   Gersende Fort, et al.
0

This paper introduces a novel algorithm, the Perturbed Proximal Preconditioned SPIDER algorithm (3P-SPIDER), designed to solve finite sum non-convex composite optimization. It is a stochastic Variable Metric Forward-Backward algorithm, which allows approximate preconditioned forward operator and uses a variable metric proximity operator as the backward operator; it also proposes a mini-batch strategy with variance reduction to address the finite sum setting. We show that 3P-SPIDER extends some Stochastic preconditioned Gradient Descent-based algorithms and some Incremental Expectation Maximization algorithms to composite optimization and to the case the forward operator can not be computed in closed form. We also provide an explicit control of convergence in expectation of 3P-SPIDER, and study its complexity in order to satisfy the epsilon-approximate stationary condition. Our results are the first to combine the composite non-convex optimization setting, a variance reduction technique to tackle the finite sum setting by using a minibatch strategy and, to allow deterministic or random approximations of the preconditioned forward operator. Finally, through an application to inference in a logistic regression model with random effects, we numerically compare 3P-SPIDER to other stochastic forward-backward algorithms and discuss the role of some design parameters of 3P-SPIDER.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/02/2016

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization

Here we study non-convex composite optimization: first, a finite-sum of ...
research
02/11/2015

Proximal Algorithms in Statistics and Machine Learning

In this paper we develop proximal methods for statistical learning. Prox...
research
05/27/2021

OpReg-Boost: Learning to Accelerate Online Algorithms with Operator Regression

This paper presents a new regularization approach – termed OpReg-Boost –...
research
02/22/2018

Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem

We study sampling as optimization in the space of measures. We focus on ...
research
10/02/2020

A variable metric mini-batch proximal stochastic recursive gradient algorithm with diagonal Barzilai-Borwein stepsize

Variable metric proximal gradient methods with different metric selectio...
research
08/29/2019

Multi-Level Composite Stochastic Optimization via Nested Variance Reduction

We consider multi-level composite optimization problems where each mappi...
research
02/20/2023

A One-Sample Decentralized Proximal Algorithm for Non-Convex Stochastic Composite Optimization

We focus on decentralized stochastic non-convex optimization, where n ag...

Please sign up or login with your details

Forgot password? Click here to reset