research
∙
08/13/2023
Law of Balance and Stationary Distribution of Stochastic Gradient Descent
The stochastic gradient descent (SGD) algorithm is the algorithm we use ...
share
research
∙
03/27/2023
On the stepwise nature of self-supervised learning
We present a simple picture of the training process of self-supervised l...
research
∙
03/23/2023
The Probabilistic Stability of Stochastic Gradient Descent
A fundamental open problem in deep learning theory is how to define and ...
research
∙
10/03/2022
Sparsity by Redundancy: Solving L_1 with a Simple Reparametrization
We identify and prove a general principle: L_1 sparsity can be achieved ...
research
∙
10/02/2022
What shapes the loss landscape of self-supervised learning?
Prevention of complete and dimensional collapse of representations has r...
research
∙
05/25/2022
Exact Phase Transitions in Deep Learning
This work reports deep-learning-unique first-order and second-order phas...
research
∙
05/09/2022
Posterior Collapse of a Linear Latent Variable Model
This work identifies the existence and cause of a type of posterior coll...
research
∙
02/10/2022
Exact Solutions of a Deep Linear Network
This work finds the exact solutions to a deep linear network with weight...
research
∙
01/30/2022
Stochastic Neural Networks with Infinite Width are Deterministic
This work theoretically studies stochastic neural networks, a main type ...
research
∙
07/25/2021
SGD May Never Escape Saddle Points
Stochastic gradient descent (SGD) has been deployed to solve highly non-...
research
∙
06/08/2021
What Data Augmentation Do We Need for Deep-Learning-Based Finance?
The main task we consider is portfolio construction in a speculative mar...
research
∙
05/20/2021
Logarithmic landscape and power-law escape rate of SGD
Stochastic gradient descent (SGD) undergoes complicated multiplicative n...
research
∙
05/15/2021
On the Distributional Properties of Adaptive Gradients
Adaptive gradient methods have achieved remarkable success in training d...
research
∙
02/10/2021
On Minibatch Noise: Discrete-Time SGD, Overparametrization, and Bayes
The noise in stochastic gradient descent (SGD), caused by minibatch samp...
research
∙
12/07/2020
Stochastic Gradient Descent with Large Learning Rate
As a simple and efficient optimization method in deep learning, stochast...
research
∙
12/04/2020
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
The natural world is abundant with concepts expressed via visual, acoust...
research
∙
10/23/2020
An Investigation of how Label Smoothing Affects Generalization
It has been hypothesized that label smoothing can reduce overfitting and...
research
∙
06/15/2020
Neural Networks Fail to Learn Periodic Functions and How to Fix It
Previous literature offers limited clues on how to learn a periodic func...
research
∙
03/25/2020
Volumization as a Natural Generalization of Weight Decay
We propose a novel regularization method, called volumization, for neura...
research
∙
02/16/2020
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important t...
research
∙
02/12/2020
LaProp: a Better Way to Combine Momentum with Adaptive Gradient
Identifying a divergence problem in Adam, we propose a new optimizer, La...
research
∙
01/06/2020
Think Locally, Act Globally: Federated Learning with Local and Global Representations
Federated learning is an emerging research paradigm to train models on p...
research
∙
06/29/2019