b'Surya Ganguli'

research

∙ 06/26/2023

Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

Pretrained transformers exhibit the remarkable ability of in-context lea...

0 Allan Raventos, et al. ∙

research

∙ 06/07/2023

Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks

In this work, we reveal a strong implicit bias of stochastic gradient de...

0 Feng Chen, et al. ∙

research

∙ 03/16/2023

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Progress in machine learning has been driven in large part by massive in...

0 Amro Abbas, et al. ∙

research

∙ 11/16/2022

Holistic Evaluation of Language Models

Language models (LMs) are becoming the foundation for almost all major l...

21 Percy Liang, et al. ∙

research

∙ 10/15/2022

Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution

Neuroscience has long been an important driver of progress in artificial...

23 Anthony Zador, et al. ∙

research

∙ 10/11/2022

What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries

Deep neural network classifiers partition input space into high confiden...

0 Stanislav Fort, et al. ∙

research

∙ 10/07/2022

The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

In this work, we explore the maximum-margin bias of quasi-homogeneous ne...

0 Daniel Kunin, et al. ∙

research

∙ 10/06/2022

Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?

Modern deep learning involves training costly, highly overparameterized ...

12 Mansheej Paul, et al. ∙

research

∙ 09/30/2022

Disentangling with Biological Constraints: A Theory of Functional Cell Types

Neurons in the brain are often finely tuned for specific task variables....

0 James C. R. Whittington, et al. ∙

research

∙ 06/29/2022

Beyond neural scaling laws: beating power law scaling via data pruning

Widely observed neural scaling laws, in which error falls off as a power...

12 Ben Sorscher, et al. ∙

research

∙ 06/02/2022

Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks

A striking observation about iterative magnitude pruning (IMP; Frankle e...

0 Mansheej Paul, et al. ∙

research

∙ 03/22/2022

MetaMorph: Learning Universal Controllers with Transformers

Multiple domains like vision, natural language, and audio are witnessing...

2 Agrim Gupta, et al. ∙

research

∙ 07/19/2021

Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations, and anomalous diffusion

In this work we explore the limiting dynamics of deep neural networks tr...

0 Daniel Kunin, et al. ∙

research

∙ 07/15/2021

Deep Learning on a Data Diet: Finding Important Examples Early in Training

The recent success of deep learning has partially been driven by trainin...

0 Mansheej Paul, et al. ∙

research

∙ 07/13/2021

How many degrees of freedom do we need to train deep networks: a loss landscape perspective

A variety of recent works, spanning pruning, lottery tickets, and traini...

0 Brett W. Larsen, et al. ∙

research

∙ 02/12/2021

Understanding self-supervised Learning Dynamics without Contrastive Pairs

Contrastive approaches to self-supervised learning (SSL) learn represent...

9 Yuandong Tian, et al. ∙

research

∙ 02/03/2021

Embodied Intelligence via Learning and Evolution

The intertwined processes of learning and evolution in complex environme...

10 Agrim Gupta, et al. ∙

research

∙ 12/08/2020

Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics

Predicting the dynamics of neural network parameters during training is ...

0 Daniel Kunin, et al. ∙

research

∙ 10/28/2020

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel

In suitably initialized wide networks, small learning rates transform de...

0 Stanislav Fort, et al. ∙

research

∙ 10/22/2020

Identifying Learning Rules From Neural Network Observables

The brain modifies its synaptic strengths during learning in order to be...

7 Aran Nayebi, et al. ∙

research

∙ 10/15/2020

RNNs can generate bounded hierarchical languages with optimal memory

Recurrent neural networks empirically generate natural language with hig...

23 John Hewitt, et al. ∙

research

∙ 10/01/2020

Understanding Self-supervised Learning with Dual Deep Networks

We propose a novel theoretical framework to understand self-supervised l...

52 Yuandong Tian, et al. ∙

research

∙ 06/25/2020

Predictive coding in balanced neural networks with noise, chaos and delays

Biological neural networks face a formidable task: performing reliable c...

8 Jonathan Kadmon, et al. ∙

research

∙ 06/09/2020

Pruning neural networks without any data by iteratively conserving synaptic flow

Pruning the parameters of deep neural networks has generated intense int...

6 Hidenori Tanaka, et al. ∙

research

∙ 02/28/2020

Two Routes to Scalable Credit Assignment without Weight Symmetry

The neural plausibility of backpropagation has long been disputed, prima...

5 Daniel Kunin, et al. ∙

research

∙ 12/12/2019

From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction

Recently, deep feedforward neural networks have achieved considerable su...

0 Hidenori Tanaka, et al. ∙

research

∙ 10/14/2019

Emergent properties of the local geometry of neural loss landscapes

The local geometry of high dimensional neural network loss landscapes ca...

6 Stanislav Fort, et al. ∙

research

∙ 07/19/2019

Universality and individuality in neural dynamics across large populations of recurrent networks

Task-based modeling with recurrent neural networks (RNNs) has emerged as...

1 Niru Maheswaranathan, et al. ∙

research

∙ 06/29/2019

Fast Convolutive Nonnegative Matrix Factorization Through Coordinate and Block Coordinate Updates

Identifying recurring patterns in high-dimensional time series data is a...

5 Anthony Degleris, et al. ∙

research

∙ 06/25/2019

Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics

Recurrent neural networks (RNNs) are a widely used tool for modeling seq...

1 Niru Maheswaranathan, et al. ∙

research

∙ 01/03/2019

A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

The visual system is hierarchically organized to process visual informat...

0 Jack Lindsey, et al. ∙

research

∙ 10/23/2018

A mathematical theory of semantic development in deep neural networks

An extensive body of empirical research has revealed remarkable regulari...

26 Andrew M. Saxe, et al. ∙

research

∙ 10/23/2018

Statistical mechanics of low-rank tensor decomposition

Often, large, high dimensional datasets collected across multiple modali...

8 Jonathan Kadmon, et al. ∙

research

∙ 09/27/2018

An analytic theory of generalization dynamics and transfer learning in deep linear networks

Much attention has been devoted recently to the generalization puzzle in...

8 Andrew K. Lampinen, et al. ∙

research

∙ 06/20/2018

Task-Driven Convolutional Recurrent Models of the Visual System

Feed-forward convolutional neural networks (CNNs) are currently state-of...

0 Aran Nayebi, et al. ∙

research

∙ 02/27/2018

The Emergence of Spectral Universality in Deep Networks

Recent work has shown that tight concentration of the entire spectrum of...

0 Jeffrey Pennington, et al. ∙

research

∙ 11/13/2017

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

It is well known that the initialization of weights in deep neural netwo...

0 Jeffrey Pennington, et al. ∙

research

∙ 11/07/2017

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net

We propose a novel method to directly learn a stochastic transition oper...

0 Anirudh Goyal, et al. ∙

research

∙ 05/31/2017

SuperSpike: Supervised learning in multi-layer spiking neural networks

A vast majority of computation in the brain is performed by spiking neur...

0 Friedemann Zenke, et al. ∙

research

∙ 03/27/2017

Biologically inspired protection of deep networks from adversarial attacks

Inspired by biophysical principles underlying nonlinear dendritic comput...

0 Aran Nayebi, et al. ∙

research

∙ 03/13/2017

Continual Learning Through Synaptic Intelligence

While deep learning has led to remarkable advances across diverse applic...

0 Friedemann Zenke, et al. ∙

research

∙ 02/06/2017

Deep Learning Models of the Retinal Response to Natural Scenes

A central challenge in neuroscience is to understand neural computations...

0 Lane T. McIntosh, et al. ∙

research

∙ 11/24/2016

Survey of Expressivity in Deep Neural Networks

We survey results on neural network expressivity described in "On the Ex...

0 Jojo Yun, et al. ∙

research

∙ 11/04/2016

Deep Information Propagation

We study the behavior of untrained neural networks whose weights and bia...

0 Samuel S. Schoenholz, et al. ∙

research

∙ 09/22/2016

An equivalence between high dimensional Bayes optimal inference and M-estimation

When recovering an unknown signal from noisy measurements, the computati...

0 Madhu Advani, et al. ∙

research

∙ 07/14/2016

Random projections of random manifolds

Interesting data often concentrate on low dimensional smooth manifolds i...

0 Subhaneil Lahiri, et al. ∙

research

∙ 06/16/2016

Exponential expressivity in deep neural networks through transient chaos

We combine Riemannian geometry with the mean field theory of high dimens...

0 Ben Poole, et al. ∙

research

∙ 06/16/2016

On the Expressive Power of Deep Neural Networks

We propose a new approach to the problem of neural network expressivity,...

0 Jojo Yun, et al. ∙

research

∙ 03/24/2016

A universal tradeoff between power, precision and speed in physical communication

Maximizing the speed and precision of communication while minimizing pow...

0 Subhaneil Lahiri, et al. ∙

research

∙ 01/18/2016

Statistical Mechanics of High-Dimensional Inference

To model modern large-scale datasets, we need efficient algorithms to in...

0 Madhu Advani, et al. ∙

Surya Ganguli

Featured Co-authors