AI Chat AI Image Generator AI Video Text to Speech

Recurrent Neural Network Regularization

09/08/2014

∙

by Wojciech Zaremba, et al.

∙

∙

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.

Wojciech Zaremba
27 publications
Ilya Sutskever
53 publications
Oriol Vinyals
101 publications

page 1

page 2

page 3

page 4

research

∙ 02/05/2014

Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) archit...

0 Hasim Sak, et al. ∙

research

∙ 11/05/2013

Dropout improves Recurrent Neural Networks for Handwriting Recognition

Recurrent neural networks (RNNs) with Long Short-Term memory cells curre...

0 Vu Pham, et al. ∙

research

∙ 08/07/2017

Regularizing and Optimizing LSTM Language Models

Recurrent neural networks (RNNs), such as long short-term memory network...

0 Stephen Merity, et al. ∙

research

∙ 08/23/2020

Variational Inference-Based Dropout in Recurrent Neural Networks for Slot Filling in Spoken Language Understanding

This paper proposes to generalize the variational recurrent neural netwo...

0 Jun Qi, et al. ∙

research

∙ 09/15/2017

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Model compression is significant for the wide adoption of Recurrent Neur...

0 Wei Wen, et al. ∙

research

∙ 11/05/2016

Quasi-Recurrent Neural Networks

Recurrent neural networks are a powerful tool for modeling sequential da...

0 James Bradbury, et al. ∙

research

∙ 10/17/2014

Learning to Execute

Recurrent Neural Networks (RNNs) with Long Short-Term Memory units (LSTM...

0 Wojciech Zaremba, et al. ∙