Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge

by   Marc Dymetman, et al.

We introduce LL-RNNs (Log-Linear RNNs), an extension of Recurrent Neural Networks that replaces the softmax output layer by a log-linear output layer, of which the softmax is a special case. This conceptually simple move has two main advantages. First, it allows the learner to combat training data sparsity by allowing it to model words (or more generally, output symbols) as complex combinations of attributes without requiring that each combination is directly observed in the training data (as the softmax does). Second, it permits the inclusion of flexible prior knowledge in the form of a priori specified modular features, where the neural network component learns to dynamically control the weights of a log-linear distribution exploiting these features. We conduct experiments in the domain of language modelling of French, that exploit morphological prior knowledge and show an important decrease in perplexity relative to a baseline RNN. We provide other motivating iillustrations, and finally argue that the log-linear and the neural-network components contribute complementary strengths to the LL-RNN: the LL aspect allows the model to incorporate rich prior knowledge, while the NN aspect, according to the "representation learning" paradigm, allows the model to discover novel combination of characteristics.


page 1

page 2

page 3

page 4


Label-Dependencies Aware Recurrent Neural Networks

In the last few years, Recurrent Neural Networks (RNNs) have proved effe...

A Factorized Recurrent Neural Network based architecture for medium to large vocabulary Language Modelling

Statistical language models are central to many applications that use se...

k-FFNN: A priori knowledge infused Feed-forward Neural Networks

Recurrent neural network (RNN) are being extensively used over feed-forw...

Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks

Natural language understanding (NLU) is a core component of a spoken dia...

The Neural State Pushdown Automata

In order to learn complex grammars, recurrent neural networks (RNNs) req...

What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?

In neural image captioning systems, a recurrent neural network (RNN) is ...

Discovering Novel Customer Features with Recurrent Neural Networks for Personality Based Financial Services

The micro-segmentation of customers in the finance sector is a non-trivi...

Please sign up or login with your details

Forgot password? Click here to reset