Word-order biases in deep-agent emergent communication

by   Rahma Chaabouni, et al.

Sequence-processing neural networks led to remarkable progress on many NLP tasks. As a consequence, there has been increasing interest in understanding to what extent they process language as humans do. We aim here to uncover which biases such models display with respect to "natural" word-order constraints. We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies. We study how the controlled characteristics of our miniature languages affect individual learning and their stability across multiple network generations. The results draw a mixed picture. On the one hand, neural networks show a strong tendency to avoid long-distance dependencies. On the other hand, there is no clear preference for the efficient, non-redundant encoding of information that is widely attested in natural language. We thus suggest inoculating a notion of "effort" into neural networks, as a possible way to make their linguistic behavior more human-like.


page 1

page 2

page 3

page 4


Examining the Inductive Bias of Neural Language Models with Artificial Languages

Since language models are used to model a wide variety of languages, it ...

Modeling rapid language learning by distilling Bayesian priors into artificial neural networks

Humans can learn languages from remarkably little experience. Developing...

What makes a language easy to deep-learn?

Neural networks drive the success of natural language processing. A fund...

Language Model Metrics and Procrustes Analysis for Improved Vector Transformation of NLP Embeddings

Artificial Neural networks are mathematical models at their core. This t...

"I'm" Lost in Translation: Pronoun Missteps in Crowdsourced Data Sets

As virtual assistants continue to be taken up globally, there is an ever...

MLRegTest: A Benchmark for the Machine Learning of Regular Languages

Evaluating machine learning (ML) systems on their ability to learn known...

Please sign up or login with your details

Forgot password? Click here to reset