Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

by   Gašper Beguš, et al.

The performance of large language models (LLMs) has recently improved to the point where the models can generate valid and coherent meta-linguistic analyses of data. This paper illustrates a vast potential for analyses of the meta-linguistic abilities of large language models. LLMs are primarily trained on language data in the form of text; analyzing their meta-linguistic abilities is informative both for our understanding of the general capabilities of LLMs as well as for models of linguistics. In this paper, we propose several types of experiments and prompt designs that allow us to analyze the ability of GPT-4 to generate meta-linguistic analyses. We focus on three linguistics subfields with formalisms that allow for a detailed analysis of GPT-4's theoretical capabilities: theoretical syntax, phonology, and semantics. We identify types of experiments, provide general guidelines, discuss limitations, and offer future directions for this research program.


page 6

page 7

page 9

page 10

page 12

page 15

page 16

page 17


Large language models and (non-)linguistic recursion

Recursion is one of the hallmarks of human language. While many design f...

Password Guessers Under a Microscope: An In-Depth Analysis to Inform Deployments

Password guessers are instrumental for assessing the strength of passwor...

Are Emergent Abilities of Large Language Models a Mirage?

Recent work claims that large language models display emergent abilities...

BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models

The rapid development of Large Language Models (LLMs) and the emergence ...

A blind spot for large language models: Supradiegetic linguistic information

Large Language Models (LLMs) like ChatGPT reflect profound changes in th...

What GPT Knows About Who is Who

Coreference resolution – which is a crucial task for understanding disco...

Is there an aesthetic component of language?

Speakers of all human languages make use of grammatical devices to expre...

Please sign up or login with your details

Forgot password? Click here to reset