A Survey on Evaluation of Large Language Models

by   Yupeng Chang, et al.

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the `where' and `how' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.


page 1

page 2

page 3

page 4


Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

This paper presents a comprehensive and practical guide for practitioner...

Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

From pre-trained language model (PLM) to large language model (LLM), the...

Large Language Models Meet NL2Code: A Survey

The task of generating code from a natural language description, or NL2C...

A Survey on Multimodal Large Language Models

Multimodal Large Language Model (MLLM) recently has been a new rising re...

BERT: A Review of Natural Language Processing and Understanding Applications

We cover the use of BERT, one of the most well-liked deep learning-based...

A Survey on Model Compression for Large Language Models

Large Language Models (LLMs) have revolutionized natural language proces...

ChatLog: Recording and Analyzing ChatGPT Across Time

While there are abundant researches about evaluating ChatGPT on natural ...

Please sign up or login with your details

Forgot password? Click here to reset