An Empirical Study of NetOps Capability of Pre-Trained Large Language Models

by   Yukai Miao, et al.

Nowadays, the versatile capabilities of Pre-trained Large Language Models (LLMs) have attracted much attention from the industry. However, some vertical domains are more interested in the in-domain capabilities of LLMs. For the Networks domain, we present NetEval, an evaluation set for measuring the comprehensive capabilities of LLMs in Network Operations (NetOps). NetEval is designed for evaluating the commonsense knowledge and inference ability in NetOps in a multi-lingual context. NetEval consists of 5,732 questions about NetOps, covering five different sub-domains of NetOps. With NetEval, we systematically evaluate the NetOps capability of 26 publicly available LLMs. The results show that only GPT-4 can achieve a performance competitive to humans. However, some open models like LLaMA 2 demonstrate significant potential.


page 3

page 6

page 9

page 10

page 11


On Pre-trained Language Models for Antibody

Antibodies are vital proteins offering robust protection for the human b...

Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage

The advancement of large language models (LLMs) brings notable improveme...

Can Pre-trained Language Models Interpret Similes as Smart as Human?

Simile interpretation is a crucial task in natural language processing. ...

OPT: Open Pre-trained Transformer Language Models

Large language models, which are often trained for hundreds of thousands...

SETI: Systematicity Evaluation of Textual Inference

We propose SETI (Systematicity Evaluation of Textual Inference), a novel...

Large Language Models for Telecom: Forthcoming Impact on the Industry

Large Language Models (LLMs) have emerged as a transformative force, rev...

Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Tickets

Over-parameterized models, typically pre-trained language models (LMs), ...

Please sign up or login with your details

Forgot password? Click here to reset