Is ChatGPT a Good Recommender? A Preliminary Study

by   Junling Liu, et al.
NetEase, Inc

Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.


page 3

page 4

page 5


LLMRec: Benchmarking Large Language Models on Recommendation Task

Recently, the fast development of Large Language Models (LLMs) such as C...

PALR: Personalization Aware LLMs for Recommendation

Large language models (LLMs) have recently received significant attentio...

Evaluating ChatGPT as a Recommender System: A Rigorous Approach

Recent popularity surrounds large AI language models due to their impres...

Enhancing E-Commerce Recommendation using Pre-Trained Language Model and Fine-Tuning

Pretrained Language Models (PLM) have been greatly successful on a board...

AUGUST: an Automatic Generation Understudy for Synthesizing Conversational Recommendation Datasets

High-quality data is essential for conversational recommendation systems...

Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models

The recent success of large language models (LLMs) has shown great poten...

RecMind: Large Language Model Powered Agent For Recommendation

Recent advancements in instructing Large Language Models (LLMs) to utili...

Please sign up or login with your details

Forgot password? Click here to reset