LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System

by   Ashish Bastola, et al.
Northern Arizona University
Clemson University

CSCW studies have increasingly explored AI's role in enhancing communication efficiency and productivity in collaborative tasks. AI tools such as chatbots, smart replies, and language models aim to optimize conversation management and improve team performance. Early AI assistants, such as Gmail smart reply, were limited by predefined knowledge bases and decision trees. However, the advent of large language models (LLMs) such as ChatGPT has revolutionized AI assistants, employing advanced deep learning architecture to generate context-aware, coherent, and personalized responses. Consequently, ChatGPT-based AI assistants provide a more natural and efficient user experience across various tasks and domains. In this paper, we formalize the concept of AI Collaborative Tools (ACT) as AI technologies in human collaborative work and discuss how the emergence of ChatGPT has transformed the AI landscape and increased focus on ACT for improving team performance. Meanwhile, we present an LLM-based Smart Reply (LSR) system utilizing the ChatGPT API to generate personalized responses in daily collaborative scenarios, considering context, tone, and communication style. Our two-step process involves generating a preliminary response type (e.g., Agree, Disagree) to provide a generalized direction for message generation, thus reducing response drafting time. We conducted an experiment in which participants completed simulated work tasks, involving Google Calendar manipulation and a double-back N-back test, while interacting with researchers posing as teammates requesting scheduling changes. Our findings indicate that the AI teammate increases perceived performance and reduces mental demand, as measured by the NASA TLX, and improves performance in the N-back task. We also provide qualitative feedback on participants' experiences working with the AI teammate.


page 4

page 7


Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Large language models are becoming increasingly pervasive and ubiquitous...

"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models

Prewriting is the process of discovering and developing ideas before a f...

Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication

Traditionally, writing assistance systems have focused on short or even ...

How does AI chat change search behaviors?

Generative AI tools such as chatGPT are poised to change the way people ...

Augmented Collective Intelligence in Collaborative Ideation: Agenda and Challenges

AI systems may be better thought of as peers than as tools. This paper e...

Please sign up or login with your details

Forgot password? Click here to reset