An Evaluation Framework for Interactive Recommender System

by   Oznur Alkan, et al.

Traditional recommender systems present a relatively static list of recommendations to a user where the feedback is typically limited to an accept/reject or a rating model. However, these simple modes of feedback may only provide limited insights as to why a user likes or dislikes an item and what aspects of the item the user has considered. Interactive recommender systems present an opportunity to engage the user in the process by allowing them to interact with the recommendations, provide feedback and impact the results in real-time. Evaluation of the impact of the user interaction typically requires an extensive user study which is time consuming and gives researchers limited opportunities to tune their solutions without having to conduct multiple rounds of user feedback. Additionally, user experience and design aspects can have a significant impact on the user feedback which may result in not necessarily assessing the quality of some of the underlying algorithmic decisions in the overall solution. As a result, we present an evaluation framework which aims to simulate the users interacting with the recommender. We formulate metrics to evaluate the quality of the interactive recommenders which are outputted by the framework once simulation is completed. While simulation along is not sufficient to evaluate a complete solution, the results can be useful to help researchers tune their solution before moving to the user study stage.


page 1

page 2

page 3

page 4


Concentrating on the Impact: Consequence-based Explanations in Recommender Systems

Recommender systems assist users in decision-making, where the presentat...

Seeker: Real-Time Interactive Search

This paper introduces Seeker, a system that allows users to interactivel...

On the Value of Bandit Feedback for Offline Recommender System Evaluation

In academic literature, recommender systems are often evaluated on the t...

Improving Recommender Systems Beyond the Algorithm

Recommender systems rely heavily on the predictive accuracy of the learn...

The HCI Aspects of Public Deployment of Research Chatbots: A User Study, Design Recommendations, and Open Challenges

Publicly deploying research chatbots is a nuanced topic involving necess...

Physics-based Mesh Deformation with Haptic Feedback and Material Anisotropy

We present a physics-based framework to simulate porous, deformable mate...

How Researchers Could Obtain Quick and Cheap User Feedback on their Algorithms Without Having to Operate their Own Recommender System

The majority of recommendation algorithms are evaluated on the basis of ...

Please sign up or login with your details

Forgot password? Click here to reset