Group Validation in Recommender Systems: Framework for Multi-layer Performance Evaluation

by   Wissam Al Jurdi, et al.

Interpreting the performance results of models that attempt to realize user behavior in platforms that employ recommenders is a big challenge that researchers and practitioners continue to face. Although current evaluation tools possess the capacity to provide solid general overview of a system's performance, they still lack consistency and effectiveness in their use as evident in most recent studies on the topic. Current traditional assessment techniques tend to fail to detect variations that could occur on smaller subsets of the data and lack the ability to explain how such variations affect the overall performance. In this article, we focus on the concept of data clustering for evaluation in recommenders and apply a neighborhood assessment method for the datasets of recommender system applications. This new method, named neighborhood-based evaluation, aids in better understanding critical performance variations in more compact subsets of the system to help spot weaknesses where such variations generally go unnoticed with conventional metrics and are typically averaged out. This new modular evaluation layer complements the existing assessment mechanisms and provides the possibility of several applications to the recommender ecosystem such as model evolution tests, fraud/attack detection and a possibility for hosting a hybrid model setup.


page 1

page 2

page 3

page 4


Distributionally-Informed Recommender System Evaluation

Current practice for evaluating recommender systems typically focuses on...

EvalRS: a Rounded Evaluation of Recommender Systems

Much of the complexity of Recommender Systems (RSs) comes from the fact ...

A Survey of Recommender System Techniques and the Ecommerce Domain

In this big data era, it is hard for the current generation to find the ...

RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems

The development of recommender systems that optimize multi-turn interact...

Simulating News Recommendation Ecosystem for Fun and Profit

Understanding the evolution of online news communities is essential for ...

It's Time to Consider "Time" when Evaluating Recommender-System Algorithms [Proposal]

In this position paper, we question the current practice of calculating ...

Validation of a recommender system for prompting omitted foods in online dietary assessment surveys

Recall assistance methods are among the key aspects that improve the acc...

Please sign up or login with your details

Forgot password? Click here to reset