Sequential Search with Off-Policy Reinforcement Learning

by   Dadong Miao, et al.

Recent years have seen a significant amount of interests in Sequential Recommendation (SR), which aims to understand and model the sequential user behaviors and the interactions between users and items over time. Surprisingly, despite the huge success Sequential Recommendation has achieved, there is little study on Sequential Search (SS), a twin learning task that takes into account a user's current and past search queries, in addition to behavior on historical query sessions. The SS learning task is even more important than the counterpart SR task for most of E-commence companies due to its much larger online serving demands as well as traffic volume. To this end, we propose a highly scalable hybrid learning model that consists of an RNN learning framework leveraging all features in short-term user-item interactions, and an attention model utilizing selected item-only features from long-term interactions. As a novel optimization step, we fit multiple short user sequences in a single RNN pass within a training batch, by solving a greedy knapsack problem on the fly. Moreover, we explore the use of off-policy reinforcement learning in multi-session personalized search ranking. Specifically, we design a pairwise Deep Deterministic Policy Gradient model that efficiently captures users' long term reward in terms of pairwise classification error. Extensive ablation experiments demonstrate significant improvement each component brings to its state-of-the-art baseline, on a variety of offline and online metrics.


page 1

page 2

page 3

page 4


Dynamic Memory based Attention Network for Sequential Recommendation

Sequential recommendation has become increasingly essential in various o...

IDNP: Interest Dynamics Modeling using Generative Neural Processes for Sequential Recommendation

Recent sequential recommendation models rely increasingly on consecutive...

SDM: Sequential Deep Matching Model for Online Large-scale Recommender System

Capturing users' precise preferences is a fundamental problem in large-s...

Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Offline RL

We study session-based recommendation scenarios where we want to recomme...

ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor

Long-term engagement is preferred over immediate engagement in sequentia...

AutoMLP: Automated MLP for Sequential Recommendations

Sequential recommender systems aim to predict users' next interested ite...

Leveraging Negative Signals with Self-Attention for Sequential Music Recommendation

Music streaming services heavily rely on their recommendation engines to...

Please sign up or login with your details

Forgot password? Click here to reset