Scalable solution to crossed random effects model with random slopes

by   Disha Ghandwani, et al.

The crossed random-effects model is widely used in applied statistics, finding applications in various fields such as longitudinal studies, e-commerce, and recommender systems, among others. However, these models encounter scalability challenges, as the computational time grows disproportionately with the number of data points, typically following a cubic root relationship (N^(3/2) or worse) with N. Our inspiration for addressing this issue comes from observing the recommender system employed by an online clothing retailer. Our dataset comprises over 700,000 clients, 5,000 items, and 5,000,000 measurements. When applying the maximum likelihood approach to fit crossed random effects, computational inefficiency becomes a significant concern, limiting the applicability of this approach in large-scale settings. To tackle the scalability issues, previous research by Ghosh et al. (2022a) and Ghosh et al. (2022b) has explored linear and logistic regression models utilizing fixed-effect features based on client and item variables, while incorporating random intercept terms for clients and items. In this study, we present a more generalized version of the problem, allowing random effect sizes/slopes. This extension enables us to capture the variability in effect size among both clients and items. Importantly, we have developed a scalable solution to address the aforementioned problem and have empirically demonstrated the consistency of our estimates. Specifically, as the number of data points increases, our estimates converge towards the true parameters. To validate our approach, we implement the proposed algorithm using Stitch Fix data.


page 1

page 2

page 3

page 4


Scalable logistic regression with crossed random effects

The cost of both generalized least squares (GLS) and Gibbs sampling in a...

Computing the Value of Data: Towards Applied Data Minimalism

We present an approach to compute the monetary value of individual data ...

Scalable Estimation of Probit Models with Crossed Random Effects

Crossed random effects structures arise in many scientific contexts. The...

Global Convergence of Federated Learning for Mixed Regression

This paper studies the problem of model training under Federated Learnin...

Exploration in two-stage recommender systems

Two-stage recommender systems are widely adopted in industry due to thei...

Backfitting for large scale crossed random effects regressions

Regression models with crossed random effect error models can be very ex...

Fitting a deeply-nested hierarchical model to a large book review dataset using a moment-based estimator

We consider a particular instance of a common problem in recommender sys...

Please sign up or login with your details

Forgot password? Click here to reset