MetaBags: Bagged Meta-Decision Trees for Regression

by   Jihed Khiari, et al.

Ensembles are popular methods for solving practical supervised learning problems. They reduce the risk of having underperforming models in production-grade software. Although critical, methods for learning heterogeneous regression ensembles have not been proposed at large scale, whereas in classical ML literature, stacking, cascading and voting are mostly restricted to classification problems. Regression poses distinct learning challenges that may result in poor performance, even when using well established homogeneous ensemble schemas such as bagging or boosting. In this paper, we introduce MetaBags, a novel, practically useful stacking framework for regression. MetaBags is a meta-learning algorithm that learns a set of meta-decision trees designed to select one base model (i.e. expert) for each query, and focuses on inductive bias reduction. A set of meta-decision trees are learned using different types of meta-features, specially created for this purpose - to then be bagged at meta-level. This procedure is designed to learn a model with a fair bias-variance trade-off, and its improvement over base model performance is correlated with the prediction diversity of different experts on specific input space subregions. The proposed method and meta-features are designed in such a way that they enable good predictive performance even in subregions of space which are not adequately represented in the available training data. An exhaustive empirical testing of the method was performed, evaluating both generalization error and scalability of the approach on synthetic, open and real-world application datasets. The obtained results show that our method significantly outperforms existing state-of-the-art approaches.


page 3

page 8


Gradient Boosting With Piece-Wise Linear Regression Trees

Gradient boosting using decision trees as base learners, so called Gradi...

Towards meta-learning for multi-target regression problems

Several multi-target regression methods were devel-oped in the last year...

Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees

Real world datasets contain incorrectly labeled instances that hamper th...

PreDisM: Pre-Disaster Modelling With CNN Ensembles for At-Risk Communities

The machine learning community has recently had increased interest in th...

Improving the compromise between accuracy, interpretability and personalization of rule-based machine learning in medical problems

One of the key challenges when developing a predictive model is the capa...

A Systematic Study of Cross-Project Defect Prediction With Meta-Learning

The prediction of defects in a target project based on data from externa...

Learning to Learn to Predict Performance Regressions in Production at Meta

Catching and attributing code change-induced performance regressions in ...

Please sign up or login with your details

Forgot password? Click here to reset