Scalable Exact Parent Sets Identification in Bayesian Networks Learning with Apache Spark

by   Subhadeep Karan, et al.
University at Buffalo

In Machine Learning, the parent set identification problem is to find a set of random variables that best explain selected variable given the data and some predefined scoring function. This problem is a critical component to structure learning of Bayesian networks and Markov blankets discovery, and thus has many practical applications, ranging from fraud detection to clinical decision support. In this paper, we introduce a new distributed memory approach to the exact parent sets assignment problem. To achieve scalability, we derive theoretical bounds to constraint the search space when MDL scoring function is used, and we reorganize the underlying dynamic programming such that the computational density is increased and fine-grain synchronization is eliminated. We then design efficient realization of our approach in the Apache Spark platform. Through experimental results, we demonstrate that the method maintains strong scalability on a 500-core standalone Spark cluster, and it can be used to efficiently process data sets with 70 variables, far beyond the reach of the currently available solutions.


page 1

page 2

page 3

page 4


A Traveling Salesman Learns Bayesian Networks

Structure learning of Bayesian networks is an important problem that ari...

Smoothness and Structure Learning by Proxy

As data sets grow in size, the ability of learning methods to find struc...

"Ideal Parent" Structure Learning for Continuous Variable Networks

In recent years, there is a growing interest in learning Bayesian networ...

PGMHD: A Scalable Probabilistic Graphical Model for Massive Hierarchical Data Problems

In the big data era, scalability has become a crucial requirement for an...

A Greedy, Flexible Algorithm to Learn an Optimal Bayesian Network Structure

In this report paper we first present a report of the Advanced Machine L...

Mining Massive Hierarchical Data Using a Scalable Probabilistic Graphical Model

Probabilistic Graphical Models (PGM) are very useful in the fields of ma...

Please sign up or login with your details

Forgot password? Click here to reset