A framework for large-scale distributed AI search across disconnected heterogeneous infrastructures

by   Lars Kotthoff, et al.

We present a framework for a large-scale distributed eScience Artificial Intelligence search. Our approach is generic and can be used for many different problems. Unlike many other approaches, we do not require dedicated machines, homogeneous infrastructure or the ability to communicate between nodes. We give special consideration to the robustness of the framework, minimising the loss of effort even after total loss of infrastructure, and allowing easy verification of every step of the distribution process. In contrast to most eScience applications, the input data and specification of the problem is very small, being easily given in a paragraph of text. The unique challenges our framework tackles are related to the combinatorial explosion of the space that contains the possible solutions and the robustness of long-running computations. Not only is the time required to finish the computations unknown, but also the resource requirements may change during the course of the computation. We demonstrate the applicability of our framework by using it to solve a challenging and hitherto open problem in computational mathematics. The results demonstrate that our approach easily scales to computations of a size that would have been impossible to tackle in practice just a decade ago.


page 1

page 2

page 3

page 4


Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning

The relational data model was designed to facilitate large-scale data ma...

V-BOINC: The Virtualization of BOINC

The Berkeley Open Infrastructure for Network Computing (BOINC) is an ope...

Distributed solving through model splitting

Constraint problems can be trivially solved in parallel by exploring dif...

A Distributed Hybrid Hardware-In-the-Loop Simulation framework for Infrastructure Enabled Autonomy

Infrastructure Enabled Autonomy (IEA) is a new paradigm that employs a d...

Recursion, Probability, Convolution and Classification for Computations

The main motivation of this work was practical, to offer computationally...

EXPERT: Public Benchmarks for Dynamic Heterogeneous Academic Graphs

Machine learning models that learn from dynamic graphs face nontrivial c...

Institutional Metaphors for Designing Large-Scale Distributed AI versus AI Techniques for Running Institutions

Artificial Intelligence (AI) started out with an ambition to reproduce t...

Please sign up or login with your details

Forgot password? Click here to reset