Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning

by   Chengfei Lv, et al.
Shanghai Jiao Tong University

To break the bottlenecks of mainstream cloud-based machine learning (ML) paradigm, we adopt device-cloud collaborative ML and build the first end-to-end and general-purpose system, called Walle, as the foundation. Walle consists of a deployment platform, distributing ML tasks to billion-scale devices in time; a data pipeline, efficiently preparing task input; and a compute container, providing a cross-platform and high-performance execution environment, while facilitating daily task iteration. Specifically, the compute container is based on Mobile Neural Network (MNN), a tensor compute engine along with the data processing and model execution libraries, which are exposed through a refined Python thread-level virtual machine (VM) to support diverse ML tasks and concurrent task execution. The core of MNN is the novel mechanisms of operator decomposition and semi-auto search, sharply reducing the workload in manually optimizing hundreds of operators for tens of hardware backends and further quickly identifying the best backend with runtime optimization for a computation graph. The data pipeline introduces an on-device stream processing framework to enable processing user behavior data at source. The deployment platform releases ML tasks with an efficient push-then-pull method and supports multi-granularity deployment policies. We evaluate Walle in practical e-commerce application scenarios to demonstrate its effectiveness, efficiency, and scalability. Extensive micro-benchmarks also highlight the superior performance of MNN and the Python thread-level VM. Walle has been in large-scale production use in Alibaba, while MNN has been open source with a broad impact in the community.


The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development

As machine learning is applied more and more widely, data scientists oft...

Looper: An end-to-end ML platform for product decisions

Modern software systems and products increasingly rely on machine learni...

ML-EXray: Visibility into ML Deployment on the Edge

Benefiting from expanding cloud infrastructure, deep neural networks (DN...

MLModelCI: An Automatic Cloud Platform for Efficient MLaaS

MLModelCI provides multimedia researchers and developers with a one-stop...

VeML: An End-to-End Machine Learning Lifecycle for Large-scale and High-dimensional Data

An end-to-end machine learning (ML) lifecycle consists of many iterative...

Scalable End-to-End ML Platforms: from AutoML to Self-serve

ML platforms help enable intelligent data-driven applications and mainta...

Petuum: A New Platform for Distributed Machine Learning on Big Data

What is a systematic way to efficiently apply a wide spectrum of advance...

Please sign up or login with your details

Forgot password? Click here to reset