XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse

by   Hyoukjun Kwon, et al.

Real-time multi-model multi-task (MMMT) workloads, a new form of deep learning inference workloads, are emerging for applications areas like extended reality (XR) to support metaverse use cases. These workloads combine user interactivity with computationally complex machine learning (ML) activities. Compared to standard ML applications, these ML workloads present unique difficulties and constraints. Real-time MMMT workloads impose heterogeneity and concurrency requirements on future ML systems and devices, necessitating the development of new capabilities. This paper begins with a discussion of the various characteristics of these real-time MMMT ML workloads and presents an ontology for evaluating the performance of future ML hardware for XR systems. Next, we present XRBench, a collection of MMMT ML tasks, models, and usage scenarios that execute these models in three representative ways: cascaded, concurrent, and cascaded-concurrency for XR use cases. Finally, we emphasize the need for new metrics that capture the requirements properly. We hope that our work will stimulate research and lead to the development of a new generation of ML systems for XR use cases.


page 2

page 6

page 7

page 9


SDRM3: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads

Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and dro...

Real-Time Scheduling of Machine Learning Operations on Heterogeneous Neuromorphic SoC

Neuromorphic Systems-on-Chip (NSoCs) are becoming heterogeneous by integ...

Reasonable Scale Machine Learning with Open-Source Metaflow

As Machine Learning (ML) gains adoption across industries and new use ca...

FleXR: A System Enabling Flexibly Distributed Extended Reality

Extended reality (XR) applications require computationally demanding fun...

Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems

The real-world use cases of Machine Learning (ML) have exploded over the...

Neural Network Inference on Mobile SoCs

The ever-increasing demand from mobile Machine Learning (ML) application...

Evaluation Methodologies for Code Learning Tasks

There has been a growing interest in developing machine learning (ML) mo...

Please sign up or login with your details

Forgot password? Click here to reset