R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics

by   Zexin Li, et al.

Autonomous robotic systems, like autonomous vehicles and robotic search and rescue, require efficient on-device training for continuous adaptation of Deep Reinforcement Learning (DRL) models in dynamic environments. This research is fundamentally motivated by the need to understand and address the challenges of on-device real-time DRL, which involves balancing timing and algorithm performance under memory constraints, as exposed through our extensive empirical studies. This intricate balance requires co-optimizing two pivotal parameters of DRL training – batch size and replay buffer size. Configuring these parameters significantly affects timing and algorithm performance, while both (unfortunately) require substantial memory allocation to achieve near-optimal performance. This paper presents R^3, a holistic solution for managing timing, memory, and algorithm performance in on-device real-time DRL training. R^3 employs (i) a deadline-driven feedback loop with dynamic batch sizing for optimizing timing, (ii) efficient memory management to reduce memory footprint and allow larger replay buffer sizes, and (iii) a runtime coordinator guided by heuristic analysis and a runtime profiler for dynamically adjusting memory resource reservations. These components collaboratively tackle the trade-offs in on-device DRL training, improving timing and algorithm performance while minimizing the risk of out-of-memory (OOM) errors. We implemented and evaluated R^3 extensively across various DRL frameworks and benchmarks on three hardware platforms commonly adopted by autonomous robotic systems. Additionally, we integrate R^3 with a popular realistic autonomous car simulator to demonstrate its real-world applicability. Evaluation results show that R^3 achieves efficacy across diverse platforms, ensuring consistent latency performance and timing predictability with minimal overhead.


page 1

page 10

page 11


Dependability Analysis of Deep Reinforcement Learning based Robotics and Autonomous Systems

While Deep Reinforcement Learning (DRL) provides transformational capabi...

A Memory Efficient Deep Reinforcement Learning Approach For Snake Game Autonomous Agents

To perform well, Deep Reinforcement Learning (DRL) methods require signi...

Associative Memory Based Experience Replay for Deep Reinforcement Learning

Experience replay is an essential component in deep reinforcement learni...

Soft Hindsight Experience Replay

Efficient learning in the environment with sparse rewards is one of the ...

Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning

Recent deep neural networks based techniques, especially those equipped ...

Sim and Real: Better Together

Simulation is used extensively in autonomous systems, particularly in ro...

Please sign up or login with your details

Forgot password? Click here to reset