Adaptive control of a mechatronic system using constrained residual reinforcement learning

by   Tom Staessens, et al.
Ghent University

We propose a simple, practical and intuitive approach to improve the performance of a conventional controller in uncertain environments using deep reinforcement learning while maintaining safe operation. Our approach is motivated by the observation that conventional controllers in industrial motion control value robustness over adaptivity to deal with different operating conditions and are suboptimal as a consequence. Reinforcement learning on the other hand can optimize a control signal directly from input-output data and thus adapt to operational conditions, but lacks safety guarantees, impeding its use in industrial environments. To realize adaptive control using reinforcement learning in such conditions, we follow a residual learning methodology, where a reinforcement learning algorithm learns corrective adaptations to a base controller's output to increase optimality. We investigate how constraining the residual agent's actions enables to leverage the base controller's robustness to guarantee safe operation. We detail the algorithmic design and propose to constrain the residual actions relative to the base controller to increase the method's robustness. Building on Lyapunov stability theory, we prove stability for a broad class of mechatronic closed-loop systems. We validate our method experimentally on a slider-crank setup and investigate how the constraints affect the safety during learning and optimality after convergence.


page 1

page 9


Data-Driven Robust Control Using Reinforcement Learning

This paper proposes a robust control design method using reinforcement-l...

Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles with Uncertainties

This paper presents a novel model-reference reinforcement learning contr...

Probabilistic Guarantees for Safe Deep Reinforcement Learning

Deep reinforcement learning has been successfully applied to many contro...

Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learning

Reinforcement learning (RL) techniques have been developed to optimize i...

Self-adaptive Torque Vectoring Controller Using Reinforcement Learning

Continuous direct yaw moment control systems such as torque-vectoring co...

DeepCAS: A Deep Reinforcement Learning Algorithm for Control-Aware Scheduling

We consider networked control systems consisting of multiple independent...

Shielded Reinforcement Learning for Hybrid Systems

Safe and optimal controller synthesis for switched-controlled hybrid sys...

Please sign up or login with your details

Forgot password? Click here to reset