Model-Free Adaptive Optimal Control of Sequential Manufacturing Processes using Reinforcement Learning

by   Johannes Dornheim, et al.

A self-learning optimal control algorithm for sequential manufacturing processes with time-discrete control actions is proposed and evaluated with simulated deep drawing processes. The necessary control model is built during consecutive process executions under optimal control via Reinforcement Learning, using the measured product quality as reward after each process execution. Prior model formation, which is required by state-of-the-art algorithms like Model Predictive Control and Approximate Dynamic Programming, is therefore obsolete. This avoids the difficulties in system identification and accurate modelling, which arise with processes subject to non-linear dynamics and stochastic influences. Also runtime complexity problems of these approaches are avoided, which arise when more complex models and larger control prediction horizons are employed. Instead of using pre-created process- and observation-models, Reinforcement Learning algorithms build functions of expected future reward during processing, which are then used for optimal process control decisions. The learning of such expectation functions is realized online by interacting with the process. The proposed algorithm also takes stochastic variations of the process conditions into consideration and is able to cope with partial observability. A method for the adaptive optimal control of partially observable fixed-horizon manufacturing processes, based on Q-learning is developed and studied. The resulting algorithm is instantiated and then evaluated by application to a time-stochastic optimal control problem in metal sheet deep drawing, where the experiments use FEM-simulated processes. The Reinforcement Learning based control shows superior results over the model-based Model Predictive Control and Approximate Dynamic Programming approaches.


page 1

page 2

page 3

page 4


Multiobjective Reinforcement Learning for Reconfigurable Adaptive Optimal Control of Manufacturing Processes

In industrial applications of adaptive optimal control often multiple co...

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

In this paper we provide framework to cope with two problems: (i) the fr...

SMPL: Simulated Industrial Manufacturing and Process Control Learning Environments

Traditional biological and pharmaceutical manufacturing plants are contr...

Using Approximate Models in Robot Learning

Trajectory following is one of the complicated control problems when its...

Probabilistic inverse optimal control with local linearization for non-linear partially observable systems

Inverse optimal control methods can be used to characterize behavior in ...

Stochastic optimal well control in subsurface reservoirs using reinforcement learning

We present a case study of model-free reinforcement learning (RL) framew...

Average Cost Optimal Control of Stochastic Systems Using Reinforcement Learning

This paper addresses the average cost minimization problem for discrete-...

Please sign up or login with your details

Forgot password? Click here to reset