On Entropic Optimization and Path Integral Control

by   Tom Lefebvre, et al.
Ghent University

This article is motivated by the question whether it is possible to solve optimal control (OC) or dynamic optimization problems in a similar fashion to how static optimization problems can be addressed with Evolutionary Strategies (ES). The latter maintain a sequence of Gaussian search distributions that converge to the optimum. For the moment, this question has been answered partially by a set of algorithms that are known as Path Integral Control (PIC). Those maintain a sequence of locally linear Gaussian feedback controllers. So far PIC methods have been derived solely from the theory of Linearly Solvable OC, which includes only a narrow subset of optimal control problems and has only limited application potential as a consequence. We aim to address this question within a more general mathematical setting. Therefore, we first identify the framework of entropic inference as a suitable setting to synthesise stochastic search algorithms. Therewith we establish the formal framework of entropic optimization and provide a compelling justification for the inclusion of entropy measures in stochastic optimization. From this theory follows a formal optimal search distribution sequence which converges monotonically to the Dirac delta distribution centred at the optimum. Then we demonstrate how this result can be used to derive Gaussian search distributions similar to existing ES. We then proceed to transfer these ideas from a static to a dynamic setting, therewith establishing the framework of Entropic OC which shares characteristics with entropy based Reinforcement Learning. From this theory we can construct a number of formal optimal path distribution sequences. Thence we derive the outlines of a generalised algorithmic framework complementing the existing PIC class. Our main ambition is to reveal how all of these fields are related in a most exciting fashion.


page 1

page 2

page 3

page 4


Convex programming in optimal control and information theory

The main theme of this thesis is the development of computational method...

Optimal Control as Variational Inference

In this article we address the stochastic and risk sensitive optimal con...

Gradient Flows for Regularized Stochastic Control Problems

This work is motivated by a desire to extend the theoretical underpinnin...

Stochastic Optimization of Linear Dynamic Systems with Parametric Uncertainties

This paper describes a new approach to solving some stochastic optimizat...

Non-local Optimization: Imposing Structure on Optimization Problems by Relaxation

In stochastic optimization, particularly in evolutionary computation and...

On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator

The convergence of policy gradient algorithms in reinforcement learning ...

Please sign up or login with your details

Forgot password? Click here to reset