Efficient Reinforcement Learning Through Trajectory Generation

by   Wenqi Cui, et al.

A key barrier to using reinforcement learning (RL) in many real-world applications is the requirement of a large number of system interactions to learn a good control policy. Off-policy and Offline RL methods have been proposed to reduce the number of interactions with the physical environment by learning control policies from historical data. However, their performances suffer from the lack of exploration and the distributional shifts in trajectories once controllers are updated. Moreover, most RL methods require that all states are directly observed, which is difficult to be attained in many settings. To overcome these challenges, we propose a trajectory generation algorithm, which adaptively generates new trajectories as if the system is being operated and explored under the updated control policies. Motivated by the fundamental lemma for linear systems, assuming sufficient excitation, we generate trajectories from linear combinations of historical trajectories. For linear feedback control, we prove that the algorithm generates trajectories with the exact distribution as if they are sampled from the real system using the updated control policy. In particular, the algorithm extends to systems where the states are not directly observed. Experiments show that the proposed method significantly reduces the number of sampled data needed for RL algorithms.


page 1

page 2

page 3

page 4


Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control

Reinforcement learning (RL) is a promising approach for deriving control...

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

Among the reasons hindering reinforcement learning (RL) applications to ...

Model-based Trajectory Stitching for Improved Offline Reinforcement Learning

In many real-world applications, collecting large and high-quality datas...

Smooth and Efficient Policy Exploration for Robot Trajectory Learning

Many policy search algorithms have been proposed for robot learning and ...

Symmetry Detection in Trajectory Data for More Meaningful Reinforcement Learning Representations

Knowledge of the symmetries of reinforcement learning (RL) systems can b...

Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures

Model-free or learning-based control, in particular, reinforcement learn...

Residual Reinforcement Learning for Robot Control

Conventional feedback control methods can solve various types of robot c...

Please sign up or login with your details

Forgot password? Click here to reset