Uncovering Instabilities in Variational-Quantum Deep Q-Networks

by   Maja Franz, et al.

Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantumclassical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproduciblity of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.


page 8

page 10


Reinforcement-Learning-Based Variational Quantum Circuits Optimization for Combinatorial Problems

Quantum computing exploits basic quantum phenomena such as state superpo...

Variational Quantum Reinforcement Learning via Evolutionary Optimization

Recent advance in classical reinforcement learning (RL) and quantum comp...

Quantum Deep Reinforcement Learning for Robot Navigation Tasks

In this work, we utilize Quantum Deep Reinforcement Learning as method t...

Efficient and practical quantum compiler towards multi-qubit systems with deep reinforcement learning

Efficient quantum compiling tactics greatly enhance the capability of qu...

Batch Quantum Reinforcement Learning

Training DRL agents is often a time-consuming process as a large number ...

Quantum Policy Iteration via Amplitude Estimation and Grover Search – Towards Quantum Advantage for Reinforcement Learning

We present a full implementation and simulation of a novel quantum reinf...

Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learning

With the development of experimental quantum technology, quantum control...

Please sign up or login with your details

Forgot password? Click here to reset