An Empirical Study of the Effectiveness of Using a Replay Buffer on Mode Discovery in GFlowNets

by   Nikhil Vemgal, et al.

Reinforcement Learning (RL) algorithms aim to learn an optimal policy by iteratively sampling actions to learn how to maximize the total expected return, R(x). GFlowNets are a special class of algorithms designed to generate diverse candidates, x, from a discrete set, by learning a policy that approximates the proportional sampling of R(x). GFlowNets exhibit improved mode discovery compared to conventional RL algorithms, which is very useful for applications such as drug discovery and combinatorial search. However, since GFlowNets are a relatively recent class of algorithms, many techniques which are useful in RL have not yet been associated with them. In this paper, we study the utilization of a replay buffer for GFlowNets. We explore empirically various replay buffer sampling techniques and assess the impact on the speed of mode discovery and the quality of the modes discovered. Our experimental results in the Hypergrid toy domain and a molecule synthesis environment demonstrate significant improvements in mode discovery when training with a replay buffer, compared to training only with trajectories generated on-policy.


The Role of Diverse Replay for Generalisation in Reinforcement Learning

In reinforcement learning (RL), key components of many algorithms are th...

Large Batch Experience Replay

Several algorithms have been proposed to sample non-uniformly the replay...

Rethinking Population-assisted Off-policy Reinforcement Learning

While off-policy reinforcement learning (RL) algorithms are sample effic...

Event Tables for Efficient Experience Replay

Experience replay (ER) is a crucial component of many deep reinforcement...

Replay For Safety

Experience replay <cit.> is a widely used technique to achieve efficient...

Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations

Reinforcement Learning (RL) has achieved significant success in applicat...

Nonmyopic Multiclass Active Search for Diverse Discovery

Active search is a setting in adaptive experimental design where we aim ...

Please sign up or login with your details

Forgot password? Click here to reset