Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

by   Tuomas Haarnoja, et al.

We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. We first trained individual skills in isolation and then composed those skills end-to-end in a self-play setting. The resulting policy exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and transitions between them in a smooth, stable, and efficient manner - well beyond what is intuitively expected from the robot. The agents also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. The full range of behaviors emerged from a small set of simple rewards. Our agents were trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer, despite significant unmodeled effects and variations across robot instances. Although the robots are inherently fragile, minor hardware modifications together with basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way. Indeed, even though the agents were optimized for scoring, in experiments they walked 156 less time to get up, and kicked 24 efficiently combining the skills to achieve the longer term objectives. Examples of the emergent behaviors and full 1v1 matches are available on the supplementary website.


page 3

page 8

page 12

page 14

page 15

page 17

page 18

page 23


Training an Interactive Humanoid Robot Using Multimodal Deep Reinforcement Learning

Training robots to perceive, act and communicate using multiple modaliti...

Learning agile and dynamic motor skills for legged robots

Legged robots pose one of the greatest challenges in robotics. Dynamic a...

Learning Agile Locomotion Skills with a Mentor

Developing agile behaviors for legged robots remains a challenging probl...

Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

We investigate the use of prior knowledge of human and animal movement t...

Learning Whole-body Motor Skills for Humanoids

This paper presents a hierarchical framework for Deep Reinforcement Lear...

Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs

Quadrupedal robots are skillful at locomotion tasks while lacking manipu...

Design strategies for controlling neuron-connected robots using reinforcement learning

Despite the growing interest in robot control utilizing the computation ...

Please sign up or login with your details

Forgot password? Click here to reset