We investigated whether deep reinforcement learning (deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies. We used deep RL to train a humanoid robot to play a simplified one-versus-one soccer game. The resulting agent exhibits robust and dynamic movement skills, such as rapid fall recovery, walking, turning, and kicking, and it transitions between them in a smooth and efficient manner. It also learned to anticipate ball movements and block opponent shots. The agent’s tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. Our agent was trained in simulation and transferred to real robots zero-shot. A combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training enabled good-quality transfer. In experiments, the agent walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline.

中文翻译：

---

通过深度强化学习为双足机器人学习敏捷足球技能


我们研究了深度强化学习（深度 RL）是否能够为低成本、微型人形机器人合成复杂且安全的运动技能，并将其组合成复杂的行为策略。我们使用深度强化学习来训练一个人形机器人来玩简化的一对一足球游戏。由此产生的智能体表现出强大而动态的运动技能，例如快速跌倒恢复、行走、转身和踢腿，并以平稳有效的方式在它们之间过渡。它还学会了预测球的运动并阻止对手的投篮。代理的战术行为以手动设计不切实际的方式适应特定的游戏环境。我们的代理接受了模拟培训，并零次转移到真实的机器人上。足够高频的控制、有针对性的动态随机化和训练过程中的扰动相结合，可以实现高质量的传输。在实验中，与脚本基线相比，智能体行走速度快了 181%，转身速度快了 302%，起身时间缩短了 63%，踢球速度快了 34%。