Springer Handbook of Robotics - Multimedia Extension

Chapter 15 — Robot Learning

Jan Peters, Daniel D. Lee, Jens Kober, Duy Nguyen-Tuong, J. Andrew Bagnell and Stefan Schaal

Machine learning offers to robotics a framework and set of tools for the design of sophisticated and hard-to-engineer behaviors; conversely, the challenges of robotic problems provide both inspiration, impact, and validation for developments in robot learning. The relationship between disciplines has sufficient promise to be likened to that between physics and mathematics. In this chapter, we attempt to strengthen the links between the two research communities by providing a survey of work in robot learning for learning control and behavior generation in robots. We highlight both key challenges in robot learning as well as notable successes. We discuss how contributions tamed the complexity of the domain and study the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, a particular focus of our chapter lies on model learning for control and robot reinforcement learning. We demonstrate how machine learning approaches may be profitably applied, and we note throughout open questions and the tremendous potential for future research.

Read this chapter on SpringerLink

Inverted helicopter hovering

Author Pieter Abbeel

Video ID : 352

An example of simulation-based optimization using a learned forward model. This brief video shows a successful application of reinforcement learning to the design of a controller for sustained inverted flight of an autonomous helicopter. The authors began by learning a stochastic, nonlinear forward model of the helicopter’s dynamics. Then, a reinforcement learning algorithm was applied to automatically learn a controller for autonomous inverted hovering. The video illustrates Section 15.2.5 -- Applications of Model Learning, Springer Handbook of Robotics, 2nd ed (2016); Reference: A.Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, E. Liang: Autonomous inverted helicopter flight via reinforcement learning, IX Int. Symp. Exp. Robot. 2004, Springer Tract. Adv. Robot. 21, 363-372 (2006)

Inverse reinforcement

Author Pieter Abbeel

Video ID : 353

This video shows a successful example of inverse reinforcement learning for acrobatic helicopter maneuvers. It illustrates apprenticeship learning algorithms, which leverage expert demonstrations to efficiently learn good controllers for tasks as demonstrated by an expert. The experimental results captured here include the first autonomous execution of a wide range of maneuvers and a complete airshow. The controllers perform as well as, and often even better than, the human expert pilot. The video illustrates a solution to the "Curse of Goal Specification" in Sect 15.3.6 Challenges in Robot Reinforcement Learning. Reference: P. Abbeel, A. Coates, A.Y. Ng: Autonomous helicopter aerobatics through apprenticeship learning, Int. J. Robot. Res. 29(13), 1608–1639 (2010)

Machine learning table tennis

Author Jan Peters, Katharina Mülling, Jens Kober, Oliver Kroemer, Zhikun Wang

Video ID : 354

The video shows recent successful demonstrations of using machine learning for robot table tennis. The first part shows learning of motor primitives for forehand strikes by training a robot with a mixture of imitation and reinforcement learning. The second part shows how the robot can anticipate an opponent's intended targets based on both forehand and backhand primitives. The video illustrates Sect. 15.3.5 Policy Search of the Springer Handbook of Robotics, 2nd edn (2016). Reference: K. Mülling, J. Kober, O. Kroemer, J. Peters: Learning to select and generalize striking movements in robot table tennis, Int. J. Robot. Res. 32(3), 263-279 (2013)

Learning motor primitives

Author Jens Kober, Jan Peters

Video ID : 355

The video shows recent success in robot learning for two basic motor tasks, namely, ball-in-a-cup and ball paddling. The video illustrates Section 15.3.5 -- Policy Search, of the Springer Handbook of Robotics, 2nd edn (2016). Reference: J. Kober, J. Peters: Imitation and reinforcement learning - Practical algorithms for motor primitive learning in robotics, IEEE Robot. Autom. Mag. 17(2), 55-62 (2010)

View Chapter

Chapter 15 — Robot Learning

Inverted helicopter hovering

Inverse reinforcement

Machine learning table tennis

Learning motor primitives