Goal Density-based Hindsight Experience Prioritization for Multi-Goal Robot Manipulation Reinforcement Learning

Yingyi Kuang, Abraham Itzhak Weinberg, George Vogiatzis, Diego R. Faria

Research output: Chapter in Book/Report/Conference proceedingConference publication

Abstract

Reinforcement learning for multi-goal robot manipulation tasks is usually challenging, especially when sparse rewards are provided. It often requires millions of data collected before a stable strategy is learned. Recent algorithms like Hindsight Experience Replay (HER) have accelerated the learning process greatly by replacing the original desired goal with one of the achieved points (substitute goals) alongside the same trajectory. However, the selection of previous experience to learn is naively sampled in HER, in which the trajectory selection and the substitute goal sampling is completely random. In this paper, we discuss an experience prioritization strategy for HER that improves the learning efficiency. We propose the Goal Density-based hindsight experience Prioritization (GDP) method that focuses on utilizing the density distribution of the achieved points and prioritizes achieved points which are rarely seen in the replay buffer. These points are used as substitute goals for HER. In addition, we propose an Prioritization Switching with Ensembling Strategy (PSES) method to switch different experience prioritization algorithms during learning, which allows to select the best performance during each learning stage. We evaluate our method with several OpenAI Gym robotic manipulation tasks. The results show that GDP accelerates the learning process in most tasks and can be improved when combining with other prioritization methods using PSES.

Original languageEnglish
Title of host publication29th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2020
PublisherIEEE
Pages432-437
Number of pages6
ISBN (Electronic)9781728160757
ISBN (Print)978-1-7281-6076-4
DOIs
Publication statusPublished - 14 Oct 2020
Event29th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2020 - Virtual, Naples, Italy
Duration: 31 Aug 20204 Sep 2020

Publication series

Name29th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2020
PublisherIEEE
ISSN (Print)1944-9445
ISSN (Electronic)1944-9437

Conference

Conference29th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2020
CountryItaly
CityVirtual, Naples
Period31/08/204/09/20

Bibliographical note

Funding Information:
ACKNOWLEDGMENT This work is supported by EPSRC-UK InDex project (EU CHIST-ERA programme), with reference EP/S032355/1.

Publisher Copyright:
© 2020 IEEE.

Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.

Fingerprint Dive into the research topics of 'Goal Density-based Hindsight Experience Prioritization for Multi-Goal Robot Manipulation Reinforcement Learning'. Together they form a unique fingerprint.

Cite this