Hand-Object Interaction: From Human Demonstrations to Robot Manipulation

Alessandro Carfì; Timothy Patten; Yingyi Kuang; Ali Hammoud; Mohamad Alameh; Elisa Maiettini; Abraham Itzhak Weinberg; Diego Faria; Fulvio Mastrogiovanni; Guillem Alenyà; Lorenzo Natale; Véronique Perdereau; Markus Vincze; Aude Billard

doi:10.3389/frobt.2021.714023

Hand-Object Interaction: From Human Demonstrations to Robot Manipulation

Alessandro Carfì^*, Timothy Patten, Yingyi Kuang, Ali Hammoud, Mohamad Alameh, Elisa Maiettini, Abraham Itzhak Weinberg, Diego Faria, Fulvio Mastrogiovanni, Guillem Alenyà, Lorenzo Natale, Véronique Perdereau, Markus Vincze, Aude Billard

^*Corresponding author for this work

College of Engineering and Physical Sciences

Research output: Contribution to journal › Article › peer-review

Abstract

Human-object interaction is of great relevance for robots to operate in human environments. However, state-of-the-art robotic hands are far from replicating humans skills. It is, therefore, essential to study how humans use their hands to develop similar robotic capabilities. This article presents a deep dive into hand-object interaction and human demonstrations, highlighting the main challenges in this research area and suggesting desirable future developments. To this extent, the article presents a general definition of the hand-object interaction problem together with a concise review for each of the main subproblems involved, namely: sensing, perception, and learning. Furthermore, the article discusses the interplay between these subproblems and describes how their interaction in learning from demonstration contributes to the success of robot manipulation. In this way, the article provides a broad overview of the interdisciplinary approaches necessary for a robotic system to learn new manipulation skills by observing human behavior in the real world.

Original language	English
Article number	714023
Journal	Frontiers in Robotics and AI
Volume	8
DOIs	https://doi.org/10.3389/frobt.2021.714023
Publication status	Published - 1 Oct 2021

Bibliographical note

Copyright © 2021 Carfì, Patten, Kuang, Hammoud, Alameh, Maiettini, Weinberg, Faria, Mastrogiovanni, Alenyà, Natale, Perdereau, Vincze and Billard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Funding: This work is supported by the CHIST-ERA (2014-2020) project InDex and received funding from the Italian Ministry of Education and Research (MIUR), Austrian Science Fund (FWF) under grant agreement No. I3969-N30, Engineering and Physical Sciences Research Council (EPSRC-UK) with reference EP/S032355/1 and Agence Nationale de la Recherche (ANR) under grant agreement No. ANR-18-CHR3-0004. This work is also supported by the ERA-net CHIST-ERA project BURG (PCIN2019-103447).

Keywords

Robotics and AI
hand-object interaction
learning from demonstration
imitation learning
transfer learning
grasping
manipulation
anthropomorphic hands
data extraction

Access to Document

10.3389/frobt.2021.714023Licence: CC BY 3.0

frobt-08-714023.pdf
Copyright © 2021 Carfì, Patten, Kuang, Hammoud, Alameh, Maiettini, Weinberg, Faria, Mastrogiovanni, Alenyà, Natale, Perdereau, Vincze and Billard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Final published version, 801 KBLicence: CC BY 3.0

Cite this

@article{93dd564418fd4b29a7afe981061a8e45,

title = "Hand-Object Interaction: From Human Demonstrations to Robot Manipulation",

abstract = "Human-object interaction is of great relevance for robots to operate in human environments. However, state-of-the-art robotic hands are far from replicating humans skills. It is, therefore, essential to study how humans use their hands to develop similar robotic capabilities. This article presents a deep dive into hand-object interaction and human demonstrations, highlighting the main challenges in this research area and suggesting desirable future developments. To this extent, the article presents a general definition of the hand-object interaction problem together with a concise review for each of the main subproblems involved, namely: sensing, perception, and learning. Furthermore, the article discusses the interplay between these subproblems and describes how their interaction in learning from demonstration contributes to the success of robot manipulation. In this way, the article provides a broad overview of the interdisciplinary approaches necessary for a robotic system to learn new manipulation skills by observing human behavior in the real world.",

keywords = "Robotics and AI, hand-object interaction, learning from demonstration, imitation learning, transfer learning, grasping, manipulation, anthropomorphic hands, data extraction",

author = "Alessandro Carf{\`i} and Timothy Patten and Yingyi Kuang and Ali Hammoud and Mohamad Alameh and Elisa Maiettini and Weinberg, {Abraham Itzhak} and Diego Faria and Fulvio Mastrogiovanni and Guillem Aleny{\`a} and Lorenzo Natale and V{\'e}ronique Perdereau and Markus Vincze and Aude Billard",

note = "Copyright {\textcopyright} 2021 Carf{\`i}, Patten, Kuang, Hammoud, Alameh, Maiettini, Weinberg, Faria, Mastrogiovanni, Aleny{\`a}, Natale, Perdereau, Vincze and Billard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. Funding: This work is supported by the CHIST-ERA (2014-2020) project InDex and received funding from the Italian Ministry of Education and Research (MIUR), Austrian Science Fund (FWF) under grant agreement No. I3969-N30, Engineering and Physical Sciences Research Council (EPSRC-UK) with reference EP/S032355/1 and Agence Nationale de la Recherche (ANR) under grant agreement No. ANR-18-CHR3-0004. This work is also supported by the ERA-net CHIST-ERA project BURG (PCIN2019-103447).",

year = "2021",

month = oct,

day = "1",

doi = "10.3389/frobt.2021.714023",

language = "English",

volume = "8",

journal = "Frontiers in Robotics and AI",

issn = "2296-9144",

publisher = "Frontiers Media S.A.",

}

TY - JOUR

T1 - Hand-Object Interaction: From Human Demonstrations to Robot Manipulation

AU - Carfì, Alessandro

AU - Patten, Timothy

AU - Kuang, Yingyi

AU - Hammoud, Ali

AU - Alameh, Mohamad

AU - Maiettini, Elisa

AU - Weinberg, Abraham Itzhak

AU - Faria, Diego

AU - Mastrogiovanni, Fulvio

AU - Alenyà, Guillem

AU - Natale, Lorenzo

AU - Perdereau, Véronique

AU - Vincze, Markus

AU - Billard, Aude

N1 - Copyright © 2021 Carfì, Patten, Kuang, Hammoud, Alameh, Maiettini, Weinberg, Faria, Mastrogiovanni, Alenyà, Natale, Perdereau, Vincze and Billard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. Funding: This work is supported by the CHIST-ERA (2014-2020) project InDex and received funding from the Italian Ministry of Education and Research (MIUR), Austrian Science Fund (FWF) under grant agreement No. I3969-N30, Engineering and Physical Sciences Research Council (EPSRC-UK) with reference EP/S032355/1 and Agence Nationale de la Recherche (ANR) under grant agreement No. ANR-18-CHR3-0004. This work is also supported by the ERA-net CHIST-ERA project BURG (PCIN2019-103447).

PY - 2021/10/1

Y1 - 2021/10/1

N2 - Human-object interaction is of great relevance for robots to operate in human environments. However, state-of-the-art robotic hands are far from replicating humans skills. It is, therefore, essential to study how humans use their hands to develop similar robotic capabilities. This article presents a deep dive into hand-object interaction and human demonstrations, highlighting the main challenges in this research area and suggesting desirable future developments. To this extent, the article presents a general definition of the hand-object interaction problem together with a concise review for each of the main subproblems involved, namely: sensing, perception, and learning. Furthermore, the article discusses the interplay between these subproblems and describes how their interaction in learning from demonstration contributes to the success of robot manipulation. In this way, the article provides a broad overview of the interdisciplinary approaches necessary for a robotic system to learn new manipulation skills by observing human behavior in the real world.

AB - Human-object interaction is of great relevance for robots to operate in human environments. However, state-of-the-art robotic hands are far from replicating humans skills. It is, therefore, essential to study how humans use their hands to develop similar robotic capabilities. This article presents a deep dive into hand-object interaction and human demonstrations, highlighting the main challenges in this research area and suggesting desirable future developments. To this extent, the article presents a general definition of the hand-object interaction problem together with a concise review for each of the main subproblems involved, namely: sensing, perception, and learning. Furthermore, the article discusses the interplay between these subproblems and describes how their interaction in learning from demonstration contributes to the success of robot manipulation. In this way, the article provides a broad overview of the interdisciplinary approaches necessary for a robotic system to learn new manipulation skills by observing human behavior in the real world.

KW - Robotics and AI

KW - hand-object interaction

KW - learning from demonstration

KW - imitation learning

KW - transfer learning

KW - grasping

KW - manipulation

KW - anthropomorphic hands

KW - data extraction

UR - https://www.frontiersin.org/articles/10.3389/frobt.2021.714023/full

U2 - 10.3389/frobt.2021.714023

DO - 10.3389/frobt.2021.714023

M3 - Article

SN - 2296-9144

VL - 8

JO - Frontiers in Robotics and AI

JF - Frontiers in Robotics and AI

M1 - 714023

ER -

Hand-Object Interaction: From Human Demonstrations to Robot Manipulation

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this