Dense Visual Learning for Robot Manipulation PDF Download

Author: Peter Raymond Florence
Publisher:
ISBN:
Category :
Languages : en
Pages : 141

View

Book Description
We would like to have highly useful robots which can richly perceive their world, semantically distinguish its fine details, and physically interact with it sufficiently for useful robotic manipulation. This is hard to achieve with previous methods: prior work has not equipped robots with the scalable ability to understand the dense visual state of their varied environments. The limitations have both been in the state representations used, and how to acquire them without significant human labeling effort. In this thesis we present work that leverages self-supervision, particularly via a mix of geometrical computer vision, deep visual learning, and robotic systems, to scalably produce dense visual inferences of the world state. These methods either enable robots to teach themselves dense visual models without human supervision, or they act as a large multiplying factor on the value of information provided by humans. Specifically, we develop a pipeline for providing ground truth labels of visual data in cluttered and multi-object scenes, we introduce the novel application of dense visual object descriptors to robotic manipulation and provide a fully robot-supervised pipeline to acquire them, and we leverage this dense visual understanding to efficiently learn new manipulation skills through imitation. With real robot hardware we demonstrate contact-rich tasks manipulating household objects, including generalizing across a class of objects, manipulating deformable objects, and manipulating a textureless symmetrical object, all with closed-loop, real-time vision-based manipulation policies.

Author: Lucas Manuelli (Ph. D.)
Publisher:
ISBN:
Category :
Languages : en
Pages : 187

View

Book Description
We would like to have robots which can perform useful manipulation tasks in real-world environments. This requires robots that can perceive the world with both precision and semantic understanding, methods for communicating desired tasks to these systems, and closed loop visual feedback controllers for robustly executing manipulation tasks. This is hard to achieve with previous methods: prior work hasn’t enabled robots to densely understand the visual world with sufficient precision to perform robotic manipulation or endowed them with the semantic understanding needed to perform tasks with novel objects. This limitation arises partly from the object representations that have been used, the challenge in extracting these representations from the available sensor data in real-world settings, and the manner in which tasks have been specified. This thesis presents a family of approaches that leverage self-supervision, both in the visual domain and for learning physical dynamics, to enable robots to perform manipulation tasks. Specifically we (i) develop a pipeline to efficiently annotate visual data in cluttered and multi-object environments (ii) demonstrate the novel application of dense visual object descriptors to robotic manipulation and provide a fully self-supervised robot system to acquire them (iii) introduce the concept of category-level manipulation tasks and develop a novel object representation based on semantic 3D keypoints along with a task specification that uses these keypoints to define the task for all objects of a category, including novel instances, (iv) utilize our dense visual object descriptors to quickly learn new manipulation skills through imitation and (v) use our visual object representations to learn data-driven models that can be used to perform closed loop feedback control in manipulation tasks.

Author: Yen-Chen Lin (Researcher in computer science)
Publisher:
ISBN:
Category :
Languages : en
Pages : 0

View

Book Description
Humans are remarkable at manipulating unfamiliar objects. For the past decades of robotics, tremendous efforts have been dedicated to endow robot manipulation systems with such capabilities. As classic solutions typically require prior knowledge of the objects (e.g., 3D CAD models) which are not available in the unstructured environments, data-driven solutions that learn from robot-environment interactions (e.g., trial and error) have emerged as a promising approach for autonomously acquiring complex skills for manipulation. For data-driven methods, the ability to do more with less data is incredibly important, since data collection through physical interaction between the robots and the environment can be both time consuming and expensive. In this thesis, we develop transfer learning algorithms for robotic manipulation in order to reduce the amount of robot-environment interactions needed to adapt to different environments. With real robot hardware, we show that our algorithms enable robots to learn to pick and grasp arbitrary objects with 10 minutes of trial and error, and help robots learn to push unfamiliar objects with 5 interactions.

Author: Bin Fang
Publisher: Springer Nature
ISBN: 9811551243
Category : Technology & Engineering
Languages : en
Pages : 219

View

Book Description
Over the next few decades, millions of people, with varying backgrounds and levels of technical expertise, will have to effectively interact with robotic technologies on a daily basis. This means it will have to be possible to modify robot behavior without explicitly writing code, but instead via a small number of wearable devices or visual demonstrations. At the same time, robots will need to infer and predict humans’ intentions and internal objectives on the basis of past interactions in order to provide assistance before it is explicitly requested; this is the basis of imitation learning for robotics. This book introduces readers to robotic imitation learning based on human demonstration with wearable devices. It presents an advanced calibration method for wearable sensors and fusion approaches under the Kalman filter framework, as well as a novel wearable device for capturing gestures and other motions. Furthermore it describes the wearable-device-based and vision-based imitation learning method for robotic manipulation, making it a valuable reference guide for graduate students with a basic knowledge of machine learning, and for researchers interested in wearable computing and robotic learning.

Author: Geoffrey Taylor
Publisher: Springer
ISBN: 3540334556
Category : Technology & Engineering
Languages : en
Pages : 231

View

Book Description
This book moves toward the realization of domestic robots by presenting an integrated view of computer vision and robotics, covering fundamental topics including optimal sensor design, visual servo-ing, 3D object modelling and recognition, and multi-cue tracking, emphasizing robustness throughout. Covering theory and implementation, experimental results and comprehensive multimedia support including video clips, VRML data, C++ code and lecture slides, this book is a practical reference for roboticists and a valuable teaching resource.

Author: Alexandros Iosifidis
Publisher: Academic Press
ISBN: 0323885721
Category : Computers
Languages : en
Pages : 638

View

Book Description
Deep Learning for Robot Perception and Cognition introduces a broad range of topics and methods in deep learning for robot perception and cognition together with end-to-end methodologies. The book provides the conceptual and mathematical background needed for approaching a large number of robot perception and cognition tasks from an end-to-end learning point-of-view. The book is suitable for students, university and industry researchers and practitioners in Robotic Vision, Intelligent Control, Mechatronics, Deep Learning, Robotic Perception and Cognition tasks. Presents deep learning principles and methodologies Explains the principles of applying end-to-end learning in robotics applications Presents how to design and train deep learning models Shows how to apply deep learning in robot vision tasks such as object recognition, image classification, video analysis, and more Uses robotic simulation environments for training deep learning models Applies deep learning methods for different tasks ranging from planning and navigation to biosignal analysis

Author: Sulabh Kumra
Publisher:
ISBN:
Category : Computer vision
Languages : en
Pages : 0

View

Book Description
"Multi-step manipulation tasks in unstructured environments are extremely challenging for a robot to learn. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. A model-free deep reinforcement learning method is proposed to learn multi-step manipulation tasks. This work introduces a novel Generative Residual Convolutional Neural Network (GR-ConvNet) model that can generate robust antipodal grasps from n-channel image input at real-time speeds (20ms). The proposed model architecture achieved a state-of-the-art accuracy on three standard grasping datasets. The adaptability of the proposed approach is demonstrated by directly transferring the trained model to a 7 DoF robotic manipulator with a grasp success rate of 95.4% and 93.0% on novel household and adversarial objects, respectively. A novel Robotic Manipulation Network (RoManNet) is introduced, which is a vision-based model architecture, to learn the action-value functions and predict manipulation action candidates. A Task Progress based Gaussian (TPG) reward function is defined to compute the reward based on actions that lead to successful motion primitives and progress towards the overall task goal. To balance the ratio of exploration/exploitation, this research introduces a Loss Adjusted Exploration (LAE) policy that determines actions from the action candidates according to the Boltzmann distribution of loss estimates. The effectiveness of the proposed approach is demonstrated by training RoManNet to learn several challenging multi-step robotic manipulation tasks in both simulation and real-world. Experimental results show that the proposed method outperforms the existing methods and achieves state-of-the-art performance in terms of success rate and action efficiency. The ablation studies show that TPG and LAE are especially beneficial for tasks like multiple block stacking."--Abstract.

Author: Arunkumar Byravan
Publisher:
ISBN:
Category :
Languages : en
Pages : 157

View

Book Description
The emergence of deep learning, access to large amounts of data and powerful computing hardware have led to great strides in the state-of-the-art in robotics, computer vision, and AI. Unlike traditional methods that are strongly model-based with priors and explicit structural constraints, these newer learning approaches tend to be data-driven and often neglect the underlying problem structure. As a consequence, while they usually outperform their traditional counterparts on many problems, achieving good generalisation, interpretability, task transfer and data-efficiency has been challenging. Combining the strengths of the two paradigms, the flexibility of modern learning techniques, and the domain knowledge and structure of traditional methods should help bridge this gap. In this thesis, we will present work that combines these two paradigms, specifically in the context of learning visual dynamics models for robot manipulation tasks. This thesis is divided into two parts. In the first part, we discuss a structured approach to designing visual dynamics models for manipulation tasks. We propose a specific class of deep visual dynamics models (SE3-Nets) that explicitly encode strong physical and 3D geometric priors (specifically, rigid body physics) in their structure. As opposed to deep models that reason about motion a pixel level, SE3-Nets model the dynamics of observed scenes at the object level - they identify objects in the scene and predict rigid body rotation and translation per object. This leads to an interpretable architecture that can robustly model the dynamics of complex interactions. Next, we discuss SE3-Pose-Nets, an extension of SE3-Nets that additionally learns to estimate a latent, globally-consistent pose representation for objects and use the corresponding representation for real-time closed-loop visuomotor control of a Baxter robot. We show that the structure inherent in SE3-Pose-Nets allows them to be robust to visual perturbations and noise, generalizing to settings significantly different than seen during training. We also briefly discuss Dynamics-Nets, a recurrent extension to SE3-Pose-Nets that can be used for the control of dynamical systems. In the second part of the thesis, we present an approach towards solving long-horizon manipulation tasks, using reinforcement learning; we combine the flexibility of modern model-free RL approaches with model-based reasoning. Our approach, Imagined Value Gradients (IVG), learns a predictive model of expected future observations, rewards and values from which a policy can be derived by following the gradient of the estimated value along imagined trajectories. We show how robust policy optimization can be achieved even with approximate models on robot manipulation tasks, learned directly from vision and proprioception. We evaluate the efficacy of our approach in a transfer learning scenario, re-using previously learned models on tasks with different reward structures and visual distractors. The structure inherent in our system, i.e. the model, allows us to transfer knowledge across tasks, achieving significant improvements in learning speed compared to strong model-free baselines. Finally, we conclude with a discussion of the proposed work and future directions.

Author: Tamim Asfour
Publisher: Springer Nature
ISBN: 3030954595
Category : Technology & Engineering
Languages : en
Pages : 1023

View

Book Description
This book contains the papers that were presented at the 17th International Symposium of Robotics Research (ISRR). The ISRR promotes the development and dissemination of groundbreaking research and technological innovation in robotics useful to society by providing a lively, intimate, forward-looking forum for discussion and debate about the current status and future trends of robotics with great emphasis on its potential role to benefit humankind. The symposium contributions contained in this book report on a variety of new robotics research results covering a broad spectrum organized into the categories: design, control; grasping and manipulation, planning, robot vision, and robot learning.

Author: Mario Prats
Publisher: Springer
ISBN: 3642332412
Category : Technology & Engineering
Languages : en
Pages : 187

View

Book Description
Robot manipulation is a great challenge; it encompasses versatility -adaptation to different situations-, autonomy -independent robot operation-, and dependability -for success under modeling or sensing errors. A complete manipulation task involves, first, a suitable grasp or contact configuration, and the subsequent motion required by the task. This monograph presents a unified framework by introducing task-related aspects into the knowledge-based grasp concept, leading to task-oriented grasps. Similarly, grasp-related issues are also considered during the execution of a task, leading to grasp-oriented tasks which is called framework for physical interaction (FPI). The book presents the theoretical framework for the versatile specification of physical interaction tasks, as well as the problem of autonomous planning of these tasks. A further focus is on sensor-based dependable execution combining three different types of sensors: force, vision and tactile. The FPI approach allows to perform a wide range of robot manipulation tasks. All contributions are validated with several experiments using different real robots placed on household environments; for instance, a high-DoF humanoid robot can successfully operate unmodeled mechanisms with widely varying structure in a general way with natural motions. This research was recipient of the European Georges Giralt Award and the Robotdalen Scientific Award Honorary Mention.