Incremental and Transfer Learning of Contextual Skill Model for Robots

dc.contributorAalto Universityen
dc.contributor.authorHazara, Murtaza
dc.contributor.departmentSähkötekniikan ja automaation laitosfi
dc.contributor.departmentDepartment of Electrical Engineering and Automationen
dc.contributor.labIntelligent Roboticsen
dc.contributor.schoolSähkötekniikan korkeakoulufi
dc.contributor.schoolSchool of Electrical Engineeringen
dc.contributor.supervisorKyrki, Ville, Prof., Aalto University, Department of Electrical Engineering and Automation, Finland
dc.description.abstractThe thesis studies building blocks for robot skill learning. Using these key components, learning frameworks can be constructed which provide robots with the capability to acquire a motion and manipulation skill autonomously. We study skill learning in two contexts: in-contact and free-space motions. In brief, this thesis investigates how to: (1) learn a policy for in-contact tasks; (2) generalize a free-space motion policy to new situations using a contextual skill model (CSM); and (3) transfer the CSM from simulation to real world. Learning an in-contact task such as wood planing from scratch can be time-consuming and dangerous. This problem can be avoided by imitating a policy from a human demonstration. However, a mere imitation may not satisfy the objective of the corresponding in-contact task. The thesis proposes a reinforcement learning (RL) framework for improving the performance of an imitated in-contact policy. The policy search for in-contact tasks has been achieved by making the motion compliant which allows for exploration in the force profile. Generalizing a policy to new situations is fundamental to skill learning as it alleviates the need to learn a new policy in every novel situation. Generalizing a policy refers to synthesizing a function mapping the policy to new situations. The function is referred to as a contextual policy or contextual skill model (CSM). The thesis proposes a parametric CSM. Experiments demonstrated that the parametric CSM can extract a global pattern from a database (DB) of policy parameters leading to significantly better extrapolation capability than with non-parametric CSMs. Furthermore, the underlying model of the CSM is fitted to the DB using a novel model selection approach to better represent the underlying regularities of the task. In order to speed the process of learning, the prediction uncertainty of the CSM is calculated using empirical Bayes (EB) and employed for guiding the exploration process of a model-free policy search. In addition, the most promising task is selected using a novel task manager, allowing for better future generalization performance achieved with minimum effort. In essence, the thesis presents an incremental learning framework,the main components of which are as follows: CSM, policy search, model selection, DB, EB, and a task manager implemented using active learning. Learning a policy in a simulated environment and transferring it to the real world will alleviate the need to learn from scratch or from a demonstration. The thesis proposes to transfer a CSM instead of transferring a single control policy. We developed a simulation-to-real transfer framework which learns a source CSM in simulation incrementally and transfers it to the real world incrementally. Transference of the source CSM has been achieved using sample policies from the target environment. Experiments indicated that one sample policy is sufficient to transfer a CSM to the target environment. The target CSM improved the extrapolation capability significantly better than zero-shot transfer.en
dc.format.extent200 + app. 70
dc.identifier.isbn978-952-60-8791-7 (electronic)
dc.identifier.isbn978-952-60-8790-0 (printed)
dc.identifier.issn1799-4942 (electronic)
dc.identifier.issn1799-4934 (printed)
dc.identifier.issn1799-4934 (ISSN-L)
dc.opnAsfour, Tamim, Prof., Karlsruhe Institute of Technology, Germany
dc.publisherAalto Universityen
dc.relation.haspart[Publication 1]: Murtaza Hazara, Ville Kyrki. Reinforcement learning for improving imitated in-contact skills. In International Conference on Humanoid Robots (Humanoids), Cancun,Mexico. pp. 194-201, 11, 2016. Full text in Acris/Aaltodoc: DOI: 10.1109/HUMANOIDS.2016.7803277
dc.relation.haspart[Publication 2]: Jens Lundell, Murtaza Hazara, Ville Kyrki. Generalizing Movement Primitives to New Situations. In Towards Autonomous Robotic Systems (TAROS), pp. 16-31, 7, 2017. Full text in Acris/Aaltodoc: DOI: 10.1007/978-3-319-64107-2_2
dc.relation.haspart[Publication 3]: Murtaza Hazara, Ville Kyrki. Model selection for incremental learning of generalizable movement primitives. In 18th International Conference on Advanced Robotics (ICAR), pp. 359-366, 7, 2017. Full text in Acris/Aaltodoc: DOI: 10.1109/ICAR.2017.8023633
dc.relation.haspart[Publication 4]: Murtaza Hazara, Ville Kyrki. Speeding Up Incremental Learning Using Data Efficient Guided Exploration. In International Conference on Robotics and Automation (ICRA), pp. 1-8, 5, 2018. Full text in Acris/Aaltodoc: DOI: 10.1109/ICRA.2018.8461241
dc.relation.haspart[Publication 5]: Murtaza Hazara, Ville Kyrki. Transferring Generalizable Motor Primitives From Simulation to Real World. Robotics and Automation Letters, pp. 2172-2179, 4, 2019. Full text in Acris/Aaltodoc: DOI: 10.1109/LRA.2019.2900768
dc.relation.haspart[Publication 6]: Murtaza Hazara, Xiaopu Li, Ville Kyrki. Active Incremental Learning of a Contextual Skill Model. Submitted to International Conference on Intelligent Robots and Systems (IROS), 2019 IEEE.
dc.relation.ispartofseriesAalto University publication series DOCTORAL DISSERTATIONSen
dc.revKroemer, Oliver B., Prof., Carnegie Mellon University, USA
dc.revTaylor, Matthew, E., Dr., Borealis AI, USA
dc.subject.keywordreinforcement learningen
dc.subject.keywordactive incremental learningen
dc.subject.keywordtransfer learningen
dc.subject.otherElectrical engineeringen
dc.titleIncremental and Transfer Learning of Contextual Skill Model for Robotsen
dc.typeG5 Artikkeliväitöskirjafi
dc.type.ontasotDoctoral dissertation (article-based)en
dc.type.ontasotVäitöskirja (artikkeli)fi
local.aalto.acrisexportstatuschecked 2019-12-14_1024
Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
32.17 MB
Adobe Portable Document Format