Regularized covariance estimation for weighted maximum likelihood policy search methods

Abdolmaleki, A. and Lau, N. and Reis, L. P. and Neumann, G. (2015) Regularized covariance estimation for weighted maximum likelihood policy search methods. In: Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on, 3 - 5 November 2015, Korea Institute of Science and Technology (KIST), Seoul, Korea (South).

Documents
Abdolmaleki_humanoids2015.pdf
[img]
[Download]
[img]
Preview
PDF
Abdolmaleki_humanoids2015.pdf - Whole Document

584kB
Item Type:Conference or Workshop contribution (Paper)
Item Status:Live Archive

Abstract

Many episode-based (or direct) policy search algorithms, maintain a multivariate Gaussian distribution as search distribution over the parameter space of some objective function. One class of algorithms, such as episodic REPS, PoWER or PI2 uses, a weighted maximum likelihood estimate (WMLE) to update the mean and covariance matrix of this distribution in each iteration. However, due to high dimensionality of covariance matrices and limited number of samples, the WMLE is an unreliable estimator. The use of WMLE leads to over-fitted covariance estimates, and, hence the variance/entropy of the search distribution decreases too quickly, which may cause premature convergence. In order to alleviate this problem, the estimated covariance matrix can be regularized in different ways, for example by using a convex combination of the diagonal covariance estimate and the sample covariance estimate. In this paper, we propose a new covariance matrix regularization technique for policy search methods that uses the convex combination of the sample covariance matrix and the old covariance matrix used in last iteration. The combination weighting is determined by specifying the desired entropy of the new search distribution. With this mechanism, the entropy of the search distribution can be gradually decreased without damage from the maximum likelihood estimate.

Keywords:Stochasic Search, Policy Search, Robotics
Subjects:G Mathematical and Computer Sciences > G760 Machine Learning
H Engineering > H671 Robotics
Divisions:College of Science > School of Computer Science
Related URLs:
ID Code:25748
Deposited On:02 Feb 2017 16:41

Repository Staff Only: item control page