Reinforcement learning method via meta-learning the exploring latent variable

The traditional work to improve the exploration ability is mostly realized by adding random noise to the agents’ action space, which is lack of "structured" information reflecting the characteristics of the task, so the exploration effect and the utilization of interactive data are low; or adding noise to the policy network parameter space will bring huge parameter overhead.

A learnable exploration latent variable was introduced into the basic model-free RL algorithm, so that the agent not only relies on task independent preset jitter exploration strategies or probability matching strategies when exploring in the environment, but also utilized task related exploration latent variables for structured exploration. Exploring latent variables was defined as sampling from a learnable Gaussian distribution, where the mean value and variance value of Gaussian distribution were learnable parameters. In the current online learning of single task, there were mainly three types of network parameter to update: Actor Critic learning in PPO, environment model learning, and meta learning for exploring latent variables (mean and variance). The overall meta-learning can be expressed as a bi-level optimization: outer layer learning refers to the learning of exploration latent variables based on environmental models, while inner layer learning refers to the PPO learning guided by exploration latent variables and the learning of environmental models. Base on this, agents can explore in the real environment based on the sample from the latent distribution, and thus this method can help improve the online reinforcement learning performance without introducing additional steps to interact with the environment.

The experiments are conducted on MuJoCo (V2 version) tasks in the OpenAI Gym among 5 random seeds. According to the original experimental setup of PPO, the number of steps the agent interacts with the environment in each task is 1000000, and the agent's current performance is evaluated 10 times every 2048 steps and averaged. In our method, an exploration latent variable value is sampled from the current latent variable distribution for the agent to run during each evaluation. Our method usually outperforms the baseline method and the general model-based method in terms of learning speed and asymptotic performance, and the direct model-based method may even cause performance degradation; in addition, our methods typically have small variances. Overall, the experiment results show that our method can achieve an average performance improvement of about 30% in continuous control applications based on the basic on-policy PPO reinforcement learning algorithm.

This work starts from the perspective of implicit variable meta learning and focuses on improving the exploratory ability of reinforcement learning agents in continuous control applications. The agent can learn based on a learnable environment model to explore the distribution of latent variables, achieving efficient structured exploration of the environment and improving the agent's learning performance without the need for additional interaction steps with the environment. The novel meta reinforcement learning design proposed in this paper can provide performance benefits for learning online tasks.