关键词: 离线强化学习, 分布偏移, 引导性探索

Abstract: Offline Reinforcement Learning aims to enable agents to leverage historical data for policy learning without online interaction, thereby reducing costs and avoiding risks in real-world scenarios. However, due to the lack of environmental feedback, policies learned from offline datasets encounter the issue of data distribution shift. Most existing methods are based on conservatism, which alleviates the distribution shift problem to some extent by confining policy learning within the distribution of the offline dataset, yet this approach overly restricts the agent's exploration capabilities. To address the above issue, this paper proposes a method based on guided exploration. This method utilizes a guided state network to generate neighboring states with high potential value, guiding the agent to explore states outside the distribution. Meanwhile, a behavior cloning term is introduced to dynamically adjust the gap between the behavioral strategy and the learning strategy, ensuring the stability of the learning process. Experimental results on the D4RL benchmark dataset demonstrate that this algorithm outperforms existing mainstream algorithms on multiple tasks. 

Key words: Offline reinforcement learning, Distribution shift, Guided exploration