Robust Reinforcement Learning Under Dimension-Wise State Information Drop

Abstract

Recent advancements in offline reinforcement learning (RL) have showcased the potential for leveraging static datasets to train optimal policies. However, real-world applications often face challenges due to missing or incomplete state information caused by imperfect sensor performance or intentional interlaces. We propose the Dimension-Wise Drop Decision Transformer (D3T), a novel framework designed to address dimension-wise data loss in sensor observations, enhancing the robustness of RL algorithms in real-world scenarios. D3T innovatively incorporates dimension-wise drop information embeddings within the Transformer architecture, facilitating effective decision-making even with incomplete observations. Our evaluation in the D4RL MuJoCo domain demonstrates that D3T significantly outperforms existing methods such as the Decision Transformer, particularly with substantial dimension-wise drops of observations. These results confirm D3T’s capability in managing real-world imperfections in state observations and illustrate its potential to substantially expand the applicability of RL in more complex and dynamic environments.

Keywords

• Dimension-wise state information drop
• masked observation
• drop information embedding
• reinforcement learning
• robust learning