An Optimized Path Planning Method for Container Ships in Bohai Bay Based on Improved Deep Q-Learning

Abstract

In response to the limitations of the DQN algorithm in adaptability, which result in a low success rate in ship path planning, this paper introduces an improved algorithm based on Deep Q-learning (DQN) to enhance path planning. The proposed algorithm aims to plan a reasonable and cost-effective route to the destination based on all historical track, regardless of the current location of ship within the environment. Firstly, the k-means clustering algorithm is employed to cluster the historical ship locations. The value of k is progressively increased to include more locations, allowing the model to interact with the environment of increasing complexity. This approach enhances the generalization capability of the model by enabling it to autonomously devise a route from any starting point to the destination port. On the other hand, for the ship path problem, the DQN algorithm is enhanced through the optimization of the reward function. This improvement targets the challenges of convergence difficulty and low learning efficiency, which greatly improves the rate of convergence for the model. Finally, the effectiveness of the method is verified by comparing it experimentally in terms of the effectiveness of path planning and model convergence trend. The results demonstrate that the improved DQN algorithm achieves a convergence speed improvement of over 25%. Additionally, with the same training time, the success rate of path planning from any position to destination within the environment is enhanced by 44%. It has better effect on the path planning of ships.

Keywords

• Ship path planning
• AIS
• deep reinforcement learning
• neural network
• k-means