This study aims to develop a deep reinforcement learning-based search algorithm for unmanned surface vehicles (USVs) in submarine detection tasks.

First, a search environment and a kinematic model are constructed, and a sonar detection probability model incorporating the effects of both distance and angle is developed, with well-defined criteria for determining detection success. Based on this framework, the problem is formulated as a Markov decision process (MDP), and a deep Q-network (DQN) is employed to solve it. The state space explicitly includes detection probability, while a multi-objective reward function is designed to integrate detection probability, distance, and angle. To enhance learning efficiency, a temporal difference DQN with probabilistic sensing (TS-DQN) algorithm is proposed, combining a double-dueling network architecture with prioritized experience replay. Additionally, a probabilistic perception-based ε-greedy exploration strategy is implemented, enabling dynamic adjustment of exploration behavior based on real-time detection states, thereby significantly improving policy learning efficiency.

Extensive simulation experiments demonstrate that the proposed method achieves a detection success rate of 38.85%, approximately 18 times that of the second-best method, Dueling DQN. The approach also reduces the average path length to 334.36 steps, shortening the search trajectory by more than 9.5% compared to other algorithms.

The proposed algorithm exhibits significant advantages in detection efficiency and effectiveness, offering an effective approach for autonomous USV-based submarine search.