Adaptive optimal tracking control for underactuated surface vessels using extended state observer and reinforcement learning

This paper investigates the adaptive optimal tracking control (AOTC) for underactuated surface vessels (USVs). Compared to the majority of existing studies, the control strategy in this paper innovatively combines an extended state observer (ESO) with reinforcement learning (RL). The designed ESO has high estimation accuracy and robust disturbance rejection capabilities for the unmeasurable information for USVs. To obtain the AOTC, the actor–critic (AC) networks based on RL are constructed to solve the Hamilton–Jacobi–Bellman (HJB) equations. Due to the uncertainties, it is challenging to obtain the optimal controller by directly solving the HJB equations. To address this issue, this paper employs neural networks (NNs) to approximate the uncertainties and solves the optimal controller via AC-RL and ESO. In addition, the adaptive parameters of the optimal controller is trained in parallel with AC networks, which can ensure that the trained networks can further improve tracking performance. The boundedness of AOTC for USVs is shown by Lyapunov stability theorem. Finally, simulation results demonstrate the effectiveness of the proposed algorithm.