SLAM algorithm for agricultural robots based on deep reinforcement learning and dynamic fusion perception

Agricultural robots have been widely used in precision agriculture in recent years. However, the stability and accuracy of their localization and mapping typically rely on fixed multi-sensor fusion, due to the significant scene dependency. The sensors, such as visual cameras, light detection and ranging (LiDAR), and inertial measurement units (IMU), are prone to degradation during multiple scenarios, leading to unstable frontend estimation and error accumulation of backend optimization. In this study, a simultaneous localization and mapping (SLAM) algorithm was proposed for the agricultural robot using deep reinforcement learning and dynamic fusion perception. The algorithm frontend comprised the visual-inertial, the LiDAR-inertia, and the adaptive weight adjustment module. Specifically, the visual-inertial module utilized image feature extraction and optical flow tracking to obtain feature correspondences. IMU pre-integration constraints were combined to minimize visual reprojection errors and inertial measurement errors within a sliding window. A continuous visual-inertial state was achieved after estimation. LiDAR depth information was introduced to further improve system robustness in weak-texture and low-illumination environments. Thereby, depth compensation and filtering of feature point were enhanced scale observability and initialization robustness. Additionally, initial pose estimation was provided for the LiDAR-inertial module during system initialization. Feature matching and global optimization were assisted during loop closure detection. The LiDAR-inertial module was responsible for processing IMU-undistorted point cloud data. Edge and planar features were extracted to perform feature matching with the local map in the sliding window. LiDAR constraint factors were constructed to improve matching convergence speed and accuracy. IMU data was used as initial value estimation on the undistorted point cloud during scan matching. Once visual information degraded or failed, the LiDAR-inertial module operated independently for the pose tracking of the system. An Actor-Critic framework was constructed in an adaptive weight adjustment module using the twin delayed deep deterministic policy gradient (TD3) algorithm, to realize dynamic weighting of visual and LiDAR factors in the factor graph. The state inputs were taken as the visual feature observation matrix and LiDAR geometric feature statistics. The reciprocal of relative pose error was employed as the reward function. The reinforcement learning was implemented to alternatively update the Actor and Critic. Actor output continuous weights were applied as the camera and LiDAR factors using input states. The visual and LiDAR constraints were adaptively fused using the weighting terms of factor graph residuals. A factor graph model was constructed after optimization on the algorithm backend. The global optimization of the factor graph was attributed to the laser constraints from the LiDAR-inertial module, the visual and inertial constraints from the visual-inertial module, and the loop detection factors. The tight coupling fusion of multi-source information was achieved in the Maximum A Posteriori (MAP) pose estimation and dense 3D environment map construction. Localization accuracy experiments in complex orchard environments demonstrated that better performance was achieved in the maximum error of 1.583 m, a mean error of 0.803 m, a root mean square error (RMSE) of 0.830 m, and a standard deviation of 0.212 m. While tightly coupled lidar-visual-inertial odometry via smoothing and mapping (LVI-SAM) exhibited the maximum error, mean error, RMSE, and standard deviation of 3.764, 1.540, 1.732, and 0.792 m, respectively, in complex orchard environments. The pose estimation accuracy also outperformed mainstream algorithms, indicating the high accuracy and efficiency under complex scenarios of modern agriculture