3D LiDAR and image data-level fusion for traffic vehicle detection

Vehicle detection is a crucial part of perception technologies for unmanned vehicles, which provides perceptive guarantees for route plans and vehicle control. Current multimodal fusion approaches for image and light detection and ranging (LiDAR) data primarily include feature-level fusion, object-level fusion, and data-level fusion strategies. However, feature-level and object-level fusion methods fail to fundamentally address the core challenges of low-quality sparse point cloud data with insufficient information and high noise levels, resulting in low confidence in fused features or detection results. Research on data-level fusion methods remains limited. Aiming at the ineffective representation of distant vehicles by sparse LiDAR clouds, this study proposes a vehicle detection method based on the data-level fusion of image and LiDAR data. First, this method estimates the depth of image pixels to generate the pseudo point clouds. Then, the K-dimensional (KD) tree is employed to reduce the noise and eliminate the outlier of pseudo points to fuse with LiDAR data. Finally, the fused point clouds are fed to the point-based region-based convolutional neural network (PointRCNN) to achieve vehicle detection. The advantage of the proposed method is that it utilizes pseudo point clouds to compensate for sparse LiDAR data, which contributes to improving the detection accuracy of distant vehicles. This study utilizes the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset to conduct comparative experiments and ablation studies. The experimental results reveal that the proposed method improves 7.8% and 4.13% in the aspects of average precision for three-dimensional objects (AP3D) and average precision for bird’s eye view (APBEV) with comparison methods, respectively. Meanwhile, in terms of distant vehicle detection, the proposed method raises 9.13% and 9.58% in the aspects of AP3D and APBEV respectively by comparison with frustum-based PointNet (F-PointNet).