Image2Occupancy: An environment perception method based on improved 3D Gaussian splatting for occupancy prediction

With the rapid development of autonomous driving technology, accurate perception of the surrounding environment has become increasingly critical, and 3D environment perception has emerged as a major research focus in this field. Traditional 3D perception systems rely heavily on expensive sensors such as LiDAR, which offer high accuracy but incur substantial costs and computational demands, limiting their scalability in large autonomous vehicle fleets. Although more recent 3D occupancy prediction methods rely solely on multicamera inputs, they typically require supervised learning with annotated 3D occupancy data, which is costly to obtain and consumes substantial memory. To address these challenges, this article proposes Image2Occupancy, an improved 3D Gaussian-splatting-based occupancy prediction method that uses only 2D surround-view camera images. The method enables effective semantic occupancy prediction of 3D scenes while reducing the need for annotated data and large memory capacity.

The Image2Occupancy framework consists of two components: (1) 2D-to-3D feature extraction and spatial mapping, and (2) self-supervised 3D occupancy representation learning. In the first component, BEVStereo and Swin Transformer modules extract 2D features from panoramic input images. These features are then interpolated and mapped to 3D space using the intrinsic and extrinsic parameters of the camera, yielding voxel-level feature representations. This process converts 2D image information into 3D semantic occupancy cues, providing accurate input for subsequent self-supervised learning. In the second component, an improved Gaussian splatting technique projects 3D voxel features back onto the 2D image plane while preserving semantic information. Gaussian points placed at each voxel center approximate scene occupancy, enabling rendering of semantic and depth maps by computing pixel-level depth and semantic information. A novel self-supervised learning framework generates pseudo-labels from the predicted depth and semantic maps of the model, eliminating the need for real 3D occupancy labels. A specialized loss function, combining cross-entropy and depth losses, minimizes discrepancies between rendered and ground-truth semantic and depth maps, optimizing prediction accuracy.

Experiments on the NuScenes dataset show that Image2Occupancy achieves an mIoU of 27.87, improving performance by 3.94 percentage points (a 16.5% increase) over existing 2D-input methods and performing comparable to, or better than, several 3D-input methods. Compared with NeRF-based approaches, GPU memory usage is reduced by 54.7% while maintaining the same number of Gaussian points. Ablation studies further validate the effectiveness of the core components of the method.

Image2Occupancy reduces hardware dependence and substantially decreases the need for large annotated datasets through self-supervised learning, offering a cost-effective and scalable 3D environment perception solution for autonomous driving systems with strong potential for practical deployment.