Artificial intelligence (AI) is vastly superior to human analysis in many ways—particularly in processing, analyzing and synthesizing massive quantities of information at incredibly fast speeds—but AI model performance is only as good as the data used to train it.

Much like humans, AI models need to be taught to perform specific tasks, and oftentimes, the more quality data used to train a model, the more proficient the model will be at recognizing patterns or processing data, for example.

Currently, one of the biggest bottlenecks for AI model implementation is the training phase, where models are taught to perform specific tasks using data. Prior to training, large amounts of data often require humans to annotate or structure data into specific formats that algorithms require. As models get larger, the amount of labor and time required to transform or appropriately label data can grow exponentially, delaying model training and deployment.

To address this issue, a group of researchers from Fudan University and Nanyang Technological University wrote a comprehensive literature review outlining how AI models can supervise their own learning using point cloud data, or raw, three-dimensional (3D) spatial coordinates that correlate to the surface of an object, greatly reducing or eliminating the amount of human labor required to annotate large datasets for model training.

The team published their review on April 22 in the journal Computational Visual Media, published by Tsinghua University Press.

“Just as self-supervised learning has already revolutionized how AI understands natural language and 2D images, it is now becoming the key to unlocking 3D data. 3D point clouds are essential for technologies like self-driving cars and robotics, but training AI to understand them traditionally required humans to manually label millions of 3D coordinates, creating a great bottleneck. This technology is attracting immense attention because it allows the AI to learn generic, powerful 3D representations directly from raw, unlabeled data, paving the way for the emergence of true 3D foundation models,” said Ben Fei, research fellow at the Chinese University of Hong Kong and first author of the review paper.

One popular self-supervised AI learning approach uses deep neural networks (DNNs) and assigns various pretext tasks for the network to solve. The pretext task serves as a temporary, artificially created problem to train an AI model on unlabeled data. This task forces the model to understand the fundamental structure and patterns of the data, which can later be applied to downstream real-world applications.

Several different pretext tasks have been proposed for self-supervised learning, including point cloud reconstruction, or the conversion of unstructured, 3D data into a usable digital 3D model; contrastive learning, which teaches models to understand data through comparison; and multi-modal learning, which integrates many different types of data into a single unified model.  Pretext tasks share two common properties: 1) The visual features of point clouds must be captured by DNNs to solve the pretext task, and 2) the supervisory signal is generated from the data itself, which results in self-supervision, by exploiting its structure.

“The main takeaway is that effective 3D representation learning is closely tied to choosing the right pre-training tasks that leverage the data itself rather than human labels. For instance, these self-supervised approaches allow a single model to learn rich, generic geometric features from the data's own structure. This comprehensive understanding serves as a powerful prior that can be easily transferred to various applications, from indoor robotics to outdoor autonomous driving. Our paper provides a comprehensive, unified roadmap that categorizes these learning schemes, showing how to systematically build and adapt these frameworks toward capable 3D foundation models,” said Fei.

Despite these advances, hurdles still exist for self-supervised 3D model training.

“Our next step is to overcome the challenges unique to 3D representation learning, including optimizing the massive computing power required and establishing higher-quality, standardized 3D pre-training datasets. Our ultimate goal is to break down the barriers between different modalities. By successfully scaling up 3D foundation models and aligning them with large language models and 2D vision, we hope to provide the crucial spatial intelligence needed to achieve true Artificial General Intelligence,” said Fei.

Jingyi Xu, Yixuan Li, Weidong Yang, Qingyuan Zhou, Liwen Liu and Tianyue Luo from the School of Computer Science at Fudan University in Shanghai, China; and Ying He from the College of Computing and Data Science at Nanyang Technological University in Singapore, Singapore also contributed to this research.

This research was supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust, the MTR Research Funding (MRF) Scheme (CHU-24003), the Research Grants Council of Hong Kong (CUHK14213224) and the Ministry of Education, Singapore, Academic Research Fund Grant (RT19/22).