ThinGPT: describing sedimentary rock thin section images with a multimodal large language model

Rock thin section description is an essential method for examining lithology, structure, diagenesis, and sedimentary environment, playing a pivotal role in fields such as geology, geophysics, and petroleum exploration. To overcome the challenges of subjectivity, low efficiency, and high expertise requirements in describing rock thin sections, we design a multimodal mapping network, ThinGPT, which aligns the feature spaces of the contrastive language-image pre-training (CLIP) and Generative Pre-trained (GPT-2) through network training. Given the high frequency of keywords and the structured sentence patterns in thin-section descriptions, we introduce a tokenization method tailored for rock thin sections. This approach enhances GPT-2's ability to effectively encode text and produce text feature vectors. We conducted comparative experiments using ThinGPT and other models on common sedimentary rocks. The results demonstrate that ThinGPT exhibits excellent potential in generating thin-section feature descriptions of rocks. Based on the geological expert evaluation criteria proposed in this study, ThinGPT achieved a score of 1.62 on the test set. For model complexity, ThinGPT avoids heavy initial training of large language models (LLMs). This training strategy makes the model lighter and improves the efficiency of rock thin section descriptions. As an innovative application of a LLMs within a lightweight architecture for rock thin section description, ThinGPT has significant implications for intelligent geology, geophysics, and petroleum exploration.