Pre-training large language models based on Transformer architecture for building industry application: A review

The construction field is known for its high professional requirements, complexity, and diversity. Over the past few decades, CO₂ emissions and energy consumption have increased significantly, making it necessary to improve the efficiency of building design and the retrofitting for energy conservation imperatively. Pre-trained large language models (LLMs) assist industry professionals in architectural design and renovation through natural language processing. This study reviews research papers on Transformer architecture pre-training LLMs from the perspective of model types and application directions. It discusses their potential future application directions along with factors affecting their performance. The assessment outcomes indicate that LLMs excel in tasks including building information detection, modeling automation, and fault diagnosis. Specifically, LLMs achieve median accuracy of approximately 89% for building information detection, 91% for auxiliary modeling, and 98% for fault detection. However, current applications are primarily limited to processing built environment data (e.g., structural data and operational/maintenance records). The input prompt is an essential factor that affects the task execution capability of LLMs. Existing studies have not fully explored the interpretability, transfer learning potential, or contextual learning methods of LLMs, and most research is limited to the use of GPT models. Future work should expand the applications of LLMs in construction, develop task-specific model adaptations, enhance model interpretability, and establish new evaluation metrics for construction-associated tasks.