Industrial fault diagnosis is crucial for ensuring the safety and efficiency of modern production systems. Industrial big data, particularly large-scale tabular data capturing multivariate time-series processes, offer valuable operational insights. Existing methods face significant challenges due to extreme label scarcity and massive unlabeled data volumes. Large Language Models (LLMs) hold great potential to address these issues due to their strong heterogeneous and few-shot learning capabilities. However, the application of LLMs to fault diagnosis with industrial big data, especially for tabular data, remains unexplored. In view of this, we propose a novel semi-supervised prefix tuning of LLMs for fault diagnosis with industrial big data. We first generate auxiliary prediction tasks based on the unlabeled data as the semi-supervised training materials for LLMs. Then we design a prefix-based soft embedding layer to fine-tune the LLMs, so that the model is able to learn the task-specific information in a parameter-efficient way. To make the model applicable to industrial big data, we also implement the Sparse Gaussian Processes (SGP) to filter the most informative samples to relieve the computational cost. Finally, we design a hybrid prompt template to effectively combine the hard and soft prompts and formulate the final prediction prompt for the industrial diagnosis tasks. The experiments have proven the superiority of the proposed method.