Accurate load forecasting is critical for efficient and reliable grid operations, particularly given increasing volatility caused by atypical events such as extreme weather, policy changes, and social disruptions. Traditional forecasting methods based solely on historical numerical data often fail to account for these anomalies in cyber-physical-social systems (CPSS), leading to significant prediction errors. To address this limitation, a novel hybrid forecasting framework is proposed, integrating large language models (LLMs), retrieval-augmented generation (RAG), and chain-of-thought (CoT) prompting. The framework leverages LLMs to systematically extract structured event-driven features from unstructured information in CPSS. These enriched contextual features are integrated into a specially designed forecasting architecture, termed Loadformer to optimize numerical prediction accuracy. Experimental validation using real-world electricity load data from multiple Australian regions demonstrates the superior performance of the proposed approach. The results highlight the substantial benefits of integrating qualitative contextual understanding from textual data into quantitative forecasting models, improving adaptability and robustness in scenarios characterized by volatility and unpredictability.