Amid global climate change and the goals of carbon peak and carbon neutrality, photovoltaic-driven air conditioning (PVAC) systems were critical for low-carbon building cooling, but existing controls suffered from poor adaptability, high computation costs, and inadequate real-time performance. This study proposed a hierarchical supervisory control framework integrating the twin delayed deep deterministic policy gradient (TD3) algorithm with large language model (LLM)-based adaptive reward weight adjustment. The TD3 controller performed real-time compressor speed regulation to balance multiple operational objectives. Meanwhile, LLM was used as a high-level module for adaptive reward tuning between control cycles. It adaptively adjusted the reward function weights based on historical operational performance, thereby reducing reliance on manual parameter tuning. Two LLM variants, DeepSeek-R1 and DeepSeek-V3, were employed as supervisory decision agents. The reasoner model DeepSeek-R1 demonstrated strong capability in mechanism-oriented reasoning and optimization exploration, whereas the chat model DeepSeek-V3 provided a lower latency compared to the reasoner model. Results showed that the supervisory mechanism enabled stable convergence of reward weights and improved the overall balance among thermal comfort, energy utilization, and operational cost. By using an adaptive supervisory decision from the LLM, The R1 model achieved a maximum comfort time ratio of 92.75%. The V3 model showed superior economic performance, with a maximum daily cost saving of 3.94 CNY and a carbon emission reduction of 2.8 kgCO₂e. Compared with the fixed-weight case, these results represented improvements of 18%, 14.2%, and 35.88%, respectively. The framework also reduced dependence on domain expertise while enhancing system adaptability.