A model-based reinforcement learning framework for building heating management with branched rollout strategy and time-series prediction model

Reinforcement learning (RL) has emerged as a promising approach for building energy management (BEM). However, most existing research focuses on model-free reinforcement learning (MFRL) approaches, which can encounter the learning challenge for heating, ventilation and air conditioning (HVAC) control due to extensive trial-and-error explorations and lengthy training times. To address this challenge, we propose a model-based reinforcement learning (MBRL) framework that incorporates a virtual environment to augment the agent’s exploration. By leveraging the branched rollout strategy to generate short rollout predictions branched from the experience trajectory, the MBRL method mitigates compounding errors introduced by the time-series prediction model, enabling robust and efficient policy updates. Evaluated in an EnergyPlus testbed with real-world data verification, the proposed method demonstrates significant advantages: (1) RL-based controllers outperform the rule-based control (RBC) baseline after one training episode, (2) MBRL reduces training time by over 50% compared to MFRL while maintaining comparable control performance, and (3) an equal mix of real and synthetic data for MBRL training achieves an optimal trade-off between efficiency and control outcomes. This study contributes an efficient model-based training method for RL development in HVAC control, offering insights into advanced control strategies for BEM applications.