Microgrid energy management strategy based on two-stage deep reinforcement learning

With the gradual transformation of the global energy structure and the rapid development of renewable energy technologies, microgrid technology has emerged as an important and rapidly growing area in the energy sector. As the core of microgrid operation and control, the energy management system ensures the efficient use of renewable energy and the stable operation of microgrids through precise monitoring and intelligent control. Traditional energy management methods struggle to effectively handle the complex interrelationships among variables in microgrids, whereas deep reinforcement learning (DRL) enables intelligent decision-making by learning through interaction with the environment and adjusting strategies based on feedback signals. To address the high exploration cost and low training efficiency of existing DRL algorithms, this study proposes a microgrid energy management strategy based on a two-stage DRL framework.

The proposed strategy includes two stages: offline and online. First, in the offline stage, linear programming is used to obtain the optimal scheduling results of typical days to construct an expert experience library, and imitation learning is then used to pretrain the agents. This stage involves extracting key information from historical data, such as photovoltaic power, wind power, load demand, and electricity prices, and transforming the data into state–action pairs, thereby forming the pretraining foundation for the agents. Subsequently, in the online stage, the agents interact with the real environment to learn the optimal scheduling strategies for nontypical days. During this stage, having accumulated sufficient knowledge in the offline stage, the agents can significantly improve the environmental tracking accuracy and operational economy. A cliff-walk reward mechanism is introduced to ensure that the agents immediately stop exploring after making decisions that violate constraints, thereby reducing the training cost associated with invalid exploration. Concurrently, the proximal policy optimization (PPO) algorithm is introduced to meet the requirements of continuous action spaces and further improve the performance of the agents.

The proposed algorithm has been validated in a typical microgrid system with three PV stations, one wind turbine, one storage system, and flexible loads. The simulation results show that the convergence speed is significantly improved, and the average daily operating cost is reduced by approximately 14% compared with that of single-stage PPO. A comparison with Double Deep Q-Network, Dueling Deep Q-Network, and Distributed Dueling Deep Q-Network further demonstrates the advantages of the proposed method in achieving optimal performance.

In the two-stage DRL, pretraining in the offline stage enables agents to learn general features and strategies, enabling them to adapt more quickly to new environments and tasks in subsequent missions. Training in the online stage helps agents avoid overfitting to task-specific training data, reducing reliance on such data, lowering the risk of overfitting, and ultimately endowing the strategies with better generalization capability and robustness. Overall, compared with single-stage DRL-based microgrid energy management algorithms, the proposed two-stage DRL-based strategy significantly improves the training efficiency and optimal performance of the agents.