Journal Home > Volume 9 , Issue 3
CSEE Journal of Power and Energy Systems 2023, 9(3): 949-962 https://doi.org/10.17775/CSEEJPES.2021.06120
Open Access | Issue | Published: 06 May 2022

Deep Reinforcement Learning Based Bi-layer Optimal Scheduling for Microgrids Considering Flexible Load Control

Show Author's Information Zitong Zhang,  Jing Shi( ),  Wangwang Yang,  Zhaofang Song,  Zexu Chen,  Dengquan Lin
State Key Laboratory of Advanced Electromagnetic Engineering and Technology, School of Electronic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China
Keywords:
deep reinforcement learning, demand response, Bi-layer optimal scheduling, microgrid scheduling
Cite this article:
Zhang Z, Shi J, Yang W, et al. Deep Reinforcement Learning Based Bi-layer Optimal Scheduling for Microgrids Considering Flexible Load Control. CSEE Journal of Power and Energy Systems, 2023, 9(3): 949-962. https://doi.org/10.17775/CSEEJPES.2021.06120
Download citation
EndNote(RIS)
BibTeX

244

Views

2

Downloads

Citations

2

Crossref

6

WoS

6

Scopus

0

CSCD

Abstract Full text About this article

In this paper, the bi-layer scheduling method for microgrids, based on deep reinforcement learning, is proposed to achieve economic and environmentally friendly operations. First, considering the uncertainty of renewable energy, the framework of day-ahead and intra-day scheduling is established, and the implementation scheme for both price-based and incentive-based demand response (DR) for the flexible load is determined. Then, comprehensively considering the operating characteristics of the microgrid in the day-ahead and intra-day time scales, a bi-layer scheduling model of the microgrid is established. In terms of algorithms, since day-ahead scheduling has no strict requirement for dispatching time, the particle swarm optimization (PSO) algorithm is used to optimize the time-of-use electricity price and distributed power output for the next day. Considering the environmental fluctuations and requirements for rapidity of intra-day online scheduling, the deep reinforcement learning (DRL) algorithm is adopted for optimization. Finally, based on the data from the actual microgrid, the rationality and effectiveness of the proposed scheduling method is verified. The results show that the proposed bi-layer scheduling based on the PSO and DRL algorithms achieves the optimization of scheduling cost and calculation speed, and is suitable for microgrid online scheduling.


menu
Abstract
Full text
Outline
About this article

Deep Reinforcement Learning Based Bi-layer Optimal Scheduling for Microgrids Considering Flexible Load Control

Show Author's information Zitong Zhang, Jing Shi( ), Wangwang Yang, Zhaofang Song, Zexu Chen, Dengquan Lin
State Key Laboratory of Advanced Electromagnetic Engineering and Technology, School of Electronic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China

Abstract

In this paper, the bi-layer scheduling method for microgrids, based on deep reinforcement learning, is proposed to achieve economic and environmentally friendly operations. First, considering the uncertainty of renewable energy, the framework of day-ahead and intra-day scheduling is established, and the implementation scheme for both price-based and incentive-based demand response (DR) for the flexible load is determined. Then, comprehensively considering the operating characteristics of the microgrid in the day-ahead and intra-day time scales, a bi-layer scheduling model of the microgrid is established. In terms of algorithms, since day-ahead scheduling has no strict requirement for dispatching time, the particle swarm optimization (PSO) algorithm is used to optimize the time-of-use electricity price and distributed power output for the next day. Considering the environmental fluctuations and requirements for rapidity of intra-day online scheduling, the deep reinforcement learning (DRL) algorithm is adopted for optimization. Finally, based on the data from the actual microgrid, the rationality and effectiveness of the proposed scheduling method is verified. The results show that the proposed bi-layer scheduling based on the PSO and DRL algorithms achieves the optimization of scheduling cost and calculation speed, and is suitable for microgrid online scheduling.

Keywords: deep reinforcement learning, demand response, Bi-layer optimal scheduling, microgrid scheduling

References(45)

[1]

A. Chaouachi, R. M. Kamel, R. Andoulsi, and K. Nagasaka, “Multiobjective intelligent energy management for a microgrid,” IEEE Transactions on Industrial Electronics, vol. 60, no. 4, pp. 1688–1699, Apr. 2013.
DOI Google Scholar
[2]

T. Strasser, F. Andrén, J. Kathan, C. Cecati, C. Buccella, P. Siano, P. Leitão, G. Zhabelova, V. Vyatkin, P. Vrba, and V. Mařík, “A review of architectures and concepts for intelligence in future electric energy systems,” IEEE Transactions on Industrial Electronics, vol. 62, no. 4, pp. 2424–2438, Apr. 2015.
DOI Google Scholar
[3]

G. R. Ren, J. F. Liu, J. Wan, Y. F. Guo, and D. R. Yu, “Overview of wind power intermittency: impacts, measurements, and mitigation solutions,” Applied Energy, vol. 204, pp. 47–65, Oct. 2017.
DOI Google Scholar
[4]

E. O. Arwa and K. A. Folly, “Reinforcement learning techniques for optimal power control in grid-connected microgrids: a comprehensive review,” IEEE Access, vol. 8, pp. 208992–209007, Nov. 2020.
DOI Google Scholar
[5]

M. Parsa Moghaddam, A. Abdollahi, and M. Rashidinejad, “Flexible demand response programs modeling in competitive electricity markets,” Applied Energy, vol. 88, no. 9, pp. 3257–3269, Sep. 2011.
DOI Google Scholar
[6]

D. S. Callaway and I. A. Hiskens, “Achieving controllability of electric loads,” Proceedings of the IEEE, vol. 99, no. 1, pp. 184–199, Jan. 2011.
DOI Google Scholar
[7]
I. R. S. da Silva, J. E. A. de Alencar, and R. de Andrade Lira Rabêlo, “A preference-based multi-objective demand response mechanism,” in Proceedings of 2020 IEEE Congress on Evolutionary Computation, 2020, pp. 1–8.
DOI
[8]

B. Roberts, “Capturing grid power,” IEEE Power and Energy Magazine, vol. 7, no. 4, pp. 32–41, Jul./Aug. 2009.
DOI Google Scholar
[9]

Y. Riffonneau, S. Bacha, F. Barruel, and S. Ploix, “Optimal power flow management for grid connected PV systems with batteries,” IEEE Transactions on Sustainable Energy, vol. 2, no. 3, pp. 309–320, Jul. 2011.
DOI Google Scholar
[10]

A. Parisio, E. Rikos, and L. Glielmo, “A model predictive control approach to microgrid operation optimization,” IEEE Transactions on Control Systems Technology, vol. 22, no. 5, pp. 1813–1827, Sep. 2014.
DOI Google Scholar
[11]

R. Palma-Behnke, C. Benavides, F. Lanas, B. Severino, L. Reyes, J. Llanos, and D. Sáez, “A microgrid energy management system based on the rolling horizon strategy,” IEEE Transactions on Smart Grid, vol. 4, no. 2, pp. 996–1006, Jun. 2013.
DOI Google Scholar
[12]

D. E. Olivares, C. A. Cañizares, and M. Kazerani, “A centralized energy management system for isolated microgrids,” IEEE Transactions on Smart Grid, vol. 5, no. 4, pp. 1864–1875, Jul. 2014.
DOI Google Scholar
[13]

S. A. Pourmousavi, M. H. Nehrir, C. M. Colson, and C. S. Wang, “Real-time energy management of a stand-alone hybrid wind-microturbine energy system using particle swarm optimization,” IEEE Transactions on Sustainable Energy, vol. 1, no. 3, pp. 193–201, Oct. 2010.
DOI Google Scholar
[14]

S. A. Alavi, A. Ahmadian, and M. Aliakbar-Golkar, “Optimal probabilistic energy management in a typical micro-grid based-on robust optimization and point estimate method,” Energy Conversion and Management, vol. 95, pp. 314–325, May 2015.
DOI Google Scholar
[15]

M. Marzband, E. Yousefnejad, A. Sumper, and J. L. Domínguez-García, “Real time experimental implementation of optimum energy management system in standalone Microgrid by using multi-layer ant colony optimization,” International Journal of Electrical Power & Energy Systems, vol. 75, pp. 265–274, Feb. 2016.
DOI Google Scholar
[16]

M. Marzband, M. Ghadimi, A. Sumper, and J. L. Domínguez-García, “Experimental validation of a real-time energy management system using multi-period gravitational search algorithm for microgrids in islanded mode,” Applied Energy, vol. 128, pp. 164–174, Sep. 2014.
DOI Google Scholar
[17]

A. Askarzadeh, “A memory-based genetic algorithm for optimization of power generation in a microgrid,” IEEE Transactions on Sustainable Energy, vol. 9, no. 3, pp. 1081–1089, Jul. 2018.
DOI Google Scholar
[18]

A. Lorestani and M. M. Ardehali, “Optimal integration of renewable energy sources for autonomous tri-generation combined cooling, heating and power system based on evolutionary particle swarm optimization algorithm,” Energy, vol. 145, pp. 839–855, Feb. 2018.
DOI Google Scholar
[19]

Z. K. Li and M. Ierapetritou, “Process scheduling under uncertainty: review and challenges,” Computers & Chemical Engineering, vol. 32, no. 4–5, pp. 715–727, Apr. 2008.
DOI Google Scholar
[20]

J. L. Yi, P. F. Lyons, P. J. Davison, P. F. Wang, and P. C. Taylor, “Robust scheduling scheme for energy storage to facilitate high penetration of renewables,” IEEE Transactions on Sustainable Energy, vol. 7, no. 2, pp. 797–807, Apr. 2016.
DOI Google Scholar
[21]

H. Y. Wu, M. Shahidehpour, Z. Y. Li, and W. Tian, “Chance-constrained day-ahead scheduling in stochastic power system operation,” IEEE Transactions on Power Systems, vol. 29, no. 4, pp. 1583–1591, Jul. 2014.
DOI Google Scholar
[22]

Z. J. Bao, Q. Zhou, Z. H. Yang, Q. Yang, L. Z. Xu, and T. Wu, “A multi time-scale and multi energy-type coordinated microgrid scheduling solution—Part Ⅰ: model and methodology,” IEEE Transactions on Power Systems, vol. 30, no. 5, pp. 2257–2266, Sep. 2015.
DOI Google Scholar
[23]

F. Arasteh and G. H. Riahy, “MPC-based approach for online demand side and storage system management in market based wind integrated power systems,” International Journal of Electrical Power & Energy Systems, vol. 106, pp. 124–137, 2019.
DOI Google Scholar
[24]

S. Tewari, C. J. Geyer, and N. Mohan, “A statistical model for wind power forecast error and its application to the estimation of penalties in liberalized markets,” IEEE Transactions on Power Systems, vol. 26, no. 4, pp. 2031–2039, Nov. 2011.
DOI Google Scholar
[25]

Z. Zhang, D. Zhang and R. C. Qiu, “Deep reinforcement learning for power system applications: An overview,” CSEE Journal of Power and Energy Systems, vol. 6, no. 1, pp. 213–225, Mar. 2020,
Google Scholar
[26]

M. Khodayar, G. Liu, J. Wang and M. E. Khodayar, “Deep learning in power systems research: A review,” CSEE Journal of Power and Energy Systems, vol. 7, no. 2, pp. 209–220, Mar. 2021.
Google Scholar
[27]

H. Tang, S. P. Wang, K. J. Chang, and J. Y. Guan, “Intra-day dynamic optimal dispatch for power system based on deep Q-learning,” IEEJ Transactions on Electrical and Electronic Engineering, vol. 16, no. 7, pp. 954–964, Jul. 2021.
DOI Google Scholar
[28]

S. Y. Zhou, Z. J. Hu, W. Gu, M. Jiang, M. Chen, Q. T. Hong, and C. Booth, “Combined heat and power system intelligent economic dispatch: a deep reinforcement learning approach,” International Journal of Electrical Power & Energy Systems, vol. 120, pp. 106016, Sep. 2020.
DOI Google Scholar
[29]

E. Mocanu, D. C. Mocanu, P. H. Nguyen, A. Liotta, M. E. Webber, M. Gibescu, and J. G. Slootweg, “On-line building energy optimization using deep reinforcement learning,” IEEE Transactions on Smart Grid, vol. 10, no. 4, pp. 3698–3708, Jul. 2019.
DOI Google Scholar
[30]

B. Wang, Y. Li, W. Y. Ming, and S. R. Wang, “Deep reinforcement learning method for demand response management of interruptible load,” IEEE Transactions on Smart Grid, vol. 11, no. 4, pp. 3146–3155, Jul. 2020.
DOI Google Scholar
[31]

S. Bahrami, Y. C. Chen, and V. W. S. Wong, “Deep reinforcement learning for demand response in distribution networks,” IEEE Transactions on Smart Grid, vol. 12, no. 2, pp. 1496–1506, Mar. 2021.
DOI Google Scholar
[32]

Y. Ji, J. H. Wang, J. C. Xu, X. K. Fang, and H. G. Zhang, “Real-time energy management of a microgrid using deep reinforcement learning,” Energies, vol. 12, no. 12, pp. 2291, Jun. 2019.
DOI Google Scholar
[33]

E. Foruzan, L. K. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Transactions on Power Systems, vol. 33, no. 5, pp. 5749–5758, Sep. 2018.
DOI Google Scholar
[34]

T. A. Nakabi and P. Toivanen, “Deep reinforcement learning for energy management in a microgrid with flexible demand,” Sustainable Energy, Grids and Networks, vol. 25, pp. 100413, Mar. 2021.
DOI Google Scholar
[35]

J. Valenzuela, P. R. Thimmapuram, and J. Kim, “Modeling and simulation of consumer response to dynamic pricing with enabled technologies,” Applied Energy, vol. 96, pp. 122–132, Aug. 2012.
DOI Google Scholar
[36]

B. Zhao, X. S. Zhang, J. Chen, C. S. Wang, and L. Guo, “Operation optimization of standalone microgrids considering lifetime characteristics of battery energy storage system,” IEEE Transactions on Sustainable Energy, vol. 4, no. 4, pp. 934–943, Oct. 2013.
DOI Google Scholar
[37]

D. S. Kirschen, G. Strbac, P. Cumperayot, and D. de Paiva Mendes, “Factoring the elasticity of demand in electricity prices,” IEEE Transactions on Power Systems, vol. 15, no. 2, pp. 612–617, May 2000.
DOI Google Scholar
[38]

R. X. Yin, E. C. Kara, Y. P. Li, N. DeForest, K. Wang, T. Y. Yong, and M. Stadler, “Quantifying flexibility of commercial and residential loads for demand response using setpoint changes,” Applied Energy, vol. 177, pp. 149–164, Sep. 2016.
DOI Google Scholar
[39]

A. Hossain, H. R. Pota, S. Squartini, and A. F. Abdou, “Modified PSO algorithm for real-time energy management in grid-connected microgrids,” Renewable Energy, vol. 136, pp. 746–757, Jun. 2019.
DOI Google Scholar
[40]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proceedings of the 4th International Conference on Learning Representations, 2015.
[41]
J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel, “Trust region policy optimization,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015.
[42]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, Proximal policy optimization algorithms, 2017, [Online] Available: https://arxiv.org/abs/1707.06347
[43]

S. M. Zhang, J. Q. Rong, and B. Y. Wang, “An optimal scheduling scheme for smart home electricity considering demand response and privacy protection,” International Journal of Electrical Power & Energy Systems, vol. 132, pp. 107159, Nov. 2021.
DOI Google Scholar
[44]
Elia Group. Transparency on grid data.[Online]. Available: http://www.elia.be/en/grid-data.
[45]

Y. Zeng, C. Li, and H. M. Wang, “Scenario-set-based economic dispatch of power system with wind power and energy storage system,” IEEE Access, vol. 8, pp. 109105–109119, Jun. 2020.
DOI Google Scholar
Publication history
Copyright
Rights and permissions

Publication history

Received: 18 August 2021
Revised: 18 November 2021
Accepted: 29 December 2021
Published: 06 May 2022
Issue date: May 2023

Copyright

© 2021 CSEE.

Rights and permissions

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Return