Journal Home > Volume 8 , Issue 4

Employing the novel Deep Reinforcement Learning approach, this paper addresses the active power corrective control in modern power systems. Seeking to minimize the joint effect engendered by operation cost and blackout penalty, this correction strategy focuses on evaluating the robustness and adaptability aspects of the control agent. In Part I of this paper, where robustness is the primary focus, the agent is developed to handle unexpected incidents and guide the stable operation of power grids A Simulation-driven Graph Attention Reinforcement Learning method is proposed to perform robust active power corrective control. The aim of the graph attention networks is to determine the representation of power system states considering the topological features. Monte Carlo tree search is adopted to select the best suitable action set out of the large action space, including generator redispatch and topology control actions. Finally, driven by simulation, a guided training mechanism along with a long-short-term action deployment strategy are designed to help the agent better evaluate the action set while training and to operate more stably when deployed. The efficacy of the proposed method has been demonstrated in the "2020 Learning to Run a Power Network - Neurips Track 1" global competition and the associated cases. Part II of this paper deals with the adaptability case, where the agent is equipped to better adapt to a grid that has an increasing share of renewable energies through the years.


menu
Abstract
Full text
Outline
About this article

Active Power Correction Strategies Based on Deep Reinforcement Learning—Part I: A Simulation-driven Solution for Robustness

Show Author's information Peidong XuJiajun DuanJun Zhang ( )Yangzhou PeiDi ShiZhiwei WangXuzhu DongYuanzhang Sun
School of Electrical Engineering and Automation, Wuhan University, Wuhan 430072, China
GEIRI North America, San Jose, CA 95134, USA

Abstract

Employing the novel Deep Reinforcement Learning approach, this paper addresses the active power corrective control in modern power systems. Seeking to minimize the joint effect engendered by operation cost and blackout penalty, this correction strategy focuses on evaluating the robustness and adaptability aspects of the control agent. In Part I of this paper, where robustness is the primary focus, the agent is developed to handle unexpected incidents and guide the stable operation of power grids A Simulation-driven Graph Attention Reinforcement Learning method is proposed to perform robust active power corrective control. The aim of the graph attention networks is to determine the representation of power system states considering the topological features. Monte Carlo tree search is adopted to select the best suitable action set out of the large action space, including generator redispatch and topology control actions. Finally, driven by simulation, a guided training mechanism along with a long-short-term action deployment strategy are designed to help the agent better evaluate the action set while training and to operate more stably when deployed. The efficacy of the proposed method has been demonstrated in the "2020 Learning to Run a Power Network - Neurips Track 1" global competition and the associated cases. Part II of this paper deals with the adaptability case, where the agent is equipped to better adapt to a grid that has an increasing share of renewable energies through the years.

Keywords: deep reinforcement learning, Active power corrective control, graph attention networks, simulation-driven

References(32)

[1]
A. Y. M. Abbas, S. E. G. M. Hassan, and Y. H. Abdelrahim, “Transmission lines overload alleviation by generation rescheduling and load shedding,” Journal of Infrastructure Systems, vol. 22, no. 4, pp. A4016001, May 2016.
[2]
M. Sahraei-Ardakani and K. W. Hedman, “Day-ahead corrective adjustment of FACTS reactance: a linear programming approach,” IEEE Transactions on Power Systems, vol. 31, no. 4, pp. 2867–2875, Jul. 2016.
[3]
R. Y. Bi, T. Lin, R. S. Chen, J. Ye, X. M. Zhou, and X. L. Xu, “Alleviation of post-contingency overloads by SOCP based corrective control considering TCSC and MTDC,” IET Generation, Transmission & Distribution, vol. 12, no. 9, pp. 2155–2164, May 2018.
[4]
B. Gou and H. M. Zhang, “Fast real-time corrective control strategy for overload relief in bulk power systems,” IET Generation, Transmission & Distribution, vol. 7, no. 12, pp. 1508–1515, Dec. 2013.
[5]
E. A. Goldis, P. A. Ruiz, M. C. Caramanis, X. G. Li, C. R. Philbrick, and A. M. Rudkevich, “Shift factor-based SCOPF topology control MIP formulations with substation configurations,” IEEE Transactions on Power Systems, vol. 32, no. 2, pp. 1179–1190, Mar. 2017.
[6]
J. D. Fuller, R. Ramasra, and A. Cha, “Fast heuristics for transmission-line switching,” IEEE Transactions on Power Systems, vol. 27, no. 3, pp. 1377–1386, Aug. 2012.
[7]
T. Ding and C. Y. Zhao, “Robust optimal transmission switching with the consideration of corrective actions for N-k contingencies,” IET Generation, Transmission & Distribution, vol. 10, no. 13, pp. 3288–3295, Oct. 2016.
[8]
M. Khanabadi, H. Ghasemi, and M. Doostizadeh, “Optimal transmission switching considering voltage security and N-1 contingency analysis,” IEEE Transactions on Power Systems, vol. 28, no. 1, pp. 542–550, Feb. 2013.
[9]
E. A. Goldis, X. G. Li, M. C. Caramanis, B. Keshavamurthy, M. Patel, A. M. Rudkevich, and P. A. Ruiz, “Applicability of topology control algorithms (TCA) to a real-size power system,” in Proceedings of 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2013, pp. 1349–1352.
[10]
M. Abdi-Khorsand, M. Sahraei-Ardakani, and Y. M. Al-Abdullah, “Corrective transmission switching for N-1–1 contingency analysis,” IEEE Transactions on Power Systems, vol. 32, no. 2, pp. 1606–1615, Mar. 2017.
[11]
A. G. Bakirtzis and A. P. S. Meliopoulos, “Incorporation of switching operations in power system corrective control computations,” IEEE Transactions on Power Systems, vol. 2, no. 3, pp. 669–675, Aug. 1987.
[12]
A. A. Mazi, B. F. Wollenberg, and M. H. Hesse, “Corrective control of power system flows by line and bus-bar switching,” IEEE Transactions on Power Systems, vol. 1, no. 3, pp. 258–264, Aug. 1986.
[13]
W. Shao and V. Vittal, “Corrective switching algorithm for relieving overloads and voltage violations,” IEEE Transactions on Power Systems, vol. 20, no. 4, pp. 1877–1885, Nov. 2005.
[14]
B. Li and G. Sansavini, “Effective multi-objective selection of inter-subnetwork power shifts to mitigate cascading failures,” Electric Power Systems Research, vol. 134, pp. 114–125, May 2016.
[15]
L. D. Arya and A. Koshti, “Anticipatory load shedding for line overload alleviation using Teaching learning based optimization (TLBO),” International Journal of Electrical Power & Energy Systems, vol. 63, pp. 862–877, Dec. 2014.
[16]
M. Y. Li, P. B. Luh, L. D. Michel, Q. C. Zhao, and X. C. Luo, “Corrective line switching with security constraints for the base and contingency cases,” IEEE Transactions on Power Systems, vol. 27, no. 1, pp. 125–133, Feb. 2012.
[17]
A. S. Korad and K. W. Hedman, “Robust corrective topology control for system reliability,” IEEE Transactions on Power Systems, vol. 28, no. 4, pp. 4042–4051, Nov. 2013.
[18]
A. A. Abou EL Ela, A. Z. El-Din, and S. R. Spea, “Optimal corrective actions for power systems using multiobjective genetic algorithms,” in Proceedings of 42nd International Universities Power Engineering Conference, Brighton, 2007, pp. 365–376.
[19]
X. P. Li, P. Balasubramanian, M. Sahraei-Ardakani, M. Abdi-Khorsand, K. W. Hedman, and R. Podmore, “Real-time contingency analysis with corrective transmission switching,” IEEE Transactions on Power Systems, vol. 32, no. 4, pp. 2604–2617, Jul. 2017.
[20]
D. Ram, L. Srivastava, M. Pandit, and J. Sharma, “Corrective action planning using RBF neural network,” Applied Soft Computing, vol. 7, no. 3, pp. 1055–1063, Jun. 2007.
[21]
M. Glavic, R. Fonteneau, and D. Ernst, “Reinforcement learning for electric power system decision and control: past considerations and perspectives,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 6918–6927, Jul. 2017.
[22]
Z. Zhang, D. Zhang and R. C. Qiu, “Deep reinforcement learning for power system applications: An overview,” CSEE Journal of Power and Energy Systems, vol. 6, no. 1, pp. 213–225, Mar. 2020, .
[23]
F. Ruelens, B. J. Claessens, P. Vrancx, F. Spiessens and G. Deconinck, “Direct load control of thermostatically controlled loads based on sparse observations using deep reinforcement learning,” CSEE Journal of Power and Energy Systems, vol. 5, no. 4, pp. 423–432, Dec. 2019, .
[24]
J. J. Duan, D. Shi, R. S. Diao, H. F. Li, Z. W. Wang, B. Zhang, D. S. Bian, and Z. H. Yi, “Deep-reinforcement-learning-based autonomous voltage control for power grid operations,” IEEE Transactions on Power Systems, vol. 35, no. 1, pp. 814–817, Jan. 2020.
[25]
T. Lan, J. J. Duan, B. Zhang, D. Shi, Z. W. Wang, R. S. Diao, and X. H. Zhang, “AI-based autonomous line flow control via topology adjustment for maximizing time-series ATCs,” in Proceedings of the IEEE Power & Energy Society General Meeting (PESGM), 2020, pp. 1–5.
[26]
P. D. Xu, Y. Z. Pei, X. H. Zheng, and J. Zhang, “A simulation-constraint graph reinforcement learning method for line flow control,” in Proceedings of the 4th IEEE Conference on Energy Internet & Energy System Integration, Wuhan, China, 2020, pp. 319–324.
[27]
Universidad Nacional de Colombia. (2020, Oct. 20). L2RPN-NEURIPS-2020 [Online]. Available: https://github.com/unaioperator/l2rpn-neurips-2020.
[28]
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” arXiv e-prints arXiv: 1710.10903, Oct. 2017. [Online]. Available: https://arxiv.org/pdf/1710.10903.pdf.
[29]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedins of the 31st International Conference on Neural Information Processing Systems (NIPS’17), New York, USA, 2017, pp. 6000–6010.
[30]
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, Jan. 2016.
[31]
RTE-France. (2021, Aug. 22). Grid2Op [Online]. Available: https://github.com/rte-france/Grid2Op.
[32]
BDonnot. (2021, Oct. 8). L2RPN NEURIPS 2020 - robustness track [Online]. Available: https://competitions.codalab.org/competitions/25426.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 30 December 2020
Revised: 18 February 2021
Accepted: 14 April 2021
Published: 10 September 2021
Issue date: July 2022

Copyright

© 2020 CSEE

Acknowledgements

Acknowledgements

The work is supported by the National Key R&D Program of China under Grant 2018AAA0101504 and the Science and technology project of SGCC (State Grid Corporation of China): fundamental theory of human-in-the-loop hybrid-augmented intelligence for power grid dispatch and control.

Rights and permissions

Return