Journal Home > Volume 9 , Issue 1
CSEE Journal of Power and Energy Systems 2023, 9(1): 197-208 https://doi.org/10.17775/CSEEJPES.2020.04610
Open Access | Issue | Published: 13 November 2021

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Show Author's Information Hongtai Zeng1,  Yanzhen Zhou1,  Qinglai Guo1( ),  Zhongmin Cai2,  Hongbin Sun1
State Key Laboratory of Power Systems, Department of Electrical Engineering, Tsinghua University, Beijing 100084, China
Department of Automation Science and Technology, Xi'an Jiaotong University, Xi'an 710000, China
Keywords:
Deep reinforcement learning, transient stability, preventive Control
Cite this article:
Zeng H, Zhou Y, Guo Q, et al. Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints. CSEE Journal of Power and Energy Systems, 2023, 9(1): 197-208. https://doi.org/10.17775/CSEEJPES.2020.04610
Download citation
EndNote(RIS)
BibTeX

316

Views

14

Downloads

Citations

3

Crossref

3

WoS

5

Scopus

0

CSCD

Abstract Full text About this article

Preventive transient stability control is an effective measure for the power system to withstand high-probability severe contingencies. It is mathematically an optimal power flow problem with transient stability constraints. Due to the constraints involved for differential algebraic equations of transient stability, it is difficult and time-consuming to solve this problem. To address these issues, this paper presents a novel deep reinforcement learning (DRL) framework for preventive transient stability control of power systems. A distributed deep deterministic policy gradient is utilized to train a DRL agent that can learn its control policy through massive interactions with a grid simulator. Once properly trained, the DRL agent can instantaneously provide effective strategies to adjust the system to a safe operating position with a near-optimal operational cost. The effectiveness of the proposed method is verified through numerical experiments conducted on a New England 39-bus system and NPCC 140-bus system.


menu
Abstract
Full text
Outline
About this article

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Show Author's information Hongtai Zeng1, Yanzhen Zhou1, Qinglai Guo1( ), Zhongmin Cai2, Hongbin Sun1
State Key Laboratory of Power Systems, Department of Electrical Engineering, Tsinghua University, Beijing 100084, China
Department of Automation Science and Technology, Xi'an Jiaotong University, Xi'an 710000, China

Abstract

Preventive transient stability control is an effective measure for the power system to withstand high-probability severe contingencies. It is mathematically an optimal power flow problem with transient stability constraints. Due to the constraints involved for differential algebraic equations of transient stability, it is difficult and time-consuming to solve this problem. To address these issues, this paper presents a novel deep reinforcement learning (DRL) framework for preventive transient stability control of power systems. A distributed deep deterministic policy gradient is utilized to train a DRL agent that can learn its control policy through massive interactions with a grid simulator. Once properly trained, the DRL agent can instantaneously provide effective strategies to adjust the system to a safe operating position with a near-optimal operational cost. The effectiveness of the proposed method is verified through numerical experiments conducted on a New England 39-bus system and NPCC 140-bus system.

Keywords: Deep reinforcement learning, transient stability, preventive Control

References(38)

[1]
D. Ruiz-Vega and M. Pavella,"A comprehensive approach to transient stability control Part 1: Near optimal preventive control," in Proceedings of 2003 IEEE Power Engineering Society General Meeting, 2003.
DOI
[2]

IEEE/CIGRE Joint Task Force on Stability Terms and Definitions, "Definition and classification of power system stability," IEEE Transactions on Power Systems, vol. 19, no. 2, pp. 1387–1401, May 2004.
DOI Google Scholar
[3]

Y. Xu, Z. Y. Dong, J. H. Zhao, Y. S. Xue, and D. J. Hill, "Trajectory sensitivity analysis on the equivalent one-machine-infinite-bus of multi-machine systems for preventive transient stability control," IET Generation, Transmission & Distribution, vol. 9, no. 3, pp. 276–286, Feb. 2015.
DOI Google Scholar
[4]

R. Zarate-Minano, T. Van Cutsem, F. Milano, and A. J. Conejo, "Securing transient stability using time-domain simulations within an optimal power flow," IEEE Transactions on Power Systems, vol. 25, no. 1, pp. 243–253, Feb. 2010.
DOI Google Scholar
[5]

M. C. Passaro, A. P. A. da Silva, and A. C. S. Lima, "Preventive control stability via neural network sensitivity," IEEE Transactions on Power Systems, vol. 29, no. 6, pp. 2846–2853, Nov. 2014.
DOI Google Scholar
[6]

C. X. Liu, K. Sun, Z. H. Rather, Z. Chen, C. L. Bak, P. Thøgersen, and P. Lund, "A systematic approach for dynamic security assessment and the corresponding preventive control scheme based on decision trees," IEEE Transactions on Power Systems, vol. 29, no. 2, pp. 717–730, Mar. 2014.
DOI Google Scholar
[7]

I. Genc, R. S. Diao, V. Vittal, S. Kolluri, and S. Mandal, "Decision tree-based preventive and corrective control applications for dynamic security enhancement in power systems," IEEE Transactions on Power Systems, vol. 25, no. 3, pp. 1611–1619, Aug. 2010.
DOI Google Scholar
[8]

Y. Xu, Z. Y. Dong, R. Zhang, and K. P. Wong, "A decision tree-based on-line preventive control strategy for power system transient instability prevention," International Journal of Systems Science, vol. 45, no. 2, pp. 176–186, Feb. 2014.
DOI Google Scholar
[9]

Y. Xu, Z. Y. Dong, L. Guan, R. Zhang, K. P. Wong, and F. J. Luo, "Preventive dynamic security control of power systems based on pattern discovery technique," IEEE Transactions on Power Systems, vol. 27, no. 3, pp. 1236–1244, Aug. 2012.
DOI Google Scholar
[10]

Y. Z. Zhou, J. Y. Wu, L. Y. Ji, Z. H. Yu, K. J. Lin, and L. L. Hao, "Transient stability preventive control of power systems using chaotic particle swarm optimization combined with two-stage support vector machine," Electric Power Systems Research, vol. 155, pp. 111–120, Feb. 2018.
DOI Google Scholar
[11]

D. Gan, R. J. Thomas, and R. D. Zimmerman, "Stability-constrained optimal power flow," IEEE Transactions on Power Systems, vol. 15, no. 2, pp. 535–540, May 2000.
DOI Google Scholar
[12]

Y. Yuan, J. Kubokawa, and H. Sasaki, "A solution of optimal power flow with multicontingency transient stability constraints," IEEE Transactions on Power Systems, vol. 18, no. 3, pp. 1094–1102, Aug. 2003.
DOI Google Scholar
[13]

Q. Y. Jiang and G. C. Geng, "A reduced-space interior point method for transient stability constrained optimal power flow," IEEE Transactions on Power Systems, vol. 25, no. 3, pp. 1232–1240, Aug. 2010.
DOI Google Scholar
[14]

Q. Y. Jiang and Z. G. Huang, "An enhanced numerical discretization method for transient stability constrained optimal power flow," IEEE Transactions on Power Systems, vol. 25, no. 4, pp. 1790–1797, Nov. 2010.
DOI Google Scholar
[15]

W. J. Huang, W. Y. Zheng, and D. J. Hill, "Distributionally robust optimal power flow in multi-microgrids with decomposition and guaranteed convergence," IEEE Transactions on Smart Grid, vol. 12, no. 1, pp. 43–55, Jan. 2021.
DOI Google Scholar
[16]

W. Y. Zheng, W. J. Huang, and D. J. Hill, "A deep learning-based general robust method for network reconfiguration in three-phase unbalanced active distribution networks," International Journal of Electrical Power & Energy Systems, vol. 120, pp. 105982, Sep. 2020.
DOI Google Scholar
[17]

H. C. Xu, A. D. Domínguez-García, V. V. Veeravalli, and P. W. Sauer, "Data-driven voltage regulation in radial power distribution systems," IEEE Transactions on Power Systems, vol. 35, no. 3, pp. 2133–2143, May 2020.
DOI Google Scholar
[18]

H. C. Xu, A. D. Domínguez-García, and P. W. Sauer, "Data-driven coordination of distributed energy resources for active power provision," IEEE Transactions on Power Systems, vol. 34, no. 4, pp. 3047–3058, Jul. 2019.
DOI Google Scholar
[19]

Z. D. Zhang, D. X. Zhang, and R. C. Qiu, "Deep reinforcement learning for power system applications: An overview," CSEE Journal of Power and Energy Systems, vol. 6, no. 1, pp. 213–225, Mar. 2020.
Google Scholar
[20]
T. Lan, J. J. Duan, B. Zhang, D. Shi, Z. W. Wang, R. S. Diao, and X. H. Zhang,"AI-based autonomous line flow control via topology adjustment for maximizing time-series ATCs," in Proceedings of the 2020 IEEE Power & Energy Society General Meeting, 2020, pp. 1–5.
DOI
[21]
J. Y. Zhang, C. Lu, C. Fang, X. Ling, and Y. Zhang,"Load shedding scheme with deep reinforcement learning to improve short-term voltage stability," in Proceedings of 2018 IEEE Innovative Smart Grid Technologies – Asia (ISGT Asia), Singapore, 2018, pp. 13–18.
DOI
[22]

W. Liu, D. X. Zhang, X. Y. Wang, J. X. Hou, and L. P. Liu, "A decision making strategy for generating unit tripping under emergency circumstances based on deep reinforcement learning," Proceedings of the CSEE, vol. 38, no. 1, pp. 109–119, Jan. 2018.
Google Scholar
[23]

Q. H. Huang, R. K. Huang, W. T. Hao, J. Tan, R. Fan, and Z. Y. Huang, "Adaptive power system emergency control using deep reinforcement learning," IEEE Transactions on Smart Grid, vol. 11, no. 2, pp. 1171–1182, Mar. 2020.
DOI Google Scholar
[24]

J. J. Duan, D. Shi, R. S. Diao, H. F. Li, Z. W. Wang, B. Zhang, D. S. Bian, and Z. H. Yi, "Deep-reinforcement-learning-based autonomous voltage control for power grid operations," IEEE Transactions on Power Systems, vol. 35, no. 1, pp. 814–817, Jan. 2020.
DOI Google Scholar
[25]

Z. M. Yan and Y. Xu, "Data-driven load frequency control for stochastic power systems: A deep reinforcement learning method with continuous action search," IEEE Transactions on Power Systems, vol. 34, no. 2, pp. 1653–1656, Mar. 2019.
DOI Google Scholar
[26]
Y. N. Hou, L. F. Liu, Q. Wei, X. D. Xu, and C. L. Chen,"A novel DDPG method with prioritized experience replay," in Proceedings of 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Banff, Canada, 2017, pp. 316–321.
DOI
[27]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra,"Continuous control with deep reinforcement learning," in Proceedings of the 4th International Conference on Learning Representations, 2016.
[28]

T. Y. Guo and J. V. Milanović, "Online identification of power system dynamic signature using PMU measurements and data mining," IEEE Transactions on Power Systems, vol. 31, no. 3, pp. 1760–1768, May 2016.
DOI Google Scholar
[29]
S. Fujimoto, H. van Hoof, and D. Meger,"Addressing function approximation error in actor-critic methods," in Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 1582–1591.
[30]

G. B. Huang, "Learning capability and storage capacity of two-hidden-layer feedforward networks," IEEE Transactions on Neural Networks, vol. 14, no. 2, pp. 274–281, Mar. 2003.
DOI Google Scholar
[31]
A. Krizhevsky, I. Sutskever, and G. E. Hinton,"ImageNet classification with deep convolutional neural networks," in Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012, pp. 1097–1105.
[32]
M. A. Pai, Energy Function Analysis for Power System Stability. Boston: Kluwer Academic Publishers, 1989.
DOI
[33]
W. Y. Ju, J. J. Qi, and K. Sun,"Simulation and analysis of cascading failures on an NPCC power system test bed," in Proceedings of 2015 IEEE Power & Energy Society General Meeting, Denver, USA, 2015, pp. 1–5.
[34]

J. H. Chow and K. W. Cheung, "A toolbox for power system dynamics and control engineering education and research," IEEE Transactions on Power Systems, vol. 7, no. 4, pp. 1559–1564, Nov. 1992.
DOI Google Scholar
[35]

N. Mo, Z. Y. Zou, K. W. Chan, and T. Y. G. Pong, "Transient stability constrained optimal power flow using particle swarm optimisation," IET Generation, Transmission & Distribution, vol. 1, no. 3, pp. 476–483, May 2007.
DOI Google Scholar
[36]
E. Parisotto, J. L. Ba, and R. Salakhutdinov,"Actor-mimic: Deep multitask and transfer reinforcement learning," in Proceedings of the 4th International Conference on Learning Representations, 2016.
[37]
P. Ammanabrolu and M. O. Riedl,"Transfer in deep reinforcement learning using knowledge graphs," in Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing, 2019, pp. 1–10.
DOI
[38]
S. Gamrian and Y. Goldberg,"Transfer learning for related reinforcement learning tasks via image-to-image translation," in Proceedings of the 36th International Conference on Machine Learning, 2019.
Publication history
Copyright
Rights and permissions

Publication history

Received: 31 August 2020
Revised: 18 December 2020
Accepted: 29 January 2021
Published: 13 November 2021
Issue date: January 2023

Copyright

© 2020 CSEE

Rights and permissions

Return