AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home CSEE Journal of Power and Energy Systems Article
PDF (4.1 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Special Section Paper | Open Access

Hierarchical Task Planning for Power Line Flow Regulation

Chenxi Wang1, Youtian Du1( ), Yanhao Huang2, Yuanlin Chang1, Zihao Guo1
Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 713599, China
State Key Laboratory of Power Grid Safety and Energy Conservation, China Electric Power Research Institute, Beijing 100192, China
Show Author Information

Abstract

The complexity and uncertainty in power systems cause great challenges to controlling power grids. As a popular data-driven technique, deep reinforcement learning (DRL) attracts attention in the control of power grids. However, DRL has some inherent drawbacks in terms of data efficiency and explainability. This paper presents a novel hierarchical task planning (HTP) approach, bridging planning and DRL, to the task of power line flow regulation. First, we introduce a three-level task hierarchy to model the task and model the sequence of task units on each level as a task planning-Markov decision processes (TP-MDPs). Second, we model the task as a sequential decision-making problem and introduce a higher planner and a lower planner in HTP to handle different levels of task units. In addition, we introduce a two-layer knowledge graph that can update dynamically during the planning procedure to assist HTP. Experimental results conducted on the IEEE 118-bus and IEEE 300-bus systems demonstrate our HTP approach outperforms proximal policy optimization, a state-of-the-art deep reinforcement learning (DRL) approach, improving efficiency by 26.16% and 6.86% on both systems.

Keywords

Knowledge graphpower line flow regulation reinforcement learningtask planning

References

[1]

A. N. Venkat, I. A. Hiskens, J. B. Rawlings, and S. J. Wright, “Distributed MPC strategies with application to power system automatic generation control,” IEEE Transactions on Control Systems Technology, vol. 16, no. 6, pp. 1192–1206, Nov. 2008.
Crossref Google Scholar
[2]

S. J. Russell, Artificial Intelligence: a Modern Approach. 3rd ed., Upper Saddle River, NJ: Prentice Hall, 2010.
[3]

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, “Mastering Atari, Go, chess and Shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, Dec, 2020.
Crossref Google Scholar
[4]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv: 1707.06347, 2017.
[5]

H. J. Zhang, N. Yang, W. Huangfu, K. P. Long, and V. C. M. Leung, “Power control based on deep reinforcement learning for spectrum sharing,” IEEE Transactions on Wireless Communications, vol. 19, no. 6, pp. 4209–4219, Jun. 2020.
Crossref Google Scholar
[6]
B. Zhang, X. Lu, R. S. Diao, H. F. Li, T. Lan, D. Shi, and Z. W. Wang, “Real-time autonomous line flow control using proximal policy optimization,” in Proceedings of 2020 IEEE Power & Energy Society General Meeting, 2020, pp. 1–5.
Crossref
[7]

H. Tang, K. Lv, B. Bak-Jensen, J. R. Pillai, and Z. F. Wang, “Deep neural network-based hierarchical learning method for dispatch control of multi-regional power grid,” Neural Computing and Applications, vol. 34, no. 7, pp. 5063–5079, Apr. 2022.
Crossref Google Scholar
[8]

Y. Ji, J. H. Wang, J. C. Xu, X. K. Fang, and H. G. Zhang, “Real-time energy management of a microgrid using deep reinforcement learning,” Energies, vol. 12, no. 12, pp. 2291, Jun. 2019.
Crossref Google Scholar
[9]

Y. F. Zhang, Q. Ai and Z. Y. Li, “Intelligent demand response resource trading using deep reinforcement learning,” CSEE Journal of Power and Energy Systems, doi: 10.17775/CSEEJPES.2020.05540.
Crossref Google Scholar
[10]

M. Kamel, R. C. Dai, Y. W. Wang, F. X. Li and G. Y. Liu, “Data-driven and model-based hybrid reinforcement learning to reduce stress on power systems branches,” CSEE Journal of Power and Energy Systems, vol. 7, no. 3, pp. 433–442, May 2021.
Crossref Google Scholar
[11]

J. Li, S. Chen, X. Y. Wang and T. J. Pu, “Load Shedding Control Strategy in Power Grid Emergency State Based on Deep Reinforcement Learning,” CSEE Journal of Power and Energy Systems, vol. 8, no. 4, pp. 1175–1182, Jul. 2022.
Google Scholar
[12]

R. S. Sutton and A. G. Barto, Reinforcement Learning: an Introduction, 2nd ed., Cambridge: MIT Press, 2018.
[13]
D. M. Lyu, F. K. Yang, B. Liu, and S. Gustafson, “SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning,” in Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019, pp. 2970–2977.
Crossref
[14]
F. K. Yang, D. M. Lyu, B. Liu, and S. Gustafson, “PEORL: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making,” in: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, pp. 4860–4866.
Crossref
[15]
A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson, “PRM-RL: long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning,” in Proceedings of 2018 IEEE International Conference on Robotics and Automation, 2018, pp. 5113–5120.
Crossref
[16]

T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker, “A unifying framework for reinforcement learning and planning,” Frontiers in Artificial Intelligence, vol. 5, p. 908353, 2022.
Crossref Google Scholar
[17]
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: learning skills without a reward function,” in Proceedings of the 7th International Conference on Learning Representations, 2018.
[18]
A. Sharma, S. X. Gu, S. Levine, V. Kumar, and K. Hausman, “Dynamics-aware unsupervised discovery of skills,” in Proceedings of the 8th International Conference on Learning Representations, 2020.
[19]

X. M. Dong, H. Sun, C. F. Wang, Z. H. Yun, Y. M. Wang, P. H. Zhao, Y. Y. Ding, and Y. Wang, “Power flow analysis considering automatic generation control for multi-area interconnection power networks,” IEEE Transactions on Industry Applications, vol. 53, no. 6, pp. 5200–5208, Nov./Dec. 2017.
Crossref Google Scholar
[20]
P. Kundur, G. K. Morison, and L. Wang, “Techniques for on-line transient stability assessment and control,” in Proceedings of 2000 IEEE Power Engineering Society Winter Meeting. Conference Proceedings, 2000, pp. 46–51.
[21]

R. S. Sutton and A. G. Barto, “Introduction to Reinforcement Learning,” Cambridge: MIT Press, 1998.
[22]

M. Hanheide, M. Göbelbecker, G. S. Horn, A. Pronobis, K. Sjöö, A. Aydemir, P. Jensfelt, C. Gretton, R. Dearden, M. Janicek, H. Zender, G. J. Kruijff, N. Hawes, and J. L. Wyatt, “Robot task planning and explanation in open and uncertain worlds,” Artificial Intelligence, vol. 247, pp. 119–150, Jun. 2017.
Crossref Google Scholar
[23]

L. X. Yang, Q. Y. Sun, N. Zhang, and Z. W. Liu, “Optimal energy operation strategy for we-energy of energy internet based on hybrid reinforcement learning with human-in-the-loop,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 1, pp. 32–42, Jan. 2022.
Crossref Google Scholar
[24]

J. J. Duan, D. Shi, R. S. Diao, H. F. Li, Z. W. Wang, B. Zhang, D. S. Bian, and Z. H. Yi, “Deep-reinforcement-learning-based autonomous voltage control for power grid operations,” IEEE Transactions on Power Systems, vol. 35, no. 1, pp. 814–817, Jan. 2020.
Crossref Google Scholar
[25]

R. Rocchetta, L. Bellani, M. Compare, E. Zio, and E. Patelli, “A reinforcement learning framework for optimal operation and maintenance of power grids,” Applied Energy, vol. 241, pp. 291–301, May 2019.
Crossref Google Scholar
[26]

Q. H. Huang, R. K. Huang, W. T. Hao, J. Tan, R. Fan, and Z. Y. Huang, “Adaptive power system emergency control using deep reinforcement learning,” IEEE Transactions on Smart Grid, vol. 11, no. 2, pp. 1171–1182, Mar. 2020.
Crossref Google Scholar
[27]

C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of Monte Carlo tree search methods,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 4, no. 1, pp. 1–43, Mar. 2012.
Crossref Google Scholar
[28]
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis, “Mastering chess and shogi by self-play with a general reinforcement learning algorithm” arXiv: 1712.01815, 2017.
[29]
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” arXiv: 1812.05905, 2019.
[30]
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 1587–1596.
CSEE Journal of Power and Energy Systems
Volume 10 Issue 1,
January 2024
Pages 29-40
DOI: 10.17775/CSEEJPES.2023.00620
Cite this article:
Wang C, Du Y, Huang Y, et al. Hierarchical Task Planning for Power Line Flow Regulation. CSEE Journal of Power and Energy Systems, 2024, 10(1): 29-40. https://doi.org/10.17775/CSEEJPES.2023.00620

180

Views

5

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 03 February 2023
Revised: 19 May 2023
Accepted: 06 July 2023
Published: 28 December 2023
© 2023 CSEE.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Return