AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Article Link
Collect
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Full Length Article | Open Access

Cooperative decision-making algorithm with efficient convergence for UCAV formation in beyond-visual-range air combat based on multi-agent reinforcement learning

Yaoming ZHOUa, Fan YANGa, Chaoyue ZHANGa, Shida LIa, Yongchao WANGb( )
School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China
Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China

Peer review under responsibility of Editorial Committee of CJA.

Show Author Information

Abstract

Highly intelligent Unmanned Combat Aerial Vehicle (UCAV) formation is expected to bring out strengths in Beyond-Visual-Range (BVR) air combat. Although Multi-Agent Reinforcement Learning (MARL) shows outstanding performance in cooperative decision-making, it is challenging for existing MARL algorithms to quickly converge to an optimal strategy for UCAV formation in BVR air combat where confrontation is complicated and reward is extremely sparse and delayed. Aiming to solve this problem, this paper proposes an Advantage Highlight Multi-Agent Proximal Policy Optimization (AHMAPPO) algorithm. First, at every step, the AHMAPPO records the degree to which the best formation exceeds the average of formations in parallel environments and carries out additional advantage sampling according to it. Then, the sampling result is introduced into the updating process of the actor network to improve its optimization efficiency. Finally, the simulation results reveal that compared with some state-of-the-art MARL algorithms, the AHMAPPO can obtain a more excellent strategy utilizing fewer sample episodes in the UCAV formation BVR air combat simulation environment built in this paper, which can reflect the critical features of BVR air combat. The AHMAPPO can significantly increase the convergence efficiency of the strategy for UCAV formation in BVR air combat, with a maximum increase of 81.5% relative to other algorithms.

Electronic Supplementary Material

Download File(s)
cja-37-8-311_ESM.pdf (1.9 MB)

References

【1】
【1】
 
 
Chinese Journal of Aeronautics
Pages 311-328

{{item.num}}

Comments on this article

Go to comment

< Back to all reports

Review Status: {{reviewData.commendedNum}} Commended , {{reviewData.revisionRequiredNum}} Revision Required , {{reviewData.notCommendedNum}} Not Commended Under Peer Review

Review Comment

Close
Close
Cite this article:
ZHOU Y, YANG F, ZHANG C, et al. Cooperative decision-making algorithm with efficient convergence for UCAV formation in beyond-visual-range air combat based on multi-agent reinforcement learning. Chinese Journal of Aeronautics, 2024, 37(8): 311-328. https://doi.org/10.1016/j.cja.2024.04.008

1680

Views

25

Crossref

21

Web of Science

27

Scopus

6

CSCD

Received: 04 September 2023
Revised: 04 December 2023
Accepted: 07 February 2024
Published: 12 April 2024
© 2024 Chinese Society of Aeronautics and Astronautics.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).