With the rapid development of mobile communication technology, the contradiction between the limited spectrum utilization resources and the demand of a lot of spectrum communication is increasingly aggravated. New intelligent methods are needed to improve the utilization rate of spectrum. A multi-user dynamic spectrum access method based on distributed priority experience pool and double deep Q network is proposed. This method can help the secondary users to continuously learn according to their perceived environment information in the dynamic environment, and choose the idle channel to complete the spectrum access task for improving the spectrum utilization rate. In this method, a distributed reinforcement learning framework is adopted, and each secondary user is regarded as an agent. Each agent learns by using standard single-agent reinforcement learning method to reduce the underlying computing overhead. In addition, the method adds priority sampling on the basis of neural network training, and then optimizes the training efficiency of neural network to help sub-users choose the optimal strategy. The simulation results show that this method can improve the success rate, reduce the collision rate and improve the communication rate.
- Article type
- Year
- Co-author
Open Access
Issue
Open Access
Issue
A dynamic spectrum allocation scheme is proposed to address the growing number of vehicles and limited spectrum resources in vehicular networks. It integrates the self-attention mechanism and monotonic value function factorisation into deep multi-agent reinforcement learning to optimize channel selection and power levels for vehicle-to-vehicle (V2V) links. The global objective is to maximize the sum throughput of vehicle-to-infrastructure (V2I) links, while meeting the latency and reliability constraints of V2V links. To handle incomplete real-time channel state information due to dynamic environment, a deep recurrent Q-network is deployed for each V2V link, enabling autonomous decision-making based on local observations. To align the local strategy optimization of each V2V link with the global objective, a global mixing network with monotonicity constraints is designed to guide the algorithm training. Additionally, an information interaction model based on the self-attention mechanism further optimizes collaboration between V2V links. Compared with the baselines, the proposed algorithm increases the sum throughput of V2I links by 1.44~8.24 percentage point and reduces the transmission delay of V2V links by 1.93~15.04 percentage point . These results confirm its effectiveness in optimizing channel selection and power levels for improved communication quality.
As a common reinforcement learning algorithm, proximal policy optimization (PPO) performs well in terms of stability, sample utilization, and wide applicability. However, its training effect in the mobile robotic arm environment is not ideal. To solve the problem of learning difficulty and falling into local optimal in the near-end strategy optimization algorithm when training a mobile manipulator, six feasible improvement methods are introduced in this paper, and the original algorithm is adjusted. These methods include advantage normalization, state normalization, reward scaling, policy entropy, gradient clipping, and standard deviation limitation.
They adjust the steps of the PPO algorithm in each stage of data acquisition and training and optimize the stability and learning efficiency of the algorithm. These improved methods have universal applicability, and several schemes can be appropriately selected to adjust the original algorithm in different environments. Before the training, the training environment of the mobile robot arm was modeled, including state space design, action space design, and reward function design. The main flow of the PPO algorithm applied in this environment was analyzed, and the points that can be improved in the training steps were clarified. To verify the effect of each improved PPO algorithm on training the mobile robot arm to pick up objects, relevant experiments were designed for each improvement point. Six improvements were added to the PPO algorithm to train the mobile manipulator, and its reward curve was compared and analyzed with the original algorithm.
The experimental results showed that (1) the native PPO algorithm performs poorly in the mobile manipulator environment, and its reward function often decays to a small value in the late training period. (2) PPO rises faster and reaches a stable value in fewer rounds after adding the advantage normalization. (3) Although state normalization has the greatest improvement on the reward range, it brings unstable factors, making the reward curve exhibit a significant downward trend in the late training period. (4) Reward scaling will bring unstable factors to the training of the mobile robot arm in the early stage; however, the PPO algorithm is robust enough to exert its effect of reward scaling adjustment in the later stage of the training. (5) A policy entropy of 0.2 is more suitable for PPO algorithm training on picking up objects by a mobile robot arm. (6) Gradient clipping prevents excessive gradient updates from causing training instability or dramatic performance fluctuations and has a significant positive effect on the rise of the reward curve.
Through analysis of the reward curve of the PPO algorithm improved by each method, the six improvement methods have different degrees of improvement on the PPO algorithm in the task of training the mobile robot arm to pick up objects. The improved PPO6+ algorithm makes the reward curve of the mobile robot arm better than the original PPO algorithm in terms of stability, reward rising speed, training completion time, etc. It can quickly converge to the ideal result and complete the task goal.
京公网安备11010802044758号