A Method for Joint Optimization of Signal Timing and Vehicle Trajectories at Intersections Based on Hierarchical Soft Actor-Critic Reinforcement Learning

This study proposes a joint optimization method for intersection signal timing and vehicle trajectory based on the Soft Actor-Critic (SAC) reinforcement learning framework. The model consists of two layers: signal timing optimization and vehicle trajectory optimization. The state space for both layers includes vehicle position, speed, and traffic signal status, while the reward function is a weighted sum of traffic efficiency, safety, and fuel consumption. In the signal timing optimization layer, the action is the duration of the signal phase, and in the vehicle trajectory optimization layer, the action is the vehicle acceleration. Each optimization layer has independent value networks and policy networks. The value network outputs the state-action value and assesses the policy network's performance according to the current state and action. The policy network generates the mean and standard deviation of a Gaussian distribution based on the current state and samples actions from this parameterized Gaussian distribution. The loss function of the policy network includes entropy and temperature coefficients to automatically adjust the breadth and depth of policy exploration, reducing the model's sensitivity to hyperparameter variations. To address the inconsistency in the intervals between signal timing optimization and vehicle trajectory optimization, this study designed an asynchronous training algorithm for the signal timing layer and vehicle trajectory optimization layer. Both the value network and the policy network of the same layer were trained simultaneously using backpropagation. The model was trained and evaluated with SUMO, and experimental results indicate that the proposed method reduces vehicle fuel consumption by an average of 24.24%, 5.39%, and 22.23%, compared to mathematical programming methods, signal-timing-only optimization methods, and trajectory-only optimization methods, respectively. It can achieve energy optimization without significantly reducing average speed, while maintaining performance deviations within 5% under state observation disturbances, demonstrating good robustness.